This is largely a non-issue. REP matters, not raw votes. Some comments have 100 Up Votes, but 0 reputation, and others display -10 but are actually positive on reputation. If your personal REP is neutral or going up you are doing fine. If it is really going down, you are running afoul of higher-REP members and Admins.