You are taking the points system entirely too seriously, Making it public that anonymous negative votes, gets under your skin, probably delights the "coward" to no end. Personally, I like negative votes.... it let's me know that someone is annoyed and upset by me, and that gives me great satisfaction. Negative votes are more fun than positives.
It's kind of ironic that a place named Unity employs a system that encourages the negative. Though I do like your thinking.