Understand that the point of the moderation system here is to promote civil discussion. The protocol most people (try) to follow here is a natural result from that.

I have a policy of only downmodding posts that are not a part of legitimate discussion. This includes obvious trolls, but also any post where I don't think the post contributes anything, such as an ad-hominem attack.

I do not downmod a post just because I disagree with it. If I disagree with the post, but it's not an obvious troll or incendiary in any way, I leave it alone. I see downmodding as a means to keep discussions civil, not to show disapproval with a point of view.

In contrast, I upmod posts I agree with. This means that a single control - voting up or down - has an asymmetric meaning for me.