The data doesn't surprise me. Downvotes feel bad. There is a special kind of frustration that you get after composing a response, and coming back to see -2: WTF is that?! -2? Minus two what? Did the OP downvote me or was it someone else? Two people? Wait, -3 now? Are people just piling on? Did I say something that this community as a whole doesn't like?

They make you guarded and defensive, and that alters your mood. After being downvoted, I have found myself away from the keyboard, minutes later, still feeling slighted. They definitely alter my approach to commenting.

From a functional perspective, not only can you sort without downvotes, their presence increases the variables in the sort, which makes sorting for one signal more difficult. A downvote is not the opposite of an upvote. Also, what would be the equivalent of a downvote in a face-to-face conversation? I imagine an upvote might be someone saying 'good point'. But people don't say 'bad point'. Disapproval without explanation is not the converse of approval without explanation.