Free speech is meant to prevent government limitations on speech. Free speech is not meant to limit judging the speaker as morally, ethically or socially bankrupt. I feel like there's a conversation or interaction I'm missing here. I agree it's important to remember others disagree with us. That doesn't add legitimacy to a hateful position, of course. Remembering there are different, opposing views can help us hone our own discussion, shape our own opinions and, in some cases, reassert the ludicrousness and hatefulness of racism, misogyny or other stereotypes.
Neither (lack of) popularity of one's opinion nor views opposing one's own have anything to do with the hatred one's message holds. Those ought to be judged upon separately. It doesn't matter if 10 or 1000 people follow me, nor does it matter if I don't agree with someone over a topic: if I spit hateful remarks, I'm an asshole, even if some portion of what I say is correct or meaningful.