I think I can understand that fear - but I like being in control. I actually follow people I disagree with, by conscious choice. It adds to the discussion. It keeps things fresh and interesting. I could cocoon away with some people that make me feel good about myself. But instead I keep a group of people in my feed that challenge the way I think and disbelieve the things I believe in.
If I miss an intelligent post made by some one I ignore because they usually post pictures of unicorns, cats and narwhals, well, then I miss out. But I also miss all of the noise. (not that there's anything wrong with unicorns, cats, and narwhals).