I've always found this to be an overwhelming cause of these issues. Dunning–Kruger effect strikes again. However, more importantly, I've never found a way to get people out of this situation. How do you encourage humility of knowledge? It seems like a combination of early education emphasizing humility and uncertainty, diverse experiences growing-up, and extensive study in at least a couple of areas to demonstrate the limits of ones own knowledge in a given field. As this article's opening quote points out, simply going to college isn't necessarily going to help.What mucks it all up when a narrow set of information is assumed to be wider than it is. There is a difference between a belief and things you just didn’t know.
I think the best way to foster humility of knowledge, as you call it, is to create an environment where things must have sufficient evidence before they're presented as fact. For instance, lots of math curriculum simply tell kids the pythagorian theorem without ever making an attempt to prove it. Imagine how different history class would be if, instead of reading out of a textbook in 5th grade and accepting everything in them as fact, we were given a collection of primary and secondary sources and were told to come up with our own ideas? Sure, it would be ridiculous to expect little elementary or middle school kids to be able to do that, and I am no educator. But I think if we tried to foster that attitude from the very start, where everything has to have evidence backing it, people would be more receptive to new knowledge and would base their beliefs upon reason more often.
I like math. I haven't had many good teachers but my 7th grade teacher (or 8th, I don't remember) taught the Pythagorean theorem with m&m's. The bases were 3cm and 4cm, the hyp was 5cm. She had us draw the right triangle and then do squares on each side. Filled the 9 and 16 squares with m&m's and then put them in the 25 square. Easy, visual, and I still remember it. This is how teaching should be done.