I'm glad that somebody much better at writing than me has noticed this, because I've seen this trend, and it's part of the reason I left academia after getting my B.S. in physics.
I've sort of become a black sheep by arguing with professors and other students that we should fund science less, not more, and that if anything, we should fund large scale projects like Kepler or the LHC, and not individual abstract scientists working in their respective labs. While it seems like having more individuals working on what they find interesting might produce more innovation, that's in general the opposite of what I've seen.
With all of the individual scientists that I've worked with, all of whom work in their own laboratory at a university, the most important thing that they needed to do,first, was to get tenure, so that they can have job security. The two things that a young scientist needs to get tenure are grants and papers. Usually, it's easiest to stay in the research area where one has done postdoc or PhD work, as this work is known to be fundable, and its the area of expertise that a scientist can put on their resume. Also, the young scientist would already have a network of collaborators in that field who will either work with them on papers or at the very least cite them, which will boost their h-index, which will definitely help with getting tenure. So, there's a strong incentive to work in an already established field, on something tangentially related to previous work.
Next, the big problem is grant writing. I've read many grants that have been written for projects that I've worked on, and I've found a basic formula for how they are written. First, they mention all of the work that has been done previously in their field. It's important to have this, because work will not be funded by the NIH or the NSF it can't be established that the work can be used by a number of other researchers. This happens all the way down to studying specific materials: I've seen a grant get rejected because the PI wanted to study a different magnetic alloy that the main one people were studying those days. Second, in order to get funding, a PI needs to outline the potential applications that their work might have. Now, perhaps I've met a disproportionate amount of mediocre professors, but none of the professors I've ever talked to about research have been able to mention in the slightest amount how their work might apply to the technologies they mention. These technologies have included earthquake detection, magnetic tape, hard drive storage space, cataracts, and worst of all, cancer. These PIs do not know a thing about these technologies, and they have no idea how their work relates, but they have been taught by their advisers that they need to include some applications in their grant proposals, so they do it.
Yeah, I got out when I realized that it was normal in science to lie about being able to cure cancer in order to get money.
And it's worse, too. The combination of those two above means not only are there a couple professors at my alma mater who lie about curing cancer for money, but they also have hundreds of collaborators all around the globe who do similar work. None of these people have any idea about metastasis or carcinogens, all they do is create molecular dynamics simulations in FORTRAN with many simplifying assumptions, and put low resolution plots of their data in papers. So all of this "cancer money" that could be spent in research and development gets spent on an inward facing circle of professors who all just want tenure. But it's not even fraud, so there's no way to report it.