I'm sorry artie, this turned into a monster.
you know what the fun thing about the history of Jazz is? The way I see it, it's basically the same story as classical music, condensed into 150 years or so. The story of a continued "harmonic breakdown" - basically things get more chromatic over time.
This is, of course, incredibly simplified and leaves out a lot - but the parallels are there.
We start off with a rigidly structured music, the Blues. Even though there are improvisatory elements in Blues music, the chord progression is almost always identical: I--IV-V-I. If you're not musically literate, those roman numerals won't make sense to you, but they're essentially the building blocks of all western art music.
This mirrors, in a lot of ways, the earliest periods of music in Europe that are in the scope of Classical tradition - Gregorian Chant. You know, this stuff:
very harmonically simple, and the words are the most important part. Blues is the same - a repeating chord rhythm, where the words are what differentiate each song.
After this, Classical Music had this intensely complicated period called Baroque. I'm no scholar of Jazz specifically, but as far as I know, Jazz never had a whole lot like that, but this period does have one element in common with baroque - the solidification of group sizes, and the creation of musical "forms"(like blueprints) . In the 1800s, The US gains Louisiana and Florida in a series of diplomatic sales and land grabs. On this land are slaves previously under Spanish and French rule - Creole (some of whom have been absorbing traditions from their owners, or even playing European Classical music to entertain them) -, and a group of people who had been expelled from Upper Canada in the 1700s - Acadians, or as we know them, Cajun. Over time, with lower class intermingling, Blues music begins to absorb all of these traditions. This is where we get this:
At the same time, after the Civil War, dance halls are becoming popular. African Americans, having more free time since, you know, not being slaves, begin to blend their music with dance culture. Musicians, employed to process in funerals, start to form bands.
You'll notice harmony is still pretty simple, but has a little bit more going on. There's even a few short solos. Armstrong was a relic of an older kind of Jazz. Even though he lived a long time, and performed until quite late in life, his style is always very reminiscent of 20's Jazz. Indeed, his style of trumpet playing defined how trumpeters played in Jazz for a long time, until he wasn't cool anymore. Because he was so popular with white audiences, he unintentionally became a kind of "Uncle Tom" figure in jazz late in his life, at which point people really began to pull away from his playing style. Just like Haydn, he started ahead of his time, and ended as a relic.
Cue: The Great War, the subsequent depression, and WW2. Music, and dancing fuelled by Live bands in particular was cheap and popular. Prohibition also helped things along, giving lots of venues for bands to play in. Majority white audiences are starting to catch on to this Jazz thing. Jazz composers such as Ellington start to appear (I'm jumping over a lot here to try to keep this short), and they begin creating larger bands, to give themselves more music tools for composition. To pander to Audiences, "Big Band" jazz is much more... genteel?
Like Beethoven before him, Ellington is not content to simply walk in the back entrance like a servant. He wants for be treated like the musical genius he knows he is. He and his band might be going town to town in beat up old cars, but damn if they don't look fine doing it.
We've moved into Jazz's " romantic period" here (depending on how one defines "romantic period" in classical music) the first threads in the sweater of Jazz's tonality are being pulled, and solos are getting longer, becoming more integral to the music.
After the war, Big bands start to lose steam. Trying to keep a 20-piece band employed is a lot more difficult than keeping a 4-piece employed. When you've just got 4 or 5 dudes, it's also a lot easier to experiment.
Then, just as Jazz is getting intensely chromatic and complicated, it simplifies - Like Classical, it's "neoclassical" period hearkens back to an earlier time. In concept, this piece:
and this piece:
have a lot in common, hearkening back to old music, old traditions. It's reactionary to the increasing chromaticism and complication. the chord progressions are simpler (though still tinged with more modern harmonies) and so are the forms. This strain still exists in a lot of ways, just like neoclassical music never really went away, even through the seriously avant-garde shit.
avant gard shit like this:
or alteratively this:
(the singing is out of tune on purpose)
It's really in this avant garde stuff that has the elements you're talking about - extreme technicality, "free" chromaticism, and stuff that is kind of hard to understand if you're not steeping yourself in it constantly. A lot of jazz being played right now is not like that - just like a lot of classical Music is not like that.
Regarding Buddy Rich: Buddy Rich was a great drummer. He was also a supreme asshole, who grew up in a time when conductors and band leaders were god when it came to the ensemble. Arturo Toscanini is the most egregious example of this in the 20th century - Messiah Complex if there ever was one. So he acted like God, and bossed a lot of people around. So did Stevie Wonder, so did Billy Joel - They made a lot more money, and were at least as good musicians.