That's a really good question. And I'll attempt to illustrate it with probably the simplest example, although I apologize if this is too high (or too low) level!

Let's start first by defining quite what I meant when I said

if one leaves the system alone for a sufficiently long time, it should settle and become hot

By this I mean that the number of particles with a given energy has a thermal distribution. This thermal distribution for electrons (or more generally, for fermions) is given by the Fermi-Dirac distribution and looks like the below for a number of temperatures

The lower axis here is the energy (E) minus the "Fermi energy" (E_F) which is defined as the energy of the highest-energy electron at absolute zero temperature (so don't worry about seeing a negative axis!).

Now, lets consider a bunch of non-interacting electrons -- the electrons just float around, not seeing one-another or anything else. Of course, this isn't realistic, but we're theorists, so we can get away with thinking about such things. Imagine now that I "dump some energy" into my system by adding an electron with energy 1; what happens? Well, we have some electrons that float around, not seeing one-another and not interacting. This means that there's no way to reduce the energy of the electron you've added, so no matter how long I wait, there'll be an electron with energy 1, and I'll have a non-thermal distribution (it'll look like the Fermi-Dirac distribution above with a jump at energy 1). In physics, we like to say that there is a conservation law -- the number of particle at each energy is conserved in this simple case. Of course, this isn't very interesting so far as everything is non-interacting and not terribly realistic.

Now, what happens if we turn on interactions between the electrons in our system? Interactions may allow us to redistribute energy: if we have an electron with energy Ea and another with energy Eb we can collide them and scatter to energies Ec and Ed provided Ea + Eb = Ec + Ed, e.g. energy is conserved. Notice now that we only really have one conservation law -- that total energy is conserved. In general, it is expected that such processes will eventually lead to thermalization (e.g., the Fermi-Dirac distribution at a suitably higher temperature, fixed by the energy we dumped into the system).

Now, as a theorist, I want to test this expectation (let's call it a conjecture). So I turn to my favorite interacting model that I know how to exactly-solve (there are not many of these) and test this conjecture. What do I find? I find that my exactly-solvable model doesn't thermalize: when I inject energy into the system I do not recover the thermal distribution. What gives?! Well, it comes down to what I previously mentioned -- conservation laws. These special exactly-solvable models are solvable precisely because they have lots of conservation laws (in fact, they have the same number of conservation laws as particles) and this puts very strong restrictions on how the particles can redistribute energy around and eventually leads to a non-thermal distribution. Figuring out what this non-thermal distribution is and how to compute the values of "measurable quantities" are serious areas of research at the moment.

This comment ended up much longer than I anticipated, and I'm not sure of an adequate tldr!