> The OthelloGPT world-model story faced a new complication when, in mid-2024 a group of student researchers released a blog post entitled “OthelloGPT Learned a Bag Of Heuristics.” The authors were part of a training program created by DeepMind’s Neel Nanda, and their project was to follow up on Nanda’s own work, and do careful experiments to look more deeply into OthelloGPT’s internal representations. The students reported that, while OthelloGPT’s internal activations do indeed encode the board state, this encoding is not a coherent, easy-to-understand model like, say, an orrery, but rather a collection of “many independent decision rules that are localized to small parts of the board.” As one example, they found a particular neuron (i.e., neural network unit) at one layer whose activation represents a quite specific rule: “If the move A4 was just played AND B4 is occupied AND C4 is occupied, then update B4 and C4 and D4 to ‘yours’ [assuming the mine, yours, or empty classification labels]”. Another neuron’s activation represents the rule “if the token for B4 does not appear before A4 in the input string, then B4 is empty.”

I’ve never understood why this is a problem. This is how all thinking actually works. When I’m solving a problem, im not inventing a solution de novo every time I do it. I’m using heuristics. This thing X generally leads to thing Y and therefore I need to do A, B and C to correct for it. That’s a heuristic. It’s also how we predict the weather. Low pressure meets high pressure means rain likely so wear a poncho or carry an umbrella. We’d call that thinking, but it’s basically using heuristics X -> Y requiring solution A. When I solve math equations, it’s nothing but procedures and heuristics. PEMDAS as order or operations, the math operations being basically procedures. And when using mathematics to solve a problem, then you’ll basically be deciding which heuristics of mathematics to use. Even making political predictions are based on heuristics gleaned from history. XY and Z happen during the rise of revolutionary thinking. Therefore if you see this predict revolution.

To be Frank, even a world model is basically a systematically constructed bag of heuristics. The religious world view: God exists, gave us rule book X, and those who follow rule book X get rewarded. Therefore do what rule book X says. The secular world view replaces Rule Book X with principles derived from science and neoliberalism, but the basic building blocks are the same. These heuristic principles lead to good outcomes, thus doing them is a good idea. It covers more domains than your ad hoc heuristics as AI is using them today, but the difference isn’t the approach, it’s the scale. A person living by Torah or KJV Bible is still using heuristics to figure out how to live, the question is scale, as the sacred book in question covers everything where AI tends to be bound to whatever applicable training sets it was given.