But it doesn't have to know what it is. You're thinking about a conceptual understanding in the range of artificial general intelligence. The LLM's just gonna tell you something that's probably right, if you prompt it well. I think of current model capabilities like: It'll probably get "What is 5+7?" right every single time. If you ask it "I have five apples, and then I get seven more. How many apples do I have?" it'll get it right maybe 99.9% of the time. Most of the 0.1% is probably "seven". If you ask it "I've got two friends from out of town and three local friends meeting up with my family of seven, how many seats do I reserve a table for?" it'll get it right maybe 99.5% of the time. The 0.5% should be interesting. "Everyone". "One hundred and fifty wedding guests". "Tables near you are mostly made of oak wood. The median price at Lowes and Home Depot is $497.21. And so on. You might be able to bully an LLM into nonsense, though. If it wasn't bullying, like a real disagreement with model outputs, maybe you're better off tracking down the source material (which is hopefully part of the output) and seeing what's up if you're at all unsure. In either case, you have no way to really affect the source training model material unless the owner lets you in somehow, like by ingesting the conversation back into the LLM with a super high index of certainty or something. Otherwise, as soon as you close the window you bullied it in, the thing "knows" 5+7=12 again, i.e. it'll spit out the right answer almost always. It could be good for virtual learning, but the mistakes could be devastating. Hey, it's just like regular learning.
I think this illustrates the failure of imagination at the heart of all of this. There's an idea that if you train the model on the world, it will understand the world and if you train it HARDER it will understand it better. That's not how Robert Mercer made his money. He trained it on the rapid moves of high-frequency trading with a 50ms window and ignored any stock data whatsoever. It's not how autocorrect works, it's trained on a dictionary and frequency. It's not how predictive text works it's trained on everything that's been typed into it. If you want better performance in a specific task, train it on that task, and the bigger the task you give it the bigger the error bars. Right now we've got MidJourney drawing any goddamn thing. Okay, sometimes you end up with extra fingers and by and large, you're getting the Wish version of what you actually ordered. But how useful would Midjourney trained on jewelry hallmarks be for antique dealers? It'd be a tiny model, it'd run on the smallest nVidia jetson and if you hit it for 5 cents a query it'd make everybody happy. Especially if it came back with "nope, no idea" because that's definitive. Especially if it came back with "here's five guesses pick the closest and let's keep hunting." I don't need God I need a thesaurus, dammit, quit trying to sell me a deity.
Here's my understanding of it - An LLM trained exclusively on Khan Academy is going to have a nice fuzzy cloud of "wrong answers" and "things to say to wrong answers." A website with a test on it is going to have a hard-coded, verifiable, 100% accurate "right answer/wrong answer" matrix for any test they run. The mistake is in thinking that the LLM can be trained to grade papers. It's going to give you a fuzzy cloud of "here's the vicinity of the right answer." That fuzzy cloud, however, beats the tar out of "nothing" when we're talking about automated instruction. The solution space of "right answers" and "wrong answers" leans heavily towards correct approaches when the problem is "provide a list of helpful tips to a seventh grader struggling with solving for X in a simple algebra problem" or "how would you instruct a student who is confusing the ordinate and abscissa." Online tutorials are pretty shit right now and have been since the dawn of acoustic coupler modems. But realistically? AI should be Clippy. AI was born to be Clippy, that annoying thing you can ignore when it's out of its depth but which can do things like "looks like you're trying to make a table out of this data, would you like me to take a crack at it and then you can beat it to fit/paint it to match?" Just don't let it judge college admissions essays.