Do models understand the world the way experts do?
Tony Chen · Princeton University
Today: how to test whether models understand the world the way experts do.
🔬Doing research
💼Running a business
AI is starting to take on this kind of work, and it has to make important decisions in a complex world.
AI is starting to take on open-ended work, like research or running a business.
In this work it makes important decisions in a complex world.
The end result is hard to check
?
decision
result
it can take years to know
🎲 and luck plays a large role
Whether a research direction or a business plan was right is hard to check.
It can take years to find out.
And luck plays a large role.
Grade the steps instead?
Check whether a model makes the same decisions an expert made along the way.
1Each decision is hard to set upWhat did the person know? What were the options?
2Decisions happen over many stepsNo single right order, and the reasons are often never recorded
3Important decisions are rareMost of what people do day to day is routine
4People don’t like being recordedBut many are happy to talk about their work
One alternative: grade the steps, not the result.
Problem 1: you have to decide what information and options the person had.
Problem 2: decisions unfold over many steps, in different orders, and depend on things never recorded.
Problem 3: important decisions are rare.
Problem 4: people do not like being recorded, but they will talk, for example on podcasts.
Problem 1
Setting up a decision sets its difficulty
In 2021, Zhuang Liu decided to test whether ConvNets had really fallen behind vision transformers, or whether the gap came from small design details.
Tell the model about the design details
The decision follows almost directly.
Too easy
Leave them out
The model has to guess what Zhuang knew at the time.
Too hard
Example: the decision behind ConvNeXt.
If you tell the model about the design details, the answer is easy.
If you do not, it has to guess what Zhuang knew.
Long chains of decisions: research projects, companies, advising.
Every step draws on the same understanding of the world. It is reused everywhere, and it is often what separates experts with good taste.
And unlike the decisions, it can be tested directly with one standalone question. No environment needed.
Asked what to do, experts first explain how the world works
Ravid Shwartz-Ziv, The Information BottleneckIn a world that we have more and more compute and more and more data, why not just build agents and solve all the problems?
Zhuang LiuAgents still make mistakes. A lot of mistakes it had for me was not remembering something that's obvious.Even if you have unlimited context, in theory you can access everything you told the model, but if it forgets, it still doesn't have good memory.How can we have unlimited memory? This is like the continual learning problem. How can we not forget?That's something more important, and would be more fruitful if we have success, than how to build collaborative agents.The fact that we need a lot of agents is exactly because one agent cannot remember everything.
81% how the world works19% what to do
The host asks a what-to-do question.
Zhuang answers.
Most of the answer, 81 percent, is about how agents fail. Only the end says what to do.
So we tested it
🎙️
3 podcast interviews
Karthik Narasimhan, Zhuang Liu, Ilya Sutskever
→
❓
119 standalone questions
about how they see their field
→
💡
281 ideas
from the experts’ actual answers
→
🤖
13 models answer as the guest
6 Claude, 7 GPT
→
⚖️
A judge checks each idea
corrected by hand where it erred
One podcast interview each.
We wrote standalone questions and listed the ideas in each real answer.
Each model sees only a one-line intro and the question.
A judge model checks which ideas each answer contains.
Answers are fluent and reasonable. The misses follow five patterns.
1Hedging“X is important, but it’s one part of the whole system.”
2Going along with the question“I think there’s a lot of truth to that.”
3Talking around the question“I think the important question is…”
4The textbook answer“The biggest challenge is the reward signal.”
5Describing AI tools from the outside“Writing code was never the core of a PhD.”
The answers read well. But when they miss the expert’s idea, they fail in one of five ways.
Pattern 1 of 5
Hedging
“X is important, but it’s one part of the whole system.”
ClaudeGPT
How do you assess the importance of neural network architecture choice for making AI more useful?
Zhuang Liu
From a practical point of view, what data we train the model on matters more than the architecture choice.
The idea: For making AI useful, training data matters more than architecture.
×Sol 6.1“I think architecture is important, but it’s one part of the whole system.”
×Luna 6“I think architecture matters, but it’s only one part of the story.”
×Astra 6“I think architecture is important, but I would judge it by what it enables.”
✓Opus 5“I have a somewhat contrarian view on this, which is that architecture gets more credit than it deserves for where we are today.”
✓Haiku 5.5“I think architecture matters less for usefulness than people often assume.”
5 of 13 models’ answers contain the idea.
The model says every factor matters instead of saying which one matters most.
Typical answers miss the idea.
Some models get it.
Pattern 2 of 5
Going along with the question
“I think there’s a lot of truth to that.”
ClaudeGPT
How do you view the idea that established neural-network architecture components are historical accidents?
Zhuang Liu
Residual connections are very essential, and I do still believe they are essential. They are not historical accidents.
The idea: Residual connections are essential, not a historical accident.
×Opus 5.5“I think there's a lot of truth to it, and a lot of my own work has come from taking that idea seriously.”
×Sol 6.1“I think there’s a lot of truth to that.”
×Astra 6“Yeah, I think there’s some truth to that.”
✓Fable 5.1“Sometimes the answer is that the component is genuinely essential. Residual connections seem like that.”
✓Haiku 5.5“Residual connections, for example, keep gradients flowing through deep networks, and it's hard to imagine training very deep models without some version of them.”
3 of 13 models’ answers contain the idea.
The model accepts the assumption built into the question, even when the expert rejects it.
Typical answers miss the idea.
Some models get it.
Pattern 3 of 5
Talking around the question
“I think the important question is…”
ClaudeGPT
How do you judge when improvements on an AI benchmark are worth pursuing?
Karthik Narasimhan
If you're at 20% and then you go up to 80%, that's a very clear signal. But if everyone's at 80% and you're climbing to 81 and 82, at that point it's probably worth your time to actually look at the next thing.
The idea: Once models score near the top of a benchmark, small gains mean little.
×Sol 6.1“I think the first question is: what are we trying to learn from this benchmark?”
×Luna 6“I’d ask whether the improvement tells us something useful about what the system can actually do.”
✓Opus 5.5“If it's at 90-plus, you're often fitting noise, label errors, or quirks of the test set, and the marginal gain tells you little about the underlying capability.”
✓Fable 5.1“If a benchmark is at 95 percent, the last few points are almost always noise or label errors.”
6 of 13 models’ answers contain the idea.
The model lists what to consider instead of giving an answer.
Typical answers miss the idea.
Some models get it.
Pattern 4 of 5
The textbook answer
“The biggest challenge is the reward signal.”
ClaudeGPT
What do you see as the main challenge in applying reinforcement learning to enterprise problems?
Karthik Narasimhan
You want to be using RL for the right things, but you also need to have the whole setup, the environment, all of that. And that requires a lot more work than the actual RL piece.
The idea: Building the setup around RL, like the environment and the reward, takes more work than the RL itself.
×Sol 6.1“I think the biggest challenge is that reinforcement learning typically needs a lot of trial and error, and in an enterprise setting, that trial and error has a real cost.”
×Luna 6“The main challenge is that enterprise problems rarely come with a clean, immediate reward signal.”
✓Opus 5“Honestly, it's not the learning algorithm. That's the thing people get wrong. The algorithms we have are good enough — PPO, GRPO, whatever flavor you like. The bottleneck is everything around them.”
7 of 13 models’ answers contain the idea.
The model gives the standard explanation and misses the practical reason the expert gives.
Typical answers miss the idea.
Some models get it.
Pattern 5 of 5
Describing AI tools from the outside
“Writing code was never the core of a PhD.”
ClaudeGPT
How do you view the claim that AI coding agents make research students unnecessary?
Zhuang Liu
I keep [telling Claude Code] to use this partition of GPUs. It may follow that for a few hours and then after that job is done, it kind of forgot that thing.
The experiments [Claude Code] did, it's not comprehensive. It's not enough to support the conclusions.
The idea: Claude Code forgets instructions it followed earlier in a long project.
×Opus 5.5“Writing code was never the core of a PhD.”
×Sonnet 5.5“I don't think writing code was ever the core of research.”
×Fable 5.1“What a coding agent is good at is the part of research that was never really the bottleneck.”
×Sol 6.1“If you think their job is just to take a specification and write the code, then, sure, a coding agent could do more and more of that. But in research, figuring out the specification is often the hard part.”
No model’s answer contains the idea.
Asked about AI tools, the model gives the general take. It never mentions how the tools actually fail in use.
Typical answers miss the idea.
No model gets it.
Takeaways
1Understanding drives the decisionsand it is often what separates experts with good taste
2Standalone questions test it directlyno environment, no waiting years for a result
3Models sound right but stay generichedging, agreeing, talking around, textbook answers, no first-hand experience
Understanding is what drives decisions.
Standalone questions are a cheap, direct probe.
Models are fluent, but they miss the specific ideas experts have.
Thank you
tonychen.xyz/research-notes/taste-gap
Thanks. The full set of examples is in the research note.