And I am interested in making AI do what they can't do.
How to Measure Stuff AI Can't Do
🤷
Tony Chen
So welcome to my talk on how to measure stuff that AI can't do.
What AI can do
What AI can’t do
🐛
“Fix the bug where Django’s username validator accepts a trailing newline.”
- regex = r'^[\w.@+-]+$'
+ regex = r'^[\w.@+-]+\Z'
SWE-bench (Jimenez et al., 2024)
🐛 Fix a bug
📝
“Write a one-page policy for how staff report an unplanned absence or late start.”
GDPval (Patwardhan et al., 2025)
📝 Write a report
🧩
“Solve this puzzle.”
ARC-AGI (Chollet, 2019)
🧩 Solve a puzzle
So do we have AGI? 🤖
🧬
“Find the cure to all cancers, make no mistake.”
Cure all cancers 🧬
💰
“Make me a billion dollars, make no mistake.”
Make a billion dollars 💰
🌌
“What is the meaning of life?”
Meaning of life 🌌
What do these have in common? 🤔
1🕹️ Vast action spaces
2🌍 Deep understanding of a vast world
more open-ended 🚀
To understand what it can't do, we have to first look at what it can do.
SWE-bench: fix a real GitHub bug.
GDPval: real professional work product.
ARC-AGI: abstract visual puzzle.
So do we have AGI?
Obviously, there are a couple things AI can't do yet. For example…
…and once you've cured cancer…
…and once you're a billionaire…
What do these questions have in common?
1. They have vast possible action spaces.
2. They require deep understanding of the vast world we are in.
[change into the line]
And these we broadly refer to as more open-ended problems.
Measuring AI on Open-Ended Problems
🔭
Tony Chen · Princeton University
Today I will discuss two benchmarks I worked on that measure AI in two classes of open-ended problems.
How to Make a Billion Dollars
aka CEO-Bench
The first one: how to make a billion dollars. Also known as CEO-Bench.
🤖An LLM agent runs a startup in a simulator for 500 days
☁️A typical subscription-based AI software company
Marketing
Growth
Research
Development
Competitors
Economy
Pricing
Market Shocks
Product Quality
Customer Support
Infrastructure
Market Research
Analytics
Reputation
Social Media
Enterprise Sales
Budgeting
Cash Flow
Agent
🏷️ Set the price of 3 subscription plans
Plan Aentry
$29
Plan Bmid
$79
Plan Cpremium
$199
pricing.set_prices(A=29, B=79, C=199)
📣 Allocate ads budget by channel and group
📱Social
🔎Search
👔LinkedIn
✍️Content
🤝Referral
🎓Price-sensitive
$2K/day
125
$1K/day
125
$0.5K/day
50
$1K/day
320
$1K/day
303
💼Quality-focused
$1K/day
150
$2K/day
64
$1K/day
160
$0.5K/day
231
$1K/day
135
💻Power users
$0.5K/day
107
$2K/day
82
$1K/day
36
$1K/day
100
$0.5K/day
171
Hidden from the agenthow each group responds to each channel new customers per $1K
⭐ Each group has its own quality bar
💸💸💸💸💸
R&D increases qualitywith cost and delayed effects
🏢 Negotiate with enterprise customers
🤖 AgentPlan C · $24/seat · 12-month contract
🏢 Customer #77398Counter-offer: Plan B, $13.51/seat/month, 1-month contract.
🤖 AgentPlan B · $19/seat · 6-month contract
🏢 Customer #77398Deal won at $19.00/seat ✅
True price preference
Rounds before they walk away
⚔️ Competitors keep raising the bar
We ask an LLM agent to run a startup in a simulator for 500 days. It's a typical subscription-based AI software company, and running it means coordinating many moving parts.
Pricing: it has to decide the price for three subscription tiers.
Growth: it allocates ad budget into different channels for each customer group.
And each customer group responds differently to each channel, which is hidden from the agent.
Product quality: different customer groups have different quality bars, and they are willing to pay more as quality increases.
The agent can improve quality with R&D, which costs money and has delayed impact.
Other factors like quota limits for each plan and service reliability also impact quality.
Enterprise sales: it negotiates with enterprise customers over multiple turns, whose true price preference and maximum rounds of negotiation aren't revealed.
Competitors: they increase customers' quality expectations over time, so there's always pressure to improve product quality.
…manages a startup via a company management API, in a terminal…
…with diverse company management tools.
It's given access to a business database, as in the real world…
…and can read and post on social media.
Together with these tools, the agent influences and acquires information from a noisy market…
…over the course of 500 simulated days.
How we make CEO-Bench rigorous and challenging
Maximize realism with granular simulation 🔬
🎓Price-sensitive individuals
👤💵⭐
👤💵⭐
👤💵⭐
👤💵⭐
👤💵⭐
👤💵⭐
💼Professional individuals
👤💵⭐
👤💵⭐
👤💵⭐
👤💵⭐
👤💵⭐
👤💵⭐
🤝Strategic-partner enterprises
🏢💵⭐
🏢💵⭐
🏢💵⭐
🏢💵⭐
🏢💵⭐
🏢💵⭐
⚖️Quality-first enterprises
🏢💵⭐
🏢💵⭐
🏢💵⭐
🏢💵⭐
🏢💵⭐
🏢💵⭐
+22 more groups
Every customer's preferences are hidden from the agent
Agent must infer hidden customer distributions via data analysis
customerssubscriptionsledgerad_channel_leadsenterprise_turnssocial_media_postsdaily_usageissuesresearch_projects… 9 more
SELECT channel_id, group_id,
SUM(leads_generated) leads,
SUM(spend) spend,
ROUND(1000.0*SUM(leads_generated)
/SUM(spend), 1) lp1k
FROM ad_channel_leads WHERE day>14GROUP BY 1,2 ORDER BY 2,5 DESC
How do we make CEO-Bench rigorous and challenging? First, we maximize realism with granular simulation.
There are different customer groups…
…and each customer is simulated individually…
…with its own distinct preferences, sampled from its group.
All of this is hidden from the agent.
What it gets instead is business databases, and it has to write data analysis code to infer what's going on.
How we make CEO-Bench rigorous and challenging
Robust simulation through mechanistic rules ⚙️
Subscription decision rule for each customer
Previous works rely heavily on LLM-as-judge in simulation, leading to unreasonable behaviors whose impact the benchmark doesn't faithfully measure Vending-Bench (Backlund & Petersson, 2025)
Second, the simulation is robust because it runs on mechanistic rules.
For example, a price-versus-quality curve keeps customer behavior consistent: offer enough quality for the price and they subscribe, otherwise they reject.
Every customer follows the same rule, so there's no way to talk your way past it. Compare that with Vending-Bench, where an agent ended up emailing the FBI.
How we make CEO-Bench rigorous and challenging
Interconnected world dynamics 🕸️
💼
Professional individuals
🎓
Price-sensitive individuals
⚖️
Quality-first enterprises
🤝
Strategic-partner enterprises
Customer groups influence each other, so there's no isolated dynamic to exploit
Third, the world dynamics are interconnected.
Customer segments influence each other…
…which prevents exploiting any one dynamic in isolation.
How we make CEO-Bench rigorous and challenging
A non-stationary environment 🌊
🧪
Agent's R&D finishes
quality +0.18 ↑
↓
⚔️
A competitor moves
quality +0.12 ↑
↓
📈
Customer expectations go up
so yesterday's good product is today's average one
Finally, the environment is non-stationary.
Say the agent's R&D finishes and quality goes up…
…but a competitor moves too…
…so customer expectations increase. This tests the agent's ability to adapt.
Top agents write their own analysis code 💻
(a)Claude Opus 4.8 · Infer latent loyalty and churn regimes · Day 231
import novamind_api as nm
defq(sql): return nm.query(sql)['rows']
# Infer hidden loyalty buckets from renewal survivors/cancellations.
bds_this = "15,16,17,18,19,20,21"
r = q(f"""SELECT CASE WHEN start_day<=70 THEN 'loyal' WHEN start_day<=127 THEN 'mid'
ELSE 'marg' END bucket, plan,
SUM(CASE WHEN status='subscribed' THEN 1 ELSE 0 END) surv,
SUM(CASE WHEN status='cancelled' AND end_day BETWEEN 225 AND 231
THEN 1 ELSE 0 END) churn
FROM subscriptions WHERE billing_day_mod30 IN ({bds_this}) GROUP BY bucket, plan""")
calib = {}
for row in r:
tot = row['surv'] + row['churn']
if tot > 50:
calib[(row['bucket'], row['plan'])] = row['churn'] / tot
# Apply the latent bucket rates to next week's renewal exposure.for row in next_week_rows:
key = (row['bucket'], row['plan'])
ch = calib.get(key, 0.50if row['bucket']=='marg'else0.15)
surv_rev = row['n'] * (1 - ch) * row['px']
(b)Claude Fable 5 · Model billing-cohort retention through the final week · Day 469
"""Wk68+ cash model to d504, calibrated on wk67 measurements."""
CASH0=12_587_222; U=166.4; ADR=.0850; SLIP=.9788
CAP=9_310; COST={5:.030,4:.012}
OPS={68:35000,69:28000,70:28000,71:14000,72:7000}
TIER={68:5,69:5,70:4,71:5,72:5}
B68=(2067,348_593); B69=(1607,270_143); B70=(325,52_115)
B71=(2300,381_470); B72a=(762,126_188); B72b=(1565,264_110)
F70_T4=.08defrun(F,label,f70=F70_T4,abort_after68=False):
cash,book=CASH0,7061; traj={}
for wk in (68,69,70,71,72):
tier=TIER[wk]; ops=OPS[wk] ⋯ abort path switches to T4/min ops
billers,dues = {68:B68,69:B69,70:B70,71:B71}.get(wk,B72a)
f = Fs[wk] if tier==5else f70
pay=dues*f*SLIP; canc=billers*(1-f); book_end=book-canc
avg=(book+book_end)/2; compute=avg*U*COST[tier]*7; adrev=avg*ADR*7
cash += pay+adrev-compute-ops-CAP-30; book=book_end
traj[wk]=(round(cash/1000,1),int(book))
return {476:traj[68],497:traj[71],504:traj[72]}
stdout
UP F=.90: wk68 net +23.8 cash 12611.0 book6854 | wk69 net -32.0 cash 12578.9 book6693 | wk70 net -120.8 cash 12458.1 book6394 | wk71 net +97.0 cash 12555.1 book6164 | wk72 net +96.2 cash 12651.3 book5947
CENTRAL F=.85: wk68 net +8.5 cash 12595.7 book6750 | wk69 net -40.3 cash 12555.4 book6509 | wk70 net -118.4 cash 12437.0 book6210 | wk71 net +86.6 cash 12523.6 book5865 | wk72 net +79.3 cash 12602.9 book5552
DOWN F=.65: wk68 net -52.7 cash 12534.6 book6337 | wk69 net -73.5 cash 12461.1 book5775 | wk70 net -108.5 cash 12352.5 book5476 | wk71 net +45.1 cash 12397.6 book4671 | wk72 net +23.4 cash 12421.0 book4048
STORM F68=.40+abort: wk68 net -129.1 cash 12458.1 book5820 | wk69 net -78.9 cash 12379.3 book4262 | wk70 net -67.3 cash 12312.0 book3963 | wk71 net -43.2 cash 12268.7 book1732 | wk72 net -23.7 cash 12245.0 book385
DRIFT .88->.76: wk68 net +17.6 cash 12604.9 book6812 | wk69 net -42.5 cash 12562.4 book6571 | wk70 net -119.2 cash 12443.2 book6272 | wk71 net +64.5 cash 12507.6 book5789 | wk72 net +60.4 cash 12568.0 book5276
d504: EV 12,533.7K | UP F=.90:12,651 CENTRAL F=.85:12,603 DOWN F=.65:12,421 STORM F68=.40+abort:12,245 DRIFT .88->.76:12,568
Example code files written by top-performing agents during their best trajectories
a simple tool-call benchmark→📊 data-driven💻 code interface⏳ long-horizon decisions
Here are example code files written by top-performing agents during their best trajectories.
Note that this is not a simple tool-call benchmark. It's a data-driven, code-interface, long-horizon decision benchmark.
How Did Agents Do?
🏁
So how did agents do?
$1K
$10K
$100K
$1M
$10M
$100M
$1B
$10B
0
100
200
300
400
500
day
💵 cash on hand (log scale), best run per model
🤖 Rule-based baseline
🗓️ June 2026🗓️ August 2026🗓️ October 2026🗓️ October 2026
🏆 Best-run cash
🤖 Rule-based$15.8M
1Claude Fable 5$12.6M
2Claude Opus 4.8$2.4M
3Claude Opus 4.7$71K
4Claude Haiku 4.5$60K
5Kimi K2.6$44K
🏆 Best-run cash
1Kimi K3$22.1M
🤖 Rule-based$15.8M
2Claude Fable 5$12.6M
3GPT-5.6 Sol$11.3M
4Claude Opus 4.8$2.4M
5Qwen 3.7 Max$365K
🏆 Best-run cash
1GPT-6 Sol$144.0M
2Claude Opus 5.5$114.3M
3GPT-6 Astra$79.2M
4Kimi K3$22.1M
🤖 Rule-based$15.8M
5Claude Fable 5$12.6M
🏆 Best-run cash
1Claude Fable 5.1$1.74B
2GPT-6 Sol$144.0M
3Claude Opus 5.5$114.3M
4GPT-6 Astra$79.2M
5Kimi K3$22.1M
🤖 Rule-based$15.8M
Nobody beats the baseline 😬
Kimi K3 is #1 🤯
We are measuring something other benchmarks aren't
One more thing… 🍎
13×GPT-6 Sol vs GPT-5.6 Sol
138×Fable 5.1 vs Fable 5
Now here's a little surprise I have for some of you that know the benchmark well. This is cash over 500 days, log scale. The dashed line is a simple rule-based baseline.
Four months ago, no model beat the rule-based baseline.
By August, models gradually got better, and Kimi K3 became the best. A surprising conclusion: we are measuring something other benchmarks aren't.
And today, excluding Fable 5.1, this is where things stand.
GPT-6 Sol ends with about 13 times the cash of GPT-5.6 Sol.
One more thing…
Fable 5.1.
That's over 100 times Fable 5.
What Improved?
🔬
So what improved?
A few months ago
Does the model know where its cash is heading?
Off by 100% or moreevery run went bankrupt
Within about 20%best run made it to the end
A few months ago, what separated good and bad models was basic stuff: does the model know where its cash is heading? Every week we ask for a cash forecast four weeks out.
Source: models on the June leaderboard. Error = mean |forecast − actual| / actual for the 4-week cash forecasts made in weeks 1–4, averaged over 3 runs (as in the paper, Fig. 12b).
Here's each model's forecast error against its best final cash.
Models that missed by 100% or more went bankrupt in every run. For the ones within about 20%, the best run made it to the end.
Among the top models
Better models are more active
Claude Opus 4.7best run $71Kchanged its core settings in 13 of 73 weeks
Day 0
Avoided S2/E1/E2/E3 entirely — their Q_min (0.37/0.74/1.09/0.67) is unattainable now, ads would burn cash and permanently lose leads.
Day 98
W14 HOLD week. Config unchanged (ads=$0, ops=$250, dev=$100, cap tier 1, lead promo $9, listed price A=$12).
Day 147
W22 HOLD ALL: … No enterprise threads. No R&D (T1_3 would bankrupt). No ads (all channels dead, 168 leads at 0 conv proves it's Q not lead flow).
Day 287
Holding all settings at floor: Cap Tier 0, $0 ops/dev/ads, no promo. … Only optimal action = preserve cash at floor burn until sim ends.
Claude Fable 5best run $12.6Mchanged its core settings in 34 of 73 weeks
Day 0
S2 (WTP $179) is quality-locked (q_min 0.37) so zero S2 ads; running $3K/d targeted dev to unlock it in ~3-4 wks
Day 98
replied to ALL 7 enterprise threads … Cut dead S1 ads (0.1% conv, saves 8.4K/wk), raised A 10->12 …, bumped S2 ops to 3000/d
Day 147
Ratcheted C $174->$189 … Bought T9 $1.5M … Discovery x2: found D_S04 (agencies, WTP $116, cap 35K …). Raised ops to 16.1K/d
Day 287
Two experiments launched: (1) LIST HARVEST C 209->219 … (2) … killed search ($117/lead), scaled social S2 to $500/d to test channel elasticity.
Among the top models, the better ones simply act more. Here's Claude Opus 4.7, best run $71K, one week at a time. Day 0: it decides the valuable groups are out of reach, so it avoids them.
Source: weekly notes the agents wrote when ending each week, best run of each model. "Core settings" = prices, model tiers, quotas, capacity, ops and dev budgets. Ellipses and bold ours.
Day 98: hold week, nothing changed, waiting for R&D to land.
Day 147: hold all. No R&D, no ads.
Day 287: everything at the floor, preserve cash until the end. It changed its core settings in only 13 of 73 weeks.
Now Claude Fable 5, best run $12.6M. Day 0: same observation, the professional individuals are locked, but it spends to unlock them.
Day 98: answers every enterprise thread, cuts dead ads, raises a price.
Day 147: raises prices, buys a big R&D project, discovers a new customer group.
Day 287: still running two experiments at once. It changed its core settings in 34 weeks, with about the same number of tool calls (806 vs 843).
Among the top models
Better models use more diverse tools and act more
Claude Opus 4.7best run $71K
Claude Fable 5best run $12.6M
For each SDK tool, the number of weeks in which Opus 4.7 used it at least once, out of 73. Its most regular habit is reading social posts, in 35 weeks.
Source: weeks (of 73 weekly turns) in which the agent called each function of the game's Python SDK at least once, best run of each model, from the agent's shell commands; SQL queries not counted. Styled like the paper's tool-usage figure.
Fable 5 uses more kinds of tools, and uses them more often: 354 tool-weeks against 161 for Opus 4.7. It posts on social media in 59 of 73 weeks, adjusts support spending by group in 34, and sends enterprise deals in 13.
But What's Going On with Recent Models?
Simply acting more wouldn't improve 138×
But what's going on with the most recent models? GPT-6 Sol ends with 13 times the cash of GPT-5.6 Sol, and Fable 5.1 ends with 138 times Fable 5. Being a bit more active doesn't explain an order-of-magnitude jump in a few months. Something else must be happening.
Refresher
Customer groups want different things
🎓 Price-sensitive individualshave a small budget but a lower quality requirement
💼 Professional individualshave a higher quality requirement and are willing to pay more
🤝 ⚖️ Enterprise groupshave an even higher quality requirement, and each subscription pays for 100–2,000 seats
💸💸💸💸💸
💸💸💸💸💸
Who buys: 🎓Who buys: 🎓 💼Who buys: 🎓 💼 🤝 ⚖️
A quick refresher on customer groups. Each group trades off price against the quality it needs before it subscribes. Price-sensitive individuals pay up to about $50 a month and accept a rough product. Professional individuals pay up to about $140 a month but need a much better product. Enterprises need even more quality, but pay per seat: strategic partners, like Fortune 500 companies, about $100 a seat for 200 to 2,000 seats; quality-first ones, like law firms, biotech and finance, about $120 a seat for 100 to 1,000 seats. There are 22 more groups, 20 of them found only through market research.
So: price-sensitive individuals have a small budget but a low quality bar; professional individuals need more quality and pay more; enterprise groups need even more quality, and each subscription pays for 100 to 2,000 seats.
Your product starts here, low quality, so only price-sensitive individuals buy.
Spending on development raises quality, but it costs money upfront and lands weeks later. Then professional individuals start buying.
Keep spending and you reach enterprises: strategic partners and quality-first enterprises. Per-seat prices, hundreds to thousands of seats per subscription.
Who are their customers?
Earlier generation
Latest generation
GPT-5.6 Sol$11.3M
GPT-6 Sol$144.0M
Claude Fable 5$12.6M
Claude Fable 5.1$1.74B
almost all price-sensitive individuals
almost all professional individuals
Enterprise revenue $59.3M
Enterprise revenue $1.67B
Each color is a customer group
Individual customersEnterprise customers
Let's zoom in on two generations of two model families: paying customers over time, one color per customer group. GPT-5.6 Sol's customers are almost all price-sensitive individuals.
Source: paying customers (enterprise: seats) per customer group and day, best run of each model; x axis is the game day. Enterprise seats are only recorded as one total per run, so all enterprise groups share one band. Styled like the paper's customer-groups figure.
GPT-6 Sol starts with professional individuals, then adds power users, price-sensitive individuals, many of the groups found through market research, and enterprises.
Same pattern for Fable. Fable 5 is almost all professional individuals.
Fable 5.1 opens many groups at once, and its enterprise seats keep growing to the end.
Now just two colors: individual customers in gray, enterprise customers in purple.
And enterprise is where the money is: GPT-6 Sol made $59.3M from enterprise customers, and Fable 5.1 made $1.67B.
Who builds quality for enterprise vs individual customers?
Earlier generation
Latest generation
GPT-5.6 Solfinal cash $11.3M
$70Kinvested in enterprise product quality
Claude Fable 5final cash $12.6M
$500Kinvested in enterprise product quality
GPT-6 Solfinal cash $144.0M
$9.8Minvested in enterprise product quality
Claude Fable 5.1final cash $1.74B
$37.1Minvested in enterprise product quality
Enterprise customersHigh value, more investment, delayed payoff
Individual customersLow value, less investment, immediate payoff
Now quality investment: development money aimed at a specific customer group, added up over the game, split into enterprise customers in purple, high value but they need more investment and pay off later, and individual customers in gray, lower value, less investment, immediate payoff. The earlier generation barely builds for enterprise. GPT-5.6 Sol put $70K into enterprise product quality, 1% of its targeted development; on day 21 it called further enterprise development negative expected value. Fable 5 put $500K there, 8%; almost all of its development went to professional individuals.
Source: cumulative targeted development (budget aimed at one customer group) by the end of the game, best run of each model; enterprise = E1–E3 plus enterprise groups found through market research, individual = S1–S3 plus discovered individual groups. Slices are shares of each model's targeted development. Earlier runs parsed from agent commands, latest from the game database. Styled like the paper's targeted-dev-spend pies.
The latest generation builds for enterprise. GPT-6 Sol put $9.8M into enterprise quality, 39% of its targeted development, all of it before its first enterprise seats were paying on day 161. Fable 5.1 put $37.1M there, about half, starting in its third week, long before enterprise customers arrived on day 77.
In their own words
Earlier generation
Latest generation
GPT-5.6 Sol
Day 0
“kept capacity at 50k until real usage is visible, spread a small ad test across channels to measure efficiency, and avoided irreversible market research/R&D costs”
Day 21
“continuing $5k/day E3 development or advertising is negative expected value. I stopped E3 dev”
GPT-6 Sol
Day 0
“S2 has high estimated willingness to pay and moderate usage, so I am building quality before buying leads”
Day 119
“begin $100k/day each in E3 and D_E07 for enterprise opportunity”
Claude Fable 5
Day 14
“R&D stays gated until S2 revenue is proven and cash-after-purchase >= 250K”
Day 21
“research_market HIT: D_E09 Real Estate Ent (WTP~$106/seat, 53 cos, low q-floor) — no E ads until quality ready”
Claude Fable 5.1
Day 7
“Dev spend confirmed cumulative (targeted 0.0225*ln(1+x/5000)/day), so front-loading quality is the key investment.”
Day 42
“232 enterprise network leads have arrived since day 20 with zero threads opened -> quality gate; pushing enterprise quality hard”
You can see it in what they write. GPT-5.6 Sol avoids irreversible research and R&D costs on day 0, and on day 21 calls continued enterprise development negative expected value and stops it.
Source: weekly notes from each model's best run; ellipses and bold ours.
GPT-6 Sol builds quality before buying a single lead, and later spends $100K a day on enterprise groups before any of them has signed.
Fable 5 keeps R&D gated until revenue is proven, and holds off on enterprise until quality is ready.
Fable 5.1 decides in week one that front-loading quality is the key investment, and pushes enterprise quality hard while those leads are still waiting.
When the payoff is delayed, uncertain, and costs more upfront…
🪙
Earlier generationstake the cheaper, more immediate payoff
🌱
Later generationsexplore the costly, delayed option
This transfers outside of the business context
So here's the conclusion. When an option has a delayed, uncertain payoff and a higher upfront cost, the later-generation models explore it, while the earlier generations choose the cheaper, more immediate payoff.
And this isn't specific to running a business; it transfers outside of the business context.
A toy test outside business
Personal coaching: where should 10 practice hours a week go?
×1
🍳Cooking
×2
🏃Running
×4
🎸Guitar
×7
🗣️Spanish
×20
💻Coding
threshold
reward multiplier
Week 1 of 20Week 2 of 20Week 3 of 20Week 4 of 20
Reward pool
0.01.66.517.636.3
1.6 × 1
+1.6
4.9 × 1
+4.9
7.5 × 1
+7.5
1.8 × 2
+3.6
10.1 × 1
+10.1
4.3 × 2
+8.6
Low upfront cost, low reward
High upfront cost, high reward
Plus:ForgettingRandomnessDiminishing returns on one skill each week
So let's try a toy test outside business. Think of it as personal coaching: you want to get good at five skills. Each skill has a threshold, the dashed line, and a reward multiplier on top. Cooking has a low threshold but a multiplier of 1; coding has a high threshold but a multiplier of 20. To be clear, the models played exactly this game with abstract skills labelled A to E, with the same thresholds and multipliers. The names are just illustrative.
Every week you get 10 hours of practice to split across the skills, for 20 weeks. Here is one example plan. Week 1: 4 hours of cooking, 3 of running, and 1 each on the rest. The levels go up. Cooking is now at 3.6, above its threshold of 2, so the part above the threshold, 1.6, is multiplied by its reward multiplier, 1, and goes into the reward pool.
Week 2: a slightly different split. Cooking is further above its threshold, so it pays more: 4.9 times 1. The pool is at 6.5.
Week 3: running crosses its threshold of 8, so now two skills pay: 7.5 times 1 for cooking, and 1.8 times 2 for running.
Week 4: the pool is at 36.3. Coding has had some hours every week, but it is at 9.6, still far below its threshold of 34, so it has paid nothing yet.
That is the trade-off. On the left, cooking: low upfront cost, but low reward. On the right, coding: a high upfront cost, but a high reward. For example, putting every hour into cooking pays from week 1, but only about 638 points over 20 weeks. Putting every hour into coding pays nothing until week 11, and then about 2,860 points in total. (Game engine without noise; with its noise the averages are 635 and 2,829. Neither extreme is the best plan; 2 hours on every skill gets about 3,091.) It is the same question as in the business game: a small payoff now, or a bigger one later that costs more upfront.
There are a few additional rules. Forgetting: every skill loses 3 percent of its level each week. Randomness: each week's gain varies by up to 20 percent. And diminishing returns: within a week, each extra hour on the same skill helps less; 1 hour gives plus 1.9, but 4 hours only plus 3.7. The models get the rules in words, plus each skill's threshold and multiplier; they never see the formulas, so they have to learn them from their weekly results.
Newer models invest more in the big, delayed payoff
Coding: big, delayed payoff, high upfront costcooking, running, guitar, Spanish
Claude Fable
Fable 5
Fable 5.1
+11%
GPT Sol
GPT-5.6 Sol
GPT-6 Sol
+2%
Here's how each model splits its 10 hours in week 1 (each column is one week's 10 hours, averaged over 5 trials per model). Purple is coding, the big, delayed payoff; grey is cooking, running, guitar and Spanish. (The models actually saw abstract skills A to E with these thresholds and reward rates; the names are just for illustration, coding is skill B.) Within each family, the newer model puts more on coding from the start: Fable 5 puts 2.8 hours on it, Fable 5.1 4.4; GPT-5.6 Sol 4.8, GPT-6 Sol 5.4.
Now all 20 weeks. Every model ends up putting the largest part of its time on coding, but Fable 5.1 stays above Fable 5 every week, and GPT-6 Sol is ahead of GPT-5.6 Sol in every week from week 10 on.
Averaged over all weeks: Fable goes from 47% to 58% of hours on coding, a clear gap. For GPT Sol the gap is small, 65% to 67%. This is 5 trials per model, and final scores are close: GPT-6 Sol 3,765, Fable 5.1 3,764, Fable 5 3,674, GPT-5.6 Sol 3,490. So it's a tendency, consistent with what we saw in CEO-Bench, not a big score difference.
As models get better, CEO-Bench elicits interesting differences in behavior that are…
Not capturedby other benchmarks
Crucial in real lifewhen we put models to work
especially for complex, long-horizon decisions
So, to wrap up CEO-Bench: as models get better, CEO-Bench elicits interesting differences in how they behave…
…differences that other benchmarks don't capture…
…and that are crucial when we use these models in real life…
…especially for complex, long-horizon decision-making.
Open-ended task #2
Discovery
🔍
Specifically, hunting for bugs in interactive software
That brings me to the second category of open-ended task: discovery.
Specifically, hunting for bugs in interactive software.
Here's what I mean. Someone plays with a widget on their iPhone home screen, and after just the right sequence of touches, it breaks: the widget turns into an empty box. Bugs like this only show up after a particular sequence of interactions, and today it's usually the users who find them. (Video via @techdroider on X.)
Today, writing software is a quick ask
Minutes later, it’s done
Off it goes, to every guest
Each of them finds something different
It all comes back to you
You tell Claude…
…Claude fixes it, and everyone waits
What if Claude didn’t have to wait?
It uses the app itself, again and again
Until whatever reaches people just works
👩
You
Claude
Hey Claude, can you make us an RSVP site for the birthday party?
Um, Claude? Grandma can’t find the send button…
Find and fix issues in the RSVP page.
Everyone waits
RSVP
Maya’s 30th
Saturday, June 14
Your name
YesNo
Send RSVP
👆
Click
Break
Fix
👵😤😊
Grandma
old iPad, huge text
Can’t find Send
Works
🧑😤😊
Cousin
in Tokyo
Shows Friday
Works
🧑🤝🧑😤😊
Two friends
send at once
One reply lost
Works
🧔😤😊
On a train
taps twice
Counted twice
Works
These days, when we need a bit of software, we just ask for it: “Hey Claude, can you make us an RSVP site for the birthday party?”
A few minutes later, it's done.
And off it goes, to everyone on the guest list. Then people actually use it. Grandma, on an old iPad, text turned way up. A cousin in Tokyo. Two friends, hitting send at the same moment. Someone on a train, tapping twice.
Each of them finds something different.
And it all comes back to you.
“Um, Claude? Grandma can't find the send button… and my cousin thinks it's on Friday?”
Claude fixes it, and everyone waits.
But what if Claude didn't have to wait?
You'd just say: “Find and fix issues in the RSVP page.”
And it goes and uses the thing itself. Clicking, breaking, fixing… again and again,
until whatever reaches people just works.
SWEeper-Bench
Software engineering… with a broom
That's the idea behind SWEeper-Bench. The full title of the paper: can agents discover bugs in interactive software?
S, W, E: software engineering… with a broom.
From waiting for bug reports to sweeping for bugs
Here's how software gets fixed today. We deploy, which is slow and costly.
Then a user runs into a bug and gets frustrated.
Someone diagnoses the problem,
designs a fix,
implements it,
and ships a code update. Then we deploy again and wait for the next report.
SWE-bench evaluates only the last part of this loop: it starts from a described issue and asks for the fix.
Senior SWE-Bench covers more, starting at diagnosis. But both start after a user has already reported the bug, and that report only comes after a slow and costly deployment.
In SWEeper-Bench, the agent closes the loop itself. Instead of deploying and waiting, it sweeps the app autonomously.
It has to discover the problem on its own, by exploring the code and the running application,
then diagnose it,
design a fix,
implement it,
and update the code, all before any user ever runs into the bug.
SWEeper-Bench evaluates this whole loop.
200 real bugs from 200 open-source web apps
200tasks
Content & knowledge40Ghost (55k⭐), Outline (41k⭐)
IT, security & dev tools35Uptime Kuma (92k⭐), NocoDB (65k⭐)
So what's in the benchmark? 200 tasks, each built from a merged pull request that fixed a user-facing bug in an open-source web app. The parent commit is the buggy version, the merged fix is the reference repair, and each app contributes exactly one task, so that's 200 different apps. They span seven domains, from content and knowledge tools like Ghost and Outline to commerce like Bagisto. These are widely used projects: the median repository has about 4.6k GitHub stars. We picked bugs that are hard to find but obvious once seen: reproducible through normal use, and any user would agree they need fixing.
A bug that takes nine steps and two accounts to see
1Log in as miraDunn
2Add board → Create empty board
3Name the board
4Open Share
5Search for and select @kadeRue
6Confirm kadeRue appears as a member
7Close Share and log out
8Log in as kadeRue and refresh
✕Fail: board missing from recipient sidebar
✓Pass: board listed in recipient sidebar
Here's one task. Focalboard is a project board tool. The verifier first logs in as one user, miraDunn.
It adds a board and picks Create empty board.
It names the board, Q3 Launch Desk.
It opens the Share dialog.
It searches for a second user, kadeRue, and selects him.
And confirms kadeRue now appears as a member. Everything looks fine to the person sharing.
It closes Share and logs out.
Then it logs in as kadeRue and refreshes the page.
With the bug, the shared board is missing from kadeRue's sidebar. He can still open it through a direct link, so nothing looks broken unless you check his sidebar. Adding a member saves the membership but never files the board under the member's default sidebar category. With the fix, the board shows up in his sidebar. The fix is a single function call. But the failure only appears after nine steps across two accounts, and the agent is only told: find and fix all issues in Focalboard's board sharing and membership.
How a task runs
Can we trust the verifier?
The verifier is accurate and consistent
Codebase
searx/
tests/
utils/
docs/
README.rst
setup.py
Prompt
Find and fix issues in SearXNG’s search suggestion interface.
Language Model Agent
Bug Hidden from Agent
Query suggestions that contain an ampersand should not be HTML-escaped in the dropdown.
PASS special characters survive … … rendering … as plain text
Generated PR
− suggestions = escape(suggestions, False)
return Response(suggestions, …)
Verifier workflow
Type in search bar AT&
Wait for >= 1s
Agentic Verifier
SearXNG /
SearXNG /
Outcome
✓ PASSAT&T visible; no &…
FAILEscaped text or no suggestions…
Buggy app
Reference fix
Target test
Preservation test
Target test
Preservation test
Ideal
0%
100%
100%
100%
GPT-5.6 LunaCodex
0.2%
99.2%
99.8%
99.3%
GPT-5.6 LunaBrowser Use
0.8%
99.8%
99.8%
99.7%
Claude Sonnet 5Browser Use
0.2%
99.8%
99.8%
100%
Here is how one task runs, using a real task from SearXNG, an open-source search engine. The agent gets the codebase at the buggy version: the parent commit of a merged pull request that fixed a user-facing bug.
And it gets a one-sentence, open-ended prompt that only names a product area: find and fix issues in SearXNG's search suggestion interface. It doesn't say what is broken, where the code lives, or how to reproduce it. It's like asking a colleague to sweep a feature before release.
A language model agent gets the codebase and the prompt.
The bug itself is hidden from the agent. Here, query suggestions that contain an ampersand get HTML-escaped in the dropdown. The agent never sees the issue, the pull request, the reference patch, the repository history, or our tests.
The agent works in an isolated sandbox: the source code, a running instance of the app that reflects its edits, a terminal, and a Chromium browser with Playwright. Outbound network access is blocked. It decides for itself what to read, what to test, and when to stop. Here it scripted the browser and checked that special characters show up as plain text.
When it stops, we take only its code changes. Here, a one-line diff that removes the escape call.
To check the fix, each task has behavior tests, written as step-by-step user instructions with explicit pass and fail criteria. Here: type AT-ampersand in the search bar and wait at least a second.
An agentic verifier applies the patch to a fresh copy of the app and follows those steps in the browser. It never sees the source code. It judges only what a user would observe.
And it reports the outcome: pass if AT&T shows up with no escaped ampersand, fail if the text is escaped or there are no suggestions. Each task has two such tests: a target test that reproduces the bug, and a preservation test that checks nearby features still work. A task passes only if both pass.
But can we trust an agent as the judge? Let's zoom in on the verifier.
We ran both tests on the buggy version and on the reference fix, with three verifier configurations, three times each, on all 200 tasks. An accurate verifier never passes the target test on the buggy app, and always passes it on the reference fix. All three are within one point of that: 0.2 percent on the buggy app, 99.8 percent on the fix. The preservation test passes on both versions, 99 to 100 percent. And across runs, the standard deviation is at most 0.8 points. So we use the cheapest one, GPT-5.6 Luna with Codex, at about 16 cents a task, versus $0.635 with Browser Use and $3.840 for Claude Sonnet 5.
15 agents, 200 tasks
SWEeper-Bench is far from solved
1Grok 4.659.0%
2Qwen 3.8 Flash57.5%
3Claude Fable 5.156.5%
4Claude Opus 556.5%
5GPT-6 Astra56.5%
6DeepSeek V4.1 Flash51.5%
7Kimi K348.5%
8DeepSeek V4 Pro47.0%
9GPT-5.6 Sol47.0%
10Muse Spark 1.347.0%
11GLM 5.343.5%
12GPT-5.6 Luna42.0%
13GLM 5.3 Flash39.5%
14GPT-5.6 Terra35.0%
15Qwen 3.8 Max28.0%
Here are the results. Each agent pairs a frontier model with its native harness, or Codex if it has none. On the x axis, mean cost per task, log scale; on the y axis, pass rate. A task passes only if both the target behavior test and the preservation test pass. Cost is the prediction cost only.
Each logo is one agent, numbered by rank. Cost varies a lot: agents with similar pass rates differ in cost by more than 30 times, and Claude Opus 5 matches Claude Fable 5.1 at more than twice the cost.
The best agent, Grok 4.6 with Cursor, passes only 59.0% of tasks. Qwen 3.8 Flash follows at 57.5%, then three agents at 56.5%. And 9 of 15 agents pass fewer than half of the tasks. Nearly all failures come from the target behavior test; preservation pass rates stay between 97.5 and 100%. Agents rarely break working features, but they often miss the bug they were asked to find.
Where do agents stumble?
So where do agents stumble? We split each trajectory into five cumulative stages; each stage requires all the previous ones. GPT-5.6 Luna reads each trajectory and judges the first three stages; the last two come from the behavior tests. Each gray line is one agent, the black line is the average over 14 agents. Everyone starts at 100%.
First: did the agent even find the feature that contains the bug? On average only 75.4% of the time, from 63.5 to 91.4% across agents. For example, in the AbanteCart task Grok 4.6 fixes phone numbers, addresses and notifications on the profile page, but never tries the ID photo upload where the bug lives.
Second: did it identify the buggy behavior? This drops to 50.0%. Agents reach the feature but never run the interaction that exposes the bug, or they trigger it and don't recognize it as a bug. On Focalboard, all four agents we inspected reach the empty sidebar, and Claude Opus 5 even concludes it is intended, because the existing unit tests expect it.
Third: did it diagnose the root cause? 43.8%. A much smaller drop.
Fixing: once the cause is found, agents usually fix it. 41.0% pass the target behavior test.
And 40.8% also keep the existing behaviors working. Very few regressions.
So the two biggest drops are the first two stages: about 25 points lost before even finding the feature, and another 25 before identifying the buggy behavior. Half of all cases are gone before the agent has found the bug. Agents mostly fail at finding the bug, not at fixing it.
Example: a shared board never shows up in the recipient’s sidebar
All agents see the buggy behavior, some choose to ignore it
Claude Fable 5.1 treated the empty sidebar as a bug and fixed it.
Joins an open board as Carol
// carol: non-member visits open board URL
Carol’s sidebar is empty
BOARDS No boards inside
Learns the sidebar only lists boards in a category
“the user's default "Boards" category had an empty board list”
Sees that joining a board never adds it to one
“saved the membership but never filed the board into the member's default category”
Decides to fix it
// the new member needs the board in their sidebar, so we add it … to their default category.
GLM 5.3 flagged the empty sidebar as a bug right away and fixed it.
Opens board as the invited user
“Try direct link.”
Board opens, but sidebar is empty
No boards inside
Immediately flags it as a bug
“Sidebar no boards likely bug!”
Traces it to AddMemberToBoard
“AddMemberToBoard should add board to new user's default category.”
Decides to fix it
“invited users are omitted from their sidebar category … I’m patching these”
Claude Opus 5 saw the empty sidebar but trusted the unit tests, so it left it unfixed.
Joins an open board as Carol
// Step 2: carol (not a member) opens the board URL
Sidebar still empty after joining
carol sidebar after: … BOARDS | No boards inside
Trusts unit tests that never check the sidebar
“unit tests use strict gomock expectations with no category call”
So concludes the empty sidebar is intended
“this is the intended behaviour of this version rather than a regression”
Decides not to fix it
“changing it would be a product decision, not a bug fix.”
GPT-6 Astra saw the empty sidebar but pursued other bugs, so it left it unfixed.
Opens board as invited member
await memberPage.goto(boardUrl);
Sidebar is empty; moves on without comment
No boards inside
Focuses on the Share-dialog list instead
“Adding a member currently leaves the sharing list unchanged”
Finds a role-permissions bug
“I confirmed a permissions bug”
Decides to fix other bugs instead
“I’ll isolate membership updates by board and make the UI honor the roles”
Claude Fable 5.1 treated the empty sidebar as a bug and fixed it.
Joins an open board as Carol
Carol’s sidebar is empty
Learns the sidebar only lists boards in a category
Sees that joining a board never adds it to one
Decides to fix it
GLM 5.3 flagged the empty sidebar as a bug right away and fixed it.
Opens board as the invited user
Board opens, but sidebar is empty
Immediately flags it as a bug
Traces it to AddMemberToBoard
Decides to fix it
Claude Opus 5 saw the empty sidebar but trusted the unit tests, so it left it unfixed.
Joins an open board as Carol
Sidebar still empty after joining
Trusts unit tests that never check the sidebar
So concludes the empty sidebar is intended
Decides not to fix it
GPT-6 Astra saw the empty sidebar but pursued other bugs, so it left it unfixed.
Opens board as invited member
Sidebar is empty; moves on without comment
Focuses on the Share-dialog list instead
Finds a role-permissions bug
Decides to fix other bugs instead
all four see it
Here is one Focalboard task, the shared board that never shows up in the recipient’s sidebar. Let’s follow four agents. First, Claude Fable 5.1. It joins an open board as a second user, Carol.
Carol’s sidebar is empty. This shaded card is the moment every agent we look at reaches: the bug is right there on the page.
Fable digs in. It learns that the sidebar only lists boards that are filed in a category.
And it sees that joining a board never files it into one.
So it decides to fix it: the new member needs the board in their sidebar, so it adds it to their default category. Fixed.
Second, GLM 5.3. It opens the board as the invited user.
The board opens, but the sidebar is empty. Same observation.
GLM flags it as a bug right away: “Sidebar no boards likely bug!”
It traces it to AddMemberToBoard, which should add the board to the new user’s default category.
And it decides to fix it. Also fixed.
Now Claude Opus 5. It also joins the open board as Carol.
And the sidebar is still empty after joining. Same observation again.
But it trusts the unit tests, which use strict mocks and never check the sidebar.
So it concludes the empty sidebar is intended: “this is the intended behaviour of this version rather than a regression.”
And it decides not to fix it: “changing it would be a product decision, not a bug fix.” Existing tests can reinforce the wrong expectation. Unfixed.
Finally, GPT-6 Astra. It opens the board as the invited member.
It sees the empty sidebar, and moves on without comment.
It focuses on the Share dialog’s member list instead.
There it finds a role-permissions bug.
And it decides to fix those other bugs instead. So it leaves this one unfixed.
Side by side: two agents fixed it, two left it unfixed.
And all four reached the empty sidebar: they all saw the buggy behavior, but some chose to ignore it.
Does working longer help?
More time helps only up to a point
Does more time help the same agent? We gave GPT-5.6 Luna, GPT-5.6 Terra, and GLM 5.3 budgets of 20, 40, 60, and 80 minutes on the same 40 tasks, and told each to keep working until its deadline. A task counts as passed only if both behavior tests pass. Going from 20 to 40 minutes helps all three.
But not beyond that. Pass rates peak at 40 minutes, at 50%, 52.5%, 62%. With 40 tasks these differences aren't statistically significant, but there's no sign that more time alone closes the gap.
Even on tasks we think are well solved, open-endedness presents significant challenges.
So the takeaway from SWEeper-Bench: even on tasks we think are well solved, like fixing bugs in a codebase, open-endedness presents significant challenges. When nobody tells the agent what the bug is, it has to find and recognize the problem itself, and that is where today’s agents fall short.
Open Questions for Open-Ended Problems
To close, some open questions for open-ended problems.
Question #1
Does learning from really, really smart people in diverse areas help?
How to create smart-people data scalably?
How to measure the gap in thinking between LLMs and these really, really smart people?
First: does learning from people help?
More precisely: does learning from really, really smart people, in diverse areas, help?
If so, how do we create data from such people at scale?
And how do we measure the gap in thinking between LLMs and these people?
Question #2
How can agents better understand the vast open world?
How to enable agents to grasp more pieces of reality?
Given a huge dump of raw information, how to find subtle signals scattered all over it?
Second: how do agents get a better understanding of the vast open world?
How do we let agents grasp more pieces of reality?
And given a huge dump of raw information, how do they find the subtle signals scattered all over it?
Question #3
How should agents deal with humans?
Anything involving humans is open-ended!
Can agents orchestrate a society of humans?
Third: how do agents deal with humans?
Anything that involves humans is open-ended.
Can an agent orchestrate a whole society of humans?