T3 Stack creator Theo Browne says incorrect LLM outputs often cost more than correct ones due to repetitive token loops
Teortaxes blames the loop behavior on immature reinforcement learning.
RL maturity issue undercooked models fall into flailing loops where they don't accumulate positive signal. Well-done models have a calibrated sense of progress towards the solution.
Theo - t3.gg@theoGot some hard data - I was wrong. Had Datacurve run the numbers for "tokens used by pass/fail" for DeepSWE. Bad models use way more tokens in fail cases, but SOTA models are much closer. GPT 5.5 used ~7% MORE tokens on correct answers!
#1829Theo - t3.gg@THEOUpdate:
Theo - t3.gg@theoGot some hard data - I was wrong. Had Datacurve run the numbers for "tokens used by pass/fail" for DeepSWE. Bad models use way more tokens in fail cases, but SOTA models are much closer. GPT 5.5 used ~7% MORE tokens on correct answers!
#1829Theo - t3.gg@THEOGot some hard data - I was wrong.
Had Datacurve run the numbers for "tokens used by pass/fail" for DeepSWE.
Bad models use way more tokens in fail cases, but SOTA models are much closer. GPT 5.5 used ~7% MORE tokens on correct answers!
Theo - t3.gg@theoWeird thing about LLMs: "incorrect responses" are more expensive than correct ones. If I go to a restaurant and they screw up my food, they usually refund me and remake the meal. If an LLM gets stuck on a problem, it runs around in loops, burning tokens and costing money.
Some more cool visualizations from @PrunusSpeciosa_ at @datacurve
This bench is so cool man. I'm hyped to have actual useful data for measuring models doing realistic code work.
Theo - t3.gg@theoGot some hard data - I was wrong. Had Datacurve run the numbers for "tokens used by pass/fail" for DeepSWE. Bad models use way more tokens in fail cases, but SOTA models are much closer. GPT 5.5 used ~7% MORE tokens on correct answers!