T3 Stack creator Theo Browne says incorrect LLM outputs often cost more than correct ones due to repetitive token loops

Got some hard data - I was wrong.

Had Datacurve run the numbers for "tokens used by pass/fail" for DeepSWE.

Bad models use way more tokens in fail cases, but SOTA models are much closer. GPT 5.5 used ~7% MORE tokens on correct answers!

Some more cool visualizations from @PrunusSpeciosa_ at @datacurve

This bench is so cool man. I'm hyped to have actual useful data for measuring models doing realistic code work.

Update:

RL maturity issue undercooked models fall into flailing loops where they don't accumulate positive signal. Well-done models have a calibrated sense of progress towards the solution.

@theo Bad LLM runs are basically paid debugging sessions where the bug is the model.

@theo Oh boy you're gonna be shocked when you hear about this new thing called government contracting

@theo If you get stuck in the mud with your vehicle and you step on the gas spinning your tires, you are not refunded fuel. You learn how to get unstuck while conserving fuel.

How much of the rise of these LLMs doing thinking is actually just them wanting to increase token spend. And especially with these changes with Opus 4.7, and I've heard rumors that 4.8 is also this way, where they've increased how the tokens are being incurred because they changed the tokenizer. Like it all just seems to be, you know, a ploy to burn more compute.

@theo noticed the same thing while building bioinformatics agent. failed traces were literally 5x more expensive than successful ones

@theo ahhhh interesting. what do you think about Opus 4.8???

@theo Nothing about token economics is consumer-friendly. The providers also define quite directly the failure rate, the length of reasoning by default and if the model leaves out a few tiny annoying issues for just one more prompt.

This is a recipe for abuse.

@theo having a hard time following this exactly and what it means. is there a way to see the methodology on how this is calculated?

@theo hmm? what could be the reason? it turns out to be... opposite? but fewer iterations are required for better models no? Would love it if you dive deeper and made a video out of it

@theo agent loops ship with no exit condition by default. you need a hard token cap per task w/ an abort-and-report when it trips, else a stuck run quietly burns the meter until the invoice tells you

@theo This is why we say, thinking should not be billed to a customer. It's not the users problem, that the model needs to think, than they should make the model better.

@theo yes, but it still spends tokens on incorrect answers first; unless it literally one shots a correct answer.

@theo You’d have to tip them after each session then

@theo That’s exactly why any AI integrated system needs an observability layer.

You can’t fly blindly on a probabilistic system because it performed well a couple of times. When LLMs begin struggling internally, they usually consume more tokens, retry more often, increase latency, and

@theo This is a good analogy to compare with and in terms of service, if an llm screws up and realises above that the ans was not very good - it should have an option to sort of give a discount and return some of the tokens as compensation to the user.

@theo this is also true of deterministic software.

an infinite loop costs... infinitely more compute than a for loop.

a memory leak eventually crashes a server.

an accidentally deleted customer database is not a good day.