Northwestern's Zihan Wang introduces BAGEN, finding frontier LLM agents consistently fail to predict and manage their token budgets

Budget-aware Agents (BAGEN) study the failure modes in budget estimation:

1. Strong agents are not strong budget estimators.

2. Frontier models are often overoptimistic.

3. Budget awareness is actionable and trainable. SFT plus RL strengthens early stop and alert behavior, saving 28-64 percent of tokens on failed trajectories.

4. Upper and lower bound calibration remains hard.

https://ragen-ai.github.io/bagen/

models underestimate how much work it takes (token usage) to accomplish a task, just like us

Most real-world tasks run under a budget. Human agents know when to stop, ask for more, or change plans. But what about AI agents? Check out our new study on the budget awareness of AI agents👇

I’m BAGEN you to stop

Been struggling with this, my OpenClaw is supposed to be choosing which model to use for which task to limit unnecessary spend but I get the feeling, as is evidenced by this paper, that it does a poor job

@wzenus Cool stuff! You may want to cite our earlier work. https://arxiv.org/abs/2604.23897

@wzenus cool study. would be very helpful if they were great at estimating required token budget

Humans have the natural instinct to do constrained optimization based on the resources available, how about agents?

🏠 Project Page: http://ragen-ai.github.io/bagen 📄 Paper: http://ragen-ai.github.io/bagen/bagen.pdf 💻 Code: http://github.com/mll-lab-nu/BAGEN 📊 Data: http://huggingface.co/datasets/MLL-Lab/BAGEN

First of all, what is budget awareness? A budget-aware agent (BAGEN) doesn't just "spend less", but estimates remaining budget mid-task with uncertainty.

We formalize this with progressive interval estimation: 📌 can agents estimate budget mid-task? 📌 can agent show uncertainty with intervals [lo, hi]?

There is a lot more to be done with Budget-Aware Agents (BAGENs). After doing part of the work and realizing it can't finish within budget, a BAGEN should:

① Ask for more budget early ② Cut losses and switch to another task ③ Hand off to a stronger agent

These motivate lots of directions for future work!

What we found: 1/ Budget awareness ≠ task performance (r ≈ 0.35).

The best-performing model is NOT the best budget estimator.

On SWE-bench: Opus leads task success, Gemini leads feasibility F₁, GPT-5.2 leads interval coverage. Three different winners, three different capabilities.

@AndreyFradkin Great work. Thanks for your recommendation!

3/ Failure is recognized too late to act on.

Models predict "feasible" above 70% even after 60% of budget is consumed. The alarm fires only in the final 20%.

But good news is, stopping after impossible predictions saves 28–64% of wasted tokens at only 1.6–4.2% success cost.

2/ All models are universally too optimistic.

Most of 20 model-task pairs underestimate remaining budget. Weaker models are MORE optimistic.

The bias doesn't shrink with task progress.

4/ Budget awareness is trainable, but hits a ceiling.

SFT alone raises Qwen-7B feasibility 25.5% → ~90% (calibration problem ✅).

Interval coverage caps near 47% after SFT+RL. Half of intervals still miss the true remaining budget (reasoning problem ❌).

I found time estimation on coding especially overly conservative. I ask how long a refactoring would take than the agent says 1.5 days and we usually finish it within 1-2 hours.

I think there is some bias coming from the training data considering it could really take around 1.5 days if AI didn’t exist.

@yoheinakajima Yes! Budget-awareness would be a missing ability that people should hillclimb :)

@wzenus do they fail because they cannot price actions, or because they keep pretending the plan still works

@wzenus @grok explícame esto