/Tech3h ago

UC Berkeley’s Dawn Song launches Agents’ Last Exam, finding frontier agents score 0% on complex professional tasks

The dataset features 1,500 tasks across 55 professional occupations.

445797816062.6K
Dawn Song
Noam Brown#33
Original post
Dawn Song@dawnsongtweets#274inTech

Everyone says the latest AI agents will be "job-ready" soon, especially after the release of Fable 5 this week. But is that really the case?

Over the past many months, my group and collaborators have been building Agents' Last Exam (ALE), a benchmark designed to test exactly that claim on real digital labor-market work.

My group and collaborators previously have created many of the benchmarks the field runs on, including MMLU, MATH, CyberGym, and ExploitGym. Today, I'm excited to share Agents' Last Exam (ALE): a rolling benchmark that measures whether AI agents can actually perform economically valuable work across a broad range of real-world domains.

With ALE, we evaluated Fable 5, GPT-5.5, Composer 2.5, and other frontier agent systems across more than 1,500 expert-sourced tasks spanning 55 occupations. The result is both impressive and sobering.

Today's agents can solve a meaningful fraction of professional tasks. But when we look at the hardest tasks, the ones requiring sustained reasoning, deep domain expertise, and reliable execution over long horizons, they are still far from human-level performance.

On ALE's hardest tier, every frontier agent we tested, including Fable 5, achieved a 0% success rate. The age of useful agents is here.

The age of truly job-ready agents is not.

We hope Agents' Last Exam (ALE) will serve as a new guidepost and north star for developing agents capable of reliably performing economically valuable work across a broad range of domains. 🧵

8:36 AM · Jun 11, 2026 · 31.7K Views
Sentiment

Positive users praise the ALE benchmark for its focus on efficiency metrics and reassuring facts about AI limits, while negative users call it misleading for ignoring messy real-world production conditions.

Pos
80.0%
Neg
20.0%
12 comments with sentiment.
Cluster Engagement
Posts from X
Most Activity
Most Activity
VIEWS20.7KBOOKMARKS42LIKES243RETWEETS12REPLIES9
Noam Brown@polynoamial

I'm happy GPT-5.5 tops this eval

I'm even happier it's still doing the best when measured vs tokens, cost, or wall-clock time!

Dawn Song@dawnsongtweets

Everyone says the latest AI agents will be "job-ready" soon, especially after the release of Fable 5 this week. But is that really the case?

Over the past many months, my group and collaborators have been building Agents' Last Exam (ALE), a benchmark designed to test exactly that claim on real digital labor-market work.

My group and collaborators previously have created many of the benchmarks the field runs on, including MMLU, MATH, CyberGym, and ExploitGym. Today, I'm excited to share Agents' Last Exam (ALE): a rolling benchmark that measures whether AI agents can actually perform economically valuable work across a broad range of real-world domains.

With ALE, we evaluated Fable 5, GPT-5.5, Composer 2.5, and other frontier agent systems across more than 1,500 expert-sourced tasks spanning 55 occupations. The result is both impressive and sobering.

Today's agents can solve a meaningful fraction of professional tasks. But when we look at the hardest tasks, the ones requiring sustained reasoning, deep domain expertise, and reliable execution over long horizons, they are still far from human-level performance.

On ALE's hardest tier, every frontier agent we tested, including Fable 5, achieved a 0% success rate. The age of useful agents is here.

The age of truly job-ready agents is not.

We hope Agents' Last Exam (ALE) will serve as a new guidepost and north star for developing agents capable of reliably performing economically valuable work across a broad range of domains. 🧵

1hViews 20.7KLikes 243Bookmarks 42
Dawn Song@dawnsongtweets

ALE is built from real work, not synthetic tasks. Every task is derived from a real project that a human expert previously completed, and converted into a verifiable evaluation with objective grading.

No vibes. No human judges. Fully reproducible.

ALE spans 55 non-physical occupations, grounded in the O*NET / SOC 2018, the U.S. federal occupation taxonomy.

Built with 300+ experts from 100+ institutions across science, engineering, medicine, law, finance, education, and many other fields.

Dawn Song@dawnsongtweets

Everyone says the latest AI agents will be "job-ready" soon, especially after the release of Fable 5 this week. But is that really the case?

Over the past many months, my group and collaborators have been building Agents' Last Exam (ALE), a benchmark designed to test exactly that claim on real digital labor-market work.

My group and collaborators previously have created many of the benchmarks the field runs on, including MMLU, MATH, CyberGym, and ExploitGym. Today, I'm excited to share Agents' Last Exam (ALE): a rolling benchmark that measures whether AI agents can actually perform economically valuable work across a broad range of real-world domains.

With ALE, we evaluated Fable 5, GPT-5.5, Composer 2.5, and other frontier agent systems across more than 1,500 expert-sourced tasks spanning 55 occupations. The result is both impressive and sobering.

Today's agents can solve a meaningful fraction of professional tasks. But when we look at the hardest tasks, the ones requiring sustained reasoning, deep domain expertise, and reliable execution over long horizons, they are still far from human-level performance.

On ALE's hardest tier, every frontier agent we tested, including Fable 5, achieved a 0% success rate. The age of useful agents is here.

The age of truly job-ready agents is not.

We hope Agents' Last Exam (ALE) will serve as a new guidepost and north star for developing agents capable of reliably performing economically valuable work across a broad range of domains. 🧵

3hViews 1.4KLikes 27Bookmarks 2
Dawn Song@dawnsongtweets

Why do ALE's results look different from some other benchmarks, especially for Fable 5?

Because there is no universally best agent.

Every frontier model, including Fable 5, has domains where it shines and domains where it struggles.

Aggregate scores average over 55 occupations and 1,500+ tasks, causing many models to cluster together. But the average is not the story.

The real signal lies in where agents succeed, where they fail, and how those patterns differ across domains. On identical tasks, different models often fail for very different reasons.

Explore the interactive breakdown in our blog → 👉 https://agents-last-exam.org/blogs/agent-showdown

Dawn Song@dawnsongtweets

In ALE, Fable 5 joins GPT-5.5 and Composer 2.5 in the same overall performance cluster.

But performance is only half the story.

Cost per task: → Fable 5: ~$15.70 → GPT-5.5: ~$3.80 → Composer 2.5: ~$1.33

At current pricing, Fable 5 delivers similar performance while costing roughly 4–12× more per completed task.

3hViews 615Likes 13Bookmarks 2
Dawn Song@dawnsongtweets

In ALE, Fable 5 joins GPT-5.5 and Composer 2.5 in the same overall performance cluster.

But performance is only half the story.

Cost per task: → Fable 5: ~$15.70 → GPT-5.5: ~$3.80 → Composer 2.5: ~$1.33

At current pricing, Fable 5 delivers similar performance while costing roughly 4–12× more per completed task.

Dawn Song@dawnsongtweets

ALE-CLI is a CLI-only subset of ALE. Compared to Terminal-Bench and SWE-bench-Pro, it is broader, longer-horizon, and substantially more challenging:

• Broader. Tasks span 40 of ALE's 55 industry subdomains, compared to just 6 in Terminal-Bench and 5 in SWE-bench-Pro.

• Longer-horizon. Human completion times range from hours to weeks, rather than minutes to days.

• Harder. The best-performing agent achieves only a 25.2% pass rate, compared to 82.0% on Terminal-Bench and 59.1% on SWE-bench-Pro.

There's still a long way to go, and plenty of headroom left to climb. 📊👇

3hViews 541Likes 12Bookmarks 1
Dawn Song@dawnsongtweets

How does ALE compare to existing agent benchmarks? Many of today's agent benchmarks are rapidly saturating as frontier systems improve.

ALE is designed to measure a different capability frontier: sustained, economically valuable work in real-world professional domains.

• 55 industry domains • 1,500+ expert-sourced tasks • Full GUI + CLI environments • Outcome-based, verifiable evaluation

If your agent only operates in the terminal, we've also released ALE-CLI: a CLI-only subset of the benchmark.

Dawn Song@dawnsongtweets

ALE is built from real work, not synthetic tasks. Every task is derived from a real project that a human expert previously completed, and converted into a verifiable evaluation with objective grading.

No vibes. No human judges. Fully reproducible.

ALE spans 55 non-physical occupations, grounded in the O*NET / SOC 2018, the U.S. federal occupation taxonomy.

Built with 300+ experts from 100+ institutions across science, engineering, medicine, law, finance, education, and many other fields.

3hViews 914Likes 15Bookmarks 0
Dawn Song@dawnsongtweets

Why "Last Exam"? The name has two meanings: "Last" as the bar to clear：passing these exams means an agent can actually do the job and continue to deliver economically-valuable work in that profession.

"Last" as the frontier of difficulty：tasks are real, complex, long-horizon, and require professional expertise to execute. ALE sits right at the edge of what today's agents can reliably accomplish.

Come test your agent on ALE → Website: https://agents-last-exam.org Tasks: https://agents-last-exam.org/demo Leaderboard: https://agents-last-exam.org/leaderboard Paper: https://arxiv.org/abs/2606.05405 Dataset: https://huggingface.co/datasets/agents-last-exam/agents-last-exam Code: https://github.com/rdi-berkeley/agents-last-exam

Dawn Song@dawnsongtweets

The most common failure mode remains a familiar one: Agents declare success before they've truly verified their work.

A typical completion reads: "Done. All checks pass." Yet the output may be missing required files, contain incorrect counts, omit key fields, or violate explicit constraints in the task specification.

These failures occur far more often than many people expect. You can explore concrete examples in https://agents-last-exam.org/blogs/agent-showdown.

3hViews 581Likes 9Bookmarks 1
Dawn Song@dawnsongtweets

The most common failure mode remains a familiar one: Agents declare success before they've truly verified their work.

A typical completion reads: "Done. All checks pass." Yet the output may be missing required files, contain incorrect counts, omit key fields, or violate explicit constraints in the task specification.

These failures occur far more often than many people expect. You can explore concrete examples in https://agents-last-exam.org/blogs/agent-showdown.

Dawn Song@dawnsongtweets

Why do ALE's results look different from some other benchmarks, especially for Fable 5?

Because there is no universally best agent.

Every frontier model, including Fable 5, has domains where it shines and domains where it struggles.

Aggregate scores average over 55 occupations and 1,500+ tasks, causing many models to cluster together. But the average is not the story.

The real signal lies in where agents succeed, where they fail, and how those patterns differ across domains. On identical tasks, different models often fail for very different reasons.

Explore the interactive breakdown in our blog → 👉 https://agents-last-exam.org/blogs/agent-showdown

3hViews 426Likes 9Bookmarks 1
Dawn Song@dawnsongtweets

ALE-CLI is a CLI-only subset of ALE. Compared to Terminal-Bench and SWE-bench-Pro, it is broader, longer-horizon, and substantially more challenging:

• Broader. Tasks span 40 of ALE's 55 industry subdomains, compared to just 6 in Terminal-Bench and 5 in SWE-bench-Pro.

• Longer-horizon. Human completion times range from hours to weeks, rather than minutes to days.

• Harder. The best-performing agent achieves only a 25.2% pass rate, compared to 82.0% on Terminal-Bench and 59.1% on SWE-bench-Pro.

There's still a long way to go, and plenty of headroom left to climb. 📊👇

Dawn Song@dawnsongtweets

How does ALE compare to existing agent benchmarks? Many of today's agent benchmarks are rapidly saturating as frontier systems improve.

ALE is designed to measure a different capability frontier: sustained, economically valuable work in real-world professional domains.

• 55 industry domains • 1,500+ expert-sourced tasks • Full GUI + CLI environments • Outcome-based, verifiable evaluation

If your agent only operates in the terminal, we've also released ALE-CLI: a CLI-only subset of the benchmark.

3hViews 536Likes 13Bookmarks 0
Brad Spahn@BradSpahn

@dawnsongtweets Fable 5’s intentional underperformance in Life Sciences is probably bringing down some of the aggregate scores though, right?

1hViews 148Likes 7Bookmarks 1
Dawn Song@dawnsongtweets

ALE is truly a community effort.

Huge thanks to a distinguished advisory committee guiding our industry landscape and task collection: @gallantlab, @thg_lab, Tarek Zohdi, Carl Boettiger & @ksteinfe (@UCBerkeley) Laure Zanna, @kaanozbay (@nyuniversity) George Em Karniadakis (@BrownUniversity) Tapio Schneider (@Caltech) @Idasim (@UCSF) Arvind Rao (@UMich) @yannakakis (@UMmalta) Patrick Bryant (@scilifelab) @yaminirangan (@HubSpot) @brad_rothenberg (@nTopology)

We are also deeply grateful to @BerkeleyRDI, RDI Foundation, @ChenInstitute, @UniPat_AI, @SnorkelAI (Open Benchmarks Grants program) for their support.

A huge thank you as well to our incredible organizing and execution team, and to all of the experts and contributors who donated their time, expertise, and real-world projects to make ALE possible.

This simply would not have happened without you.

Dawn Song@dawnsongtweets

Why "Last Exam"? The name has two meanings: "Last" as the bar to clear：passing these exams means an agent can actually do the job and continue to deliver economically-valuable work in that profession.

"Last" as the frontier of difficulty：tasks are real, complex, long-horizon, and require professional expertise to execute. ALE sits right at the edge of what today's agents can reliably accomplish.

Come test your agent on ALE → Website: https://agents-last-exam.org Tasks: https://agents-last-exam.org/demo Leaderboard: https://agents-last-exam.org/leaderboard Paper: https://arxiv.org/abs/2606.05405 Dataset: https://huggingface.co/datasets/agents-last-exam/agents-last-exam Code: https://github.com/rdi-berkeley/agents-last-exam

3hViews 544Likes 7Bookmarks 1
Robert Yang@GuangyuRobert

@dawnsongtweets I feel like lots of these tasks are forcing agents to operate traditional software. What's the rationale behind it?

55mViews 280Likes 3
G, MD@DrBeavisAI

@polynoamial We expect OpenAI to also have the best & largest models, some are willing to pay more :-) Do accelerate faster please

1hViews 222Likes 1
itchy@itchy_est

@polynoamial Can you make it top this one?

1hViews 219
the law of non-contradiction@1nterchangeable

@Dan_Jeffries1 This kind of thing is actually reassuring to me, because we're clearly seeing diminishing returns at this point despite a massive increase in parameter counts and token costs.

57mViews 1
Patty@Patty_H93

@dawnsongtweets

1hViews 153
Jad@Egbariaj

@polynoamial 1 benchmark from 30 benchmark doesn't help . we want smarter model . smarter than Fable .

1hViews 150
the.PM@thePM_001

@dawnsongtweets Try with AEP (Agent Element Protocol) correctly implemented.

45mViews 43Likes 1
Strata@ChainZenit

@polynoamial that is some serious efficiency, love to see it

1hViews 98
Hira@Hiraweb3

@dawnsongtweets facts over vibes, finally

3hViews 39
chainoflack@chainoflack

@dawnsongtweets There "real" tasks are a fraction of the real everyday job for most people. Real tasks are way messier, and this benchmark is deeply missleading. There is no way someone with good faith will think that composer 2.5 is similar to gpt 5.5 and not to mention even Fable...

10mViews 9
Load more posts