Jeff Dean and Andy Konwinski launch ALE, an agent benchmark finding frontier models score 0% on complex professional tasks

@dawnsongtweets Whoever is saying agents will be ready to take jobs soon is dreaming, the harness and the model are no where near being able to complete non structured tasks autonomously maybe in this generation but not anytime soon

@GuangyuRobert @dawnsongtweets From the tasks I've seen it seems like they're presented with industry standard tools in the descriptions.

We have decades of (software) tools that people use daily, and you can't really put a HTTP server and JSON codec in all of them and call it MCP.

@dawnsongtweets The fact that ALE-CLI is already harder than SWE-bench Pro and Terminal-Bench says everything. Great to see frontier models doing meaningful work on mid-tier tasks, but the long-horizon expert tier is clearly where the real challenge lies.

@dawnsongtweets Is the benchmark open source? Id like to give a crack at applying my private Harness/Environment/Orchestration engineering systems toward it.

Confident that, with a few iterations, today’s SOTA models may be able to hit 90-100%.

@dawnsongtweets I think your approach is wrong. We don’t need AGI-like agents. Agents good at specific jobs already exist, they are already working, and they will only improve

@dawnsongtweets but job readiness has nothing to do with human competence, it's about automatable tasks

@dawnsongtweets How many of the tasks did Fable 5 refuse to do? (fwiw, marking a refusal as zero is the correct methodology, but the lower score would be surprising otherwise)

@Tigger0000 @dawnsongtweets @grok composer 2.5 is the ai built by cursor in partnership with Xai from elon musk to build a new set of kimi k2.6 base models but further trained on data from cursor on the datacenter for grok

@dawnsongtweets Thanks for this test , proves my point that humans are better long term.

@BradSpahn @dawnsongtweets "We only look bad because we throttled ourselves" is an excuse any vendor could use for any benchmark. We wrote up exactly where Fable 5 falls short (mostly not following simple instructions + calling unverified work "done"): https://agents-last-exam.org/blogs/agent-showdown.

@Tigger0000 @dawnsongtweets @grok no idea why grok didnt reply to you bro rip

@JeffSte17327059 @dawnsongtweets @grok interesting, there's much more happening with grok than i had any idea about until recently :D thanks!

@dawnsongtweets @scaling01 where is the comparison with human on the charts?

@dawnsongtweets We’ve been wanting to try a test like this for a while. Can we?

@dawnsongtweets Hi Dawn... I just LinkedIn with you and tried your mail at your jobs email address, I might like to write this story up. Can you connect on LinkedIn or reply on email? Adrian

@dawnsongtweets

@dawnsongtweets amazing work!

working on getting to 100% on similar benchmark from day 1.

@dawnsongtweets Great! But does it have Opus 4.6 results? Fable 5 was a waste of money and time, only receiving some downgrade responses, like a scam.

@dawnsongtweets