Agents' Last Exam Benchmark Tests AI Agents On 1,000+ Economic Tasks

@dair_ai I think that (2.6% pass rate) is a sobering but incredibly useful data point.

Of course the question is what the human pass rate is.

We know (ok I’m assuming knowing humans) it’s not 100% but it matters if it’s -20% or +80%.

@dair_ai 2.6% pass rate just means we're still miles away from autonomous yield farmers.

@dair_ai kind of wild that only 2.6% pass the hardest tier

wonder how much of that is just orchestration failing

@dair_ai Tier list benchmarks are cute but 2.6% on real-world tasks feels more like a filter than a metric

what happens when models train on the test set?

@dair_ai industry mapping is the interesting part here. the taxonomy approach might actually filter out lab-designed gotchas.

@dair_ai 2.6% makes sense - real work is genuinely hard. But how much is agent design versus the test being too strict? If the test is too rigid, agents optimize for passing it instead of solving actual problems.