Anthropic evaluated the model's tool capabilities using Zapier AutomationBench
Somebody high up at Anthropic went to Michigan
Fable 5 is state-of-the-art on nearly all tested benchmarks, with exceptional performance in software engineering, knowledge work, scientific research, and vision.
The longer and more complex the task, the larger Fable 5’s lead over our other models.
Some mindblowing SWE-bench numbers from Anthropic's Fable 5 release.
Also super excited about how Anthropic is using ProgramBench to explore different multi-coding agent set ups. PB is a far more effective discriminator than SWE-bench for this.
Fable 5 is state-of-the-art on nearly all tested benchmarks, with exceptional performance in software engineering, knowledge work, scientific research, and vision.
The longer and more complex the task, the larger Fable 5’s lead over our other models.
Mythos/Fable 5 outperforms Opus 4.8 with Multiagents on Programbench. Single agent baseline seems to be only slightly higher. This is from the Anthropic system card.
Zapier AutomationBench being used to report Tool Use performance on Fable 5's model card
Fable 5 is state-of-the-art on nearly all tested benchmarks, with exceptional performance in software engineering, knowledge work, scientific research, and vision.
The longer and more complex the task, the larger Fable 5’s lead over our other models.
Source: https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c342ee809620.pdf
Look out for open source SWE-bench multimodal and ProgramBench leaderboard, coming soon!
Some mindblowing SWE-bench numbers from Anthropic's Fable 5 release.
Also super excited about how Anthropic is using ProgramBench to explore different multi-coding agent set ups. PB is a far more effective discriminator than SWE-bench for this.
@KLieret @jyangballin @OfirPress @parth007_96 @18jeffreyma Am I correctly concluding from this that the bleeding edge of codegen models still cannot reconstruct a single one of these program perfectly (<1% resolved)? How much do we think the 1M token limit is capping the performance?
@jyangballin @OfirPress @parth007_96 @18jeffreyma ProgramBench is a joint effort across Meta FAIR, Meta TBD, Stanford, Harvard
@jyangballin (co-first author)@18jeffreyma @parth007_96 @dpedch @sten_sootla @micmylin @pengchengyin @magpie_rayhou @syhw @Diyi_Yang @OfirPress
@davidlee ??
@mushroomify @jyangballin @OfirPress @parth007_96 @18jeffreyma No, unfortunately they don't use the official metric (% resolved instances), so there's no way of knowing how many instances they solved. It seems like Opus 4.8 is capable of fully solving 2 instances (1%). Seems like we'll have to run Mythos ourselves to find out
@jyangballin @OfirPress @parth007_96 @18jeffreyma Source: https://www-cdn.anthropic.com/d00db56fa754a1b115b6dd7cb2e3c342ee809620.pdf
Anthropic evaluated the model's tool capabilities using Zapier AutomationBench
Somebody high up at Anthropic went to Michigan
Fable 5 is state-of-the-art on nearly all tested benchmarks, with exceptional performance in software engineering, knowledge work, scientific research, and vision.
The longer and more complex the task, the larger Fable 5’s lead over our other models.
