Cognition releases FrontierCode, a coding benchmark built by open-source maintainers to evaluate models on complex software maintenance
Opus 4.8-medium solved 32% of tasks at a 40x speedup
Opus 4.8-medium solved 32% of tasks at a 40x speedup
Opus 4.8-medium solved 32% of tasks at a 40x speedup
Opus 4.8-medium solved 32% of tasks at a 40x speedup

@cognition Fake!! You're manipulating people. Opus 4.8 is nowhere near Codex 5.5 it's far inferior!!
@ScottWu46 Awesome
It's finally out!!! @METR_Evals found that more than half of SWEBench results is unmergeable slop. FrontierCode represents over 1000+ hours of maintainer validated software engineering work most frontier models cannot yet solve, much less solve with high quality. Cog had IOI Gold medalists and top code maintainers Look At The Data — FrontierCode includes 3000+ rubrics covering code quality and anticheat reward hacking plaguing other benchmarks. FC Diamond is so hard that Opus 4.8 scores 13.8%. Three eras of AI coding: Three eras of benchmarks 2021 • Autocomplete: HumanEval 2023 • Passing Tests: SWEBench, TerminalBench 2026 • Maintainable Code: FrontierCode to me the most beautiful chart when I requested a special historical run into all extant old models, the data was finding that the easiest third of FC tasks (in FC Extended) were rapidlly and suddenly solved over late 2025 - Opus almost doubled from a 41% pass rate to 74% in 4 months. This describes the "WTF happened in Dec 2025" vibe shift that a lot of folks from @dhh to @karpathy have called out: it is the difference between getting 95% success in 2 rerolls vs 6, making it finally feasible to go up the next layer of abstraction in agentic coding, eg @GeoffreyHuntley's ralph loops or @bcherny's /goals or @steipete's "loops that prompt your agents" without fearing too much that things go off the rails. My guess: as AI accelerates from here, each FrontierCode tier will saturate in sequence, hopefully ~annually. I've already asked the team to prepare FrontierCode 2027.... The old mountains will be destroyed. Their rubble becomes regolith. And from that regolith, the next model forest grows. Circle of life.
SWE-Bench style grading has been the standard for years now - you ask the agent to solve an issue and then run its code on a pre-constructed unit test. The problem is that passing a unit test is only one part of writing production-ready code. You also want to evaluate agents on a number of other axes, including scope, coding style, and unintended side effects. The result is our new benchmark FrontierCode - which has ~80% fewer false positives and for which the best model (Opus 4.8) only scores 13%! "Where others grade like a CI, FrontierCode grades like a tech lead.
Opus 4.8 is the best coding model out there FrontierCode by Cognition is probably the highest quality coding benchmark we have seen so far it moves beyond just using unit-testing for scoring, it also tests for regression safety, mechanical cleanliness, test correctness, scope and code quality 20+ open-source developers handcrafted 150 tasks, each of which took over 40 hours to construct it also tests a more diverse set of programming languages
Mythos is live! so excited to have our FrontierCode recognized as the next frontier coding bench. on FC Diamond, BOTH Opus 4.8 and GPT 5.5 don't meaningfully scale with effort, which many of you caught yesterday. Mythos/Fable posttraining have really applied that test time compute toward solving very, very long running problems - dozens of human hour equivalents, hundreds of dollars per task, for the first time ever measured. Available now in @Cognition @Devin for only 1.4x ACUs too! (I never thought i'd see this launch lol)
Oh my God! @METR_Evals’s coding benchmarks are saturated! 🤯 Mythos broke the METR graph 🤯 4 weeks later, out comes a new coding task, this time from @cognition: “FrontierCode Diamond remains unsaturated: the best performing model, Claude Opus 4.8, achieves a score of only 13.4%. There is still a lots of headroom. *Note that METR itself never panicked. It’s the Twitterverse that has egg on its face.
just finished rerunning FC Diamond on my historical charts. none of the official tables/charts are capturing the degree of takeoff. https://x.com/karpathy/status/2064409694761054332 its this same chart all the way down difficulty classes (below) breaks every curve fit because Fable is a diffferent CLASS of model, with beeeeeg model smell.
A lot of people are already deploying AI into production codebases, but until now we didn’t really have a good eval for whether it writes code that is actually high-quality and maintainable. Pretty cool grading here: https://x.com/karinanguyen/status/2064090762774528299/photo/1 https://twitter.com/cognition/status/2064061031912288715
Ask a question below.
Published answers will appear here.