Ask a question below.
Published answers will appear here.
GPT‑5.6 lost our coding benchmark. I switched to it anyway. On the Superconductor "Custom SWE-Bench", which evaluates agents on our own Ruby on Rails codebase, the new Sol, Terra, and Luna models dominate the cost Pareto frontier. They are literally 5x faster than Opus and Fable. But on quality, Fable 5 is still the clear winner on our repo, and even Opus 4.8 beats GPT 5.6 Sol at all effort levels, consistent with what some other benchmarks have shown. Composer 2.5 Fast remains a standout surprise, matching 5.6 Sol's performance at roughly the same cost -- but even the Fast version is slower. Yesterday's Grok 4.5 is still on the speed Pareto frontier. We don't know how much it costs, unfortunately. Digging into failure cases, GPT 5.6 -- even Sol! -- occasionally writes code that just isn't valid, and its "taste" just isn't on par with Fable 5, or even Opus 4.8. One meta-observation about company strategy: Anthropic releases a new model only when it's clearly better — and our benchmark shows that clean progression from 4.6 to 4.7 to 4.8 to 5. OpenAI ships more variants, more often, which adds noise: older models sometimes beat newer ones. That said, I still switched my daily driver to 5.6 Sol High. The speed makes a huge difference, and since we're all constantly running out of Fable 5 usage, it is the more workable model right now. Lastly, don't take any single benchmark result at face value! Build your own at http://superconductor.com/benchmark to see how these agents perform on YOUR codebase.
Ask a question below.
Published answers will appear here.