A benchmark by Stanford NLP's Chenglei Si finds Claude-Fable-5 leads on autoresearch, while open-weight Kimi-K2.7-Code tops ML engineering

Really really strange set of results throughout I can believe that Fable was nerfed and Kimi K2.7 is relatively great at ML engineering but they also get Gemini 3.1 at/near the top on may tasks

Surprisingly though, we found that a recent open model, Kimi-K2.7, performed very well on ML engineering. And Fable performed even worse than Opus. This could be either because of the inflated cost, or the guardrails put on ML tasks. (4/5)

Overall, it seems like the model supply chain will be less stable in the autoresearch space.

On the Weco side, we’ll stay model-neutral and provide more options for our users. Today, we just added support for Kimi-2.7 (5/5)

If you’re interested: https://www.weco.ai/

Overall, Fable is a really strong model for autoresearch. It dominates harness/prompt engineering and algorithmic discovery tasks. We were especially surprised by the algorithmic discovery results, because the eval cost is low and cheaper models can run many more steps. (3/5)

@sanmking @OpenRouter Yes it's possible I believe @SakanaAILabs had some research on this https://sakana.ai/ab-mcts/

Benchmark protocol: - cost (LLM + eval cost) constrained, not steps constrained. This means an agent can run more steps if its model or solutions are cheaper to run - All of the models use the autonomous research harness behind the @WecoAI service - The scores should be interpreted as how good the final solution is compared to a naive ReAct agent. (2/5)

@zhengyaojiang Did you do an analysis on similarity of answers or scores across problems. I’m specially curious given the recent results from @OpenRouter Fusion API:

Maybe a bag of models, would perform better. Specially, in a verifiable domain as autoresearch.

@alokbishoyi97 yeah it's not quite good at heuristic engineering & more conventional algorithm design

On MLE, the Opus vs. GPT-5.5 gap is very small, so I wouldn’t read too much into it (can be noise).

On harness tuning, that’s a fair concern. It has been tuned for models from different providers throughout the development process. It’s possible there’s some bias here but it’s hard to tell which specific model gets an advantage.

Thanks Davide! It is quite noisy but we ran a lot of seeds, the aggregated number should be rather robust

I think a key difference between our benchmark and others is that we’re cost-bound. Claude models are generally quite expensive, which leads to fewer iteration steps. Also, Claude is somewhat weaker at some niche tasks that differ from conventional software engineering. For example, it was quite bad at MLE until Opus 4.6, and is still bad at algorithmic/heuristic engineering.

@zhengyaojiang Wow, opus 4-8 so down the list!?

@zhengyaojiang Quite interesting, thanks! Can you share an example of ML engineering task?

I’m curious if it’s tradML, deep learning fine tuning or pipeline, etc

@zhengyaojiang Super interesting

@zhengyaojiang interesting, the vibes with opus 4.7 for ml engineering seemed better than 5.5 to me - just vibes tho i guess. reckon your harness is tuned to 1 vs the other?

@TheGrizztronic yes similar as in: https://arxiv.org/html/2605.21384v1

@zhengyaojiang Very cool stuff! Surprised to see that despite other benchmarks, Opus 4.8 is lagging the rest in this. How replicable is this/how noisy?

@zhengyaojiang Do you have cheating mitigation in place?

@zhengyaojiang Big if true

@zhengyaojiang Was this using the original nerfed Fable model before they walked it back and started routing to Opus 4.8?

@zhengyaojiang What about considering cost?