Andrew Trask argues centralized frontier AI companies will never exceed the capability frontier of ensembled models

VIEWS5.8KLIKES34RETWEETS3

merve@mervenoyann

ensembling is so back?

⿻ Andrew Trask@iamtrask

This is a *way* bigger deal than it seems...

Frontier AI companies will *never* own the frontier again

I kid you not... I've been waiting for someone to show this result for like 4 years... this is a huge deal.

The short reason: combinations of models will *always* outperform individual models

The long reason: this is the gateway to a million times more data... and huge leaps in compute efficiency.

The AI scaling laws always win.

More in article below 👇

3d5.8K342

BOOKMARKS8

⿻ Andrew Trask@iamtrask

@TheZvi Nope - that still works. In fact ensembling models with themselves is such a reliable means to achieve SOTA it has been shadowbanned at top AI conferences for a decade

details https://andrewtrask.substack.com/p/breaking-todays-frontier-ai-companies

Zvi Mowshowitz@TheZvi

This says 'Opus 4.8 + Opus 4.8 > Fable' so in terms of being meaningful I pretty much just flat out don't believe it?

2d1.4K168

REPLIES4

⿻ Andrew Trask@iamtrask

@davidmanheim @alexolegimas there's enough randomness in all models that you get a natural diversity

David Manheim@davidmanheim

@iamtrask @alexolegimas I don't think the thread supports the claim that this is mostly about having different models. They explicitly showed it's mostly synthesis lift, i.e. using tokens more cleverly via multiple sessions and a judge, not diversity.

2d57970

rohit@krishnanrohit

@iamtrask Well, here's some I've shown before against the solo frontier as the only option.

rohit@krishnanrohit

🚨 Fable benchmark update for hub vs spoke!

Fable 5 is the first model I've tested whose self-assessments are genuinely calibrated! But even a simply designed market seems to do better.

- Solo Fable 5 beat *every* topology on quality (8.1 avg, 87% pass vs the market's 7.2/76%), at 2.6x the market's cost! - A pricier market of frontier coding agents (Opus 4.7 and GPT-5.5) bought no quality at 4x the cost - Routing based on track record - give Fable the tasks the cheap pool has failed - hits 93%, beating solo Fable on quality and cost!

What this means is that we really can make much better topologies that actually manage both cost and effort, even with older models that aren't as good at being calibrated!

3d99284

Nathan Labenz@labenz

@TheZvi

OpenRouter@OpenRouter

Note: we have only evaluated one deep research benchmark so far, which did not include long-horizon tasks.

Fable's long-horizon abilities were extremely impressive, and it calls for future work to benchmark both on long-horizon tasks.

2d1.8K232

⿻ Andrew Trask@iamtrask

deeper fundamentals on why this paradigm is going to win https://attribution-based-control.ai/

2d2K63

Martin Kemka@mkemka_

@iamtrask Is it the actual ensemble of models or different system prompts and context? Keep thinking about this from last year https://openreview.net/forum?id=saDOrrnNTz

3d60913

-@tixtacs

@iamtrask hell yeah dude. this article pumped me tf up lol genuinely inspiring.

I hope you’re right. I think a lot of us have naively thought, “just combine the models, 5head, that will obviously be better” but it seems like such low hanging fruit. hopefully these gains are real

3d49911

Artale@artale93

@iamtrask Ensembles > Solo heavyweights, the meta game just changed 🤝

3d2391

⿻ Andrew Trask@iamtrask

@AndrewCritchPhD 💯

Andrew Critch (🤖🩺🚀)@AndrewCritchPhD

A big deal, but old news if you use theMultiplicity ;)

3d1.1K20

David Manheim@davidmanheim

@iamtrask @alexolegimas @AndrewCritchPhD I might be wrong about this, but I'd be interested in your thoughts about whether the model diversity is what matters, as opposed to, e.g. just giving a single LLM different persona prompts.

2d6733

OPENheimer (𝔦, 𝔦)@JR_Openheimer

@iamtrask this has been well known for years https://arxiv.org/abs/2305.14325 https://arxiv.org/abs/2406.04692

if its so revolutionary, why didn't they do it before?

2d1304

⿻ Andrew Trask@iamtrask

If you want to work on this kind of frontier AI as your day job, we're hiring: https://openmined.org/careers/

2d4271

⿻ Andrew Trask@iamtrask

@TomDavidsonX @davidmanheim @alexolegimas I ran Humanity's Last Exam versions of this last winter and the difference was pretty big. 44% -> ~52%

A student at stanford also got the same leap at around that time https://sup.ai/

Tom Davidson@TomDavidsonX

@iamtrask @davidmanheim @alexolegimas I'm confused - you get a cpl % uplift from multiple models. That's small, and not a challenge to frontier dominance

2d11320

Coding with Morgan@morgan_coding

@omarsar0 the "one model to rule them all" assumption was always wishful thinking. different models fail differently. that's actually useful if you know how to design around it. most teams don't design around it. they just switch models when the current one disappoints them

3d1731

Lauro@laurolangosco

@iamtrask @TheZvi The fact that they dropped the blogpost without including that experiment makes me wonder whether the reason is that the result doesn't fit the narrative!

But yeah it's a tough request to satisfy now

2d531

David Manheim@davidmanheim

@TheZvi Makes sense to me, given the actual thread. Trask just hypes it up incorrectly.

But in general we should throw out evals that don't report critical details like token budgets. (We need better eval standards, and better reporting.)

2d1694

Mng@Mng64218162

@omarsar0 why do you guys act like models are free? companies are trying to reduce the cost of ai because using even 1 model for every task is so expensive and you guys are talking about using 3 models for 1 task

3d160

Lauro@laurolangosco

@iamtrask @TheZvi I do think that at the very least you then ought to compare against a baseline of 'Fable 5 + Fable 5' if you want to make the claim that fusions of models surpass individual models.

2d491

⿻ Andrew Trask@iamtrask

@laurolangosco @TheZvi We can ask again in a week

2d471