Andrew Trask argues compound model ensembling beats monolithic models, but David Manheim attributes the gains to query compounding

ensembling is so back?

@iamtrask Well, here's some I've shown before against the solo frontier as the only option.

@iamtrask Is it the actual ensemble of models or different system prompts and context? Keep thinking about this from last year https://openreview.net/forum?id=saDOrrnNTz

@TheZvi

@iamtrask hell yeah dude. this article pumped me tf up lol genuinely inspiring.

I hope you’re right. I think a lot of us have naively thought, “just combine the models, 5head, that will obviously be better” but it seems like such low hanging fruit. hopefully these gains are real

@iamtrask Ensembles > Solo heavyweights, the meta game just changed 🤝

@AndrewCritchPhD 💯

@iamtrask @alexolegimas @AndrewCritchPhD I might be wrong about this, but I'd be interested in your thoughts about whether the model diversity is what matters, as opposed to, e.g. just giving a single LLM different persona prompts.

@omarsar0 the "one model to rule them all" assumption was always wishful thinking. different models fail differently. that's actually useful if you know how to design around it. most teams don't design around it. they just switch models when the current one disappoints them

@TheZvi Makes sense to me, given the actual thread. Trask just hypes it up incorrectly.

But in general we should throw out evals that don't report critical details like token budgets. (We need better eval standards, and better reporting.)

@omarsar0 why do you guys act like models are free? companies are trying to reduce the cost of ai because using even 1 model for every task is so expensive and you guys are talking about using 3 models for 1 task

@AndrewCritchPhD (in the substack I screenshot/quote your tweet saying that btw)

@iamtrask

@omarsar0 Even manual combinations can compound.

For instance, if I'm planning a new tool, I ask all the top models to take a pass at it, then get one of them to composite a "best of" version by taking all the good stuff from each individual plan.

@morgan_coding That last-minute model-swapping strategy is unsustainable.

@TheZvi seems crap

Isn't that the point, though? Ensembling models is precisely the type of approach that could enable cost efficiency. I must admit that this is an unsolved problem. However, ensembling means that we could leverage smaller models (local or cloud) to bring that cost down significantly.

@omarsar0 this all makes sense when you remember that combining the weights of two models just makes them overall better. it was a weird finding that nobody could really explain at the time, but that's how it worked.

imagine if all the frontier labs just combined all their weights.

@omarsar0 If not for one model fits all, models trained on specific data by specific providers would perform way different on an isolated prompt, and would still require a fusion of all these field specific models as well

@omarsar0 @omarsar0 dude, totally agree. people forget the value of tailored models for specific needs. generic ain't always best.