Andrew Trask calls multi-model routing a primary research frontier
@iamtrask @davidmanheim @alexolegimas My experience is that scaffolding multiple models almost always gives better results, presumably partly because the "natural diversity" of one model is insufficient
@davidmanheim @alexolegimas there's enough randomness in all models that you get a natural diversity
Users expressed enthusiasm for testing combinations of multiple AI models, seeing potential for better results than single-model approaches.
@davidmanheim @herbiebradley @alexolegimas honestly i think you've just described the frontier of network-source AI research... how do we do routing? does every model get the same prompt (likley). how do we weight the results? how do we scale it? etc.
@herbiebradley @iamtrask @alexolegimas Do you give different persona prompts to the single model? (Because this could easily be more an artifact of the system prompt pre-conditioning in ways that reduce variance, rather than the model itself not supporting diversity.)
@herbiebradley @iamtrask @alexolegimas In that vein, I'd love to see work swapping the system prompts of Grok, ChatGPT, and Claude, and seeing how that changes their performance.
LLM-search doesn't find much, but is indicative: https://arxiv.org/abs/2502.06065 shows sensitivity to prompts, but cf. https://arxiv.org/abs/2509.01790
@herbiebradley @iamtrask @alexolegimas Do you give different persona prompts to the single model? (Because this could easily be more an artifact of the system prompt pre-conditioning in ways that reduce variance, rather than the model itself not supporting diversity.)
@herbiebradley @iamtrask @alexolegimas The data *isn't* that different, though the fine-tuning might be, and there might be differing curriculum effects.
Also, subagent routing seems to do work on top of but ignoring the MoE layer in the individual models, which could give additional diversity, or might not.
@davidmanheim @iamtrask @alexolegimas No, but that seems unlikely relative to the explanation of "models trained on different data will have different strengths & weaknesses"
btw this is partly the same principle that makes Devin and many other app layer products perform so well (subagent routing)
@herbiebradley @iamtrask @alexolegimas Do you give different persona prompts to the single model? (Because this could easily be more an artifact of the system prompt pre-conditioning in ways that reduce variance, rather than the model itself not supporting diversity.)
@davidmanheim @herbiebradley @alexolegimas *unlikely
@davidmanheim @herbiebradley @alexolegimas honestly i think you've just described the frontier of network-source AI research... how do we do routing? does every model get the same prompt (likley). how do we weight the results? how do we scale it? etc.
@davidmanheim @herbiebradley @alexolegimas cheers to that - this would be a very interesting project. i'd like to see that too
@herbiebradley @iamtrask @alexolegimas In that vein, I'd love to see work swapping the system prompts of Grok, ChatGPT, and Claude, and seeing how that changes their performance.
LLM-search doesn't find much, but is indicative: https://arxiv.org/abs/2502.06065 shows sensitivity to prompts, but cf. https://arxiv.org/abs/2509.01790
I know it's not that different but we should expect data differences to give significant differences in jaggedness of capabilities
The entire discourse on this platform over Opus being good at X and ChatGPT being good at Y is surely not just system prompts, it's almost certainly differences in data. To me this has been an enduring characteristic of models for years.
I agree there are probably ways to improve diversity by fiddling with MoE layers, or doing some fun sampling techniques. But realistically we're not going to get that in APIs.
@herbiebradley @iamtrask @alexolegimas The data *isn't* that different, though the fine-tuning might be, and there might be differing curriculum effects.
Also, subagent routing seems to do work on top of but ignoring the MoE layer in the individual models, which could give additional diversity, or might not.
@iamtrask @herbiebradley @alexolegimas That does sound like a potentially better approach than just scaling MoE as it's currently used, by allowing the models to diverge more. But I'd want to understand more of the (possibly Lab-internal-only) literature on returns to number of experts within MoE systems first.