Stanford study finds 70% of frontier model queries could run accurately on local devices at zero cost

i sort of dislike this way of framing it because it tells you absolutely nothing about consistency, and consistency matters extremely. only RL done by competent actors + scale fully gets you this. i HEAVILY dislike model routers for similar reasons to this at a conceptual level

@MTSlive @huggingface this is the research:

@NoemiTitarenco by what measure also; absolutely not, modern MoE is about sparse ffn subnetworks, not "multiple distinct models doing different things totally independently" finegrained sparsity is THE most maximally entangled way of doing MoE, and it's the one that won & isn't GOFAI-ish at all

the gap between "the pass@1 sample that a study tested" vs "its empirical avg@k + worst@k at nontrivial samples" is WIDE

@NoemiTitarenco i was mainly arguing about "model routing as a singleton thing that gets decided based on the whim of a classifier, and is engineered around single turn conversations, often with dubiously calibrated data for the classifier" as being the brittle thing, rather than ensembling

@kalomaze they perform better, what else do you want. i dont think consistency is a major problem with fusion models. its just MOE abstracted another layer.

They perform better by benchmarks, but also common sense will tell you that having multiple models answer the same prompt and then using the answers from all of the models in order to formulate a final answer is a more informed way to go to solve a complex problem. Model fusion is not something that is meant be used as part of a typical agentic workflow but if you're doing research or planning, I think it's great. The models were developed independently and actually the lack of consistency between models is what makes fusion work. You're not using it for genetic tasks that need consistency like tool calling.

@kalomaze https://openrouter.ai/docs/guides/features/plugins/fusion

@NoemiTitarenco ensembling is the maximally expensive thing tho, ofc you can get better results that way, that seems distinct from model routing as it was originally conceptualized?

@NoemiTitarenco @kalomaze you’re out of your league here, i would sit this one out

@MTSlive @ClementDelangue @huggingface I'm sure this is true, but I'd guess that >90% of the value creation happens on the 30% of queries that couldn't be done locally.

@MTSlive @ClementDelangue @huggingface Probably true. The annoying part is distro and updates, not inference itself.

@ClementDelangue @MTSlive @huggingface real

Are there fusion models that work like that? I think they all have a primary model that always returns the final answer. Maybe I missed the part of the video where they talked about it like that. i'm working on a fusion model where it's just a two model system where one model is this constant advisor to the primary model.

@ClementDelangue @MTSlive @huggingface Very nice to see studies and more info on this, I wrote about a possible future: https://valeanu.xyz/ai-belongs-at-the-edge/

@MTSlive @ClementDelangue @huggingface Yep. Very, very good filter needed here. Cool demos are one thing. Unit economics are another.

@MTSlive @ClementDelangue @huggingface Like like like. Women don't belong in ANY meaningful discussion, ever.