Blog post: https://machinelearning.apple.com/research/introducing-third-generation-of-apple-foundation-models

Original publication: https://machinelearning.apple.com/research/pruning-large-language

It's very cool that Apple shipped a 20B parameter on-device.

You can't put 20B parameters in RAM at any reasonable precision. To make it work they are using pretty exotic architecture by today's standards.

A small model predicts from the query (or prompt) which experts to load from Nand into RAM. The key distinction from a typical MoE is that you do this once per query and then generate all the tokens with the same experts (instead of switching the experts for every token).

This is quite amazing. First time I see explicit MoE design for the GPU-poors.

@awnihannun "Instead of forcing the entire model into DRAM, the full model is stored in flash memory (NAND)" - does this mean the entire 20B param model is stored on the device taking up ~10GB Nand Space needed?

@awnihannun apple gets it, the scalable way for AI is locally run, not with data centers

@awnihannun 20B is huge. Not sure how will they do it. At least I cannot run gpt-oss20b comfortably on my macbook.

@awnihannun Compatibility woes for older gen hardware!

@aayeinbaigan @awnihannun At least 10gb if int4 but it is on the device

@awnihannun Cool breakdown. The once-per-query loading is forced by NAND bandwidth. Swapping experts every token would stall generation on flash reads, so they load once and reuse. The tradeoff: you lose per-token routing. Every token is stuck with the expert set the prompt picked.

@Everlier @awnihannun Maybe the engineer implementing it also didn’t know and it worked somewhat okay

@awnihannun I am sure that shit will be used mainly to profile you, drain battery like crazy and make you phone hot so you will consider buying gloves🥴

@awnihannun However, the capability of system gets seriously hampered due to invoking of a single expert. memory compression is needed to have an effective single system, instead of having to stream weights for every individual invoke per prompt.

@Everlier @awnihannun There has been a lot of distilling and refinement using the latest Gemini models to get here

@awnihannun I find it funny that this is actually exactly how most think that MoE works.

The overlap between experts must be pretty high to support such an architecture. So much so, I'm questioning how much of its performance will be retained by just frankenmerging those.

@awnihannun @ivanfioravanti Genuinely amazing work!

@awnihannun per-query expert loading dodges the NAND bandwidth wall.

at 1M context the KV cache geometry matters more than active parameter count.

The expert prediction layer is the key insight here. Instead of loading all 20B params, you predict which experts matter per query and only materialize those weights. It's essentially a dynamic routing problem — and it makes on-device inference feasible at a scale that would otherwise be impossible. Curious how the routing overhead compares to the inference savings.

@awnihannun oh interesting use of on device routing to let the small model select the experts cool

@awnihannun What's fascinating is that on-device AI is forcing entirely different architectural trade-offs. Cloud models optimize around compute. On-device models optimize around memory. The constraints are changing, and architecture is evolving with them.

@aayeinbaigan @awnihannun It’s a QAT model (Gemma4 architecture) so it should be a bit less but yes.