Meituan has released the weights for LongCat 2.0 under an MIT license, giving the community access to a 1.6-trillion-parameter Mixture-of-Experts model that activates roughly 48 billion parameters per token along with 135 billion n-gram embedding parameters.
longcat 2.0 (1.6T, ~48B active) weights are now open under MIT license
interesting design choice compared to previous iteration is that they keep the same attention shape and REDUCE the number of experts from 256 to 128 (i would have expected the opposite?). they also have 135B embedding parameters with n-gram
https://huggingface.co/meituan-longcat/LongCat-2.0
The model trims the expert pool from 256 down to 128 while preserving the prior attention shape and adding LongCat Sparse Attention variants, so practitioners can test whether these tweaks improve long-context throughput on their own hardware.
Benchmarks are reported mainly from internal runs on agentic and coding tasks, leaving open how the model compares under independent evaluation or outside the Chinese training stack that used only domestic ASICs.
Users are excited about Meituan open-sourcing Longcat 2.0 1.6T MoE weights under MIT license, with some calling it finally here and better than Meta's models.
No Digg Deeper questions have been answered for this story yet.
longcat 2.0 (1.6T, ~48B active) weights are now open under MIT license
interesting design choice compared to previous iteration is that they keep the same attention shape and REDUCE the number of zero communication experts from 256 to 128 (i would have expected the opposite?). they also have 135B embedding parameters with n-gram
https://huggingface.co/meituan-longcat/LongCat-2.0
@eliebakouch maybe it's a complex tradeoff with N-grams or maybe zero-comm experts are not working that well
longcat 2.0 (1.6T, ~48B active) weights are now open under MIT license
interesting design choice compared to previous iteration is that they keep the same attention shape and REDUCE the number of zero communication experts from 256 to 128 (i would have expected the opposite?). they also have 135B embedding parameters with n-gram
https://huggingface.co/meituan-longcat/LongCat-2.0
@eliebakouch maybe this time it will stay up
longcat 2.0 (1.6T, ~48B active) weights are now open under MIT license
interesting design choice compared to previous iteration is that they keep the same attention shape and REDUCE the number of experts from 256 to 128 (i would have expected the opposite?). they also have 135B embedding parameters with n-gram
https://huggingface.co/meituan-longcat/LongCat-2.0
uh meant the number of zero communication expert (just modified the post thanks), but in general scaling the sparsity is interesting since you get more total param with the same active count, which is better in pre training and inference assuming you have a good infra lol (similar to why people do MoE over dense model)
@eliebakouch Hey Elie! Could you please explain why scaling the number of experts would have been interesting here? What’s the impact of making ur model sparser?
@eliebakouch Got it, thanks a lot!
@xeophon the weight weren't on the previous repo iirc
@teortaxesTex i don't think it's related to ngram, but i agree with the second point, my guess is that larger scale lead to more instability and this zero expert thing is sensible to that
@eliebakouch finally!
@eliebakouch 36B active already seems wild for a draft
curious how reducing the zero comm experts affects the math for inference though
@NoeFlandre tassuming those total param that you add lead to more intelligence)
@eliebakouch Chinese DoorDash just cooked a better model than @AIatMeta 🤣
@eliebakouch "num_layers": 38 👀
@eliebakouch wild that they cut the experts, very curious why they did that
@eliebakouch that shift in the expert count is honestly such a trip.