Teortaxes▶️ (DeepSeek 推特🐋铁粉 2023 – ∞)#501
Today we're releasing Mellum2: our first "serious" LLM.
This is a 12B A2.5B MoE LLM pre-trained on ~11T tokens and post-trained with RLVR. I'm proud to be leading the team that was working on it for the last 6 months.
We release base/SFT/RL checkpoints along with a tech report
Many users congratulated the JetBrains team on releasing Mellum2, praising the open-source 12B MoE model as amazing work and expressing eagerness to test it.
@nv_pavlichenko Congrats!
@nv_pavlichenko you mention a qwen3-like architecture put then point to Gemma for SWA 3:1, what were the qwen-oriented attributes?
@nv_pavlichenko Congrats! I have to be honest, I was start to lose faith in JetBrains after being an hardcore users for years. Happy to see you are back on track on the AI coding way!
@nv_pavlichenko How many (and which) GPUs?
HF: https://huggingface.co/collections/JetBrains/mellum-2 Tech report: https://arxiv.org/pdf/2605.31268 Announcement: https://blog.jetbrains.com/ai/2026/06/mellum2-goes-open-source-a-fast-model-for-ai-workflows/
Btw, 100% European🇪🇺
@nv_pavlichenko cc @0xSero 👀
@bahamutru @justALEXWORTEGA Мы и старались сделать эффективную модель для инференса на гпу (это наш основной юзкейс). Согласен, что для локального инференса скорее всего это не самая эффективная модель, @liquidai делает оч крутые локальные модели + квен оптимизирует свои маленькие под локальный инференс
@nv_pavlichenko @teortaxesTex 6months total training time? On which stack you trained ?
@nv_pavlichenko thanks for not mentioning _once_ how big it is in GB...
@nmb_four Well it’s simple math from param count. 12B * 16 bit if in full precision. Half of that if fp8
@nv_pavlichenko best for Kotlin?
@nv_pavlichenko @justALEXWORTEGA Но если быть честным, на ноутбуке такое уже не запустить (LFM2.5-8B-A1B (A1.5B) гораздо ближе к реальности), нужно что-то меньше и быстрее. И гораздо умнее, не Qwen3.5-9b, а, минимум, Qwen3.6-27b. Тогда будет «вы можете программировать на ноутбуке». Не раньше.
@nv_pavlichenko llama.cpp support?
@nv_pavlichenko Amazing, congrats on the release Nikita and JetBrains team!
Going to cover this on the next @thursdai_pod recording!
https://x.com/i/spaces/1jGXggzMNpzKZ
@nv_pavlichenko Congrats ! Happy to see it come out ! 💻
Finally, the eval numbers. One thing that I learnt that small Qwen3/3.5 are absolutely unusable in thinking mode (keept thinking forever).
We needed to build everything from scratch and learn how to do things: how to run architecture search, do data ablations, fix stability issues, spin up RL infra, fix runs that keep blowing up, and do evals.
All of that with a team of just ~7 people
@nv_pavlichenko 🇪🇺 100%
We wanted to build an alternative to 7/8/9B dense models that would be much faster.
We noticed that small OSS releases are mostly targeted local inference and are very inefficient in H100 inference setup.
This is why we decided to go sparse
It is, but gated deltanet is quite an overhead on small models and short contexts + qwen3.5 4b is quite unusable in thinking mode: thinks forever for 100k+ tokens
But yeah, this is our first try, we’ve got a long way to go. Building models is quite hard. Big fan of Qwen regardless