The 12B variant uses an encoder-free architecture for raw audio.
The 12B variant uses an encoder-free architecture for raw audio.
@osanseviero way less technical than i would like it to be tbh. Need reports like from DeepSeek and ziphu
@osanseviero a gem 💎 to read
Sounds about right. Not even shocking. We know of open models with similar compute footprint and >40T pretraining tokens (gpt-oss, MiMo). This approach is also why Demis says they don't have compute for open weights. Kek. Gemma 4 project has comparable footprint to GLM 5.2/DSV4. https://twitter.com/eliebakouch/status/2074473875614712236
seems like gemma 4 was trained on MUCH more tokens than the previous iteration https://x.com/eliebakouch/status/2074473875614712236/photo/1 https://twitter.com/osanseviero/status/2074436670770868249
Happy to share we just published Gemma 4 technical report! Take a look https://x.com/osanseviero/status/2074436670770868249/photo/1
the estimation process (mfu for moe i assumed ~20% less than dense variant) https://x.com/eliebakouch/status/2074474046159200317/photo/1
💎💎💎💎 tech report out now! https://x.com/PetarV_93/status/2074453434690928647/photo/1
@teortaxesTex it's also multimodal btw, hence why more data imo
https://arxiv.org/abs/2607.02770
Ask a question below.
Published answers will appear here.
