Home
Technology
Gaming
Entertainment
World & Business
Science
Sports
AI

World & Business

Home
Technology
Gaming
Entertainment
World & Business
Science
Sports
AI

Microsoft AI's latest round of pretraining · Digg

Report

Microsoft AI's latest round of pretraining

A post linking to a Microsoft AI blog entry on the round declares, “The scaling continues!”

4 Sources, 1h ago, first seen 1h ago

TLDR

A post points to a Microsoft AI blog entry it describes as covering its latest round of pretraining, adding, “The scaling continues!”

Combined views

4 Sources, first seen 1h ago

1 reposts

Combined views

4 Sources, first seen 1h ago

1 reposts

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Featured Source

Sentiment

Positive——Negative

Summary

Not enough discussion yet.

No sentiment analysis available yet.

Today's Rank

#17

Today's Rank

#17

4 Sources

@ChengleiSiRT @david_r_so: The scaling continues! Check out our blog post on our latest round of pretraining: https://microsoft.ai/news/frontier-pre-training-at-microsoft-ai/1h

@eliebakouchwow, amazing technical blog post about microsoft's new 3T pre-training with very rare and clear insights on how to train models of this size first let's get an overview of the shape of this model, it's a sparser model than Kimi K3 (80B active vs 104B), same sliding window hybrid pattern, with this time more kv heads for the sliding window layers (a bit different from stepfun and poolside that add query head). they added a shared expert and also removed the interleaved MoE/Dense pattern of MAI 1 the scaling ladder is very similar to MAI 1 with the same mechanism, each rung depends only on the number of layers. they mention that one of the key improvements is how they handle the smaller rung data distribution where they make sure to have the same repetition as the final corpus. they already did that for MAI-1 but only for data ablation iirc their prediction is almost identical to the final run loss, very impressive they compare the model with different base models such as deepseek v4 pro and MAI 1, tho in the case of deepseek v4 pro i'd say it's quite hard to distinguish how much the data mixture "overfitting" to the scaling ladder has an impact here (they mention it's both), need to always be a bit careful about EG metrics for data in general they have VERY NICE scaling laws for weight decay and learning rate, for the learning rate they express it as a function of tokens per parameter (N/D) and steps (hence depending on the batch size). they also do the same for dropout where it's a function of active params only, they also find the usual sqrt(k) scaling for learning rate and batch size (other scaling laws often also converge to that) a bunch of other stability stuff, they use Muon this time and activation gradient clipping, the efficiency gain here is actually pretty crazy at larger scale. they also increase the training horizon to ~40T text (20T unique) + a new 10T (4.5T unique) multimodal since the model is natively multimodal imo this is a safe but very clean scaling! very exciting for the tech report and MAI-base-3, congrats!!1h

@recurseparadoxDamn word gets around fast. Score centering, agc, …1h

4 Sources

@ChengleiSiRT @david_r_so: The scaling continues! Check out our blog post on our latest round of pretraining: https://microsoft.ai/news/frontier-pre-training-at-microsoft-ai/1h

@eliebakouchwow, amazing technical blog post about microsoft's new 3T pre-training with very rare and clear insights on how to train models of this size first let's get an overview of the shape of this model, it's a sparser model than Kimi K3 (80B active vs 104B), same sliding window hybrid pattern, with this time more kv heads for the sliding window layers (a bit different from stepfun and poolside that add query head). they added a shared expert and also removed the interleaved MoE/Dense pattern of MAI 1 the scaling ladder is very similar to MAI 1 with the same mechanism, each rung depends only on the number of layers. they mention that one of the key improvements is how they handle the smaller rung data distribution where they make sure to have the same repetition as the final corpus. they already did that for MAI-1 but only for data ablation iirc their prediction is almost identical to the final run loss, very impressive they compare the model with different base models such as deepseek v4 pro and MAI 1, tho in the case of deepseek v4 pro i'd say it's quite hard to distinguish how much the data mixture "overfitting" to the scaling ladder has an impact here (they mention it's both), need to always be a bit careful about EG metrics for data in general they have VERY NICE scaling laws for weight decay and learning rate, for the learning rate they express it as a function of tokens per parameter (N/D) and steps (hence depending on the batch size). they also do the same for dropout where it's a function of active params only, they also find the usual sqrt(k) scaling for learning rate and batch size (other scaling laws often also converge to that) a bunch of other stability stuff, they use Muon this time and activation gradient clipping, the efficiency gain here is actually pretty crazy at larger scale. they also increase the training horizon to ~40T text (20T unique) + a new 10T (4.5T unique) multimodal since the model is natively multimodal imo this is a safe but very clean scaling! very exciting for the tech report and MAI-base-3, congrats!!1h

@recurseparadoxDamn word gets around fast. Score centering, agc, …1h