@natashajaques what i find fascinating is that even with publicly released data, the hparams and decisions made during pre-training and mid-training still feel opaque, like a black box we can't quite peek into

Really enjoyed reading the Microsoft MAI-Thinking-1 "Building a Hill Climbing Machine" paper. Amazing they publicly released all the info needed to train a frontier model, down to hparams.

I also thought this was pretty telling: - pre-training: 30 trillion tokens - mid-training (SFT on STEM/math/code data): 3.55 trillion tokens - RL post-training: 150 billion tokens. Looks like @ylecun was right all along with the cake analogy.

Obviously I still think something like RL (optimizing for long term goals) is fundamental to what we think of as intelligence. But it's not the volume of learning signal, it's the optimization on top of an already reasonable predictive model.

@ylecun @natashajaques Cakeology

The token distribution is probably the most important signal in the paper. It reinforces that intelligence may emerge from massive world-model formation first, while RL acts more like a refinement layer that sharpens reasoning toward specific objectives rather than creating it from scratch.

What MAI-Thinking-1 seems to suggest is that prediction remains the foundation, but optimization determines how effectively that knowledge is deployed toward long-term goals.