Marin 535B Model Starts Open Pretraining on 18.75T Tokens
Researchers begin transparent pretraining of a 535B-parameter model using public data and configs.
Posts from researchers and engineers report that the Marin project has started pretraining its 535B-parameter model on 18.75 trillion tokens. The run uses 11 NVIDIA GB200 NVL72 systems and is projected to last roughly three months. Participants describe the effort as fully open, with public pretraining mixtures, sampled documents, live loss curves, and training configs. Replies highlight the scale and transparency while comparing it to prior open initiatives. No independent verification of completion or final results appears in the packet.
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open. Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow. Before kicking off the run, we trained a 4-rung…
Reactions from ranked influencers
17 postsmarin is the most open model training initiative in the world you can visualize the exact pre training mixture per domain, look at sampled documents, follow the live training loss, dig into configs, scaling laws
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open. Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow. Before kicking off the run, we trained a 4-rung…
> marin is the most open model training initiative in the world they are also super ambitious and smart btw
@percyliang amazing job as always, gl for the hero run 🔥
@percyliang Good luck!!!!
5 years ago, almost to the day, @siddkaramcheti @laurel_orr1 @percyliang put out their first public writing on training open models. Even then, model training was not for the faint of heart. Mistral (before there was @MistralAI) was defined by 7 principles: simplicity,…
🚢 Marin 535B-A23B started training this week! As usual, the whole process is open. Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow. Before kicking off the run, we trained a 4-rung…
pre-registering your pretraining loss forecast is the ultimate confidence move. 535B moe training happening all in the open.
In addition to forecasting our final loss, we can also forecast the loss of intermediate checkpoints, so we can see if we’re on track over the voyage:
