Poolside AI releases technical report for Laguna M.1 and XS.2 models detailing its Model Factory infrastructure · Digg

Poolside AI releases technical report for Laguna M.1 and XS.2 models detailing its Model Factory infrastructure · Digg

Posts from X

Most Activity

VIEWS39.9KBOOKMARKS334LIKES454RETWEETS40REPLIES11

elie@eliebakouch

wow, amazing tech report. lots of details on every part of the pipeline, especially on data. love that they share the system design of how they train models and do research with their "model factory", and also the negative results from M1 and how they fixed them in XS.2

one of the best tech reports to get up to speed on model training

Poolside@poolsideai

Today we’re publishing the technical report behind Laguna M.1 and Laguna XS.2.

This report opens up more of what went into them: Model Factory, pre-training data, distributed training, post-training, agent RL, quantization, and evaluation.

https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf

25d39.9K454334

wh@nrehiew_

Really like this tech report from the Poolside team on the Laguna models.

Here are some of my notes on the parts I found interesting:

24d5.8K8856

Andrew Carr 🤸@andrew_n_carr

This is a great read. One thing stands out. True long context data is scarce: 30T total tokens and only 200B for long context extension.

Poolside@poolsideai

Today we’re publishing the technical report behind Laguna M.1 and Laguna XS.2.

This report opens up more of what went into them: Model Factory, pre-training data, distributed training, post-training, agent RL, quantization, and evaluation.

https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf

25d4.3K387

Niels Rogge@NielsRogge

The technical report of @poolsideai is now available here too: https://paperswithcode.co/paper/83726

See the evals at the bottom to compare to others!

Poolside@poolsideai

Today we’re publishing the technical report behind Laguna M.1 and Laguna XS.2.

This report opens up more of what went into them: Model Factory, pre-training data, distributed training, post-training, agent RL, quantization, and evaluation.

https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf

25d9.2K306

elie@eliebakouch

lots more in the tech report about rl, infra ect..

https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf

elie@eliebakouch

wow, amazing tech report. lots of details on every part of the pipeline, especially on data. love that they share the system design of how they train models and do research with their "model factory", and also the negative results from M1 and how they fixed them in XS.2

one of the best tech reports to get up to speed on model training

25d1.2K117

wh@nrehiew_

Really encourage everyone to read the paper, its super detailed and interesting and theres quite abit of info I did not include. Specifically, the paper goes into quite abit of detail about their internal "Model Factory".

Essentially, its a detailed description of the different pieces of their stack and the associated technologies/platforms they use. Not at all common in tech reports.

https://poolside.ai/assets/laguna/laguna-m1-xs2-technical-report.pdf

wh@nrehiew_

Quantization is also a heavy focus with KV Cache being done in FP8.

For INT4 quant, they find that the later layers have massive outlier activations. So they quantize the first 30 layers to INT4 and the last 10 to INT8

They were also unable to do NVFP4 directly without performance drops so they basically did distillation against the higher precision teacher specifically on the mlp weights

24d654123

wh@nrehiew_

They also kindly shared some of the issues they faced during training. - Expert collapse with standard Muon without Moonlight implementation - FP32 needed for the lm head input gradient all reduce - Padding tokens are not learnable and always routed to one particular expert. Solution is to not bother routing or load balancing padding tokens

wh@nrehiew_

Quite a bit of info on their distributed training setup.

They do EP8, fusing the dispatch and combine directly with the grouped GEMM kernel which allows the All-to-alls to overlap.

They do batch sharded attention instead of ring-style context parallel. So A2A, vanilla attention, A2A which simplifies the attention implementation across entire sequence

Muon is pretty standard, each rank gets 1 full parameter so it just needs to gather 1 full parameter

TP used only for the embeddings and LM Head

24d41783

wh@nrehiew_

Pretraining is > 30T tokens samples from 27T (so some batches are seen more than once) - High quality web data. The paper has a table on how they annotate quality. They find that quality is too nuanced to filter based on a single score so they just use the score to determine sample probability - 4.4T of synthetic data with details about the pipeline. Ton of detail in Table 2 - For the data mix, they train a bunch of small models on different mixtures. Then fit on the downstream eval score, and finally jointly optimize the coefficients specifically focusing on STEM. This leads to 30.6% math, 25.2% web, 25.4% synthetic, 9% math

wh@nrehiew_

They also kindly shared some of the issues they faced during training. - Expert collapse with standard Muon without Moonlight implementation - FP32 needed for the lm head input gradient all reduce - Padding tokens are not learnable and always routed to one particular expert. Solution is to not bother routing or load balancing padding tokens

24d39393

wh@nrehiew_

SFT is 3 epochs of 40B tokens each, with a 85% focus on agentic trajectories. - They synthetically generate coding tasks from git commits and use teacher generated ground truth trajectories - They heavily focus on instruction following for the next RL stage. Instructions are synthetically generated and applied to the coding trajectories - Trajectories are generated across different harnesses OpenHands, OpenCode and Mini-SWE-Agent

wh@nrehiew_

For midtraining, they first add special tokens. They find that in the larger model, these embeddings were too different and led to instability which required a warmup phase.

They also specifically call out the template mismatch problem. Basically the Renderers library

Overall mid training is 60B tokens at 131K sequence length, focusing on logic, reasoning, coding agents, and general chat

24d36452

wh@nrehiew_

Quite a bit of info on their distributed training setup.

They do EP8, fusing the dispatch and combine directly with the grouped GEMM kernel which allows the All-to-alls to overlap.

They do batch sharded attention instead of ring-style context parallel. So A2A, vanilla attention, A2A which simplifies the attention implementation across entire sequence

Muon is pretty standard, each rank gets 1 full parameter so it just needs to gather 1 full parameter

TP used only for the embeddings and LM Head

wh@nrehiew_

The architecture is pretty straightforward MoE SWA etc. Training is done in BF16 with FP32 RMSNorm and RoPE

They fit a scaling law to the WSD schedule to find the optimal peak learning rate across token budgets.

24d1K71

wh@nrehiew_

The architecture is pretty straightforward MoE SWA etc. Training is done in BF16 with FP32 RMSNorm and RoPE

They fit a scaling law to the WSD schedule to find the optimal peak learning rate across token budgets.

wh@nrehiew_

Really like this tech report from the Poolside team on the Laguna models.

Here are some of my notes on the parts I found interesting:

24d1.2K80

wh@nrehiew_

Quantization is also a heavy focus with KV Cache being done in FP8.

For INT4 quant, they find that the later layers have massive outlier activations. So they quantize the first 30 layers to INT4 and the last 10 to INT8

They were also unable to do NVFP4 directly without performance drops so they basically did distillation against the higher precision teacher specifically on the mlp weights

24d90

wh@nrehiew_

RL uses CISPO IS ratio clipping and a leave one out baseline calculation. The clipping is quite expansaive so only the heavily off policy tokens are clipped out

They also provide some info about the different reward functions they use.

Tasks are TerminalBench/SWE-Bench/Math reasoning using code.

Weight syncs are done directly via NCCL rather than something like filesystem/host memory offloading. They generally have between 2-3x more inference replicas than training.

Broadcasts are done every 2 steps, reseting the kv cache. They dont do full async with all current generating stages blocked until the update is finished.

Inference is done in FP8 with requantization happening on the inference workers each weight sync. They note no observable instability from this setup

24d20

wh@nrehiew_

For midtraining, they first add special tokens. They find that in the larger model, these embeddings were too different and led to instability which required a warmup phase.

They also specifically call out the template mismatch problem. Basically the Renderers library

Overall mid training is 60B tokens at 131K sequence length, focusing on logic, reasoning, coding agents, and general chat

24d15

yucel@17Ahmetyucel

@eliebakouch From my rapid skim what I see is that they do no use mup, but rather construct scaling laws for lr. I also saw some other models doing the same. But also read a lot about mup working great. Curious about what you think.

25d1211

Saylor@seylorra

@eliebakouch the "negative results from M1" section is what more reports should have tbh. easy to publish wins, rare to document what broke first

25d82

stochasm@stochasticchasm

@eliebakouch @ezyang can you open source your claude prompt for these lol

24d11

KALALA NZENIELE@cniongolo

@mgalle Quels sont les restrictions pour un SFT avec le Laguna XS.2 ?

25d11

Saylor@seylorra

@andrew_n_carr the ratio alone tells you everything about where research focus actually goes

30T sounds massive til you realize how shallow most of it is

25d6

Grim@justgrm

@andrew_n_carr so basically most of that 30T is useless for real long context training

kinda wild how little useful data there actually is

25d4