🚀New paper on Looped Transformers! Latent reasoning is fast, but struggles to match CoT-level accuracy at scale. Can looped Transformers give us both? We find: yes! A looped padded backbone turns out to be a surprisingly simple recipe that works -- It gives latent thoughts a parallel workspace that can be supervised similarly to explicit CoT. 🧵

1/ We present Looped Transformers with parallel supervision on latents (LOTUS). It inserts K latent blocks between the question and answer, then loops the base model over them R times to get the post-loop latents. Two simple losses for training: • Latents: plain cross-entropy on the gold CoT tokens, read through the model's own LM head. • Answer: standard next-token prediction, conditioned on those post-loop latents.

🚀New paper on Looped Transformers! Latent reasoning is fast, but struggles to match CoT-level accuracy at scale. Can looped Transformers give us both? We find: yes! A looped padded backbone turns out to be a surprisingly simple recipe that works -- It gives latent thoughts a parallel workspace that can be supervised similarly to explicit CoT. 🧵

5/ The latents don't just memorize the trained chains: They also surface valid steps the model was never trained on. For example, consider two valid chains reaching the same answer (108): Trained: 3×2=6 → 6×12=72 → 2×2=4 → 4×9=36 → 72+36=108 Alternate: 2×12=24 → 24×3=72 → 2×9=18 → 18×2=36 → 72+36=108 24 and 18 are numbers from the untrained chain only, while they still show up in the readout with nontrivial mass compared with random control, despite never appearing in the trained chain, question, or answer.