AIToken superposition training reportedly reaches comparable or better loss with about 2.5× less compute in a 10B MoE test
A post describes a method that first combines neighboring tokens into shared representations, then switches to standard next-token prediction. Its author flags a caveat: the approach assumes compute, rather than data, is the bottleneck.
TLDR
A post discussing Nous Research’s token superposition training describes averaging neighboring tokens’ embeddings—their numerical representations—into one latent token that predicts the next non-overlapping group. With groups of four, the transformer processes roughly four times fewer sequence positions, allowing more raw text for the same compute. Training then switches back to standard next-token prediction to learn exact ordering. The post reports about 2.5× less compute to reach comparable or better loss in a 10B mixture-of-experts (MoE) experiment. The author cautions that the approach assumes compute, not data, is the bottleneck, which they say does not match current conditions.
Combined views
5.2K
2 Sources, first seen 3h ago
Token superposition training reportedly reaches comparable or better loss with about 2.5× less compute in a 10B MoE test
A post describes a method that first combines neighboring tokens into shared representations, then switches to standard next-token prediction. Its author flags a caveat: the approach assumes compute, rather than data, is the bottleneck.