Nestor routes tokens through both attention modules and experts

Nestor’s researchers describe a transformer that routes each token to selected attention modules, then to experts within them. In pretraining tests, they report the lowest held-out loss at each tested scale and estimate that MoE models need 1.13–1.29 times as much compute to match Nestor’s loss. In four domain-adaptation tests, Nestor showed 3% less forgetting than MoE on average, they say.