Muir’s researchers report processing 10-million-token streams with a fixed-size working state, while maintaining stable next-token prediction scores. Their proposed linear-attention architecture compresses context rather than treating it as a lossless archive.
Tushaar Gangavarapu, Gil Tabak and Clemens JS Schaefer describe the design in “Muir: Linear Attention with Sensitivity-Allocated Gating and Matched-Filter Readout,” whose abstract appears on Hugging Face. At 1.34 billion parameters, they report extending from a 4K-token training length to 10-million-token streams without drift in perplexity, a measure of next-token prediction performance.
The memory figure comes from a comment by paper submitter tushaar, who describes streaming 10 million tokens in a constant 25 MB state. That figure describes the working state, not total model memory; the comment does not provide a memory-accounting breakdown.
How Muir changes the working state
The researchers identify a problem with how information is read from the state: rows that decay slowly accumulate much larger magnitudes than faster-decaying rows. Those larger rows can dominate the readout, requiring stronger erasure when new information is written to suppress interference.
Muir addresses that imbalance with a “matched-filter readout gate” that reweights the state’s rows. The authors also combine the gates that control state updates into a sensitivity-allocated, low-rank projection and put the saved parameters into a wider feed-forward network. Their abstract describes a parallel update designed to keep training stable and efficient at large chunk sizes.
What the benchmarks establish
The long-stream prediction result and the retrieval result are different tests. The authors report near-100% passkey retrieval at one million tokens using static readout; tushaar’s comment gives the figure as 99.4%. That is a specific retrieval benchmark at one million tokens, not a demonstration of perfect recall across the 10-million-token stream.
The authors also report training 2.7 times faster than their Transformer baseline at a 32K-token sequence length. The submitter comment identifies the comparison as Splash Attention. This is a training-speed result at that sequence length, not a general claim about inference speed or every Transformer implementation.
