Sangyun Lee and Giulia Fanti propose a "sleep" phase to convert LLM context into fast weights and clear KV cache
Offline consolidation passes reduce quadratic attention costs during inference.
abs: https://arxiv.org/abs/2605.26099
Tanishq Mathew Abraham, Ph.D.@iScienceLuvrLanguage Models Need Sleep "Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights before clearing its key-value cache." "increasing sleep duration N for our models improves performance, with the largest gains on examples that require deeper reasoning."
Language models need "sleep"
#885Andrey Kurenkov@ANDREY_KURENKOV"Sleep-like consolidation pattern" really is a slick way to say 'Test-time training' (which appears to be the most relevant prior work - no shade, good on them for having a solid related works section!)
Oh oops there is also this:
himanshu@himanshustwtsvery cool research (and nomenclature)