Latent Context Language Models compress context tokens up to 16x, cutting time-to-first-token by 8.8x on the RULER benchmark

We trained language models that compress massive contexts into tiny latent representations. Latent Context Language Models (LCLMs) outperform existing KV cache compression methods on the latency/accuracy frontier. 🧵1/10

@micahgoldblum Compression architecture == ConvNet-style hierarchy through pooling/stride.

Paper📝: https://arxiv.org/abs/2606.09659 Models🤖: https://huggingface.co/latent-context Code💻: https://github.com/LeonLixyz/LCLM 10/10

How far can we compress the discrete tokens in an LLM's context into compact latent vectors?

With the right training recipe at large scale, our Latent Context Language Models (LCLMs) compress context up to 16× and land on a new Pareto frontier for long-context inference. 🧵(1/n)

With the right architecture + training recipe, learned compression works far better than prior work suggested. We trained 4×, 8×, and 16× compressors jointly with pretrained decoders on 350B+ tokens, and we tested out tons of architectures and staged training pipelines. 5/10

Instead, we use a small encoder to convert raw text into compact latent representations in small chunks. The larger decoder then only takes in those compressed latent representations. This technique does not require the full prefill, and it is hardware and software friendly. 4/10

Our models establish a new Pareto frontier on long-context benchmarks like RULER, LongBench, and LongHealth. At high compression ratios, we see much lower memory consumption, substantially faster TTFT, and strong long-context accuracy. 6/10

models: https://huggingface.co/latent-context code: https://github.com/LeonLixyz/LCLM arxiv: https://arxiv.org/abs/2606.09659

We then built an agentic system where the model globally reasons over compressed context and selectively expands regions on demand. This lets an agent “skim” enormous corpora before zooming in on precise details. 7/10

Long-context inference is becoming a massive bottleneck for AI systems. Agents increasingly need to reason over huge codebases, long chats, reasoning traces, etc. Naively scaling context is prohibitively expensive. 2/10

Most existing approaches compress the KV cache after the model processes the entire context. That means you still pay the full prefill cost, latency remains high, and many of those methods are hard to deploy in real systems. 3/10

We believe that compression architectures can meet the growing context demands of agents and reasoning models. How to plug these models into complex agentic systems and how to compress text online while generating are open problems. 8/10

Huge thanks to all my amazing collaborators who made this possible: @iamleonli, @SeanMcleish, @tonychenxyz, @qw3rtman, @tingtang222, @artemg314, Suhas, and more. Ang (Leon) led the charge here and did an enormous amount of work on this project! 9/10

KV cache memory grows linearly and attention compute quadratically with context length, so longer context means explosive memory and slower decoding. We train LCLMs to fold context into a handful of latent vectors while keeping the underlying model's abilities intact. (2/n)

The setup: A 0.6B encoder compresses raw tokens into latents, an adapter maps them into the decoder's space, and a 4B decoder reads them as latent context in place of the original tokens. We train the pair end-to-end on 350B+ tokens, at 4×, 8×, and 16× compression ratios. (3/n)

We are not the first to compress context. Prior work mostly evicts KV cache entries using hand-designed heuristics. Against SnapKV, KVzip, Expected Attention, and Attention Matching, LCLMs match or beat accuracy with up to 8.8× faster TTFT and far less memory. (4/n)

Here is our vision. Long-horizon agents with large, cheap, persistent working memory that skim compressed context and zoom in on demand. In the future, we hope to see compression as a general-purpose latent memory for any long-context input: text, images, video, audios. (10/n)

The speedup is structural. KV cache methods prefill the whole context first, so their compression time barely changes with the ratio. LCLMs shrink the input before prefill, directly cuts decoder memory, scaling to 1M tokens where every baseline runs OOM on a single H200. (5/n)

The compressor can perform extremely well even on information-dense inputs. On GSM8K (avg 60 tokens per prompt) at 16× compression, with about 94% of tokens gone, a strong baseline like KVzip drops to 0% while LCLM achieves 81%. The full model scores 93%. (8/n)