We trained language models that compress massive contexts into tiny latent representations. Latent Context Language Models (LCLMs) outperform existing KV cache compression methods on the latency/accuracy frontier. 🧵1/10

Long-context inference is becoming a massive bottleneck for AI systems. Agents increasingly need to reason over huge codebases, long chats, reasoning traces, etc. Naively scaling context is prohibitively expensive. 2/10

How far can we compress the discrete tokens in an LLM's context into compact latent vectors?

With the right training recipe at large scale, our Latent Context Language Models (LCLMs) compress context up to 16× and land on a new Pareto frontier for long-context inference. 🧵(1/n)

Paper📝: https://arxiv.org/abs/2606.09659 Models🤖: https://huggingface.co/latent-context Code💻: https://github.com/LeonLixyz/LCLM 10/10

models: https://huggingface.co/latent-context code: https://github.com/LeonLixyz/LCLM arxiv: https://arxiv.org/abs/2606.09659

With the right architecture + training recipe, learned compression works far better than prior work suggested. We trained 4×, 8×, and 16× compressors jointly with pretrained decoders on 350B+ tokens, and we tested out tons of architectures and staged training pipelines. 5/10

Our models establish a new Pareto frontier on long-context benchmarks like RULER, LongBench, and LongHealth. At high compression ratios, we see much lower memory consumption, substantially faster TTFT, and strong long-context accuracy. 6/10

Most existing approaches compress the KV cache after the model processes the entire context. That means you still pay the full prefill cost, latency remains high, and many of those methods are hard to deploy in real systems. 3/10

Instead, we use a small encoder to convert raw text into compact latent representations in small chunks. The larger decoder then only takes in those compressed latent representations. This technique does not require the full prefill, and it is hardware and software friendly. 4/10

We believe that compression architectures can meet the growing context demands of agents and reasoning models. How to plug these models into complex agentic systems and how to compress text online while generating are open problems. 8/10

We then built an agentic system where the model globally reasons over compressed context and selectively expands regions on demand. This lets an agent “skim” enormous corpora before zooming in on precise details. 7/10

Huge thanks to all my amazing collaborators who made this possible: @iamleonli, @SeanMcleish, @tonychenxyz, @qw3rtman, @tingtang222, @artemg314, Suhas, and more. Ang (Leon) led the charge here and did an enormous amount of work on this project! 9/10