InfiniAI Lab releases Vortex, an agent-designed sparse attention framework that accelerates LLM inference throughput by up to 4.7x

Vortex builds a layer for the current AI agents to make themselves more efficient, if you view AI agents as a species. We are soon to see AI agents directly program on an FPGA, defining efficient and expressive architecture, writing efficient low-level implementations, and training themselves. And, I do not know what can stop this. At that moment, whether AI can conduct research on something novel (rather than simply replicate what humans can already do) will determine whether 2025-2026 is a real genesis moment.

Agents providing improvements in production environments with SGLang. Very cool work!

📢Vortex feels like an early glimpse of recursive self-improvement for ML systems: agents discovering better architectures, compiling them into real serving stacks, and shifting the bottleneck from "can it code?" to "can we still steer it?"

AI building AI just went from theory to benchmark chart.

We're excited to see Vortex built on SGLang! By integrating directly into the serving stack, Vortex turns theoretical sparse-attention speedups into real end-to-end gains. It's a great example of why a programmable, production-grade serving layer matters. Excited to build this with the community! 👏

[2/6] What if the AI agent did the research? 🤔

Hand it Vortex + a goal, and it just goes: ✍️ write a flow → 🚀 compile to real kernels → 📊 benchmark → 🔁 improve → repeat.

18 hours later, fully on its own: 92 algorithms over 23 rounds, frontier pushed way out — 3.46× faster on AIME24 (3,437 → 11,894 tok/s), zero accuracy lost. 🔥

[3/6] Not a fluke either. 🎯

Three different frontier agents — Claude Opus 4.7, Claude Sonnet 4.6, GPT-5 — each generate structurally diverse designs, and after a staged filtering pipeline, the selected ones are efficient: full-attention accuracy at 2–3.1× higher throughput across three benchmarks.

The design space is theirs to explore. 🤖

[4/6] New architecture? No problem. 🧬

MLA (the attention behind DeepSeek & GLM) is tricky — it squeezes the KV cache into one shared latent. In Vortex, we sketched a rope-aware block sparse attention for it in a few lines.

GLM-4.7-Flash result: up to 4.7× faster, matching full attention on mean@16, pass@4 & pass@8. ⚡

[5/6] Does it scale? We went to 229B. 🏔️

At this size — MiniMax-M2.7 across 4× B200 (TP=4) — even *running* a sparse-attention experiment is basically impossible without Vortex.

With it: up to 1.37× faster on AIME26, accuracy even nudging *above* full attention. Sparse attention still pays off at the frontier of model size. 💪

[6/6] The big picture 👇

Sparse-attention research should be a loop an agent can run on its own. The hard part was never the idea — it was turning math into fast, production-ready kernels. Vortex removes that wall.

When trying, an algorithm is as easy to describe as it is to implement; humans + AI agents can co-discover the next generation of efficient attention. 🌀

Thanks to @chenzhuoming911, @XinruiZhongx, Qilong Feng, @RJ_Sadhukhan, @IronSteveZhou, @michaelqshieh, @JiaZhihao, @BeidiChen Come build it with us 👇

@lmsysorg Sparse attention speedups probably die in the scheduler before they reach the application. Batch heterogeneity eats them, and that's why you can't just bolt this on at the kernel level.