The Tokyo-based lab will share research alongside Agentic Learning AI Lab.
Sakana AI is heading to #ICML2026 in Seoul (July 6–11)! 🐟🇰🇷
Our team will present 11 papers spanning multi-agent coordination, sparse and efficient LLMs, test-time scaling, long-term memory, and agent benchmarks. A thread of everything we're presenting:
Users are excited about Sakana AI presenting 11 papers at ICML 2026 because of the impressive research output and focus on improving systems around models rather than the models themselves.
No Digg Deeper questions have been answered for this story yet.
"Sparser, Faster, Lighter Transformer Language Models" will be presented at #ICML2026
Paper: https://arxiv.org/abs/2603.23198 Blog: https://pub.sakana.ai/sparser-faster-llms/
In collaboration with @NVIDIAAI, we introduce TwELL, a new sparse packing format designed to integrate with optimized tiled GPU workloads and develop custom CUDA kernels to accelerate LLM training and inference. We use our kernels to train and benchmark sparse models at billion-parameter scales, demonstrating over 20% speedups and even higher savings in peak memory and energy.
"SoftMatcha 2: A Fast and Soft Pattern Matcher for Trillion-Scale Corpora" will be presented at #ICML2026
Paper: https://arxiv.org/abs/2602.10908 Project page: https://softmatcha.github.io/v2/
What's really hiding in your trillion-token pre-training corpus? Exact string search won't tell you. SoftMatcha 2 searches 1.4T tokens in <0.3s with soft, word-order-aware matching, and finds benchmark contamination others miss.
"CoffeeBench: Benchmarking Long-Horizon LLM Agents in Heterogeneous Multi-Agent Economies" will be presented at the Failure Modes in Agentic AI Workshop on July 10 at #ICML2026
Paper: https://icml.cc/virtual/2026/77968
We introduce CoffeeBench, a benchmark for evaluating how much net income an LLM agent can generate as a coffee roaster over 90 days in a multi-agent economy with two farmers, two roasters, and two retailers. We observe diverse behaviors across models, including an idle-drift failure mode in Claude Haiku 4.5, resulting in substantial differences in business performance.
"Feedback-to-Rubrics: Can We Extract Expert Criteria from Inline Comments?" will be presented at the Workshop on Human-AI Co-Creativity: Advances, Opportunities, and Challenges on July 11 at #ICML2026
Paper: https://arxiv.org/abs/2605.29857 (Extended full-paper version)
LLMs are increasingly used for writing and review support, but their usefulness depends on context-dependent criteria, such as expert preferences or organization-specific conventions, that are often tacit, undocumented, and difficult to elicit directly. We propose a problem setting for learning reusable natural-language rubrics from accumulated inline comments on artifacts such as human-written or LLM-generated drafts.
"Doc-to-LoRA: Learning to Instantly Internalize Contexts" will be presented at #ICML2026
Paper: https://openreview.net/forum?id=iW1oBBO72S
Long-term memory is one of the most important cognitive capabilities that LLMs still lack today. Without long-term memory, users have to provide LLMs with relevant content at the start of every new session, creating friction, discontinuity, and longer time-to-response. Doc-to-LoRA tackles the long-term memory problem directly. Given a document, it generates a LoRA adapter that internalizes the document’s content into the base model’s weights. Subsequent questions can then be answered without the document ever appearing in the context window—cutting both latency and VRAM, and adding long-term memory well beyond the model’s native context limit.
"Bridging Spherical Black-Box Optimizers" will be presented at #ICML2026
Paper: https://arxiv.org/abs/2606.25761
Black-box optimizers can be used when gradients are unavailable. We show that multiple different such optimizers from different families can be written using a single unified update equation, and can use this update to design new optimizers. We then apply them to LLM model merging, reducing the training cost to a fraction.
"RePo: Language Models with Context Re-Positioning" will be presented at #ICML2026
Paper: https://openreview.net/pdf?id=RsLH9NuYKK Standard language models process information as a rigid linear sequence where the only signal for structure is a fixed token index, forcing them to treat physical proximity as semantic relevance. This is inefficient, since when key facts are buried in noise, models waste finite capacity managing disorganized inputs instead of focusing on deep reasoning. RePo breaks this bottleneck by allowing models to actively reorganize their context. Rather than using a fixed index, our module learns to assign positions based on content relevance. This flexibility yields significant gains in robustness.
"Fast-weight Product Key Memory" will be presented as an oral paper at the Continual Adaptation at Scale: Towards Sustainable AI Workshop at #ICML2026
Paper: https://arxiv.org/abs/2601.00671
Can language models build a huge memory at inference time without updating all their weights? FwPKM combines Test-Time Training (TTT) with Product Key Memory (PKM) into sparse fast weights: large storage, sparse access, and test-time writes. It reaches ~75% 5-needle NIAH accuracy at 128K context length, and quickly adapts to a new domain with up to -0.4 ΔNLL after reading 4M tokens from the target domain.
Come and chat with us in Seoul at ICML!
Agentic Learning AI Lab @agentic_ai_lab presents at #ICML2026 in Seoul✈️🇰🇷 Come check out our latest research!
📍Main Conference Papers Temporal Straightening for Latent Planning Tue Jul 7 10:30am Hall A #1509
Context Tuning for In-Context Optimization Wed Jul 8 5pm Hall A #2510
📍Workshop Papers The Surprising Effectiveness of Deleting Weights in LLM Reasoning and Adaptation Fri Jul 10 Rm 318 | Foundations of Deep Generative Models Workshop
Seeking the Unfamiliar but Memorable: Conceptual Creativity as Meta-Learning Sat Jul 11 Rm 401 | Human-AI Co-Creativity Workshop
Aligning LLMs with Human Uncertainty: A Beta-Bernoulli Calibrator for LLM Forecasting Sat Jul 11 Gr. Ballroom 104–105 | Forecasting as a New Frontier of Intelligence Workshop
Alive and Predicting: A Live Evaluation of Multi-Step Forecasting Agents Sat Jul 11 Gr. Ballroom 104–105 | Forecasting as a New Frontier of Intelligence Workshop
See threads below for details!🧵👇
"UnMaskFork: Test-Time Scaling for Masked Diffusion via Deterministic Action Branching" will be presented at #ICML2026
Paper: https://arxiv.org/abs/2602.04344
We introduce UnMaskFork, a test-time scaling framework for Masked Diffusion Language Models (MDLMs). Using Monte Carlo Tree Search, it explores diverse generation paths by dynamically switching between multiple pre-trained MDLMs to collaboratively generate the text. Evaluations on coding and mathematical reasoning tasks show that UnMaskFork consistently outperforms standard Best-of-N and other tree search baselines. The results demonstrate that deriving search space diversity from multiple distinct models is a highly effective test-time scaling strategy for MDLMs.
"How Small Can a Tandem Speech Front-End Be? Diagnosing Front-End Capacity with Layer Removal" will be presented at the Workshop on Machine Learning for Audio on July 10 at #ICML2026
Link: https://icml.cc/virtual/2026/77315
Recent tandem speech-to-speech (S2S) dialogue systems delegate knowledge and reasoning to a back-end LLM, leaving a front-end S2S transformer to handle low-latency spoken interaction. This raises a practical capacity question: how small can the front-end become, what performance is preserved, and how does its behavior change as layers are removed? To diagnose this, we randomly remove different numbers of layers from a KAME-style front-end transformer, fine-tune each variant on the same data, and evaluate on speech MT-Bench.
"Learning Multi-Agent Coordination via Sheaf-ADMM" will be presented at #ICML2026
Paper: http://arxiv.org/abs/2605.31005 Blog: https://pub.sakana.ai/sheaf-admm/
We propose a differentiable framework where agents solve local convex subproblems and coordinate via ADMM, with a network sheaf specifying which parts of neighboring solutions must agree. Agents with individually insufficient local views coordinate to produce correct global outputs, with improved robustness to distribution shift.
@SakanaAILabs showing practical use cases for your findings could spark collaborations with industry, making the research more impactful beyond academia.
@SakanaAILabs @hardmaru Amazing
@SakanaAILabs 11 papers is wild output — the multi-agent coordination and long-term memory ones are where I'd be camping out
@SakanaAILabs welcome to Korea🇰🇷
@SakanaAILabs wow nice. what is the fish on the lefts name?
@SakanaAILabs @NVIDIAAI sakana image model when
@SakanaAILabs The pattern across all 11 papers: none of them are "make the model smarter."
Every single one is "make the system around models smarter" — coordination, memory, verification, scaling at test time.
That's the whole bet, and TRINITY → Fugu already showed it pays off.
@SakanaAILabs
