/TECH

Researchers Present Anatomy of Massive Activations and Attention Sinks at ICML

Source Posts on X

Cluster sources

Why to care? Massive activations wreck quantisation: the outlier spikes crush every normal value toward zero (why SmoothQuant, LLM.int8() exist). Attention sinks enable streaming: pin the sink tokens in the KV cache and the model runs on forever (StreamingLLM). (4/5)

Combined views

3.8K

5 posts, first seen 4h ago

34 likes4 comments

Alfredo Canziani @ ICML@alfcnzTECH

Ablations pin it: swap RMSNorm for DynamicTanh and spikes vanish; add conditional gating and sinks vanish. A sink is just a junk collector for spare attention, the model's way to turn a head down. Give it a gate and the junk goes. 📄 arXiv · 💻 code · 🖼️ poster #ICML2026 (5/5)

Alfredo Canziani @ ICML@alfcnzTECH

Mechanism: ① early FFN injects a giant spike in a few fixed channels; a later one cancels it. ② RMSNorm turns that spike token into a fixed, near one-hot direction, erasing its identity. ③ fixed token → fixed key. Near the query space, every query attends to it = sink. (3/5)

Alfredo Canziani @ ICML@alfcnzTECH

Transformers have two famous quirks: • massive activations (a few residual channels blow up huge) and • attention sinks (almost every query dumps its attention on one token). Our ICML 2026 work: they’re the same thing. Same tokens, same time. 🧵 (2/5)

View all

Alfredo Canziani @ ICML@alfcnzTECH

Come and learn about «Anatomy of Massive Activations and Attention Sinks», 📆 today (Thu) at 🕥 10:30, 📍 HALL A #3002. #ICML2026 https://x.com/alfcnz/status/2075012691241157038/photo/1

13 saves

3 reposts

Sentiment

Positive0%0%Negative

Guardrails removed spam, off-topic, unclear, or duplicate replies.

Digg Deeper

Ask a question below.

Published answers will appear here.