/Tech3h ago

ICML Paper Presents RAT+ Architecture For Efficient Sparse Inference

2261433.6K

#836

Original post

Caglar Gulcehre@caglarml#836inTech

Go and check out our paper at ICML and make sure to ask any of your questions to @XiuyingWei966!

Xiuying Wei @ICML🇰🇷@XiuyingWei966

Excited to present our #ICML 🇰🇷 paper tomorrow morning at Hall A #2911!

🧩RAT+: an upstream dense architecture that enables more flexible and accurate downstream sparse inference.

Compared to attention: 1) RAT+ enables flexible dilated pattern at inference (up to 64x KVCache reduction). 2) Query-aware sparsity patterns, e.g., top-k, perform much better with RAT+.

Happy to chat about anything!

5:20 AM · Jul 6, 2026 · 1.2K Views

Sentiment

Sentiment building, check back later.

Cluster Engagement

Digg Deeper

No Digg Deeper questions have been answered for this story yet.

Posts from X

Most Activity

RETWEETS7

Xiuying Wei @ICML🇰🇷@XiuyingWei966

Excited to present our #ICML 🇰🇷 paper tomorrow morning at Hall A #2911!

🧩RAT+: an upstream dense architecture that enables more flexible and accurate downstream sparse inference.

Compared to attention: 1) RAT+ enables flexible dilated pattern at inference (up to 64x KVCache reduction). 2) Query-aware sparsity patterns, e.g., top-k, perform much better with RAT+.

Happy to chat about anything!

5h2K121