With two tweaks, Kimi Delta Attention is claimed to handle 2D rotations that Gated DeltaNet can't
A user compares two forms of linear attention used in LLMs, saying Kimi Delta Attention replaces Gated DeltaNet's scalar gate with a channel-wise gate. They claim two further tweaks let it handle 2D rotations.
TLDR
A user identifies Gated DeltaNet with Qwen3.8 and Kimi Delta Attention with Kimi K3. They say Kimi Delta Attention uses a channel-wise gate instead of Gated DeltaNet's scalar gate, and claim that, with two tweaks, it can perform 2D rotations while Gated DeltaNet cannot.
Combined views
10
1 Source, first seen 2h ago
reposts