GDN-2 Adds Dual Gates to Boost Long-Context Retrieval Performance

I started a new blog, so you can find the full write-up there - https://www.the-information-bottleneck.com/p/editing-a-compressed-memory

Tagging the expert @ahatamiz1

12/ The results are about what you'd expect: small gains on language modeling, larger ones on long-context retrieval with many keys, where interference is worst. In the ablation, most of the improvement comes from the erase gate, not the write gate.

13/ The point is to manage a fixed-size memory better, not make it bigger, and none of it matches softmax's exact recall. That's why current models are hybrid: they use several linear blocks per full-attention block.

@ziv_ravid This is pretty informative. Thanks for the nice blog.

9/ The erase factor makes each step depend on the previous one, so the naive version is sequential and slow on a GPU. DeltaNet runs it in chunks: within a chunk, it's matrix multiplication and a triangular solve, and only the state is carried over between chunks.

10/ The rest are refinements that keep that chunked solve working:

Gated DeltaNet: a scalar decay before each write

KDA (Kimi Linear): a per-channel decay

Gated DeltaNet-2: separate erase and write gates

11/ GDN-2 splits the write strength into two channel-wise gates: one for erasing (key side), one for writing (value side). The forward pass is the same as KDA. The backward pass needs a new kernel, because the gate sits inside a sum over channels and can't be pulled out.

7/The delta rule (Widrow-Hoff 1960; DeltaNet 2024) reads what the key currently returns, then writes the difference toward the target:

S ← (I − β kkᵀ)S + β kvᵀ

When we have a new key, the read is ~0, and it acts as a plain add. When we have a key that is similar to the previous one, the old value is subtracted and replaced.

8/ β is the write strength, learned per token. At 1 it fully overwrites. Below 1 it writes more gently, which matters because keys aren't orthogonal, so a hard write along one key also perturbs the keys that overlap it.

@ahatamiz1 Thanks! Please let me know if you have comments/errors