DeltaNet Parallelizes Chunk Training Using Matrix Ops And Triangular Solves

I started a new blog, so you can find the full write-up there - https://www.the-information-bottleneck.com/p/editing-a-compressed-memory

Tagging the expert @ahatamiz1

10/ The rest are refinements that keep that chunked solve working:

Gated DeltaNet: a scalar decay before each write

KDA (Kimi Linear): a per-channel decay

Gated DeltaNet-2: separate erase and write gates

13/ The point is to manage a fixed-size memory better, not make it bigger, and none of it matches softmax's exact recall. That's why current models are hybrid: they use several linear blocks per full-attention block.

12/ The results are about what you'd expect: small gains on language modeling, larger ones on long-context retrieval with many keys, where interference is worst. In the ablation, most of the improvement comes from the erase gate, not the write gate.

11/ GDN-2 splits the write strength into two channel-wise gates: one for erasing (key side), one for writing (value side). The forward pass is the same as KDA. The backward pass needs a new kernel, because the gate sits inside a sum over channels and can't be pulled out.