Linear Attention and the Delta Rule: Building Kimi Delta Attention

From softmax attention to linear attention to the delta rule, building up to Kimi Delta Attention (KDA): per-channel learnable forgetting via a diagonal-plus-low-rank transition, why it powers the 1M-token context of Kimi K3, implemented from first principles with a working long-context toy model.