Section 3: Enter the KV Cache — The Hero We Needed

The key observation is deceptively simple. When you move from generating token 1,001 to generating token 1,002, what changes? The new query is different — it comes from the new token. But the keys and values for the previous 1,001 tokens? They are completely unchanged. A token's key and value are a function of that token and that token alone — they do not depend on what comes after. This means all that work we did in the previous step to compute those keys and values was not wasted — we just fai