Section 5: Living With the Necessary Evil — GQA, Quantization, and the Bandwidth Fight

Once you understand that the KV cache's core problem is memory bandwidth — the truck having too much to carry — the design space for solutions becomes clear. You either make the cache smaller, or you make the cache denser (more information per byte). Every major technique in this space is doing one of those two things.