Section 1: The GPU Has Two Masters — VRAM and Bandwidth

A GPU is fundamentally a machine for doing a very large number of multiplications and additions very quickly. Modern GPUs used for deep learning can perform tens of trillions of such operations per second. That number is almost incomprehensible in its scale. If you handed a human a pencil and asked them to perform one multiplication per second, they would need millions of years to match what a GPU does in a single heartbeat.