GPU Parallelism for Large Language Models
From a single GPU to thousands — understand every parallelism strategy used to train modern LLMs, one day at a time.
Pods in this Course
Intro to GPUs and GPU Parallelism for LLMs
What GPUs are, why they matter for deep learning, and a bird's-eye view of the five parallelism strategies that make LLM training possible.
GPU Memory — The Four Major Components
Understand where your GPU memory actually goes: weights, gradients, optimizer states, and activations — the four components that determine whether your model fits.
Activation Recomputation, Gradient Accumulation, and Data Parallelism
Three foundational techniques that reduce memory and scale training: recomputing activations, accumulating gradients, and replicating your model across GPUs.
Ring-AllReduce, Choosing Batch Size, and TensorBoard GPU Profiling
How GPUs communicate, how to pick the right batch size, and how to profile your training runs to find bottlenecks.
ZeRO-1, 2, 3 — Zero Redundancy Optimizer
The elegant idea behind ZeRO: instead of replicating everything on every GPU, partition optimizer states, gradients, and parameters across GPUs.
Tensor Parallelism
Split individual weight matrices across GPUs — column-linear, row-linear, and how tensor parallelism works inside a transformer block.
Sequence Parallelism
Why LayerNorm and dropout waste memory in tensor parallelism, and how sequence parallelism fixes it by splitting the sequence dimension.
Context Parallelism
Training with million-token contexts by splitting the sequence across GPUs using Ring Attention and its optimized variants.
Pipeline Parallelism
Split transformer layers across GPUs like an assembly line — from naive schedules to 1F1B, minimizing the pipeline bubble.
Expert Parallelism
Mixture of Experts models activate only a fraction of parameters per token — learn how experts are distributed across GPUs and kept balanced.