Activation Recomputation, Gradient Accumulation, and Data Parallelism
Three foundational techniques that reduce memory and scale training: recomputing activations, accumulating gradients, and replicating your model across GPUs.
beginner~4 hours3 notebooksActivation Recomputation (Gradient Checkpointing)Gradient Accumulation: Big Batches Without Big MemoryData Parallelism from ScratchData Parallelism Optimizations: Overlap, Bucketing, and Scaling
Curator of this Module
Dr. Rajat Dandekar
Course Instructor
Dr. Rajat Dandekar is a researcher and educator specializing in AI/ML, with a passion for making complex concepts accessible through intuitive explanations and hands-on learning.
Checking access…
Learning Path
Article
1
Notebook 12
Notebook 23
Notebook 3Certificate