Expert Parallelism
Mixture of Experts models activate only a fraction of parameters per token — learn how experts are distributed across GPUs and kept balanced.
beginner~4 hours3 notebooksMixture of Experts: Sparse Models from ScratchToken Routing: How Tokens Choose Their ExpertsExpert Parallelism: Distributing Experts Across GPUsLoad Balancing, Auxiliary Losses, and Scaling MoE
Curator of this Module
Dr. Rajat Dandekar
Course Instructor
Dr. Rajat Dandekar is a researcher and educator specializing in AI/ML, with a passion for making complex concepts accessible through intuitive explanations and hands-on learning.
Checking access…
Learning Path
Article
1
Notebook 12
Notebook 23
Notebook 3Certificate