Tensor Parallelism
Split individual weight matrices across GPUs — column-linear, row-linear, and how tensor parallelism works inside a transformer block.
beginner~4 hours3 notebooksMatrix Multiplication and Sharding FundamentalsColumn-Linear and Row-Linear: The Two Sharding StrategiesTensor Parallelism in a Transformer BlockCommunication Costs and Scaling Limits of Tensor Parallelism
Curator of this Module
Dr. Rajat Dandekar
Course Instructor
Dr. Rajat Dandekar is a researcher and educator specializing in AI/ML, with a passion for making complex concepts accessible through intuitive explanations and hands-on learning.
Checking access…
Learning Path
Article
1
Notebook 12
Notebook 23
Notebook 3Certificate