Build LLM from Scratch
From BERT to GPT — build large language models from first principles.
Pods in this Course
Understanding BERT from Scratch
## How a simple idea — reading in both directions — changed NLP forever
Foundations of Language Modeling
From N-grams to Neural LMs to Transformers — trace the evolution of language modeling and understand why each breakthrough was necessary.
Self-Attention from First Principles
Queries, Keys, Values, Multi-Head Attention, and Positional Encoding — derive the self-attention mechanism from first principles.
Building a GPT-style Model from Scratch
Forward pass, loss computation, and backpropagation — build a GPT-style autoregressive language model from the ground up.
Training Pipeline Engineering
Tokenization, data loading, and optimization — engineer the full training pipeline that turns raw text into a working language model.
Inference & Scaling
KV cache, sampling strategies, fine-tuning, and alignment — take a trained model from raw outputs to production-ready inference.
Build Qwen3.5 from Scratch
Hybrid attention, Mixture-of-Experts, and local deployment — build the architecture that lets a 9B model outperform one 13x its size.
Mixture of Experts (MoE) from Scratch
Mixture of Experts (MoE): How Modern LLMs Scale to Trillions of Parameters Efficiently
Understanding Multi-Head Latent Attention (MLA) from Scratch
Multi-Head Latent Attention (MLA): DeepSeek’s Solution to the KV Cache Bottleneck
Linear Attention and the Delta Rule: Building Kimi Delta Attention
From softmax attention to linear attention to the delta rule, building up to Kimi Delta Attention (KDA): per-channel learnable forgetting via a diagonal-plus-low-rank transition, why it powers the 1M-token context of Kimi K3, implemented from first principles with a working long-context toy model.
Sparse Attention for Million-Token Context: Building DeepSeek-V4's Hybrid Attention
Sparse Attention for Million-Token Context: Building DeepSeek-V4's Hybrid Attention
Sparse Mixture-of-Experts from Scratch: How a 2.4T-Parameter Model Runs on 95B Active
Sparse Mixture-of-Experts from Scratch: How a 2.4T-Parameter Model Runs on 95B Active