Week 6: The Attention Revolution & Building Transformers

Forget everything you know about sequential loops. RNNs hit a massive wall of catastrophic memory loss and slow training speeds. We dismantle that bottleneck and step into the architecture that took over the world: the Transformer. You will learn how "Attention Is All You Need" replaced recurrence with a highly efficient database lookup. We dissect the modern Transformer block-by-block and see how it powers everything from Llama 3 to modern robotics. Core Topics: The Self-Attention Mechanism, Queries, Keys, and Values (QKV), Multi-Head Attention, Positional Encodings (Sinusoidal vs. RoPE), Feed-Forward Networks (SwiGLU), Modern Optimizations (RMSNorm, KV Cache, Grouped Query Attention), and Cross-Domain Transformers (ViT, AST, VLA). Week 6 Lab: Rip out last week's RNN and upgrade to a Transformer Decoder. Build Scaled Dot-Product Attention, Multi-Head matrices, and Positional Encodings entirely by hand to see how modern LLMs actually tick.

8 weeks · 59 lectures · free to watch