The lecture begins with important course logistics, including details about the upcoming midterm and final exams, their formats, and coverage. The midterm will cover lectures 1 to 4, featuring multiple-choice and free-form questions, while the final will focus on lectures 5 to 9. The instructor emphasizes the importance of attending office hours and reviewing the provided cheat sheet for effective preparation. After this, the lecture recaps previous topics such as mixture of experts architectures, decoding methods like greedy decoding, beam search, and sampling, as well as inference optimization techniques like KV cache.
The core of the lecture focuses on the training of large language models (LLMs). It explains the paradigm shift from training task-specific models to leveraging transfer learning, where a large pre-trained model is fine-tuned for specific tasks. Pre-training involves training on massive datasets, often comprising hundreds of billions to trillions of tokens from diverse sources like Common Crawl, Wikipedia, Reddit, and code repositories. The computational cost of pre-training is enormous, measured in floating point operations (FLOPs), and requires specialized hardware such as GPUs or TPUs. The lecture also discusses scaling laws that relate model size, dataset size, and compute budget, highlighting the importance of balancing these factors for optimal training efficiency.
To handle the massive memory and compute requirements during training, the lecture covers parallelism techniques. Data parallelism distributes batches of data across multiple GPUs, each holding a copy of the model, while model parallelism splits the model itself across devices. Advanced methods like zero redundancy optimization (ZeRO) shard parameters, gradients, and optimizer states across GPUs to reduce memory duplication. The lecture also introduces Flash Attention, a Stanford-developed technique that optimizes the attention mechanism by minimizing slow memory access through clever tiling and recomputation strategies, resulting in faster and more memory-efficient training.
The lecture then shifts to fine-tuning, explaining how supervised fine-tuning (SFT) adapts the pre-trained model to perform specific tasks or behave as a helpful assistant. Unlike pre-training, fine-tuning uses labeled input-output pairs and focuses on predicting tokens only after the input prompt. Instruction tuning is a subtype of SFT aimed at making models respond effectively to user instructions. The data for fine-tuning is smaller but higher quality, often including human-written or model-generated examples that emphasize helpfulness, safety, and alignment with user expectations. Challenges such as generalization, distribution mismatch, and evaluation complexities are discussed, along with the importance of benchmarks and user preference-based evaluations.
Finally, the lecture covers optimization techniques to make fine-tuning more efficient. LoRa (Low-Rank Adaptation) is introduced as a method that decomposes weight updates into low-rank matrices, significantly reducing the number of parameters that need to be trained. This approach allows for task-specific tuning without modifying the entire model, saving compute and memory resources. The lecture also touches on mixed precision training and quantization techniques that reduce memory usage and speed up computation by lowering the numerical precision of weights and activations. These methods collectively enable practical training and fine-tuning of large models despite their enormous size and complexity.
