In this lecture, Michael Ryan introduces the foundational concepts of large language models (LLMs) and their relevance to building AI agents. He outlines a three-layer “LLM stack”: engineering (training models on massive compute clusters), adaptation (fine-tuning and customizing models for specific contexts), and applications (building tools like chatbots and agents on top of LLMs). The course focuses primarily on the application layer, leveraging pre-built LLMs rather than building them from scratch. Ryan emphasizes understanding core design decisions in training and inference to better utilize LLMs, including how attention mechanisms and KV caches influence model capabilities and limitations.
Ryan then delves into the fundamentals of language models, explaining that they predict the probability of token sequences and are generally considered generative models because they model the likelihood of entire sequences rather than discriminating between classes. He discusses various sampling strategies such as greedy decoding, top-k, top-p, and temperature scaling, which control the diversity and creativity of generated text. These strategies impact the model’s output depending on the task, with lower temperatures favoring predictability and higher temperatures encouraging diversity.
The lecture proceeds to cover the architecture of modern LLMs, focusing on transformers and their attention mechanisms. Ryan explains how tokens are embedded and combined with positional embeddings to retain order information, followed by the multi-headed self-attention mechanism that allows the model to weigh different parts of the input sequence. He contrasts transformers with recurrent models, highlighting transformers’ parallel processing advantages but also their quadratic complexity in sequence length. He introduces linear attention as a more memory-efficient alternative, though it struggles with very long contexts, and discusses hybrid approaches that combine linear and full attention to balance efficiency and performance.
Training large language models involves several stages: pre-training on vast, filtered internet text corpora, mid-training to adjust data distributions toward target domains, and post-training fine-tuning with supervised data and reinforcement learning from human feedback or verifiable rewards. Ryan notes challenges with human feedback, such as inconsistency and bias, and highlights synthetic data generation and agent-specific training environments as promising approaches to improve model capabilities, especially for coding and reasoning tasks.
Finally, Ryan discusses inference techniques and practical considerations for using LLMs in agent systems. He explains the difference between prefill (processing the entire prompt) and decode (generating tokens one at a time), and introduces speculative decoding to improve efficiency. He covers inference-time scaling methods like chain-of-thought prompting and reasoning models that generate intermediate reasoning steps to improve accuracy. The lecture concludes with strategies for structured input/output, context engineering to manage long contexts efficiently, and recursive language models that enable agents to handle extended information by breaking down tasks recursively, thereby mitigating context length limitations and improving agent performance.
Useful Links
