The lecture focuses on inference scaling for large language models (LLMs), highlighting three main stages of LLM development: pre-training, fine-tuning, and inference. While pre-training is computationally intensive and time-consuming, inference traditionally required minimal compute. However, recent research shows that inference time compute can be scaled effectively to improve model performance without changing model parameters. One key method discussed is repeated sampling, where the same input is queried multiple times, and a verifier selects the best response. This approach can significantly boost the performance of smaller or less capable models, sometimes surpassing larger proprietary models like GPT-4.
The lecture introduces scaling laws for inference, showing that the relationship between the number of samples drawn and problem coverage follows a power law. This behavior holds across various model sizes and domains, including coding and math problems. The presence of a long tail of hard problems in datasets explains this scaling, as simpler problems are solved easily on the first try, while harder problems require more samples to find correct answers. This insight suggests that investing more compute at inference time can be a viable strategy to improve model capabilities, especially for open-source or smaller models.
A critical challenge in repeated sampling is verification—determining which generated responses are correct. In some domains, such as coding or math, automated verification is feasible through unit tests or formal proofs, enabling end-to-end automated improvement. However, in domains lacking reliable verifiers, there is a significant gap between the best generated answers and those that can be confidently identified as correct, known as the generation-verification gap. Various methods like majority voting or reward models help but often fall short, especially for harder problems where correct answers are rare among samples.
The lecture also explores advanced inference time techniques beyond repeated sampling, including sequential revisions where the model iteratively improves its answers, and the use of reward models that score responses either based on the final outcome or intermediate steps. Combining parallel sampling with sequential revisions and reward-guided search can optimize the trade-off between compute cost and accuracy. Empirical results suggest that for easier and medium-difficulty problems, scaling inference compute can be more effective than further pre-training, while for the hardest problems, larger models with more pre-training still hold an advantage.
Finally, the Archon framework is presented as a method to optimize inference architectures by combining multiple models and inference time techniques such as generation, fusion, critique, ranking, and unit test evaluation. Using Bayesian optimization, Archon searches for the best combination of these techniques under a compute budget to maximize accuracy. This approach has demonstrated the ability to outperform state-of-the-art closed-source models like GPT-4 and Claude 3.5 on various benchmarks by intelligently mixing and matching inference strategies. The lecture concludes with open research questions on improving verification, scaling strategies, and the potential of inference-time compute to complement or even substitute pre-training in certain contexts.
