The video announces a free live masterclass called the “Model Selection Playbook,” designed to teach AI engineers how to make informed decisions when choosing large language models by understanding benchmarks, creating custom evaluations, and applying advanced techniques like model routing and cascades. The session is accessible to all skill levels, lasts two to three hours, and offers practical insights to optimize model selection beyond popular rankings, with a recording available for those who cannot attend live.
The video addresses AI engineers who are building applications that utilize large language models (LLMs), emphasizing the critical importance of selecting the right model for the specific task at hand. Model selection is framed as a vital system design skill, comparable to decisions like database choice or architecture design. The speaker highlights that choosing the appropriate LLM requires careful consideration and rigor, similar to other technical decisions in system development.
Commonly, engineers might rely on popular models or benchmark leaderboards to guide their choice. However, the speaker cautions that this approach is not always reliable. Instead, they propose a more informed and rational method for model selection, which involves understanding the nature and limitations of benchmarks. The speaker promises to explain what benchmarks actually measure, how they are constructed, and what they fail to capture, helping engineers avoid being misled by superficial rankings.
To help engineers master this process, the speaker is offering a free live masterclass called the “Model Selection Playbook.” This class aims to teach participants how to make well-informed decisions about model choice by diving deep into model evaluations (evals). Attendees will learn how to create small, custom evals using their own data, gaining practical skills that go beyond buzzwords and hype.
The class will also cover advanced concepts such as the model router pattern and cascades, explaining how these techniques work behind the scenes and how they can be leveraged to optimize costs. The speaker emphasizes that the session is designed to provide genuine understanding, enabling participants to apply these insights effectively in their projects.
Finally, the masterclass is accessible to all skill levels, requires no prior background, and runs for two to three hours. Those unable to attend live can register to receive a recording. The speaker encourages viewers to bring real model selection challenges or questions to the session, promising a collaborative and informative experience. A registration link is provided in the video description.
