In “Evals Are Broken, Use Them Anyway,” Ara Khan from Cline highlights the limitations of current AI evaluation methods, advocating for their thoughtful use as approximate tools rather than definitive measures, especially emphasizing the need for realistic, complex evals in coding tasks. He encourages a balanced approach combining quantitative and qualitative insights, continuous improvement through detailed failure analysis, and collaboration to refine evaluation frameworks and build more effective AI agents.
In the talk titled “Evals Are Broken, Use Them Anyway,” Ara Khan from Cline discusses the current state of evaluation methods (evals) in AI, particularly focusing on their limitations and how to effectively use them despite their flaws. Ara begins by highlighting the common misconceptions about evals, dividing critics into two camps: those who overly trust objective metrics and those who rely solely on subjective taste. He argues that neither extreme is correct and that evals should be used thoughtfully as tools that provide approximate insights rather than definitive judgments.
Ara outlines three stages for effectively leveraging evals: using existing evals from others, employing evals to improve one’s own AI agents, and ultimately building custom evals tailored to specific use cases. He emphasizes skepticism towards headline numbers from model leaderboards, advising users not to take these scores at face value. Instead, he recommends waiting for new models to stabilize before adopting them and seeking out fresh, precise evals that better reflect real-world capabilities, as many traditional benchmarks have become outdated.
The speaker then delves into the challenges of creating meaningful evals for complex AI agents, especially those involved in coding tasks. Unlike simple one-shot questions with clear answers, coding agents operate in dynamic environments requiring multi-step problem solving, such as debugging or setting up servers. Ara shares Cline’s approach of building evals based on real user data and complex programming problems, referencing Stanford’s Terminal Bench as a similar effort to create realistic, multi-faceted coding challenges that test agents in practical scenarios.
Ara also discusses the infrastructure and methodology needed to run these comprehensive evals, including isolated environments and parallel task execution to efficiently handle numerous complex problems. He explains the importance of analyzing failure cases in detail to identify specific weaknesses in the model, the evaluation harness, or the problem set itself. This iterative process of diagnosing and addressing issues enables continuous improvement, or “hill climbing,” to enhance both the AI agent’s performance and the evaluation framework.
In conclusion, Ara encourages the audience to embrace evals as imperfect but valuable tools for AI development. He stresses the importance of balancing quantitative scores with qualitative judgment and warns against overfitting evals to achieve misleadingly high results. He invites those interested in advancing eval methodologies or working on related challenges to connect with him, underscoring the collaborative nature of this ongoing effort to refine AI evaluation and build more effective, reliable agents.
