Jess from BrainTrust highlights the importance of structured AI evaluations to make data-driven decisions, illustrating this with a comparison between vector search and agent search for coding agents using the Microsoft TypeScript repository. The evaluation revealed that while both methods had similar accuracy, agent search was more efficient and contextually aware, demonstrating the value of thorough, collaborative evals in improving AI system performance.
In this talk, Jess from BrainTrust introduces the concept of AI evaluations, or “evals,” emphasizing their importance in making data-driven decisions about AI system performance rather than relying on subjective feelings. She explains that evaluations help answer critical questions such as which large language model (LLM) best suits specific needs, how AI performs across different languages or coding languages, and how to balance performance with economic efficiency. Jess highlights that evaluations are essential for maintaining brand consistency, learning from user feedback, and detecting when AI systems degrade or break, citing a real-world example where an OpenAI update unintentionally reduced model truthfulness.
Jess outlines the four main components of an evaluation: the dataset, the task, the evaluation system, and experiments. The dataset includes real-world data, edge cases, and failure scenarios. The task defines how the AI should behave with the input data, including system prompts and model choice. The evaluation system sets criteria for judging output quality, which can be deterministic, human-verified, or judged by another LLM. Experiments involve running different configurations of datasets, tasks, and scoring systems to test hypotheses and compare results. BrainTrust’s platform supports this workflow with interfaces for querying experiments, monitoring logs, and visualizing traces to detect issues like hallucinations.
Jess stresses that evaluation is a collaborative effort involving AI engineers, product managers, domain experts, and data analysts. Each plays a role in data collection, hypothesis development, prompt tuning, and result analysis. She then presents a real-world example comparing vector search and agent search for coding agents, inspired by a tweet from her CEO. Vector search converts code into semantic embeddings stored in a vector database to retrieve relevant code snippets, while agent search mimics human exploration of codebases using commands like grep and bash to navigate and understand code contextually.
For the evaluation, Jess used the Microsoft TypeScript repository, focusing on pull requests with bug fixes. They created tasks where the AI had to locate faulty code using either vector or agent search. Implementing agent search was straightforward with Claude code, but vector search required disabling agent search tools and carefully crafting prompts. They also improved trace visibility by linking subprocesses to parent spans, enabling detailed debugging. The evaluation used a binary scoring system based on whether the Microsoft TypeScript test suite passed or failed after the AI’s fix.
The results showed that both vector and agent search achieved similar accuracy, but vector search was four times more expensive due to numerous queries and lack of contextual completeness in returned snippets. Agent search, acting more like a human, could follow function calls and understand code logic better, making it more efficient. Jess acknowledges limitations in the assessment, such as a small dataset and a relatively simple vector search implementation, but emphasizes that the goal was to demonstrate the evaluation process from start to finish. She concludes by encouraging teams to adopt structured evaluations to improve AI systems systematically.