Philipp Schmid from Google DeepMind stresses the necessity of pairing AI skills with robust evaluations to ensure their effectiveness, reliability, and cost-efficiency, especially in production environments where users lack context about underlying skills. He advises concise, well-structured skill design, thorough testing with positive and negative cases, and continuous evaluation to prevent regressions and optimize agent performance.
Philipp Schmid from Google DeepMind emphasizes the critical importance of not shipping AI skills without proper evaluations (evals). He highlights a common issue where many developers use skills in coding agents but lack corresponding evals to verify their effectiveness. Skills, which are modular components designed to extend agent capabilities, often go untested, making it difficult to determine if failures are due to the skill itself or the inherent challenges of the task. Schmid distinguishes between the agents developers use personally—where they have context and can intervene—and the agents built for customers, who are unaware of the underlying skills, underscoring the need for robust evals in production environments.
Schmid explains what skills are and how they function, describing them as folders containing a skills.md file and related assets. Skills operate on progressive disclosure, starting with a brief description that helps the model decide when to use the skill, followed by more detailed instructions and references. He categorizes skills into capability skills, which teach models new abilities and may be retired as models improve, and preference skills, which encode specific workflows or styles unique to an organization and require ongoing validation to prevent performance degradation.
The speaker shares insights from Skills Bench, a benchmarking tool that evaluates thousands of skills across various models and tasks. The data shows that well-crafted, human-written skills generally improve performance by about 15%, while AI-generated skills can sometimes harm it. Schmid advises keeping skill descriptions concise—ideally under 500 lines—and layering information so that detailed references are only accessed when necessary. He also stresses the importance of clear, directive descriptions that specify when and how a skill should be used, avoiding overly prescriptive step-by-step instructions better suited for scripts.
Schmid offers practical advice for writing and testing skills, including creating both positive and negative test cases to ensure skills trigger appropriately without overuse. He highlights the value of early testing with real-world data and warns against no-op instructions that do not affect agent behavior, which can waste tokens and increase costs. He also discusses the necessity of retiring skills when models become capable enough to perform tasks without them, thereby saving resources and reducing maintenance overhead.
Finally, Schmid presents a case study from DeepMind where they developed a skill for the Gemini Interactions API, which was unknown to the model at release. By creating 117 test cases and using simple regex-based assertions, they significantly improved the model’s ability to generate valid API code. He outlines their internal process of running evals on every skill change to prevent regressions and encourages developers to adopt similar practices. His closing advice is clear: always accompany skills with thorough evals to ensure reliability, cost-effectiveness, and continuous improvement.