Raphael Kalandadze’s talk, “The Missing Layer After Launch,” emphasizes the critical phase that begins after launching an AI agent product. While building and shipping an agent may seem like the main challenge, the real work starts post-launch. He highlights the difficulty in monitoring and understanding how the agent performs across thousands of real conversations daily. Traditional software testing and monitoring methods fall short because agents operate in unpredictable, non-deterministic environments with endless possible interactions, making it impossible to pretest every scenario before production.
Kalandadze explains that agents differ from conventional software due to their vast and dynamic capabilities, such as handling numerous tools and sub-agents during long-running tasks. This complexity means that even if an agent completes a task successfully on the surface, it might still fail in subtle ways that impact user satisfaction. Moreover, agents can recover from errors silently, masking underlying issues that require attention. This unpredictability and the sheer scale of interactions necessitate a robust feedback loop to detect, diagnose, and fix problems promptly.
To address these challenges, Kalandadze advocates for building a monitoring and feedback system—what he calls the “missing layer”—that continuously analyzes logs, conversations, and system behavior. This system uses specialized agents to parse logs, identify issues, and even generate pull requests (PRs) for fixes automatically. A review agent then critically evaluates these PRs to ensure quality before human intervention. This approach creates a fast feedback loop that helps maintain system health and accelerates problem resolution, reducing reliance on humans as bottlenecks.
In addition to the fast loop for immediate fixes, Kalandadze describes a higher-level monitoring system that provides an overview of the agent’s health. This dashboard aggregates metrics such as session scores, success rates, sentiment analysis, and tool usage, offering insights into patterns and systemic issues. It helps teams maintain visibility over large volumes of interactions and prioritize areas needing attention. Complementing this, a “computer user agent” simulates real user interactions with the UI to detect issues that logs alone cannot reveal, ensuring a comprehensive understanding of the system’s performance from multiple perspectives.
Ultimately, Kalandadze stresses that shipping an AI agent is just the beginning. To build reliable, production-ready agents, teams must implement a meta-system or harness that monitors, understands, and improves the agent continuously. This closed-loop system, combining automated analysis, review, and human oversight, is essential for detecting hidden problems, maintaining quality, and evolving the product effectively. He concludes that while many focus on building the agent itself, the true challenge and opportunity lie in managing what happens after launch.
