# From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI **URL:** \ **Category:** Content Creators\ **Tags:** agents, self-adapting, ai-engineer, automation\ **Created:** [5 October 2026 23:00 UTC](https://www.artofsm.art/t/from-vibes-to-production-evaluating-and-shipping-ai-agents-that-work-201-laurie-voss-arize-ai/25097 "2026-10-05T23:00:09Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![artesia](https://www.artofsm.art/user_avatar/www.artofsm.art/artesia/32/36_2.png) [@artesia](https://www.artofsm.art/u/artesia)\ **Post date:** [5 October 2026 23:00 UTC](https://www.artofsm.art/t/from-vibes-to-production-evaluating-and-shipping-ai-agents-that-work-201-laurie-voss-arize-ai/25097/1 "2026-10-05T23:00:09Z")
[![](https://www.artofsm.art/uploads/default/original/3X/e/a/ea3ffcb057d5dd24160b9e186483d1aeb88e07a2.jpeg "From Vibes to Production: Evaluating and Shipping AI Agents That Work 201 — Laurie Voss, Arize AI") ](https://www.youtube.com/watch?v=F0TNSmbo5hE) Laurie Voss from Arize AI discusses advancing AI agent observability from manual trace analysis to automated evaluations and self-improving agents using Arize AI’s platform, including the Signal product that detects issues and applies fixes autonomously. The session highlights a shift toward continuous, scalable AI agent monitoring and remediation, addressing challenges of trust and compliance while enabling rapid, reliable deployment. ---
**Author:** ![artesia](https://www.artofsm.art/user_avatar/www.artofsm.art/artesia/32/36_2.png) [@artesia](https://www.artofsm.art/u/artesia)\ **Post date:** [5 October 2026 23:24 UTC](https://www.artofsm.art/t/from-vibes-to-production-evaluating-and-shipping-ai-agents-that-work-201-laurie-voss-arize-ai/25097/3 "2026-10-05T23:24:32Z")
In this session titled “From Vibes to Production: Evaluating and Shipping AI Agents That Work 201,” Laurie Voss from Arize AI discusses the evolution of observability and evaluation in AI agent development. He begins by recapping the basics covered in an earlier 101 session, emphasizing the importance of traces—detailed logs of every action an AI agent takes—as the new source of truth for understanding non-deterministic AI behavior. Unlike traditional software, AI agents do not produce predictable outputs from the same inputs, making code alone insufficient to understand their behavior. Traces provide visibility into each step, enabling developers to diagnose issues such as inefficiencies or unexpected behaviors. Laurie then explains the challenge of scale: as AI agents operate at high volumes, the sheer number of traces becomes overwhelming for humans to analyze manually. To address this, Arize AI introduced evals—automated evaluations that use LLMs to assess traces and provide human-readable explanations of failures or issues. However, as agent usage grows, even evals become too numerous for humans to manage effectively. This leads to the next phase of observability, where agents themselves analyze traces and evals to detect patterns and signals of failure, enabling continuous and automated improvement at scale. The workshop portion demonstrates hands-on how to add observability to an AI-powered shopping app called Wonder Toys using Arize AI’s platform. Laurie shows how the agent can automatically instrument the app to send traces without manual coding. By analyzing these traces, the agent identifies a key problem: the app lacks price filtering functionality. The agent then autonomously generates and applies a fix, adding price filtering to the app, effectively closing the loop from observability to automated remediation without human intervention. Laurie introduces Arize AI’s product called Signal, which automates the continuous monitoring and improvement process. Signal runs periodically to analyze large volumes of traces, detect recurring issues, and suggest or even create fixes automatically, such as opening pull requests with code changes. This represents a shift from observability as passive monitoring to observability as active, automated software improvement. Signal also integrates with evals and can generate GitHub issues or PRs, streamlining the development lifecycle and reducing human bottlenecks. Throughout the session, Laurie addresses audience questions about trust, compliance, and adoption in regulated industries, emphasizing that while automated fixes raise concerns, layered abstractions and rigorous evals help maintain reliability. He highlights that Arize AI supports on-prem deployments for compliance and that Signal itself is continuously evaluated using the same observability tools. The session concludes with a vision of AI development where software not only reports its behavior but actively improves itself, enabling rapid, scalable, and reliable AI agent deployment. ## Useful Links - [Arize AI official GitHub repository for project-rosetta-stone](https://github.com/Arize-ai/project-rosetta-stone) — Directly enables hands-on replication of the workshop’s observability and agent improvement loop.