Evan Kormos from Coinbase explains that successful integration of AI agents into policy-driven workflows requires clear boundaries, rigorous quality evaluation, and phased adoption to build trust and ensure reliability within dynamic organizational processes. He emphasizes maintaining deterministic control at process boundaries while enabling adaptive AI operations internally, supported by metrics and confidence mechanisms to safely accelerate enterprise workflows.
Evan Kormos, a seasoned software professional at Coinbase, discusses the integration of AI agents into policy-driven workflows, emphasizing the importance of policy as the foundational framework that governs organizational processes, especially in risk and compliance. He distinguishes between the policy (build state) and the running system (run state), highlighting that while policies set the rules, the system is dynamic and continuously evolving. Agents operate across people, processes, and technology, but the focus here is on how they interact with processes, the boundaries set for their work, and building trust in their outputs.
Kormos explains workflows using directed acyclic graphs (DAGs) to represent sequential and parallel steps in processes such as software delivery, commercial lending, or token listing. Within these workflows, nodes represent tasks that often involve iterative loops, such as code reviews, where work is refined until it meets quality standards. He introduces the concept of directed cyclic graphs (DCGs) to manage these iterative loops and notes the complexity involved in handling exception flows and volume shifts when agents accelerate parts of the process, underscoring the need for operational visibility and metrics to identify bottlenecks.
A critical challenge in deploying AI agents is ensuring the quality of their outputs, which depends heavily on the quality of inputs or context provided to the agents. Kormos stresses the importance of injecting workflow state into agents to give them a broader understanding of their role within the process. The evaluation or “check” step is vital, as it determines whether the work is good enough to proceed. Unlike traditional software, AI outputs are probabilistic and variable, requiring statistical evaluation over multiple runs and representative datasets to build confidence in their reliability.
To build trust in AI-driven processes, Kormos advocates for confidence mechanisms similar to those in traditional software development, such as systematic testing and evaluation frameworks. He suggests a phased approach to adopting AI agents: starting with supervised operation alongside humans, moving to sampling where humans review a subset of outputs, and eventually reaching a stage where agents operate mostly unsupervised with humans focusing on exceptions and continuous refinement. This approach ensures that AI integration is safe, reliable, and aligned with policy requirements.
In conclusion, Kormos highlights that successful AI agent integration requires maintaining deterministic control at the process boundaries while allowing adaptive, iterative work within. Quality assurance through rigorous evaluation is essential to build trust, and AI engineering is a collaborative effort involving multiple disciplines. Ultimately, agents can accelerate workflows across the enterprise, but their deployment must be carefully managed with clear policies, metrics, and confidence-building practices to ensure they operate effectively and safely within organizational processes.