Andrew Dumont from Watershed Technology Inc. discusses the challenges and solutions involved in deploying AI coding agents within the sustainability sector, particularly focusing on measuring product carbon footprints. Sustainability involves numerous expert judgment calls, making it difficult to verify answers solely based on outcomes. Instead, validating the process that produces these answers is crucial, as demonstrated by a study where experts analyzing the same data arrived at significantly different emissions estimates. This complexity necessitates a focus on ensuring the correctness of the process rather than just the final result.
The talk zooms in on a specific task: helping users edit complex supply chain graphs representing products like dark wash jeans. These graphs contain thousands of nodes with detailed metadata about materials and processes. Initial attempts to automate edits using AI agents faced issues with consistency, scalability, and context management, leading to errors and hallucinations. Although coding agents later improved exploration and editing capabilities, their unconstrained nature introduced new risks such as unexpected code execution, false edit confirmations, and difficulties in reviewing the agent’s work, especially for non-coders.
To address these challenges, Watershed developed a framework that respects the process by constraining the effects of the agent’s actions rather than limiting its reasoning. They implemented a typed Software Development Kit (SDK) in TypeScript that serves as the sole interface for graph edits, ensuring edits are valid, traceable, and replayable. The system controls the final execution of the agent’s code, performing linting, conflict detection, and validation before committing changes. This deterministic execution guarantees that edits align with the intended process and produces reviewable artifacts that users can understand without needing to read the underlying code.
This harness approach effectively mitigates the three main problems encountered: preventing invalid edits through the typed SDK, catching false edit confirmations via deterministic execution, and enabling easy review of outcomes without requiring coding expertise. Despite these safeguards, continuous improvement is necessary. Watershed improved their agent’s performance on complex edits from 43% to 92% accuracy by refining prompts, providing few-shot examples, enhancing SDK ergonomics, and breaking down tasks into manageable planning and execution steps. Importantly, even when errors occur or expert judgments vary, the process validation ensures reliability and transparency.
In conclusion, Andrew emphasizes that in domains rife with expert judgment like sustainability, respecting and guaranteeing the process is essential. Coding agents are indispensable for complex tasks but come with risks that must be managed through well-scoped primitives, controlled execution, and user-friendly validation outputs. Combined with effective prompt engineering, this approach allows organizations to harness the power of AI agents while maintaining trust and accuracy in critical sustainability workflows.
