Milind Sunil Shah from Brainbase Labs shares their experience of developing an AI back-office agent for Domino’s to automate competitor price tracking across multiple platforms. Initially, Domino’s analysts manually gathered pricing data from competitor websites and delivery apps, a process prone to errors and time-consuming. The goal was to create an AI agent that could replicate this task accurately, producing a spreadsheet with the lowest prices for various items. The first version of the agent achieved only 41% accuracy, but through iterative improvements, they reached 97% accuracy by version 11, while also reducing costs and runtime significantly.
The development process followed a structured six-step framework: kickoff, ontology, benchmarks, first agent, iterating, and handing off. During kickoff, they aligned with Domino’s on the project goals, defining success metrics such as accuracy, cost, runtime, and human oversight requirements. Ontology involved defining all relevant terms and rules the agent must follow to avoid incorrect assumptions, such as distinguishing delivery from takeaway and correctly applying promotions. This detailed specification helped the agent understand exactly what data to collect and how to process it.
Benchmarking and evaluation were critical to measuring progress. They used historical spreadsheets and archived competitor website snapshots to create offline benchmarks. Automated checks ensured every item had a price, discounts were applied correctly, and significant price changes were flagged. They also employed language models and the agent itself to judge the correctness of outputs, with human analysts reviewing flagged cases. This multi-layered evaluation helped identify errors and guided improvements.
The first agent was deliberately narrow in scope to establish a baseline for accuracy and cost. Starting with a single app and item in one city, they gradually expanded to include more items, apps, and websites. Iteration involved fixing the most frequent and costly errors one at a time, refining instructions and rules before considering model changes. For example, they adjusted the agent to scan entire pages rather than stopping at linked sections and optimized browser behavior to reduce unnecessary image zooming, which cut costs and improved efficiency.
Finally, the agent was handed off to Domino’s with a system for ongoing review and improvement. Analysts now only review flagged exceptions, and corrections automatically update benchmarks, enabling continuous learning. The agent runs on a schedule or via manual triggers, supporting expansion into new markets. This framework proved effective not only for Domino’s but also for other clients dealing with complex, messy data sources, demonstrating a repeatable approach to building highly accurate, cost-effective AI agents for back-office automation.
Useful Links
