Nuno Campos from Witan Labs shared insights from their four-month journey teaching coding agents to master spreadsheets, aiming to achieve proficiency comparable to popular programming languages like Python or JavaScript. Initially, their agents had about 50% accuracy on a financial analysis benchmark, which they improved to 92%. He highlighted the inherent complexity of spreadsheets for AI, noting that unlike humans who intuitively grasp spreadsheet structure visually, large language models (LLMs) struggle to interpret the context, such as distinguishing between different types of revenue or understanding whether a number is an input or a formula.
Early attempts involved splitting tasks among multiple agents and experimenting with various spreadsheet representations like SQL and XML. While none of these representations worked perfectly alone, they contributed useful concepts, such as CSV/TSV views and HTML rendering, which helped in building a rendering engine to visualize spreadsheets as images. The major breakthrough came with consolidating numerous tools into a single Node.js REPL (Read-Eval-Print Loop), allowing the agent to execute JavaScript functions in a sandboxed environment. This approach enabled more efficient, combined tool calls, reducing timeouts and improving the agent’s ability to interleave reasoning with actions, leading to faster and more accurate results.
Campos emphasized the importance of feedback loops in spreadsheet tasks, drawing parallels to coding where compiling, linting, and testing improve outcomes. They developed high-fidelity formula and rendering engines to verify the agent’s work, ensuring that formulas were correctly calculated and visually accurate. This verification loop was crucial because incomplete engines could mislead the agent, resulting in errors. The REPL interface was praised as the best current method for agent interaction, though Campos acknowledged that future advances might favor other interfaces, such as direct computer use.
Domain knowledge integration into prompts consistently improved performance by guiding the model’s focus on relevant concepts like revenue or ARR, rather than teaching it new information. Evaluation proved challenging, initially relying on LLMs as judges but evolving towards deterministic comparisons using golden spreadsheets to validate outputs. This approach provided more reliable assessments. Campos also noted that many apparent reasoning errors were actually bugs in code, prompts, or tools, underscoring the importance of thorough debugging and trace analysis to distinguish between model limitations and implementation issues.
In conclusion, Campos shared several generalizable lessons: avoid multiple sequential tool calls by providing agents with a real scripting language interface like a REPL; build feedback loops with high-fidelity engines to enable verification; prioritize finding the best interface for agent interaction and be ready to adapt it as models evolve; leverage planning and domain knowledge to improve focus and accuracy; and invest in robust, preferably deterministic evaluation methods while carefully debugging the system. These insights offer valuable guidance for developing more capable and reliable coding agents across various domains.
