Soheil Feizi presents a framework for continual learning in AI agents that emphasizes creating replayable learning environments from production logs and feedback to enable verifiable, regression-aware improvements across model, harness, and memory layers. His approach, embodied in Rely’s Verifiable Continual Learning (VCL) engine, ensures durable, efficient, and holistic agent enhancements by balancing cost, speed, and impact while preventing regressions.
Soheil Feizi, founder and CEO of Rely and associate professor at the University of Maryland, presents on continual learning for AI agents, focusing on how to transition from failures to durable improvements. He explains that continual learning aims to enable AI agents to learn from experience by interacting with the world, receiving feedback, and improving without forgetting. This learning can occur across three layers of an agent: the model layer (modifying model weights), the harness layer (adjusting prompts, skills, tools, and workflows), and the memory layer (storing facts and skills). Two fundamental challenges in continual learning are obtaining meaningful feedback and effectively acting upon it to optimize the agent.
Feizi discusses the sources of feedback, highlighting that while benchmarks with evaluators are useful during development, production environments mostly yield session logs without explicit feedback. To address this, feedback can be automatically generated using models or human experts can provide domain-specific critiques. However, logs and feedback alone are insufficient because they are not testable. The solution is to create replayable learning environments—simulations derived from logs and feedback that allow agents to be tested and evaluated repeatedly, making learning verifiable and actionable.
When it comes to acting on feedback, Feizi outlines three layers where improvements can be made: model fine-tuning (e.g., supervised fine-tuning or reinforcement learning), harness engineering (e.g., prompt tuning and tool adjustments), and memory updates (e.g., storing corrections and distilling skills). Each layer has trade-offs in terms of cost, speed, and verifiability. Effective continual learning requires making the smallest durable change at the appropriate layer, balancing efficiency and impact.
Feizi introduces the concept of verifiable continual learning (VCL), which ensures that every improvement is proven to help without causing regressions. VCL relies on four principles: replayability (turning failures into replayable tests), holisticness (considering multiple causes and fixes across layers), lifelongness (ensuring new fixes do not break past successes), and efficiency (running frequent, cost-effective learning loops). Rely’s VCL engine embodies these principles by converting signals like logs and feedback into replayable environments, performing root cause analysis, and applying regression-aware optimization to produce verifiable agent updates.
Finally, Feizi demonstrates Rely’s approach using a fictional support agent benchmark, showing how learning environments can be created from simple instructions or production logs, enabling measurable improvements without regressions. He emphasizes three key takeaways: continual learning extends beyond model fine-tuning to include harness and memory layers; production logs must be transformed into replayable learning environments for effective learning; and regression-aware continual improvement is essential to maintain and build upon past successes. Interested users can try Rely’s VCL tools at rely.ai.