Will Brown, lead of applied research at Primordial AI, presents on the challenges and advancements in reinforcement learning without verifiable rewards (RLVR). He begins by explaining the basics of reinforcement learning (RL), where an agent interacts with an environment composed of tasks, tools, and reward mechanisms to optimize its policy for higher rewards. While traditional RL relies on clear, verifiable rewards—such as numerical answers in math or test cases in code—many real-world tasks lack such straightforward evaluation criteria, making RL more complex and less scalable in practical applications.
Brown emphasizes the difficulty of designing environments and rewards for messy, real-world tasks like report writing, booking flights, or handling customer interactions, where success is subjective and not easily quantifiable. He highlights the problem of reward hacking, where agents exploit loopholes in loosely defined reward functions, leading to undesired behaviors. To address these challenges, Primordial AI has developed a comprehensive platform that includes large-scale GPU orchestration, flexible environment and task design tools, and a training platform called Lab, all aimed at enabling users to iteratively train and evaluate models in complex, real-world settings.
A key focus of the talk is on creating reliable learning signals in environments without verifiable rewards. Brown discusses techniques such as grounding—using source material to create supervision signals—and leveraging powerful language models as judges to evaluate agent actions. He also stresses the importance of mining production traces from deployed agents to understand real-world task distributions and iteratively improve simulators that replicate complex tool use and web applications. These high-fidelity simulators allow for controlled training and verification, even when the real-world environment is partially observable or uncontrollable.
Brown further explores the role of scaling search and compute at test time to refine task difficulty, generate rubrics for evaluation, and detect reward hacking through adversarial testing and trace analysis. He advocates for integrating training experiments into environment design to monitor behavioral changes and improve reward functions continuously. Importantly, he underscores the need for human experts to remain in the loop at a high level, providing critical judgments on agent performance and guiding the refinement of tasks and rewards.
In conclusion, Brown envisions a future where continual learning enables deployed agents to autonomously improve by learning from their mistakes and evolving over time. By automating environment and reward design processes and combining reinforcement learning with supervised signals from the environment, Primordial AI aims to make post-training and continual learning more accessible and scalable. This approach empowers a broader range of users to develop specialized agents tailored to their unique real-world tasks, ultimately closing the loop between deployment, evaluation, and ongoing improvement.
