In this presentation, Dominik Kundel from OpenAI provides an in-depth look into the Codex harness, an open-source framework designed for building AI agents. He begins by outlining the two main protocols involved in the Codex system: the app server protocol, which manages communication between the user interface and the harness, and the responses API, which handles interactions between the harness and the language model inference. Both protocols are designed to support an open ecosystem, allowing developers to build custom interfaces or integrate Codex into various platforms, such as Cloud Code or even Doom.
A key focus of the talk is on how the Codex harness constructs context for the agent. Kundel emphasizes the importance of balancing context size to avoid token budget overruns and model confusion, while maintaining flexibility and performance. To manage growing context complexity, especially with multiple skills and plugins, the harness employs strategies like deferred tools and capping the description length of available skills. These mechanisms help optimize cost and efficiency, and are accessible through the responses API for developers building their own agents.
The presentation then explores the three primary types of actions that Codex agents perform: asynchronous actions, computer use, and file system interactions. Asynchronous actions include spawning sub-agents and managing background terminals, enabling multitasking within the agent. Computer use has evolved from a limited single-action model to a more flexible code execution approach, allowing agents to script interactions in languages like JavaScript or Python, exemplified by Codex’s browser automation via Playwright. For file system operations, Codex uses an apply patch tool for editing and creating files, supplemented by shell commands for navigation, all executed within sandbox environments tailored for different operating systems to ensure security.
Addressing security and usability, Kundel discusses the challenge of approval fatigue when agents require permissions for potentially risky actions. To mitigate this, OpenAI developed an auto approval mode featuring an auto review sub-agent that independently assesses the risk of actions like file deletions or network calls based on user authorization and risk taxonomies. This system aims to reduce unnecessary user prompts while maintaining safety, reflecting extensive engineering and research efforts. Additionally, the harness incorporates performance optimizations such as WebSocket mode to reduce network overhead and enable stateful, incremental updates, significantly speeding up agent interactions.
Finally, Kundel touches on advanced features like goal-oriented loops and context compaction. The harness supports iterative goal achievement by injecting continuation prompts until the agent signals completion, encouraging concise and clear goal definitions. To maintain performance over long-running sessions, Codex employs automatic compaction of conversation history, condensing context without losing essential information. He concludes by encouraging developers to explore the open-source Codex harness and responses API, highlighting their evolving capabilities as models advance, and inviting the audience to leverage these tools to build more powerful and efficient AI agents.
