Kyle Mistele from HumanLayer advocates for using control theory to design incremental, manageable AI-driven coding loops that improve complex software systems gradually, emphasizing human feedback and flow control to ensure safety and maintainability. He illustrates this approach with a practical example of migrating an RPC API using AI agents guided by deterministic sensors and controllers, promoting sustainable, team-friendly automation over large, risky AI-generated pull requests.
In this talk, Kyle Mistele from HumanLayer discusses the practical engineering of AI-driven coding loops for real-world, complex software systems. He critiques the hype around large, blind AI-generated pull requests (PRs), often tens of thousands of lines long, which are difficult to review and risky to deploy. While tools like Ralph and Claude are innovative and useful for certain scenarios, they don’t fit well with team-based development or critical systems that require careful oversight. Instead, Kyle advocates for building incremental, controlled loops that improve codebases gradually, making the process manageable and maintainable.
Kyle introduces control theory as a foundational concept for designing effective coding loops. Control theory involves measuring the current state of a system, comparing it to a desired set point, and applying incremental changes based on the measured error. This approach helps avoid large, destabilizing changes and reduces risk. He explains that many familiar technologies, such as thermostats, Kubernetes autoscaling, and React’s virtual DOM, use control loops. Applying this theory to software development means creating loops that sense code quality issues, decide on small improvements, and actuate those changes, then repeat the process continuously.
To illustrate, Kyle shares a concrete example from HumanLayer where they incrementally migrate their RPC API to use the Effect library. They start by building a sensor using tools like asgrep to detect unmigrated procedures, filtering and sorting violations deterministically. They enforce that new code adheres to the desired standard to prevent regression. The controller then selects which procedures to migrate, either deterministically or with agent assistance, prioritizing based on factors like error rates or instrumentation gaps. The actuator is an AI agent guided by carefully crafted “golden patterns” that incrementally refactor the code and create PRs with detailed descriptions.
Kyle emphasizes the importance of integrating human feedback into these loops to keep them effective and low friction. They track feedback in version-controlled markdown files and use PR comments to trigger loop iterations, allowing developers to steer the loop’s behavior without disrupting the process. Additionally, they implement flow control to prevent multiple overlapping PRs from the same loop, ensuring that only one incremental change is reviewed at a time. This avoids conflicts, duplication, and reviewer fatigue, making the loop sustainable in a team environment.
Finally, Kyle highlights ways to scale the velocity of these loops, such as batching multiple migrations per iteration or distributing work across team members. The overall message is that well-engineered control loops enable teams to harness AI coding agents effectively, producing incremental, readable, and verifiable code improvements. He invites viewers to try their skill and join HumanLayer if interested in advancing AI-assisted software development for mission-critical systems. The talk concludes with an encouragement to build loops that work for the real world, balancing automation with human oversight.