Angel Ortmann Lee, a software engineer at Duolingo, discusses the concept of human-in-the-loop AI, emphasizing that humans involved in AI decision-making often do not critically evaluate AI outputs but tend to rubber-stamp them. She explains that human-in-the-loop AI involves humans supervising or making decisions based on AI outputs to ensure accuracy, safety, and ethical considerations. However, as AI becomes more integrated into daily life, people increasingly trust AI outputs without sufficient scrutiny, a phenomenon called cognitive surrender, where humans adopt AI answers with minimal critical thinking, even when those answers are incorrect.
Lee illustrates this issue through a case study from the Duolingo English Test (DET), a high-stakes, fully online English proficiency exam that uses AI to detect cheating behaviors such as copy typing. In experiments, human proctors were given fake AI cheating flags on legitimate test sessions to see if they would discern false alarms. Despite their expertise, proctors accepted 50% of these false flags, indicating automation bias where humans defer judgment to AI without independent verification. To address this, Duolingo updated proctor guidelines to emphasize that AI signals are preliminary alerts and that proctors must find independent evidence before confirming cheating, which significantly improved decision accuracy without changing the AI model or interface.
The talk highlights the importance of designing AI-human interaction loops intentionally to foster independent human judgment rather than passive approval. Lee stresses that the interaction between AI outputs and human decisions is cyclical, not linear, and that well-structured interactions can generate high-quality labeled data to improve AI models iteratively. Poorly designed interactions risk creating a vicious cycle where humans rubber-stamp AI outputs, reinforcing model confidence without critical evaluation, whereas thoughtfully engineered interactions encourage deliberation, yielding honest feedback and better model improvements.
Lee provides practical examples of designing effective AI-human interactions, such as breaking down complex AI flags into distinct questions to improve data quality and avoid false signals, and creating user interfaces that mimic natural human feedback patterns, like inline writing tutors that provide concise, actionable feedback tied directly to specific text. She also discusses coding agents, advocating for AI tools that act as collaborative junior developers—planning, asking questions, and delivering manageable changes—rather than overwhelming users with large, unreviewable code diffs or constant approval requests, which lead to rubber-stamping and poor data collection.
Finally, Lee outlines key design principles for building AI systems that promote discernment: engineering reasoning patterns that position humans as investigators rather than validators; surfacing assumptions and trade-offs for user evaluation; matching interaction friction to the stakes involved to encourage thoughtful decisions; treating every interaction as a labeled data point for continuous improvement; and proactively defining success metrics and data needs from the start. She concludes that often the solution to AI-human interaction challenges is not better models or more oversight, but carefully engineering the interaction itself to foster critical thinking and partnership between humans and AI.
