Ornella and Joel present Ace, a live AI voice tutor that prioritizes low latency by using a small speech-focused AI model combined with external systems handling reasoning and lesson management, ensuring responses within 950 milliseconds for natural interaction. This approach contrasts with slower, larger frontier models, offering a more efficient, cost-effective, and scalable solution for real-time voice applications by separating speech generation from complex decision-making processes.
In the video, Ornella and Joel introduce Ace, a live AI voice tutor designed to operate using a small AI model intentionally. Ornella emphasizes that in voice applications, latency is critical—any pause longer than a second makes users perceive the system as unresponsive or broken. Therefore, the AI model must respond within approximately 950 milliseconds to maintain a natural conversational flow. Instead of relying on a large, complex frontier model that takes longer to process, they opted for a smaller model that focuses solely on generating speech, while offloading other complex tasks to external systems.
Joel elaborates on the challenges with frontier models like Anthropic’s Claude 4.7, which, although excellent at reasoning and problem-solving, take several seconds to generate responses. This delay is detrimental in voice applications where real-time interaction is essential. To address this, they extracted all reasoning and decision-making processes from the AI model and implemented them in a state machine and intelligent layers outside the model. This external system manages lesson flow, student knowledge tracking, and response planning, feeding only the final output to the AI model for speech generation.
The video demonstrates the difference between using a large frontier model and their approach with a smaller model, Haiku 4.5. The large model takes several seconds to respond, whereas the smaller model, supported by the external scaffolding system, responds in under a second, providing a near-instantaneous and natural conversational experience. This approach not only improves latency but also reduces costs, as smaller models are more efficient and less resource-intensive.
Joel acknowledges that this method requires significant upfront investment in building the scaffolding system, including state machines and logic to handle various scenarios and maintain organization during longer interactions. However, this cost is a one-time development effort, unlike the ongoing computational cost of running large models for every interaction. The key takeaway is to choose the fastest model that fits the latency requirements and invest in external systems to handle complexity, allowing the model to focus on what it does best—speaking.
In conclusion, Ornella and Joel advocate for a design philosophy in voice AI applications that prioritizes low latency and efficiency over raw model size and reasoning power. By separating reasoning and logic from speech generation, they achieve a responsive, cost-effective, and scalable voice tutor system. This approach is particularly beneficial for real-time, high-volume applications where user experience depends heavily on immediate feedback and natural interaction. They invite viewers to reach out with questions, highlighting their commitment to advancing practical AI voice solutions.