The video explores the limitations of AI avatars in following live instructions during conversations, highlighting a new benchmark called StreamEval designed to test this capability. Unlike traditional video benchmarks that evaluate a finished clip, StreamEval sends instructions midstream while the video is being generated, simulating real-time interaction. The benchmark covers five tasks: digital human dialogue, avatar performance with live directions, world generation with camera controls, and two types of editing tasks. It includes 300 test cases with over 1,300 planned live events, scoring models on metrics such as quality, stability, identity preservation, and crucially, the ability to follow instructions in real time.
StreamEval’s results reveal a significant gap between avatars’ visual quality and their responsiveness to live commands. Among seven avatar models tested, the highest score for following live instructions was only about 37 out of 100, meaning even the best models only correctly executed roughly one in three commands. Other metrics like identity preservation and stability scored much higher, often above 80 or 90, indicating that while avatars look consistent and stable, they struggle to react appropriately during live interactions. Similar patterns emerged in dialogue, world generation, and editing tasks, where models excelled in some areas but failed to reliably follow instructions.
The video creator ran StreamEval on their own machine and then on a rented RTX 4090 GPU to test a real avatar model with live instructions. They encountered practical challenges such as setup issues, cloud GPU availability, and unexpected costs, but successfully streamed a 13-second speech clip with four live instructions. Despite the server acknowledging all commands instantly, the avatar showed no visible response beyond mouth movement for speech, confirming the benchmark’s finding that many avatars do not truly “listen” or react in real time. This highlights the difference between receiving an instruction and actually executing it visibly.
The presenter emphasizes the importance of testing AI avatars with the specific instructions relevant to one’s product, especially sending commands during active speech when models are most likely to fail. They advise examining the actual video frames rather than relying on system acknowledgments, as the latter only confirm receipt, not execution. The StreamEval benchmark is open source and extensible, allowing developers to add new models and test them under consistent conditions. However, running the full benchmark is resource-intensive, requiring significant GPU power, storage, and API keys for scoring.
In conclusion, the video underscores that visual quality and instruction-following ability are distinct and both critical for practical AI avatars. Current models often look good but fail to respond reliably to live commands, a gap that StreamEval exposes. The presenter invites viewers to suggest other avatar models to test and shares resources for running the benchmark and accessing affordable cloud GPUs. This work provides valuable insights for developers aiming to build truly interactive and responsive AI avatars for real-time applications.
Useful Links
