The video presents an extensive evaluation of various AI coding assistants as of August 2025, focusing primarily on their ability to follow long-running specifications, execute instructions accurately, and call tools effectively. The host emphasizes that the scoring methodology involves static code analysis, unit testing, linting, and an LLM-based judge, with the latter having a smaller weighted influence due to its non-deterministic nature. Prompting and the harness used to run these models significantly impact performance, with some models like Gemini 2.5 Pro and GPT-4.1 requiring tuning to avoid laziness or incomplete task execution. The host also notes that while scores provide a quantitative measure, user experience factors such as speed and vision capabilities are equally important but not fully captured in the scoring.
Among the tested models, GitHub Copilot surprisingly emerged as the top performer in the Claw 4 Sonnet category, narrowly beating competitors like Root Code and Klein, whose scores were nearly tied. The overall scores across many assistants have converged, making it challenging to declare a definitive winner. Some models like Zed and Augment Code scored lower despite their popularity, with Zed notably consuming excessive tokens and sometimes failing to complete tasks. New entrants like Factory AI showed promise but require further exploration. The host highlights that while Claude Code remains strong, it has dropped in ranking compared to previous evaluations.
The evaluation of the 03 model revealed interesting results, with Cursor landing third place and Windsurf surprisingly taking the top spot, outperforming expectations based on previous tests with Gemini 2.5 Pro. Ader also performed well, especially considering its lower operational cost, though it is less agentic and more focused on batch code generation. The host points out that the way models are harnessed and controlled plays a crucial role in their effectiveness, with some models excelling in coding but struggling with agent-driven tasks. Several models like Open Code, Root Code, and Quinn3 coder showed strong performances, with Quinn3 coder notably achieving the highest overall score, especially when authenticated directly through Alibaba’s API.
Kimmy K2 presented challenges due to unstable providers and limited availability, making it difficult to test consistently. Despite this, some platforms like Trey, Klein, and Root Code managed to deliver solid results with Kimmy K2, provided a stable provider was used. Gemini 2.5 Pro, once a favorite, now scores lower compared to Claw 4 Sonnet models and requires specialized prompting to perform well. The host also discusses the importance of context window size and prompt caching, noting that Quinn3 coder has a slightly larger context window than Sonnet 4, which can influence performance and cost. Each model exhibits unique quirks, such as tool call tendencies or shortcut behaviors, which affect user experience.
In conclusion, the host shares personal preferences for August, favoring Open Code for its simplicity, Root Code for its advanced features and configurability, and continuing use of Claude Code despite its drop in ranking. Honorable mentions include Charm Crush for its potential, Augment for its strong agent capabilities, and AMP Code, which, despite hype, delivered average results but remains worth exploring further. The video closes by inviting viewer feedback on surprising results and encouraging likes and subscriptions, underscoring the evolving and competitive landscape of AI coding assistants.
