The video compares 10 AI models on a complex coding task, revealing significant cost disparities despite similar accuracy, with some models delivering perfect results at a fraction of the price of others. The creator highlights the importance of testing multiple models using a unified API to balance cost, speed, and performance, recommending Claude Haiku for low-cost tasks and reserving pricier models for more demanding needs.
The video explores the cost and performance differences among 10 AI models tasked with writing a regular expression engine from scratch. Using the same prompt, 80 hidden tests, and a single API key via Oxyy, the creator compared the models’ accuracy, speed, and real costs. Two models produced perfect solutions, but their costs varied drastically—one costing under half a cent and the other 28 cents, a 66-fold difference for identical results. This highlighted the often overlooked but significant price disparities between AI models despite similar output quality.
The creator developed a Python script using Oxyy’s API to run tests on multiple models with a single key, simplifying the process of comparing models. The script retrieves a catalog of 198 models with their prices per million tokens, enabling precise cost calculations based on token usage. Testing involved a challenging coding task: building a regex engine with specific features and constraints, graded against 80 hidden tests including correctness and performance traps. This task was chosen because it could be objectively graded and was complex enough to differentiate model capabilities.
Results showed that seven models achieved perfect scores, with Claude Haiku 5.5 being the cheapest at under half a cent per run, followed by Gemini 3.5 Flash, Sonnet 5.5, Gemini 3.7 Flash, and GPT 5.6 Terra, all costing between 4 and 6 cents. Speed varied, with some models finishing in about 35 seconds while others took up to three minutes. More expensive models did not necessarily deliver better accuracy, though sometimes they offered faster responses. Two models failed significantly: Deepseek V4 flash scored 60/80 and struggled with timing traps, while GLM 5.2 scored zero due to token budget issues.
The creator also noted variability in costs even for the same model and prompt, with some runs consuming more tokens and costing more. The total cost for all tests and retries was about $1 across 59 requests, factoring in a promotional discount. Some anomalies were observed in billing, such as Gemini 3.5 Flash requests being charged at a cheaper rate under a different model name. These findings emphasize that pricing and performance can fluctuate and should be monitored closely when selecting AI models for coding tasks.
In conclusion, for small, well-defined coding jobs like this regex engine, the creator recommends starting with Claude Haiku due to its low cost and speed, moving to Sonnet for more capacity, and reserving pricier models like Opus for tasks where cheaper models fail. The experiment underscores the importance of testing multiple models with consistent metrics to balance cost, speed, and accuracy. The creator plans to continue using the Oxyy runner for future tests and invites viewers to suggest other models to evaluate next.