Archana Kamath and Tyler Gillam from DigitalOcean advocate for model routing, a dynamic approach that selects AI models based on specific task requirements, balancing cost, latency, and accuracy rather than relying on a single benchmark-leading model. They present DigitalOcean’s open-source inference router, which enables customizable, efficient routing of requests to appropriate models, demonstrated to reduce costs and latency while maintaining quality in real-world applications.
The talk titled “Preferences Over Benchmarks: Model Routing” by Archana Kamath and Tyler Gillam from DigitalOcean challenges the common practice of selecting AI models solely based on benchmark performance. Archana, VP of Engineering at DigitalOcean, emphasizes that relying on a single top-performing model is often inefficient and risky. She outlines three key reasons to move beyond the one-model approach: escalating costs, the mismatch between model size and task complexity, and the risk of failure without failover options. Instead, she advocates for model routing, where the choice of model depends on the specific request, balancing factors like cost, latency, accuracy, and user preferences.
Archana explains that no single model is universally best; the optimal model varies by task. For example, small open models may suffice for classification tasks, while code completion might require faster, larger models. Critical tasks like code review and security demand frontier models. This nuanced approach cannot be captured by public leaderboards alone, as it depends on the task, system prompts, cost constraints, latency requirements, and end-user preferences. This insight sets the stage for DigitalOcean’s inference router, designed to dynamically select the right model per request.
Tyler Gillam then presents the architecture and functionality of DigitalOcean’s inference router. Unlike previous black-box auto-routing attempts, their router is transparent, customizable, and open source, avoiding vendor lock-in. Users specify workload priorities such as cost, latency, and quality, and the router intelligently routes requests to the most appropriate model. The routing model itself is lightweight and fast, operating under 200 milliseconds without extra cost. Tyler demonstrates the router’s configuration interface, showing how users can define tasks and assign multiple models with fallback and selection policies, enabling flexible and efficient routing.
In a live demo, Tyler compares a single-model setup using a premium model against the inference router in a coding agent workflow. The router dynamically selects models based on the task—code generation, bug fixing, test writing—resulting in significantly lower costs (about one-third) and reduced latency while maintaining comparable output quality. Real-time observability shows token usage, model selection, and cost accumulation, highlighting the router’s efficiency. This practical example illustrates how model routing optimizes both performance and expense in real-world applications.
Archana concludes by reinforcing that model routing is foundational, not the end goal. She outlines future enhancements such as evaluation tools to validate routing decisions, caching to avoid redundant costs, and personalization to adapt routing based on team preferences. The continuous feedback loop improves routing effectiveness over time. Ultimately, DigitalOcean’s open, preference-driven router empowers teams to move beyond benchmarks and build AI solutions tailored to their unique needs, marking a significant step forward in AI infrastructure.