The video demonstrates building a model router using Jev and OpenJevs that intelligently routes tasks to local or cloud AI models based on task type, complexity, and privacy needs, ensuring sensitive data remains local. It highlights Jev’s efficient, low-latency decision-making through probabilistic question answering, showcases a live multi-model routing system, and compares cloud-based Jev with the open-source SEM if model for cost-effective, privacy-conscious AI deployment.
In this video, the presenter demonstrates how to build a model router using Jev and OpenJevs, focusing on practical decisions, techniques, and benefits of these tools. The router acts as a local endpoint that interprets incoming requests and routes them to appropriate models based on the task type, difficulty, and privacy considerations. The local setup primarily uses the mini CPM 52B model, while cloud models like Deepseek V4.1 Flash and the newly released Quen image 2.1 are also integrated for tasks such as image generation and editing. The system is designed to handle various types of inputs, including text and images, and can decide whether sensitive data should remain local to avoid cloud exposure.
Jev, developed by Typesafe AI, is described as a “system one” model that does not generate text but instead evaluates input states against typed questions, returning probabilistic answers. This allows developers to implement conditional logic based on model outputs, such as categorizing tasks or assessing difficulty levels, without worrying about prompt engineering or JSON formatting. Jev supports three question types: choice (categorization), score (ranking difficulty), and null (yes/no questions), all of which can be queried simultaneously with minimal latency and low cost, making it efficient for real-time decision-making.
The video explains how these question types map onto the router’s functionality. Choice questions determine the task category (e.g., chitchat, code, image generation), score questions assess the complexity to decide if a local or more powerful cloud model should handle the request, and null questions check for private identifiable information (PII) to enforce local processing for sensitive data. The presenter highlights the importance of privacy by initially trusting Jev’s cloud service but later switching to open-source local versions like SEM if to keep all sensitive data on-premises.
A live demonstration showcases the router in action, routing simple conversational prompts to the local mini CPM model, code-related requests to the Deepseek cloud model, and image generation tasks to the local Quen image model. The system also detects PII, such as fake API keys or personal information, and routes those requests locally to protect privacy. The architecture involves a Next.js frontend communicating with a FastAPI backend that manages routing, logging, and health checks, integrating multiple models both locally and in the cloud. The presenter also discusses prompt rewriting for image generation and the flexibility to adjust routing thresholds and models.
Towards the end, the video compares the cloud-based Jev with the open-source SEM if model, showing similar functionality but with the advantage of full local control and faster response times. The presenter emphasizes the cost-effectiveness of this approach, noting that most requests are handled locally, significantly reducing expenses. The video concludes with an invitation for viewers to suggest use cases and expresses plans to release more code-focused content and GitHub repositories, encouraging experimentation with Jev and OpenJevs for building efficient, privacy-conscious AI routing systems.