The video demonstrates an experimental method of adding vision capabilities to a text-only Deepseek LLM by pairing it with a smaller vision sidecar model that converts images into textual descriptions for the main model to process, enabling image understanding without retraining. While effective in providing detailed image analysis and maintaining the reasoning strengths of the original model, this approach introduces memory constraints and depends on the accuracy of the vision model’s interpretations.
In this video, the presenter demonstrates an experimental method to add vision capabilities to a text-only local large language model (LLM), specifically the Deepseek version 4 flash model running on a DGX Spark. The Deepseek model is powerful for coding and reasoning tasks but lacks inherent vision functionality, meaning it cannot process images directly. The presenter initially shows how the model fails to identify the contents of an image thumbnail because it has no vision tower or image processor built into its neural network.
The solution explored is inspired by a post from Mia AI Lab, who developed an experimental setup that pairs the main Deepseek model with a smaller vision model running alongside it. Instead of retraining the large model to include vision or switching to a multimodal model, this approach uses a “sidecar” vision model to analyze images and convert them into textual descriptions. These descriptions are then passed to the Deepseek model, which processes the text and performs reasoning without ever directly seeing the pixels. This architecture treats vision as a tool the model calls, separating the roles of “eyes” (vision model) and “brain” (Deepseek model).
The vision sidecar is a smaller, deterministic 4-billion parameter model designed to provide factual descriptions, OCR text extraction, and image comparisons without speculation or creativity. It runs concurrently with the Deepseek model and communicates via an MCP server that converts images to base64 data URIs and sends them to the vision model. The Deepseek model receives only the textual output from the vision model, maintaining its text-only nature. This setup is experimental and comes with trade-offs, including a significant reduction in the Deepseek model’s context window due to increased memory usage from the vision integration.
The presenter tests the system with various images, including a stylized anime character, a claymation prehistoric scene, and a low-resolution promotional thumbnail. The vision model successfully provides detailed and accurate descriptions in most cases, even identifying subtle details like the cardboard texture of the anime character. However, limitations are evident when the vision model misinterprets or misses details, such as confusing a Spotify playlist for a plugin list in the thumbnail. Since the Deepseek model relies solely on the vision model’s textual output, any errors in vision interpretation propagate into the final reasoning.
In conclusion, this experimental vision integration offers a clever and practical way to add image understanding to a powerful text-only LLM without retraining or sacrificing its reasoning capabilities. While it introduces memory and context limitations and depends heavily on the accuracy of the vision sidecar, it allows users to maintain a single reasoning model that can handle both text and image inputs indirectly. The presenter encourages viewers interested in local AI models to explore Mia AI Lab’s work and consider this approach as a promising direction for multimodal AI on local hardware.
