The video discusses Mage-VL-Cowork, a 4 billion parameter AI model from Microsoft designed to perform real desktop tasks by intelligently choosing when to click or write code. Unlike larger models, this one claims a 66% success rate on the OSWorld benchmark, which measures task completion by verifying the final state rather than relying on subjective assessments. The model uses a vision-language approach, processing screenshots and commands, and balances between mouse clicks and shell commands to optimize task efficiency. Microsoft highlights that this smaller model approaches human-level performance on desktop tasks, with a significant advantage in speed and cost when running on-device rather than relying on cloud APIs.
Microsoft’s training process for Mage-VL-Cowork involved a five-stage curriculum, including grounding, GUI agent training, coding and terminal training, hybrid training, and online reinforcement learning. This approach improved the model’s performance from around 45% to 66%, while also reducing the computational cost per task. The model’s design emphasizes choosing the most efficient tool for each step, such as preferring shell commands over clicking when faster. Demonstrations by Microsoft show the model completing tasks much faster than larger rivals by leveraging coding rather than extensive clicking.
The video creator attempted to run the model locally on a laptop with an RTX 4060 GPU but encountered significant issues due to insufficient memory and compatibility problems with quantization and transformer versions. The model failed to load correctly, producing broken outputs and invalid commands. This highlighted the challenges of running such a model on consumer hardware and the importance of using the recommended Docker environment and sufficient GPU memory (around 10 GB or more). The creator then rented a powerful cloud GPU (NVIDIA A100) to properly test the model, which successfully loaded and performed some tasks, though with mixed results.
In the cloud test, the model completed two out of three simple tasks correctly, such as summing a CSV column and renaming a file using shell commands, but failed a line-counting task due to a mismatch between its action description and tool call. The model showed a preference for shell commands over mouse clicks for file operations, consistent with Microsoft’s claims. However, the creator could not test the model’s clicking accuracy due to technical limitations in his setup. Overall, the test was a limited smoke test rather than a full benchmark, indicating the model is functional but with some reliability and recovery issues to be explored further.
The verdict is that Mage-VL-Cowork is a promising on-device AI agent capable of handling desktop tasks by combining vision and language understanding with coding and clicking actions. It requires a suitable environment with at least 16 GB GPU memory and the official Docker setup to run correctly. While Microsoft’s reported 66% benchmark score is impressive for a 4 billion parameter model, independent testing remains limited. Users interested in experimenting with this model should be cautious about setup details, avoid running it on personal data, and consider renting cloud GPUs for initial trials. The video ends by inviting suggestions for further task tests on a proper GPU setup.
Useful Links
