The video reviews Microsoft’s Mage-VL-Cowork model, highlighting its innovative combination of vision and language to automate complex spreadsheet tasks via mouse and shell commands, but finds its performance inconsistent and slower than claimed, with significant hardware and setup challenges. Despite showing promise, the model requires careful verification of outputs and substantial resources, and the presenter encourages further testing and community feedback.
The video examines the Microsoft Mage-VL-Cowork model, a 4 billion parameter AI designed for complex computer use tasks, particularly spreadsheet manipulation. Microsoft claims the model achieves 66% accuracy on a challenging benchmark and can complete certain spreadsheet tasks significantly faster than much larger models. The model integrates screenshot-based mouse control with a persistent terminal, allowing it to decide whether to interact via mouse clicks or by running code in a shell. It uses a vision encoder combined with a language model, has a large context window, and operates by writing short thoughts followed by tool commands, either for mouse/keyboard actions or bash shell commands.
The presenter tested the model on a real-world spreadsheet task involving cleaning data, fixing region names, converting revenue to numbers, and creating a chart. The model was run three times: twice with both mouse and shell tools enabled, and once with only mouse control. The hybrid runs showed mixed results—one run passed the task checker, another failed despite the model reporting success, and the mouse-only run failed to complete the task within the step limit. The model demonstrated the ability to switch between mouse and shell commands autonomously, often preferring shell commands for efficiency.
Technical challenges were encountered during setup, including compatibility issues with the latest transformers library and the need to use an older version for the model to load correctly. The model’s weights require substantial GPU memory, making it impractical to run on typical consumer hardware without renting powerful cloud GPUs. The presenter rented an A100 GPU for testing, which cost about $150 for the hour-long session. Despite these hurdles, the model showed promise in handling complex tasks but lacked consistent reliability.
Regarding speed, the presenter could not replicate Microsoft’s claim of completing the spreadsheet task in 39 seconds. Instead, the runs took between 12 and 16 minutes, largely due to the slower inference speed on the rented cloud hardware and the longer step count in the test task. The discrepancy is attributed to differences in the testing environment and task specifics. The presenter emphasizes that Microsoft’s speed claim is based on their own demo setup and has not been independently verified.
In conclusion, the Mage-VL-Cowork model is a novel approach combining vision and language for computer automation, capable of choosing between mouse and shell interactions. However, it requires careful setup, significant hardware resources, and cautious use in sandboxed environments due to its ability to run arbitrary shell commands. The model’s performance is promising but inconsistent, and users should verify outputs carefully. The presenter encourages viewers to try the model themselves, pin the correct software version, and share suggestions for further testing.