Microsoft recently held a Windows and Surface event where they showcased significant advancements in local AI processing. A standout demo involved a coding agent handling 1.6 million tokens locally on a laptop without incurring cloud costs, highlighting the model’s ability to run entirely on-device. This hybrid intelligence approach, termed “intelligent routing,” allows Windows to decide dynamically which tasks run locally and which are sent to the cloud, optimizing performance and efficiency. The GitHub Copilot app will incorporate this technology starting October 15th, blending local and cloud models to handle coding tasks more effectively.
The local models demonstrated include Microsoft’s AI Code 1.1 Flash model, which has been quantized down to 3 bits to reduce size by about 80%, enabling it to run efficiently on laptops with a 256k context window. Additionally, Microsoft introduced a new Nemotron model running at 2 bits, optimized for the RTX Spark GPU, and a highly compressed 1.6-bit version of DeepS 4, which is notable for its memory efficiency but raises questions about maintaining model quality and accuracy, especially for complex coding and reasoning tasks. These quantization techniques aim to balance model size, precision, and performance for local deployment.
To support these models, Microsoft integrated Llama CPP into Windows ML, their runtime for deploying AI models across various hardware like GPUs, NPUs, and CPUs. This integration ensures compatibility with open models immediately upon release, allowing users and developers to run a wide range of models without waiting for official packaging. This flexibility could be a major advantage for Windows apps using local AI, potentially enabling users to plug in custom models seamlessly into the AI routing system.
Security and control were also addressed with the introduction of Microsoft Execution Containers (MXC), a sandbox environment that enforces organizational policies on AI agents. MXC attributes every action to the agent rather than the user, allowing for precise control and traceability. This setup helps prevent unauthorized actions by AI agents while maintaining user access, enhancing trust and safety in AI-assisted workflows.
Finally, Microsoft unveiled the RTX Spark hardware, a powerful GPU integrated into Windows laptops with up to 128 GB of unified memory and full CUDA support, comparable to an RTX 5070 but with efficient power consumption. Pricing starts around $2,600, with higher memory configurations increasing the cost. Alongside laptops, Microsoft teased a desktop version and a Windows DGX station, signaling a strong commitment to local AI hardware. Overall, Microsoft’s event marked a significant step toward mainstreaming local AI, joining Nvidia in pushing for on-device AI processing and signaling a future where hybrid cloud-local AI becomes standard.
Useful Links
