The video showcases a remarkable setup featuring 224 GB of GPU memory packed into a compact mini PC, combining an AMD Strix Halo APU with an Nvidia GPU via a unique Oculink port. This setup allows both GPUs to collaboratively run a massive 122 billion parameter AI model, a feat previously thought impractical due to the differing architectures and software stacks of AMD and Nvidia. The Oculink port, a high-speed external PCIe connection, enables the integration of a desktop-grade Nvidia GPU into the mini PC, opening new possibilities for AI workloads on compact systems.
The presenter revisits performance benchmarks from the previous GMK Tech EVO X2 mini PC, confirming that the newer X3 model maintains similar speeds on older models, ensuring reliability in testing. Using Llama CPP for inference, the Nvidia RTX 5080 eGPU significantly outperforms the built-in AMD GPU on smaller models, but struggles with larger models due to its limited 16 GB VRAM. Conversely, the AMD Strix Halo’s 128 GB shared memory excels with large models but is slower on smaller ones. The video also demonstrates simultaneous multi-agent processing, where the Nvidia GPU handles multiple requests efficiently, while the AMD GPU scales more slowly.
The highlight is the use of the Nvidia RTX 6000 Pro with 96 GB VRAM, which handles large models at impressive speeds and power efficiency, running a 32 billion parameter model six times faster than the built-in chip. Even more impressive is the 122 billion parameter mixture-of-experts model, which runs faster than smaller dense models because only a subset of parameters is active at any time. However, integrating such a large GPU required a workaround involving hot-plugging due to firmware limitations, showcasing the challenges of pushing mini PC hardware to its limits.
To tackle models too large for a single GPU, the presenter splits a 130 GB model across both the Nvidia and AMD GPUs, achieving a balanced workload and a combined throughput of around 37 tokens per second. This cross-vendor multi-GPU setup is enabled by using the Vulkan API, which supports both AMD’s ROCm and Nvidia’s CUDA ecosystems, allowing the model to run seamlessly despite the different underlying technologies. The presenter also notes BIOS configurations that allow memory allocation adjustments for the AMD GPU, enhancing flexibility.
Finally, the video discusses the practical considerations of purchasing such systems. While the new X3 model offers the innovative Oculink port for eGPU expansion, it lacks some of the ports and stability features of the previous X2 model, which remains a strong choice for users not needing external GPU support. Pricing comparisons reveal that the X3 is slightly more expensive but offers unique capabilities, making it a compelling option for AI enthusiasts seeking powerful, compact hardware solutions. The video concludes by encouraging viewers to weigh their needs between expansion potential and port availability when choosing between these mini PCs.
