The video chronicles the creator’s extensive journey to build a high-performance AI cluster using four AMD Ryzen AI Max Plus 395 (Strix Halo) mini PCs, each equipped with 128 GB of unified memory. Despite the impressive hardware, it took over three months to optimize the cluster’s performance due to initial misunderstandings about the bottlenecks. The creator initially believed that upgrading network bandwidth would speed up the cluster, investing in high-speed Intel 810 network cards and a MikroTik CRS A12 switch. However, testing revealed that bandwidth was not the limiting factor; instead, network latency was the critical issue affecting token generation speed during multi-node inference.
The key technical insight was the difference between pipeline parallelism and tensor parallelism in multi-node AI model inference. Pipeline parallelism, used by llama.cpp RPC, allows fitting larger models across multiple machines but does not improve speed because each token must sequentially pass through every node. In contrast, tensor parallelism splits each layer of the model across all machines, enabling simultaneous processing of tokens and true speed scaling with more nodes. The creator adopted vLLM, a framework supporting tensor parallelism, but faced challenges adapting it to AMD GPUs and the Strix Halo hardware.
A major breakthrough came with the adoption of RDMA (Remote Direct Memory Access) networking, which drastically reduces latency by allowing direct memory access between machines without CPU intervention. This reduced round-trip latency from hundreds of microseconds to about five microseconds, significantly improving cluster throughput. However, RDMA required matched network hardware, leading to complications when mixing Intel and Mellanox cards. After switching to Mellanox cards on all nodes and installing Fedora (due to better compatibility), the cluster achieved near-linear scaling on smaller dense models, with token generation speeds increasing substantially over TCP networking.
Despite hardware and software improvements, running very large models like DeepSeek R1 (671 billion parameters) remained challenging due to software bugs and quantization issues, especially with mixture-of-experts (MoE) models and certain quantization schemes. The creator experimented with various quantization methods and kernel versions, eventually upgrading to Linux kernel 6.19 and using updated ROCm drivers patched by community contributor Donato Capitella. This resolved many hangs and crashes, enabling stable multi-node tensor parallelism on large models and surpassing the performance of Minisforum’s own marketing demo by about 5%.
In conclusion, the project demonstrated that building a powerful multi-node AI cluster with AMD Strix Halo mini PCs is feasible but requires deep understanding of networking latency, parallelism strategies, and software compatibility. The Realtek NICs on the Minisforum hardware posed Linux driver challenges, but overall the hardware was praised for expansion capabilities. The creator highlighted the importance of RDMA for low-latency communication and the benefits of community-driven software patches. The video ends with a comparison of different models and cluster sizes, showcasing impressive token generation rates and encouraging viewers to consider both hardware and software factors when building AI clusters.
