The video demonstrates how two AMD Ryzen AI Halos can be clustered using either a simpler RPC method or a more complex Rickle-based tensor parallelism approach to run extremely large AI models up to 400 billion parameters locally, highlighting the trade-offs between ease of setup and scalability. It also covers the technical challenges of networking, OS unification, and performance testing, showcasing AMD’s advancements in AI workloads and potential future expansions to multi-node clusters.
The video explores the capabilities of the AMD Ryzen AI Halo, a compact APU combining CPU and GPU with 128 GB of shared memory designed for AI workloads. The presenter revisits previous benchmarks comparing the Halo to Nvidia’s DJX Spark and highlights AMD’s recent advancements, including playbooks that facilitate generating images, videos, text, and coding assistance. Notably, AMD now supports clustering two Ryzen AI Halos, theoretically enabling the running of AI models up to 400 billion parameters by combining their resources. However, clustering requires splitting the model and continuous data exchange over a 10 GB Ethernet connection, raising questions about network efficiency and practicality.
To set up the cluster, the presenter emphasizes the necessity of a proper 10 GB Ethernet switch, as directly connecting the two machines is insufficient. After acquiring a suitable switch, the challenge was to unify the operating systems since one Halo ran Windows and the other Linux. Cloning the Linux installation onto the Windows machine’s SSD ensured identical environments, drivers, and configurations, which is crucial for clustering. Network tests confirmed near-maximum throughput, preparing the system for the clustering experiments.
The first clustering method tested was using llama CPP with RPC (Remote Procedure Call), which allows one Halo to borrow GPU memory from the other. The presenter details the importance of configuring GPU memory allocation on Linux, as Windows limits this capacity. Using the GLM 4.7 model with 358 billion parameters, the cluster achieved around 8 tokens per second with single concurrency and improved performance with higher concurrency, though performance dropped with very long prompts. This method is simpler but better suited for fewer simultaneous requests.
The second, more complex method involved using Rickle (RCCL) with VLM and Ray for tensor parallelism, enabling more efficient multi-node communication and better scaling. This approach was tested with the larger Quen 3.5 397 billion parameter model, which took longer to launch but demonstrated full GPU utilization and better token generation rates, especially at higher concurrency levels. The presenter notes that this setup resembles running a mini data center and is more suitable for users needing to handle multiple simultaneous AI workloads.
In conclusion, the video showcases how two Ryzen AI Halos can be clustered to run extremely large AI models locally, with AMD providing two distinct playbooks catering to different use cases. The simpler RPC method is easier to set up but less scalable, while the Rickle method offers better performance and scalability at the cost of complexity. The presenter also hints at future possibilities with four-node clusters and encourages viewers interested in expanding their AMD AI infrastructure to explore these advanced configurations.