The presentation discusses the evolution of collective communication libraries within AMD’s ROCm ecosystem to meet the demands of increasingly complex AI workloads. Initially, AI training involved relatively simple communication patterns, primarily large-message all-reduce operations across GPUs. However, modern AI models have grown in size and complexity, introducing techniques such as tensor parallelism, pipeline parallelism, and mixture of experts. These advancements require communication libraries to support a diverse range of message sizes, latency sensitivities, and communication topologies, including point-to-point and all-to-all patterns, often simultaneously within large-scale multi-node environments.
For inference workloads, where latency is critical, the focus is on minimizing communication overhead and tightly integrating communication with computation. AMD has invested in enabling GPU-initiated communication and fusing compute and communication kernels to reduce latency. Techniques such as symmetric memory allow GPUs to directly access each other’s memory, eliminating staging buffers and further reducing delays. These innovations help improve user experience by ensuring communication happens promptly after compute kernels complete, which is vital for real-time AI inference.
Training workloads, which run over extended periods on tens of thousands of GPUs, prioritize efficiency and scalability. Challenges include network congestion, long job bootstrap times, and resource consumption by collective operations. To address these, AMD has enhanced its Rekall and Roxman libraries with features like system DMA (SDMA) engines to offload data movement from compute units, hierarchical and topology-aware algorithms for multi-node communication, and congestion-aware traffic routing. These improvements enable better overlap of communication and computation, reduce latency for small messages, and optimize resource usage across large GPU clusters.
The Rekall library has been optimized to leverage SDMA engines for data transfers, allowing compute units to focus on matrix multiplications, resulting in significant speedups when communication and computation overlap. For small message sizes common in inference, Rekall employs one-shot and two-shot algorithms that outperform traditional ring algorithms, achieving up to 3.7x speedup. Additionally, a hierarchical communication approach splits communication into scale-up (intra-node) and scale-out (inter-node) phases, improving multi-node collective performance by up to 3.6x. GPU-initiated communication is supported both within nodes and across nodes using GPU-direct asynchronous transfers, enhancing bandwidth and reducing latency.
Roxman, AMD’s GPU-initiated runtime implementing OpenCMN APIs, similarly benefits from SDMA-based data transfers, achieving comparable speedups to Rekall while using fewer compute units. It supports multiple NICs, including AMD’s Pensando Polaris, Volcanum, Broth Kontor X, and Connect7, delivering high bandwidth and low latency for inter-node communication. To mitigate congestion in large-scale training, AMD introduced congestion-aware spray (CAS) traffic routing, dynamically distributing traffic based on measured round-trip times across multiple queue pairs. This software-based technique significantly improves bandwidth utilization and reduces communication delays, contributing to more efficient large-scale AI training.
