The video explores why adding a second GPU or additional machines often doesn’t speed up local AI model inference as one might expect. Using the example of four Mac Studios clustered together running a 70-billion-parameter model, it was observed that the cluster’s speed matched that of a single machine, despite the combined hardware. This is due to the way the model is split across machines. The common method, pipeline parallelism, assigns blocks of layers to each machine sequentially, meaning only one machine works at a time while others wait, resulting in no speed gain but increased memory capacity.
An alternative approach, tensor parallelism, divides each layer’s computation across multiple machines simultaneously, which can increase speed. However, this method requires frequent communication between machines to combine partial results before moving to the next layer. This communication introduces latency, especially over typical network connections, which can negate the speed benefits. Tests showed that even with high bandwidth, the overhead of data exchange limited performance improvements, and in some cases, a single machine outperformed multiple machines connected over a network.
The solution to this bottleneck lies in reducing communication latency through technologies like RDMA (Remote Direct Memory Access), which allows direct memory writes between machines, bypassing slower networking stacks. Apple’s implementation of RDMA via Thunderbolt 5 and specialized cluster software significantly improved multi-machine performance, tripling token generation speed. Similarly, Nvidia’s NVLink provides a high-speed direct connection between GPUs within the same machine, enabling faster tensor parallelism, though this feature is now limited to data center hardware and not available on recent consumer GPUs.
Large-scale AI training clusters, such as those used by Meta and ByteDance, face similar challenges with network communication being a major bottleneck. These companies invest heavily in high-speed networking infrastructure like InfiniBand and optimize their models and training processes to minimize cross-machine communication. Innovations like Deep Seek’s expert subnet routing reduce network traffic by limiting data exchange to a few servers, improving efficiency. Despite the focus on networking, hardware failures, especially GPU-related, remain a significant cause of training interruptions.
In conclusion, adding more GPUs or machines primarily increases memory capacity, allowing larger models or more simultaneous users, but does not inherently speed up inference unless three conditions are met: software support for tensor parallelism, low-latency interconnects like NVLink or RDMA, and suitable model architecture. For most users, investing in a more powerful single GPU is more effective than adding a second one or clustering multiple machines, as the network communication overhead often outweighs the benefits of parallelism outside of specialized data center environments.
Useful Links
