The talk addresses the challenge of speeding up AI inference by enabling fast key-value (KV) movement over Ethernet networks, which are traditionally lossy and not optimized for such workloads. There are two main types of inference systems: mainstream systems where pre-fill and decode happen on one or two GPUs with external storage accessed via a lossy front-end Ethernet network, and large-scale enterprise systems where pre-fill and decode are disaggregated, allowing use of both front-end and back-end networks. The key question is how to support both TCP and RDMA traffic efficiently on these networks, given their conflicting requirements—TCP relies on congestion signals and buffering, while RDMA demands lossless, low-latency transport.
To address this, the speaker introduces a software-based shim technology that synchronizes clocks end-to-end without requiring network support, enabling precise measurement of one-way delays to detect congestion. This shim intercepts RDMA queue pair creation and sends shadow probe packets to monitor network conditions. When congestion is detected, it sends congestion notification packets (QCN) to rate limit traffic, and can dynamically reroute flows to less congested paths. This approach is transparent to applications and networks and can be deployed on SmartNICs, making it practical for large-scale hyperscale environments.
The talk also explains the concept of the Ghost Buffer, a software mechanism that acts as a virtual queue to estimate and control congestion at the network edge, particularly useful for receiver-side in-cast scenarios where multiple senders overwhelm a receiver. By combining one-way delay measurements with the Ghost Buffer, the system can proactively manage congestion, maintain low latency, and enable bandwidth slicing with priority classes (e.g., gold, silver, bronze) to prioritize critical KV reads over less urgent traffic like KV writes or demotions.
Experimental results from deployments in Oracle Cloud demonstrate the effectiveness of this approach. Without the shim, RDMA traffic on a lossy Ethernet network suffers from high packet loss and poor throughput, while TCP shows high latency and variability. With the shim and congestion control enabled, packet loss is eliminated, throughput reaches line rate, latency drops significantly, and fairness is achieved among competing flows. The system also supports bandwidth slicing, allowing different traffic classes to receive guaranteed rates without wasting bandwidth when some classes are idle.
Finally, the talk highlights the benefits of applying these techniques to both front-end and back-end networks in disaggregated inference systems. Improvements in time-to-first-token and end-to-end latency range from 2.5x to over 10x, especially at higher percentiles where latency matters most. The speaker concludes by emphasizing the importance of eliminating packet drops and creating bandwidth headroom on front-end networks to prevent GPU idling and improve inference performance. Future work includes more advanced bandwidth slicing and prioritization to further reduce completion times and enhance predictability.
