The presentation introduces ATOM, an inferencing framework developed by AMD, supported by a collection of kernels called AITER. AMD has delivered powerful hardware, but to fully leverage their GPUs, sophisticated software stacks like ATOM are essential. The speaker outlines four main challenges in large language model (LLM) inference: underutilized GPU performance with new building blocks, the complexity of tuning models with many configuration knobs, scaling models that outgrow a single machine, and the delay in optimizing new models for speed. The talk is structured as a four-story building addressing these challenges step-by-step, starting with ATOM as the execution engine.
ATOM serves as a lightweight, open-source orchestration layer that sits between popular machine learning frameworks (like PyTorch and VLM) and AMD GPUs. It handles scheduling and bookkeeping efficiently without reinventing the wheel, integrating tightly with existing inference engines to provide a seamless user experience. Behind ATOM is AITER, a collection of kernels that act as a router, selecting the most appropriate kernel implementation—ranging from high-level Python kernels to assembly-level optimizations—based on the model and workload. Additional compiler techniques such as aggressive quantization, pre-computing, and speculative decoding further enhance performance.
The second layer focuses on optimization, addressing the challenge of tuning numerous parameters and kernel combinations. This involves a two-level search: an inner loop that benchmarks candidate kernels for individual operations and an outer loop that tunes global settings like batch size and data formats. To efficiently evaluate performance, AMD employs three modes: quick estimation without GPUs, full evaluation on real hardware, and a hybrid approach combining both. Continuous integration (CI) systems ensure that kernel contributions maintain or improve performance, with multi-layer testing from operator-level to end-to-end framework integration.
Scaling beyond a single node is the third focus, as next-generation LLMs with hundreds of billions of parameters cannot fit on one machine. AMD addresses this with a disaggregated serving approach, splitting workloads between prompt reading (compute-heavy) and answer writing (memory-heavy) tasks. The communication layer, called Moray, efficiently manages data shuffling, KV cache synchronization, and chip-to-chip communication, enabling smooth multi-node operation. Performance benchmarks show AMD’s solution achieves faster data movement and higher throughput compared to alternatives, while maintaining competitive total cost of ownership by leveraging both AMD GPUs and CPUs.
Finally, the roadmap highlights AMD’s commitment to supporting upcoming hardware generations and evolving disaggregated serving techniques, including expert parallelism. The company plans to provide improved APIs for easier programming and fine-tuning, along with a “flywheel” ecosystem to accelerate testing and adoption. The presentation concludes by emphasizing three key points: ATOM is production-ready for demanding models, it self-tunes for optimal performance, and it is fully open source and available on GitHub for users to try on AMD GPUs.
