Why Inference Matters 12x More Than Training | YC Paper Club

The video “Why Inference Matters 12x More Than Training | YC Paper Club” emphasizes that inference in AI, especially for large language models, demands significantly more computational resources and optimization efforts than training, impacting cost, scalability, and performance. Experts discuss various strategies—from kernel and compiler optimizations to memory management and quality assurance—to enhance inference efficiency, reliability, and adaptability across diverse hardware and deployment scenarios.

The video titled “Why Inference Matters 12x More Than Training | YC Paper Club” features a series of expert talks focusing on the critical importance of inference in AI model deployment, particularly large language models (LLMs). The discussion begins with Francois highlighting that inference—the process of running trained models to generate outputs—is significantly more resource-intensive and commercially impactful than training. Using data from DeepSeek, he explains that inference can require up to 12 times more computational effort over a model’s useful life compared to training. This underscores why optimizing inference efficiency is crucial for cost management and scalability in AI services. Francois also outlines key technical challenges in inference, such as managing cache hits, kernel utilization, and balancing throughput with latency, emphasizing innovations in model architecture, runtime kernels, and system-level optimizations.

Dan Fu from Together AI expands on the inference lifecycle, detailing how requests flow through data centers, are routed to GPUs, tokenized, and processed through prefill and decode stages. He explains the complexity behind serving inference at scale, including the importance of routing requests to GPUs with relevant cached data to maximize efficiency. Dan also discusses the distinct computational characteristics of prefill (heavy, compute-bound) versus decode (memory-bound, latency-sensitive) workloads and the strategies to optimize each. He highlights Together AI’s focus on kernel optimization, parallelism, and quantization, sharing insights into their use of agent-driven compiler optimization and large-scale GPU clusters to improve throughput and reduce costs.

Jeff Morgan, CEO of Ollama, addresses the variability in quality and performance across different open model inference providers. He stresses that not all tokens generated by open models are equal due to differences in hardware, quantization methods, tool calling implementations, and inference engines. Jeff discusses the challenges of routing requests across multiple providers to ensure consistent quality, reliability, and cost-effectiveness. He reviews various methods for monitoring and evaluating inference quality, including benchmarks, distribution tests, and log probability comparisons, and advocates for more robust tooling and shared infrastructure to close the quality gap between open and closed models.

Joe Fiaty, CEO of Luminal, presents a unique perspective on inference optimization through compiler technology. His company leverages a technique called equality saturation to optimize model execution by exhaustively searching for the most efficient program transformations. By decomposing complex neural network operations into a small set of primitive operations and applying rewrite rules, their compiler can generate highly optimized kernels tailored to diverse hardware backends, including GPUs, TPUs, and specialized ASICs. Joe emphasizes the importance of real hardware profiling over heuristic cost models to guide optimization and highlights the scalability and flexibility of their approach in supporting a wide range of models and hardware.

Leon Shaw from SG Lang discusses innovations in memory management and scheduling for inference engines, particularly for agent workloads with long and variable context lengths. He introduces the concept of a unified radix tree for efficient prefix reuse and cache management across different memory pools (GPU memory, host memory, storage). Leon explains how their engine handles eviction policies, overlapping scheduling to reduce GPU idle time, and compaction techniques to manage fragmented memory. He also touches on architectural adaptations for long-context transformers and the integration of asynchronous execution frameworks like XLA and Miles to enhance inference performance and training synergy. The talk concludes with an invitation to the upcoming XLA summit, highlighting ongoing community efforts to advance open-source AI infrastructure.

Useful Links