The video highlights Avalon’s significant 47% performance improvement over Strata on AMD’s Radeon RX 7900 XT for running large language models, achieved through innovative VRAM caching and speculative drafting optimizations tailored to the Quinn 38B Flash Next model. However, it emphasizes that these gains are hardware- and model-specific, with Strata continuing to excel on newer GPUs and offering broader support, underscoring the importance of context when evaluating benchmark claims.
The video discusses a significant performance improvement in running local large language models (LLMs) on AMD GPUs, specifically highlighting Avalon, a fork of Llama CPP, which reportedly outperforms Strata, the previously fastest AMD-optimized inference engine, by 47% on a Radeon RX 7900 XT. The presenter shares a common frustration among AMD GPU users who, despite impressive hardware specs, experienced slow token generation speeds due to software ecosystem limitations, particularly around ROCm support and kernel tuning. Avalon’s benchmark of 91 tokens per second on a used 7900 XT contrasts sharply with Strata’s 61.9 tokens per second, sparking excitement but also skepticism about the validity and context of these numbers.
Avalon is not just a simple fork but a heavily optimized project that incorporates ideas from multiple sources, including Strata itself, and benefits from the author’s 30 years of optimization experience combined with AI-assisted code generation. It is specifically tuned for the Quinn 38B Flash Next model, a complex mixture of experts architecture with 125 billion active parameters and an additional 51 billion parameter lookup table stored outside the GPU. The main bottleneck for performance is not GPU compute power but memory bandwidth, particularly the ability to quickly access large expert weight data from system RAM and VRAM. Avalon innovates by maximizing VRAM usage as a cache for expert weights and parallelizing work between the GPU and CPU to reduce latency and improve throughput.
The performance gains come from two main innovations: an expert cache system that dynamically uses available VRAM and CPU cores to handle cache misses, and a more efficient speculative drafting mechanism that guesses multiple tokens ahead to speed up inference. These optimizations allow Avalon to achieve a 27% speed increase over Strata without speculative drafting and a total 47% improvement when drafting is enabled. Importantly, Avalon’s model file is larger and of higher quality than Strata’s, disproving the notion that the speedup comes from using a lower-quality or more compressed model.
However, the video emphasizes that these benchmark results are highly specific to the tested hardware and model. Strata has continued rapid development, especially for newer RDNA4 GPUs like the Radeon AI Pro R9700, where it outperforms Avalon. Avalon’s optimizations are finely tuned for the 7900 XT’s particular balance of compute and memory bandwidth, and performance gains diminish or reverse on other hardware configurations. Additionally, Avalon currently only supports Linux on RDNA3 GPUs and is limited to the Quinn 38B Flash Next model, whereas Strata offers broader model and platform support.
In conclusion, the video highlights the importance of context when interpreting benchmark numbers, urging viewers to consider the exact hardware, software versions, model files, and ownership of the tested systems. Avalon represents a bold, focused approach to optimization by a single developer leveraging AI assistance, achieving impressive gains on a specific setup. Meanwhile, Strata continues to evolve rapidly, especially on newer hardware. The key takeaway is that no single benchmark tells the whole story, and users should choose the engine best suited to their hardware and needs while understanding the nuances behind headline performance claims.