The video discusses a significant performance improvement in running massive 125 billion parameter Mixture of Experts (MoE) AI models on consumer GPUs, specifically through a recent update to llama.cpp. Traditionally, running these models on high-end GPUs like the 24 GB RX7900 XTX resulted in painfully slow token generation speeds—around nine tokens per second—barely faster than human reading speed. This sluggishness stems from the model’s architecture, where only a small fraction of the 512 experts per layer are active at any time, but the entire model’s weights often reside in slower system RAM, causing frequent and costly data transfers over the PCIe bus that bottleneck performance.
The breakthrough came with a new caching strategy introduced by developer Lick 77 and merged into llama.cpp on October 7th. Instead of moving entire expert layers to GPU memory, the update caches individual experts on the GPU based on recent usage, drastically reducing unnecessary data transfers. This expert-level caching leverages the model’s tendency to reuse certain experts repeatedly, akin to a busy kitchen keeping frequently used pots on the stove. As a result, token generation speeds jumped from about 9.7 to 37.2 tokens per second on the same hardware, a nearly 4x improvement, though the actual speedup varies depending on hardware and configuration.
However, the video cautions that this 4x speedup headline can be misleading. The improvement is measured against a baseline where no experts were cached on the GPU, which is an unrealistic starting point. More practical comparisons show a 1.5x to 2.2x speedup over previously optimized whole-layer caching methods. The effectiveness of the expert cache heavily depends on available GPU VRAM, PCIe bus speed, and workload type. Cards with at least 24 GB VRAM and fast PCIe 4.0 x16 slots benefit most, while smaller GPUs with limited VRAM or slower PCIe connections may see no improvement or even slower performance due to cache management overhead.
Additional caveats include the impact on prompt processing speed, which can decrease because the cache only accelerates token decoding, not the initial prompt reading. Speculative decoding techniques that predict multiple tokens ahead can also reduce cache efficiency if the cache is not sized to handle the increased expert calls. Moreover, some users have reported rare issues like repetitive output loops when using the cache, though these are likely unrelated to the caching mechanism itself. The video emphasizes the importance of benchmarking individual setups to determine if enabling the expert cache is beneficial, recommending specific steps to measure decode speed, prompt speed, and cache hit rates.
Finally, the video compares llama.cpp to Strata, another runtime optimized specifically for Flash Next models. Strata currently outperforms llama.cpp in speed and prompt processing but supports fewer model families. Llama.cpp remains the more versatile choice for users who want broad model compatibility and rapid support for new releases. The expert cache is a meaningful advancement that addresses a long-standing bottleneck in running large MoE models locally, but its benefits depend heavily on hardware and usage patterns. Viewers are encouraged to test their own systems and share results to build a clearer picture of real-world performance gains.
Useful Links
