Mixture of Experts (MoE) models enable massive AI model capacity by activating only a few expert components per token, reducing compute needs but requiring all experts to be loaded into memory simultaneously, leading to enormous RAM demands. This memory bottleneck has driven up global RAM costs and scarcity, prompting researchers to explore solutions like expert splitting, SSD offloading, and dynamic caching to balance performance and memory use.
The rising cost and scarcity of RAM, particularly DDR5 memory, is largely driven by the demands of AI data centers, but more specifically by a neural network architecture called Mixture of Experts (MoE). Unlike traditional dense models where every parameter is used for every token, MoE models have thousands of expert feed-forward blocks but activate only a few per token. This design allows models to have massive capacity without a proportional increase in compute per token. However, all experts, active or inactive, must be loaded into memory, creating a huge demand for RAM that the world is struggling to meet.
MoE routing is dynamic and token-specific, meaning the model decides in real-time which experts to activate based on the token’s hidden state. This prevents preloading or paging experts in and out easily, as the next token’s routing is unpredictable. Experts are not cleanly specialized human-labeled roles but rather learned tools shaped by training. This complexity means that the entire set of experts must be kept in memory, increasing the total parameter count and memory footprint significantly compared to dense models with fewer parameters.
The modern use of MoE as a scaling technique was popularized by Google researchers who showed that owning thousands of experts but activating only a few per token allows for much larger models without a proportional increase in compute. Models like Mixtral, Deepseek, Kamiko, Quen, and Llama 4 Maverick exemplify this trend, boasting hundreds of billions of total parameters but only a fraction active per token. While this approach boosts model capacity and efficiency, it places enormous pressure on memory resources, especially since GPU memory bandwidth has not kept pace with compute improvements.
To address the memory bottleneck, researchers and companies are exploring architectural and system-level solutions. These include splitting experts into smaller pieces, shared always-on experts, elastic expert parallelism across GPUs, and streaming model weights from SSDs rather than preloading them into RAM. While SSD offloading can enable running larger models on limited RAM, it comes with significant energy efficiency trade-offs, sometimes increasing energy consumption by up to 12 times. Other approaches involve machine-learned caching and on-demand paging of experts to balance memory use and performance.
In summary, Mixture of Experts started as a specialization concept but evolved into a scaling trick that decouples model capacity from compute cost. This innovation has driven unprecedented growth in model size but at the cost of massive memory demands, making RAM the most contested resource in AI computing today. The industry is actively seeking solutions to this memory challenge, but for now, the success of MoE models hinges on the availability of ever-larger memory capacities, which has contributed to the current global memory shortage and price surge.