The video explains the concept of Mixture of Experts (MoE) models, which are widely used in AI but often misunderstood. For example, the Kimmy K3 model contains 896 experts, but only 16 are activated for any given word. Despite having trillions of parameters, less than 2% of them perform computations at a time. This architecture allows models to be extremely large yet efficient in processing, as only a small subset of parameters is used for each word, although all parameters must be loaded into memory.
Contrary to the common misconception, the term “expert” does not refer to specialized modules like a panel of specialists. Instead, a model stage is divided into many slices, each called an expert, but these are just narrower parts of the same machinery. The router is the critical component that decides which experts to activate for each word. It evaluates the input representation of a word and selects the top experts to process it, blending their outputs. This routing decision happens independently for every word at every stage, resulting in thousands of routing decisions per prompt.
The routing choices are not based on human-like expertise but rather on patterns such as grammar and word shape. Attempts to predict which experts will be needed next show that adjacent words often activate different experts, making preloading difficult. Additionally, a small subset of experts handles most of the workload, while many remain rarely used. The router’s behavior emerges from training dynamics, where some experts get more traffic and improve faster, creating a feedback loop that can lead to uneven expert utilization.
To address uneven expert usage, newer designs implement load balancing techniques that encourage more uniform expert activation without compromising model performance. The main advantage of MoE models is computational efficiency: a model with hundreds of billions of parameters can generate output at the speed of a much smaller dense model because only a fraction of parameters are active per word. However, memory requirements remain high since all experts must be loaded simultaneously, posing challenges for running these models locally.
Finally, the video highlights the distinction between the total parameter count and the active parameter count. While the total size dictates hardware requirements and memory usage, the active parameters determine computational cost and speed. Providers hosting large MoE models pay for the entire model but only run a fraction of it per word, allowing them to offer powerful models at lower prices than smaller dense models. Users should size their machines based on the total model size, as the sparsity only applies to computation, not memory.
