The video begins by reflecting on the early expectations that AI would be trained using evolution strategies or genetic algorithms, inspired by natural evolution and early game AIs. However, despite this intuitive appeal, evolution strategies fell out of favor because they struggled to scale to deep neural networks with millions of parameters. The main challenge was that random mutations in such high-dimensional parameter spaces mostly degraded performance, making it difficult to find beneficial changes. Additionally, the interdependent nature of neural network parameters complicated the optimization process, and attempts to model these interactions were computationally infeasible.
A breakthrough came with OpenAI’s 2017 paper, which revived evolution strategies by simplifying the approach: instead of modeling complex parameter interactions, they applied Gaussian noise to perturb parameters and averaged the results over many samples. This method made evolution strategies scalable to deep reinforcement learning tasks, such as Atari games, by efficiently estimating useful update directions despite the noise. However, applying evolution strategies to large language models (LLMs) remained challenging because next-token prediction provides a rich gradient signal that evolution strategies lack, and the computational cost of multiple forward passes was prohibitive.
The video highlights that evolution strategies are particularly well-suited for reinforcement learning fine-tuning of LLMs, where only a scalar reward for the entire output is available, making token-level credit assignment difficult. Unlike gradient-based methods, evolution strategies treat the model as a black box and explore parameter space directly, potentially discovering new reasoning behaviors. A recent 2025 paper demonstrated that evolution strategies could be effectively applied to billion-parameter models using surprisingly small populations (around 30), leveraging the smoothness and low-dimensional structure of useful update directions in large neural networks. This finding challenges previous assumptions that evolution strategies could not scale to such large models.
To address the high computational cost of evolution strategies, a follow-up paper introduced Egg Roll, which uses low-rank perturbations structured as LoRA (Low-Rank Adaptation) updates. This innovation allows multiple perturbations to be evaluated efficiently by swapping small LoRA modules during a single forward pass, drastically reducing the compute and memory requirements. Egg Roll achieves comparable performance to standard evolution strategies but with much greater hardware efficiency. It also outperforms popular reinforcement learning fine-tuning methods like GRPO in terms of speed and parallelism, enabling more extensive exploration within the same computational budget.
In conclusion, the video emphasizes that evolution strategies, once considered obsolete for large neural networks, are making a surprising comeback thanks to new engineering and algorithmic advances. While more research is needed to fully compare evolution strategies with gradient-based reinforcement learning methods, the approach shows great promise for fine-tuning LLMs, especially in scenarios with sparse or coarse reward signals. The presenter encourages viewers to follow ongoing developments and explore intuitive AI education resources to deepen their understanding of these cutting-edge techniques.
