The video demonstrates training a small vision-language model to play Kirby’s Dream Land using a two-phase approach inspired by the Odysseus paper, combining supervised fine-tuning for perception with reinforcement learning guided by a lightweight CNN critic. Despite challenges like limited data and model drift, the experiment showed significant gameplay improvements, highlighting the potential of this method for long-horizon decision-making in AI agents.
In this video, the creator explores vision language models (VLMs), which combine visual perception with language understanding, and demonstrates training a small 50 million parameter VLM to play the classic Game Boy game Kirby’s Dream Land. Inspired by a recent Princeton paper called Odysseus, which tackled the challenge of training AI agents to make long sequences of decisions in games like Super Mario Land, the video breaks down the core difficulties of long-horizon reinforcement learning (RL) and how the paper’s novel training recipe overcomes them. The key challenge addressed is credit assignment—figuring out which decisions in a long chain actually impact the final outcome.
The Odysseus approach splits training into two phases: supervised fine-tuning (SFT) and reinforcement learning. In the first phase, the model learns perception by being trained on thousands of annotated game frames describing the environment, enemies, and objects, rather than directly imitating gameplay. This teaches the VLM to “see” and understand the game state. In the second phase, reinforcement learning with proximal policy optimization (PPO) is used to teach the model to act, letting it learn from trial and error and the consequences of its actions over many turns. A crucial innovation is using a lightweight convolutional neural network (CNN) as a critic to evaluate game states cheaply and efficiently, rather than relying on expensive large language models.
The video’s creator attempts to replicate this approach on a smaller scale using a laptop GPU and a smaller VLM model, applying the same two-phase training to Kirby’s Dream Land via a Python-based Game Boy emulator. They use Codex to automate much of the coding and training process, encountering challenges such as limited data, label quality, and the model drifting towards memorizing scripted actions rather than generalizing. After refining the dataset and training setup, they achieve improved results, with the trained model showing significant progress in gameplay compared to the untrained baseline, especially when sampling actions stochastically rather than deterministically.
Quantitatively, the reinforcement learning phase improved the model’s game progress by over 50% and reward signals by around 60%, demonstrating that even a small VLM combined with a CNN critic and PPO can learn to play a complex, long-horizon game better than random or scripted baselines. However, the deterministic “greedy” policy remained weak, indicating room for further improvement in producing stable, reliable gameplay decisions. The video also showcases gameplay footage comparing the untrained and trained agents, highlighting the model’s ability to navigate further into the level after training.
In conclusion, the video provides a practical demonstration of the Odysseus paper’s novel training recipe for long-horizon vision-language reinforcement learning, showing that even small models can benefit from perception-focused supervised fine-tuning followed by RL with a cheap CNN critic. While the experiment faced challenges and did not fully replicate the original paper’s scale or performance, it successfully illustrated the core concepts and potential of this approach for training embodied AI agents to perform complex sequential decision-making tasks in video games. The creator encourages viewers to engage with the content and follow for more AI research experiments.