Starting in 2022, Google advanced robotic planning by integrating large language models (LLMs) like Palm and Palm E with control algorithms such as RT1 and RT2, progressively unifying planning and control into a single multimodal system capable of handling complex tasks autonomously. By early 2024, this innovation led to the creation of the startup Physical Intelligence, which continues to develop cutting-edge robot brains that leverage LLMs for integrated perception, planning, and control.
In 2022, the release of ChatGPT coincided with a breakthrough at Google, where a team demonstrated that large language models (LLMs) could be used as robot planners. These LLMs were capable of breaking down complex tasks into pre-trained subtasks, marking a significant step in robotic planning. However, the effectiveness of these early systems was limited by the existing robot control algorithms, which acted as a bottleneck.
Towards the end of 2022, Google made a major advancement by introducing RT1, an improved control algorithm trained on over 130,000 human demonstrations and built on a larger transformer-based architecture. While the planning layer used the text-only Palm 540B model from early 2022, it lacked direct sensory input, interacting with the control layer solely through text commands. This setup highlighted the gap between planning and perception in robotic systems.
In March 2023, just before the release of GPT-4, Google unveiled Palm E, a multimodal variant of the Palm LLM that integrated images and other data sources directly into the planning process. By combining Palm E with the RT1 control layer, the team significantly enhanced robotic capabilities, enabling robots to autonomously recover from setbacks and perform more complex tasks with greater autonomy.
This progress led to a critical question: is it necessary to maintain separate models for control and planning, or could a single LLM handle both by processing raw sensor data and outputting control signals directly? In July 2023, Google answered this by developing RT2, which used the Palm E LLM to unify planning and control into one model, streamlining robotic decision-making and execution.
By early 2024, many key members of the RT2 team left Google to form the startup Physical Intelligence, which builds on and improves the RT2 approach. Physical Intelligence now produces some of the most advanced robot brains globally, pushing the frontier of robotics by leveraging large language models as integrated, multimodal control systems.