Patrick Löber from Google DeepMind presents an overview of building native multimodal agents using the Gemini API and AI Studio. He introduces the concept of “any-to-any,” highlighting Gemini’s ability to understand and generate across multiple modalities including text, code, images, audio, video, URLs, and Google Search. While Gemini currently uses different specialized models for generation tasks—such as Nano Banana for images and speech generation—the vision is to unify these capabilities into a single multimodal model. The session focuses on four key areas: multimodal understanding, native image generation, native speech generation, and real-time interactions, culminating in building a notebook LM clone as a practical example.
The first phase discussed is multimodal understanding, where Gemini can analyze diverse inputs like PDFs, videos, and audio files to generate summaries and insights. Using the Google AI SDK, developers can easily upload various file types or provide inline data, and leverage Gemini’s powerful capabilities to transcribe audio, analyze video timestamps, and combine multiple sources for cross-modal understanding. The API supports large token limits, enabling processing of extensive content such as hours of audio or video, and includes features like context caching to reduce costs during repeated queries.
Next, Patrick explains the multimodal generation phase, which uses an agentic loop where Gemini acts as the reasoning model that calls specialized native generation models via function calls. These models include Gemini 3.5 flash image preview (Nano Banana 2) for creating infographics and Gemini 2.5-based text-to-speech models capable of generating multilingual and accented speech. The agent decides autonomously which assets to generate, enhancing the study guide with visuals and audio summaries. The native generation models benefit from Gemini’s deep world understanding, enabling sophisticated outputs such as annotated math homework corrections or context-aware images.
Patrick also highlights the importance of native generation models being based on Gemini, which allows them to leverage the extensive training and understanding embedded in the main model. This results in more accurate and contextually relevant outputs across modalities. He demonstrates the versatility of the speech models by showcasing different accents and multilingual capabilities. The session then touches on the live API, featuring Gemini 3.1 flash live, a new audio-to-audio model that enables natural, real-time conversational interactions without the need for cascaded pipelines, enhancing user experience with fluid and responsive dialogue.
Finally, Patrick mentions additional advancements such as a multimodal embedding model that unifies different modalities into a single vector space for applications like multimodal search, and the availability of local multimodal models like Gemma 4. He encourages developers to explore these tools and build their own multimodal agents, emphasizing the broad applicability of the patterns demonstrated. The session concludes with an invitation to try out the live API and a call to have fun building innovative multimodal AI applications.
