In this talk, Philipp Schmid from Google DeepMind discusses how to build and code an autonomous AI co-worker, particularly for finance professionals and engineers. He outlines the evolution of AI from simple chatbots to autonomous digital employees capable of performing complex tasks independently. Schmid emphasizes that just like human co-workers, AI agents need essential components: a brain (the AI model), a computer to store and iterate work, playbooks for workflows, tools to perform tasks, credentials for access, house rules for governance, and schedules for operation. These seven elements form the foundation for creating effective AI co-workers using Google’s Gemini model and managed agents.
Schmid introduces Gemini, a powerful AI model recently made available with significant improvements, including the ability to handle up to 1 million tokens for complex reasoning and math tasks. He highlights its extensive internal use at Google, where it has accelerated projects like Chromium browser development and codebase improvements. The Gemini API offers a simple way to interact with the model, making it accessible for developers to experiment with and integrate into their workflows. Schmid encourages users to try Gemini, noting its affordability and strong performance.
The core of the talk focuses on managed agents, a framework that allows developers to create remote AI agents with their own sandboxed computing environment. This setup enables the agent to execute code, manage files, and use tools autonomously without constant back-and-forth communication. Developers can define agents with unique IDs, system instructions, and skills, which are modular pieces of knowledge or functionality. These skills can be sourced from GitHub repositories or cloud storage, allowing the agent to bootstrap its environment and improve over time by creating new skills or storing memories.
Security and control are critical aspects addressed through managed agents. Schmid explains the secure handling of credentials using bearer tokens, OAuth, and environment variables, all managed via an external network proxy that prevents credential leakage or misuse. This proxy swaps dummy tokens with real credentials only for authorized domains, protecting against prompt injection attacks. Additionally, hooks provide deterministic control over agent actions, enabling human-in-the-loop approvals or restrictions on certain commands. These features ensure that AI co-workers operate safely and within defined boundaries.
Finally, Schmid discusses proactive capabilities for AI agents, such as scheduled triggers and upcoming webhook integrations, allowing agents to perform tasks automatically or respond to external events. He demonstrates a portfolio auditing example where the agent autonomously updates a Google Sheet and posts updates to Slack channels. The entire infrastructure is designed to be easy to integrate and manage, providing a comprehensive solution for deploying AI co-workers in real-world environments. Schmid concludes by encouraging the audience to start building their own AI co-workers using Gemini and managed agents.
Useful Links
