Sai Krishna Rallabandi’s talk explores the development of agentic systems from single-user to group-based settings, highlighting challenges in privacy, security, memory management, and social dynamics when agents operate within groups like families or teams. He emphasizes the need for robust security measures, intelligent memory handling, and responsible information routing to ensure effective, context-aware, and privacy-respecting group agents embedded in always-on devices.
Sai Krishna Rallabandi’s talk, “Wearing the Agent: From Group Chats to Glasses,” explores the evolution of agentic systems from serving individual users to functioning effectively within group settings. He begins by defining an agent as a combination of a system model (the brain), orchestration, and tools that enable it to perform tasks. While current agents typically cater to single users, the future points toward agents serving groups, such as families or work teams, often embedded in always-on devices like smart glasses. This shift introduces unique challenges, as group agents must handle privacy, evolving conversations, and proactive interactions differently than single-user agents.
Rallabandi illustrates these challenges through examples from Judith, an agent deployed in group settings over eight months. Judith manages tasks like syncing calendars for event planning, maintaining privacy by selectively responding via direct messages, and proactively assisting users based on contextual cues such as sleep deprivation. The agent also curates ongoing group conversations by filtering relevant content and supports educational interactions with a child user. These examples highlight the complexity and dynamic nature of group-based agent deployment, emphasizing the need for specialized design considerations.
A major focus of the talk is on security, particularly the concept of a “guard” layer that controls what information the agent accesses and acts upon. Unlike large language models, agentic systems interact with a broader surface area, including web pages, emails, and group messages, making them vulnerable to sophisticated attacks like prompt injections. Rallabandi discusses the limitations of static security measures and advocates for a boundary-level defense that uses deterministic and learned models to detect and filter malicious inputs at the action stage, balancing security with usability and latency.
Memory management in group agents is another critical topic. Unlike personal agents, group agents must handle evolving, multi-user conversations, requiring intelligent extraction, storage, and retrieval of atomic facts rather than naive full conversation logs. Rallabandi suggests continuous relevance scoring to decide what information to retain or forget, optimizing token usage and latency. He also highlights the importance of designing memory systems that accommodate cloud or local serving while being aware of technical constraints like KV caching, ensuring efficient and contextually appropriate memory handling.
Finally, Rallabandi addresses privacy and social dynamics in shared agents, framing them as a new social contract where the same data can be public or private depending on context. He proposes using a shared memory layer with personalized machine learning adapters for each user to enforce permissions naturally. Additionally, he notes the tendency of group agents to be overly proactive and suggests training models to regulate when agents should speak or remain silent. He concludes by emphasizing three key considerations for group agents: robust security gating, intelligent memory design, and responsible information routing to maintain privacy and effectiveness in multi-user environments.