Antje Barth from the Amazon AGI Lab presents “perception agents,” AI systems that continuously observe and verify their actions across multiple applications to improve reliability and collaboration in complex workflows, addressing the challenges faced by traditional agents in coordinating tasks and ensuring trust. She highlights the lab’s open-source tools for annotation and verification, demonstrates integration with wearable devices for real-time transcription and workflow updates, and calls for community involvement to advance smarter AI development.
Antje Barth from the Amazon AGI Lab discusses the evolution and current challenges of AI agents in performing complex workflows across multiple applications. While agents have become proficient at basic tasks like clicking buttons and navigating interfaces, they struggle with end-to-end processes that involve coordinating actions across different systems. The core difficulty lies in the “seams” between applications where the real work happens, and where agents lack the ability to reliably verify and adapt their actions, limiting trust and practical deployment.
Barth highlights that although AI agents have gained impressive capabilities, reliability remains a critical hurdle. She contrasts this with the progress made in coding agents, which have become trusted because code is verifiable through testing and execution. In contrast, most knowledge work is messy and subjective, lacking clear verification methods. This makes it difficult for agents to confirm whether their actions have succeeded, especially when dealing with tasks that span multiple tools and require nuanced understanding.
To address this, Barth introduces the concept of “perception agents,” which combine perception, planning, and action in a continuous loop similar to how humans and robots operate. Unlike current agents that act and then wait for feedback, perception agents continuously observe the rendered interface and environment, enabling them to verify their work in real time and recover from errors. This shared perception also allows for more natural collaboration between humans and agents, as both can see and interact with the same context simultaneously.
Amazon AGI Lab has open-sourced two key components of their perception agent framework: an annotation tool and a verification system. The annotation tool allows users to directly select and specify changes on a screen, providing precise instructions to the agent without lengthy textual descriptions. The verification system automatically checks the agent’s output against design specifications and user flow rules, generating reports that highlight successes and failures. This approach reduces manual review and increases trust in the agent’s work.
Finally, Barth shares a demonstration of perception beyond visual input, using wearable devices to capture and transcribe a live meeting. The transcription is integrated into the workflow, allowing spoken ideas to be directly applied as changes on a website, followed by automatic verification. She invites the community to engage with and contribute to this open-source project, emphasizing that building smarter AI requires collective effort. Barth also recommends further resources on human-agent interaction and encourages attendees to explore related talks and the Amazon AGI Lab’s presence at the event.
