In this talk, Kevin Orellana from Amazon Bedrock Agent Core shares his experience of stress-testing a cloud sandbox hosting environment by running 1,000 automated coding agent tasks overnight to identify what breaks when large language models (LLMs) run code in such environments. The motivation behind this experiment was to better understand how customers use their platform, which provides agents, identity, memory, runtime environments, sandboxes, and gateways for building and hosting agents in the cloud. By creating an MCP (Model-Controller-Protocol) server interface, they enabled coding agents to interact natively with their sandbox environment, allowing them to run a wide variety of tests across multiple domains.
The testing involved a diverse set of workflows, including file system operations, software development life cycles like writing Python and Java packages, machine learning training programs, and browser automation tasks such as opening tabs, scraping content, and automating logins. These tests revealed different failure modes depending on the sandboxed application, highlighting how sandbox constraints like memory limits, file system permissions, and process capabilities significantly impact agent performance. For example, cloud browser sandboxes frequently failed due to memory exhaustion, whereas code sandboxes faced issues related to file system encoding and persistence.
Kevin emphasizes the importance of distinguishing between failures at the interface layer (MCP protocol) and those at the sandbox layer itself. Unlike traditional long-running API services, sandbox-hosted agents operate in constrained environments with varying permissions and resource limits, which can cause unexpected behavior such as missing files or corrupted data. These constraints affect the agents’ ability to execute complex workflows, including downloading packages, spawning processes, or accessing internal APIs, underscoring the need for careful sandbox configuration tailored to the agent’s workload.
The talk also touches on the broader implications for developers and organizations adopting agent-based architectures. As agents become more prevalent and are deployed in remote sandboxes rather than local environments, discrepancies between local and remote toolsets can cause failures. To address this, Amazon Bedrock Agent Core has introduced features like SSH access to sandboxes, allowing developers or agents themselves to inspect and debug sandbox environments. This capability helps identify and resolve issues that hinder agent workflows, facilitating smoother scaling of coding agents in cloud environments.
Finally, Kevin highlights academic research that supports the importance of environment controls for agent performance. Studies show that agents with well-configured environments can achieve near-perfect benchmark results, while those with broken or limited tools perform poorly. His key takeaway is a challenge to the audience: when hosting agents in sandboxes, it is crucial not only to test locally but also to perform extensive fuzzy testing with diverse domain tasks remotely. This approach helps uncover subtle sandbox limitations and ensures agents can reliably complete their intended goals in real-world cloud environments.
