Nishant Gupta from Meta highlights the challenge of creating deterministic infrastructure to reliably support inherently non-deterministic, autonomous AI agents, emphasizing the need for layered control systems, observability, and safety mechanisms to ensure consistent and safe operation at scale. He advocates treating AI workloads as complex distributed systems, focusing on robust infrastructure and operational reliability rather than solely improving AI models.
Nishant Gupta, a software engineering tech lead at Meta, discusses the critical challenge of building deterministic infrastructure to support inherently non-deterministic AI agents. While much of the AI conversation has focused on improving models, the shift from chatbots to autonomous agents introduces new reliability challenges. These agents are stateful, long-running, and dynamic, which conflicts with traditional cloud infrastructure assumptions designed for short-lived, deterministic services. Gupta emphasizes that the key issue is not intelligence but reliability—ensuring AI agents can perform tasks consistently, recover from failures, and operate safely at scale.
Gupta highlights the “great mismatch” between autonomous AI systems and existing infrastructure. Unlike typical deterministic workflows, AI agents may execute different workflows for the same inputs, leading to unpredictable behavior. This unpredictability demands a new approach to system design, where the focus shifts from showcasing AI capabilities to ensuring dependable, repeatable outcomes. He points out that many AI failures stem not from model hallucinations but from infrastructure issues such as recursive loops, workflow deadlocks, and resource exhaustion caused by uncontrolled retries.
To address these challenges, Gupta advocates for a clear separation between AI models and production systems. Models should only generate proposals, while infrastructure components like policy engines and execution gateways validate and enforce decisions. This layered control approach prevents models from directly controlling critical systems, enabling reliability despite the probabilistic nature of AI. He introduces the concept of an “agentic control plane,” an emerging infrastructure layer responsible for scheduling, memory coordination, policy enforcement, and monitoring—essentially an operating system for autonomous AI agents.
Observability and memory management are also crucial in this new paradigm. Traditional logs are insufficient; instead, multi-dimensional tracing is needed to understand the reasoning and decision-making processes of AI agents. Memory consistency issues arise when multiple agents share state, leading to failures that appear as reasoning errors but are actually distributed system problems. Gupta stresses the importance of layered safety mechanisms, including prompt controls, policy validations, human oversight, and audit systems, applying the principle of defense in depth to autonomous AI.
Finally, Gupta notes that AI workloads resemble complex cluster scheduling problems with bursty demand and variable resource needs. Efficient GPU utilization, workload placement, and elastic capacity management become vital. Many reliability patterns from distributed systems—such as circuit breakers, rate limits, and retries—can be adapted for AI infrastructure. He concludes that the future competitive advantage in AI will come not from better models or prompts but from building robust, reliable infrastructure that treats AI agents as distributed systems, ensuring deterministic control over inherently stochastic models.