Harish Gaggar, a senior staff software engineer at Intuit Credit Karma, presents a comprehensive framework for transitioning AI agents from experimental stages to production-ready enterprise environments. He begins by outlining the common challenges faced when deploying AI agents, such as lack of proper authentication gateways, insufficient tool execution controls, inadequate tracking and evaluation systems, and poor memory management. To address these issues, Harish proposes a structured approach involving an agent framework, MCP (Model Control Plane) gateway, customized MCP servers, and robust monitoring systems, alongside additional frameworks for privacy protection and context optimization.
The development process starts with registering the AI agent and defining its use case through an internal developer portal. This portal automates the creation of repositories containing essential frameworks like memory management, tracking and evaluation, context optimization, and PII (Personally Identifiable Information) blocking. Developers build and test their agents locally using Docker containers, ensuring consistent implementation across the organization. Once validated, agents undergo a CI/CD pipeline that includes security scanning and deployment to Kubernetes clusters, with all necessary resources such as workload identities and secret management configured automatically.
Harish emphasizes the importance of building custom MCP servers, especially in air-gapped environments, to maintain control over the tools used for query execution. These servers work in tandem with the MCP gateway, which acts as a centralized authentication and routing hub. The gateway authenticates requests, issues JWT tokens, and securely routes calls to the appropriate MCP servers, ensuring that no direct access to endpoints occurs without proper authorization. This architecture enhances security and operational control over AI agent interactions with backend services.
Monitoring and evaluation are critical components of the framework. The MCP gateway logs operational data to systems like ELK or CloudWatch, while traces and metrics are collected via OpenTelemetry and stored in tools such as Tempo and Prometheus. This continuous monitoring enables real-time visibility into agent performance, latency, errors, and authentication issues. Additionally, the agent framework, primarily built using LangGraph, supports complex workflows with state management, branching, and human-in-the-loop approval processes for sensitive operations, ensuring both flexibility and governance.
Finally, Harish discusses supplementary frameworks essential for enterprise-grade AI agents. Memory management preserves the state and context of interactions over time, enabling seamless recovery and continuity. Tracking and evaluation frameworks capture detailed logs of queries, responses, token usage, and quality metrics to assess agent effectiveness and ROI. A PII blocking mechanism prevents sensitive data from leaking into LLMs or storage systems. The context optimizer significantly reduces token consumption by pruning, re-ranking, deduplicating, and compressing input data, achieving up to 90% token savings in demonstrations. Harish concludes by stressing the importance of building a comprehensive, governed AI agent ecosystem rather than rushing to deploy isolated agents, and he invites viewers to access his recently published MCP book for further insights.
