Do you need an AI Gateway?

LLM applications increasingly rely on MCP servers for tools, data, and automation, but teams often hit real-world snags: trusted server discovery, inconsistent authentication, lack of built-in security features, unreliability of tool calls, tool pollution, lack of proper observability, etc. All these issues have a significant impact on the developer experience and make it hard to properly operationalize MCP for real-world production environments.

In this talk, we’ll frame these pain points from a developer and platform persona angles, and discuss how an MCP gateway can help address them and provide a solid foundation to leverage MCP at scale in enterprise environments.

Transcript

1. Do you need an AI Gateway? 2 October | London IGNASI BARRERA Founding Engineer @ Tetrate
2. ➜ Founding Engineer @ Tetrate ➜ ASF proud Member ➜ Envoy AI Gateway contributor MCP DEV SUMMIT | EUROPE
3. MCP DEV SUMMIT | EUROPE
4. MCP DEV SUMMIT | EUROPE
5. MCP DEV SUMMIT | EUROPE
6. MCP DEV SUMMIT | EUROPE
7. MCP DEV SUMMIT | EUROPE
8. MCP DEV SUMMIT | EUROPE
9. MCP DEV SUMMIT | EUROPE
10. MCP DEV SUMMIT | EUROPE Some problems are well-known • Trusted server discovery • Authentication / Authorization • Server resiliency: high availability, load balancing, failover • Observability Some are new • Context bloating • Tool confusion • Tool poisoning, security guardrails • Decision fatigue
11. MCP DEV SUMMIT | EUROPE Some problems are well-known • Trusted server discovery => MCP Registry (https://github.com/modelcontextprotocol/registry) • AuthN/ AuthZ => SEP-1036 (https://github.com/modelcontextprotocol/modelcontextprotocol/issues/1036) • Server resiliency: high availability, load balancing, failover • Observability => OpenInference (https://arize-ai.github.io/openinference/spec/) Some are new => Require deep understanding of the protocol • Context bloating • Tool confusion • Tool poisoning, security guardrails • Decision fatigue
12. MCP DEV SUMMIT | EUROPE
13. MCP DEV SUMMIT | EUROPE
14. MCP DEV SUMMIT | EUROPE Production, not experimentation
15. MCP DEV SUMMIT | EUROPE Envoy AI Gateway • Reliable Foundation with Envoy Proxy Built upon Envoy's battle-tested proxy technology to provide a stable, performant, and production-ready foundation that enterprises can rely on • Resilient Connectivity Across LLM Providers and Self-Hosted Models • Comprehensive Observability for Performance and Cost Management • Enterprise-Grade Security Features • Extensible Architecture
16. MCP DEV SUMMIT | EUROPE Envoy now gets MCP support!
17. MCP DEV SUMMIT | EUROPE Let's see it in action!
18. Title Here MCP DEV SUMMIT | EUROPE Thank you! http://router.tetrate.ai https://aigateway.envoyproxy.io