Azure AI Gateway

What it does

Azure AI Gateway provides a centralized control plane for enterprises to manage their AI infrastructure. It connects AI applications to various LLMs and MCP servers while enforcing security, reliability, and cost controls via Azure API Management and Microsoft Foundry.

Tools

The gateway acts as an orchestrator for multiple tools. Key capabilities include:

• MCP Protocol Integration: Plug-and-play tool connectivity with OAuth credential management.
• Model Routing: Dynamic routing of requests to different backends based on version or availability.
• Semantic Caching: Vector-based response caching to reduce costs and latency.
• Token Rate Limiting: Precise control over token consumption across different teams and apps.

Installation

Deployment is typically handled via Bicep templates provided in the labs. For a basic MCP integration, add the gateway endpoint to your config:

{
  "mcpServers": {
    "azure-ai-gateway": {
      "command": "npx",
      "args": ["@azure/ai-gateway-mcp-server"]
    }
  }
}

Supported hosts

Confirmed hosts include Claude Desktop, VS Code Copilot, and Gemini MCP Agents.