Building an Enterprise AI Gateway with Azure API Management and Azure AI Foundry

As organizations scale Generative AI applications, one of the biggest challenges is no longer just model access - it’s reliability, scalability, governance, and traffic management.

A single Azure OpenAI / Azure AI Foundry deployment can quickly hit:

• TPM (Tokens Per Minute) limits
• RPM (Requests Per Minute) limits
• regional throttling
• availability constraints

To solve this, I recently explored building an AI Gateway architecture using Azure API Management (APIM) in front of multiple Azure AI Foundry GPT-4.1 deployments.

The setup turned out to be a very interesting enterprise-grade architecture pattern for scalable GenAI systems.

Architecture Overview

The architecture looked like this:

Client Applications
        ↓
Azure API Management (APIM)
        ↓
Backend Pool (Load Balancer)
     ├── Azure AI Foundry Instance 1
     └── Azure AI Foundry Instance 2

The idea was simple:

• expose a single AI endpoint through APIM
• distribute requests across multiple GPT-4.1 deployments
• centralize authentication and governance
• improve throughput and resiliency

Why APIM for Azure OpenAI?

Azure API Management acts as an AI Gateway layer.

Instead of applications directly calling Azure OpenAI deployments, all traffic flows through APIM.

This enables:

• centralized authentication
• traffic routing
• rate limiting
• retries
• observability
• backend abstraction
• multi-region failover
• model governance

In enterprise environments, this becomes extremely valuable when multiple teams and applications consume AI models at scale.

Infrastructure Setup

I created:

• Azure AI Foundry Instance 1 — GPT-4.1 deployment
• Azure AI Foundry Instance 2 — GPT-4.1 deployment
• Azure API Management (APIM) — AI Gateway

Both Foundry instances had:

• Model: gpt-4.1
• Deployment Name: gpt-4.1

Using identical deployment names simplifies backend routing significantly.

Backend Pool Configuration

Inside APIM, I configured:

• Two backend services
• A backend pool called gpt41-pool
• Weighted load balancing across deployments

Configuration used:

• aoai-dev-01 → Priority: 1 → Weight: 50
• aoai-dev-02 → Priority: 1 → Weight: 50

This allowed APIM to distribute traffic approximately evenly across both GPT-4.1 deployments.

Instead of using API keys, APIM authenticated to Azure OpenAI using Managed Identity.

This is one of the cleanest patterns for enterprise deployments.

The APIM Managed Identity was assigned:

Cognitive Services OpenAI User

role on both Azure AI Foundry resources.

APIM Policy Configuration

The core of the architecture was implemented using APIM policies.

<policies>
    <inbound>
        <base />
          <set-backend-service backend-id="gpt41-pool" />
          <authentication-managed-identity
            resource="https://cognitiveservices.azure.com"
            output-token-variable-name="mi-token"
            ignore-error="false" />
        <set-header name="Authorization" exists-action="override">
            <value>@("Bearer " + (string)context.Variables["mi-token"])</value>
        </set-header>
        <rewrite-uri template="/openai/deployments/gpt-4.1/chat/completions?api-version=2025-01-01-preview" />
    </inbound>
    <backend>
        <forward-request />
    </backend>
    <outbound>
        <base />
    </outbound>
</policies>

This policy handled:

• backend pool selection
• Managed Identity authentication
• deployment-specific routing

One Important Learning About Deployment Routing

Even though the Foundry resources also contained other models such as:

• gpt-4o
• gpt-4.1-mini

APIM still routed specifically to:

gpt-4.1

because of:

<rewrite-uri template="/openai/deployments/gpt-4.1/chat/completions?api-version=2025-01-01-preview" />

This was an important insight.

The backend URL only points to the Foundry service endpoint.

The actual model/deployment selection happens through the rewritten deployment path.

Testing Load Balancing

To validate the setup properly, I used Python async load testing instead of relying only on the APIM Test Console.

Using aiohttp + asyncio, I sent concurrent requests through APIM and observed:

• request distribution
• throttling behavior
• backend utilization
• throughput scaling

Example test setup:

TOTAL_REQUESTS = 10
CONCURRENT_REQUESTS = 2
max_tokens = 10

Understanding TPM and 429 Errors

One surprising observation was how Azure OpenAI handles TPM throttling.

At first glance, it seems intuitive that:

20 requests × 10 max_tokens = 200 tokens

should easily fit within a 1K TPM deployment.

But Azure OpenAI throttling is more nuanced.

Token estimation includes:

• input prompt tokens
• output tokens
• formatting overhead
• internal token estimation
• rolling token windows
• RPM limits
• burst concurrency behavior

This means even relatively small concurrent bursts can still trigger:

429 Too Many Requests

especially with lower TPM allocations.

Why This Architecture Matters

This pattern effectively creates:

• a scalable AI gateway
• centralized governance layer
• abstraction between applications and models

Benefits include:

• horizontal TPM scaling
• multi-region resiliency
• centralized security
• retry/failover support
• easier observability
• simplified client integration

Applications only talk to APIM.

APIM handles:

• backend selection
• authentication
• routing
• retries
• scaling

Interesting New APIM AI Features

While exploring this architecture, I also came across new APIM AI capabilities:

• Language Model API
• Microsoft Foundry API
• A2A Agent support

These features suggest Microsoft is evolving APIM into a full-fledged Enterprise AI Gateway platform.

The newer Foundry integration especially looks promising because it can automate much of the manual setup:

• backend configuration
• model routing
• AI policies
• authentication

This feels like the future direction for scalable enterprise GenAI systems on Azure.

Final Thoughts

Building scalable AI systems is quickly becoming less about “calling an LLM” and more about:

• governance
• resiliency
• observability
• throughput management
• platform engineering

Azure API Management combined with Azure AI Foundry provides a powerful pattern for solving these challenges in enterprise environments.

This was a very interesting architecture to experiment with, especially around:

• APIM backend pools
• deployment routing
• TPM behavior
• AI gateway design
• concurrent traffic management

And it’s clear that AI Gateway architectures are going to become increasingly important as enterprise GenAI adoption grows.