Solo.io as Gateway for Azure Open AI — 2
This is a follow up from our previous article. We installed solo.io previously, in this article we will configure solo.io to act as a true…
Solo.io as Gateway for Azure Open AI — 2
Photo by Heather Wilde on Unsplash
This is a follow up from our previous article. We installed solo.io previously, in this article we will configure solo.io to act as a true gateway for all LLM operations.
Solo.io as a true LLM gateway — the python service knows nothing about Azure OpenAI credentials, endpoints, or models. It just talks to Gloo Gateway, and Gloo handles everything else.
Target architecture:
Client
└── FastAPI microservice (no Azure creds, no OpenAI SDK)
└── HTTP POST to Gloo Gateway (172.18.0.2:32475)
└── Solo.io Gloo Gateway
├── Stores Azure OpenAI credentials (as K8s secrets)
├── Rate limiting / token tracking
├── Guardrails (prompt injection, content policy)
├── Observability (token consumption per service)
└── Azure OpenAI (cloud)
Here’s what we can do with this implementation

Same gateway pattern, four different upstreams. Each microservice picks its LLM by calling a different path prefix — /llm/azure, /llm/bedrock, /llm/local, /llm/vertex. The VirtualService in Gloo matches on that prefix and routes to the appropriate upstream, applying a completely independent transformation for each one (different host rewrite, different auth header format, different path structure).
Per-provider secrets — each LLM gets its own K8s secret in gloo-system. AWS Bedrock uses SigV4 signing headers, Google Vertex uses OAuth bearer tokens, local LLM needs no auth at all. The microservice never knows any of this — it just POSTs JSON to the gateway.
What you already have built (Azure OpenAI) maps directly to the leftmost backend. The other three follow the exact same pattern we worked through — Upstream → VirtualService route → stagedTransformations → hostRewrite.
The local LLM path is the simplest to add next — no TLS, no credentials, just an HTTP upstream pointing to localhost:11434 (Ollama) or whatever port your local model serves on, connected via the same Docker network trick we used for FastAPI.
This is exactly the right enterprise pattern. Every microservice team just gets a gateway URL and a route — no credential management, no SDK, unified policy enforcement in one place.
Gloo Gateway 1.21 supports this via its AI Gateway feature, which has native Azure OpenAI upstream support built in.
Before we get started here are some common Kubernets commands we will be using.
Common Kubernetes commands — Reference
Section 1 — Cluster & namespace commands — get nodes, get pods -A, create namespace and the variants we used during setup and cleanup.
Section 2 — Resource inspection — get svc, get upstream, get virtualservice, get secret, get crd, api-resources, and get settings — the commands we used to understand what was happening inside Gloo.
Section 3 — Gloo resource apply — kubectl apply -f - for Upstreams and VirtualServices, plus kubectl delete virtualservice for the domain conflict fix.
Section 4 — Deployment & rollout — rollout status and rollout restart used to wait for pods and force secret reloads.
Section 5 — Secret management — create secret generic, the --dry-run | apply update pattern, and the base64 -d decode trick to read a secret value for baking into the transformation template.
Section 6 — Debugging & logs — kubectl logs for the Envoy and Gloo control plane, and kubectl describe pod for the AI Gateway feature check.
Section 7 — Exec & in-container — the config_dump Envoy admin API call that showed us the live route config, and the env | grep AZURE check for the mounted secret.
Section 8 — Patching — strategic merge patch to add the env var, and the JSON RFC 6902 patch to surgically remove it when that approach failed.
For full reference you can check this documentation
Let’s start by making sure you have the right license to start with (you can do all of this without license key too)
kubectl get upstream -n gloo-system
kubectl describe pod gloo-54bdbcc5d7-clr6d -n gloo-system | grep -i "ai\|llm\|model"
NAME AGE
default-petstore-8080 3h44m
extauth 4h9m
rate-limit 4h9m
Containers:
Container ID: containerd://5c11c2254b81c125102f09eb55bba81f1cc086609d5b763a28f4cbd2a4ce95a1
Readiness: tcp-socket :9977 delay=3s timeout=1s period=10s #success=1 #failure=3
PodReadyToStartContainers True
ContainersReady True
Type: Projected (a volume that contains injected data from multiple sources)
No AI-specific flags in the pod description, but that’s okay — Gloo EE 1.21 has AI Gateway built into the Helm chart. Let’s check if the AI Gateway CRDs are installed:
# Check for AI Gateway custom resource definitions
kubectl get crd | grep -i "ai\|llm\|ratelimit\|routeoption"
# Also check what APIs are available
kubectl api-resources | grep -i "solo\|gloo" | head -30
failoverschemes.fed.solo.io 2026-05-25T16:15:56Z
federatedratelimitconfigs.fed.ratelimit.solo.io 2026-05-25T16:15:56Z
ratelimitconfigs.ratelimit.solo.io 2026-05-25T16:15:55Z
routeoptions.gateway.solo.io 2026-05-25T16:15:55Z
authconfigs ac,gac enterprise.gloo.solo.io/v1 true AuthConfig
federatedauthconfigs fed.enterprise.gloo.solo.io/v1 true FederatedAuthConfig
federatedgateways fed.gateway.solo.io/v1 true FederatedGateway
federatedmatchablehttpgateways fed.gateway.solo.io/v1 true FederatedMatchableHttpGateway
federatedmatchabletcpgateways fed.gateway.solo.io/v1 true FederatedMatchableTcpGateway
federatedroutetables fed.gateway.solo.io/v1 true FederatedRouteTable
federatedvirtualservices fed.gateway.solo.io/v1 true FederatedVirtualService
federatedsettings fed.gloo.solo.io/v1 true FederatedSettings
federatedupstreamgroups fed.gloo.solo.io/v1 true FederatedUpstreamGroup
federatedupstreams fed.gloo.solo.io/v1 true FederatedUpstream
federatedratelimitconfigs fed.ratelimit.solo.io/v1alpha1 true FederatedRateLimitConfig
failoverschemes fed.solo.io/v1 true FailoverScheme
glooinstances fed.solo.io/v1 true GlooInstance
directresponses dr gateway.gloo.solo.io/v1alpha1 true DirectResponse
gatewayparameters gwp gateway.gloo.solo.io/v1alpha1 true GatewayParameters
gateways gw,ggw gateway.solo.io/v1 true Gateway
httpgateways hgw,ghgw gateway.solo.io/v1 true MatchableHttpGateway
httplisteneroptions hlisopts,ghlisopts gateway.solo.io/v1 true HttpListenerOption
listeneroptions lisopts,glisopts gateway.solo.io/v1 true ListenerOption
routeoptions rto,rtopts,grto gateway.solo.io/v1 true RouteOption
routetables rt,grt gateway.solo.io/v1 true RouteTable
tcpgateways tgw,gtgw gateway.solo.io/v1 true MatchableTcpGateway
virtualhostoptions vho,vhopts,gvho gateway.solo.io/v1 true VirtualHostOption
virtualservices vs,gvs gateway.solo.io/v1 true VirtualService
proxies px,gpx gloo.solo.io/v1 true Proxy
settings st,gst gloo.solo.io/v1 true Settings
upstreamgroups ug,gug gloo.solo.io/v1 true UpstreamGroup
upstreams us,gus gloo.solo.io/v1 true Upstream
kubernetesclusters multicluster.solo.io/v1alpha1 true KubernetesCluster
multiclusterrolebindings multicluster.solo.io/v1alpha1 true MultiClusterRoleBinding
We have 10 steps to setup our gateway
Photo by Moriah Wolfe on Unsplash
Step 1 — Create a Kubernetes secret with your Azure OpenAI credentials
This is the only place credentials will live. No microservice ever sees them.
kubectl create secret generic azure-openai-secret \
--from-literal=api-key="<YOUR_AZURE_OPENAI_API_KEY>" \
--from-literal=endpoint="<YOUR_AZURE_OPENAI_ENDPOINT>" \
--from-literal=deployment="<YOUR_DEPLOYMENT_NAME>" \
-n gloo-system
# Verify it was created
kubectl get secret azure-openai-secret -n gloo-system
NAME TYPE DATA AGE
azure-openai-secret Opaque 4 18s
Replace the placeholders with your actual values from your .env file. Secret is safely stored in Kubernetes.
Step 2 — Create the Azure OpenAI Upstream
Now let’s create the Upstream that points to Azure OpenAI:
kubectl apply -f - <<'EOF'
apiVersion: gloo.solo.io/v1
kind: Upstream
metadata:
name: azure-openai
namespace: gloo-system
spec:
static:
hosts:
- addr: oai-eus2-agenticai-dev-01.openai.azure.com
port: 443
useHttp2: false
sslConfig:
sni: oai-eus2-agenticai-dev-01.openai.azure.com
EOF
upstream.gloo.solo.io/azure-openai created
~$ kubectl get upstream azure-openai -n gloo-system -o jsonpath='{.status}' | python3 -m json.tool
{
"statuses": {
"gloo-system": {
"reportedBy": "gloo",
"state": "Accepted"
}
}
}
Upstream is Accepted.
Now let’s create a RouteOption that injects the Azure OpenAI API key header automatically — this is the key piece that means no microservice ever needs to know the credential:
Step 3 — Create the RouteOption to inject the API key
Actually, Gloo Edge doesn’t support secret interpolation in headers directly that way. Instead we’ll use a cleaner approach — inject the header value via a VirtualService transformation. Let’s first check if the transformation filter is available:
kubectl get settings -n gloo-system default -o jsonpath='{.spec}' | python3 -m json.tool | grep -i "transform\|filter"
"disableTransformationValidation": false,
"enableAutoWebsocketTransformationPassthrough": false,
Transformations are available. However, for injecting secrets as headers the cleanest and most secure approach in Gloo Edge is to use a VirtualService with a static header transformation. The API key gets pulled from the Kubernetes secret via an environment variable mounted into the gateway-proxy pod.
Let’s do this the right way — mount the secret into the gateway proxy, then reference it in a transformation:
Step 4— Patch the gateway-proxy to mount the Azure OpenAI secret
kubectl patch deployment gateway-proxy -n gloo-system --patch '
{
"spec": {
"template": {
"spec": {
"containers": [{
"name": "gateway-proxy",
"env": [{
"name": "AZURE_OPENAI_API_KEY",
"valueFrom": {
"secretKeyRef": {
"name": "azure-openai-secret",
"key": "api-key"
}
}
}]
}]
}
}
}
}'
Wait for the rollout:
kubectl rollout status deployment/gateway-proxy -n gloo-system
The secret is mounted into the gateway proxy. However I can see the old API key is still showing — please make sure you’ve rotated that key in Azure portal and updated the Kubernetes secret with the new one before we proceed, otherwise the route won’t work.
To update the secret with your new key:Then verify the env var is present in the pod:
kubectl exec -n gloo-system deployment/gateway-proxy -- env | grep AZURE
AZURE_OPENAI_API_KEY=7RGxxIssrZ11Jl4Nn8FJJy6gQafZsNKiWB2GiRCSyIdOw7qIVSORJQQJ99CEACYeBjFXJ3w3AAABACOG1erH
The secret is mounted into the gateway proxy. However I can see the old API key is still showing — please make sure you’ve rotated that key in Azure portal and updated the Kubernetes secret with the new one before we proceed, otherwise the route won’t work.
To update the secret with your new key:
kubectl exec -n gloo-system deployment/gateway-proxy -- env | grep AZURE
AZURE_OPENAI_API_KEY=7RGxxIssrZ11Jl4Nn8FJJy6gQafZsNKiWB2GiRCSyIdOw7qIVSORJQQJ99CEACYeBjFXJ3w3AAABACOG1erH
~$ kubectl create secret generic azure-openai-secret \
--from-literal=api-key="7RGxxIssrZ11Jl4Nn8FJJy6gQafZsNKiWB2GiRCSyIdOw7qIVSORJQQJ99CEACYeBjFXJ3w3AAABACOG1erH" \
--from-literal=endpoint="https://oai-eus2-agenticai-dev-01.openai.azure.com" \
--from-literal=deployment="gpt-4.1" \
--from-literal=api-version="2025-01-01-preview" \
-n gloo-system \
--save-config \
--dry-run=client -o yaml | kubectl apply -f -
Warning: resource secrets/azure-openai-secret is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
secret/azure-openai-secret configured
~$ hkubectl rollout restart deployment/gateway-proxy -n gloo-system
bash: hkubectl: command not found...
~$ kubectl rollout restart deployment/gateway-proxy -n gloo-system
deployment.apps/gateway-proxy restarted
~$ kubectl rollout status deployment/gateway-proxy -n gloo-system
deployment "gateway-proxy" successfully rolled out
~$ kubectl exec -n gloo-system deployment/gateway-proxy -- env | grep AZURE
AZURE_OPENAI_API_KEY=7RGxxIssrZ11Jl4Nn8FJJy6gQafZsNKiWB2GiRCSyIdOw7qIVSORJQQJ99CEACYeBjFXJ3w3AAABACOG1erH
Step 5— Create the VirtualService to route to Azure OpenAI
This routes any request to /openai through Gloo, injects the API key header from the environment variable, and rewrites the path to match what Azure OpenAI expects:
AZURE_KEY=$(kubectl get secret azure-openai-secret -n gloo-system \
-o jsonpath='{.data.api-key}' | base64 -d)
kubectl apply -f - <<EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
name: gloo-routes
namespace: gloo-system
spec:
virtualHost:
domains:
- '*'
routes:
- matchers:
- prefix: /openai
routeAction:
single:
upstream:
name: azure-openai
namespace: gloo-system
options:
timeout: 60s
hostRewrite: oai-eus2-agenticai-dev-01.openai.azure.com
stagedTransformations:
regular:
requestTransforms:
- requestTransformation:
transformationTemplate:
headers:
api-key:
text: "${AZURE_KEY}"
content-type:
text: "application/json"
":path":
text: "/openai/deployments/gpt-4.1/chat/completions?api-version=2025-01-01-preview"
passthrough: {}
- matchers:
- prefix: /api/pets
routeAction:
single:
upstream:
name: default-petstore-8080
namespace: gloo-system
EOF
Verify it’s accepted:
kubectl get virtualservice gloo-routes -n gloo-system -o jsonpath='{.status}' | python3 -m json.tool
{
"statuses": {
"gloo-system": {
"reportedBy": "gloo",
"state": "Accepted",
"subresourceStatuses": {
"*v1.Proxy.gateway-proxy_gloo-system": {
"reportedBy": "gloo",
"state": "Accepted"
}
}
}
}
}
VirtualService is Accepted by both Gloo and the proxy.
Now let’s test if Gloo can route a real request to Azure OpenAI:
kubectl get virtualservice gloo-routes -n gloo-system \
-o jsonpath='{.status}' | python3 -m json.tool
{
"statuses": {
"gloo-system": {
"reportedBy": "gloo",
"state": "Accepted",
"subresourceStatuses": {
"*v1.Proxy.gateway-proxy_gloo-system": {
"reportedBy": "gloo",
"state": "Accepted"
}
}
}
}
}
Step 6: Check transformation configuration
Looking at the Envoy config dump confirms exactly what happens to every request:
kubectl exec -n gloo-system deployment/gateway-proxy -- \
wget -q -O- http://localhost:19000/config_dump | \
python3 -m json.tool | \
grep -A 40 '"prefix": "/openai"'
{
"prefix": "/openai",
"route": {
"cluster": "azure-openai_gloo-system",
"timeout": "60s"
},
"typed_per_filter_config": {
"io.solo.transformation": {
"transformations": [{
"request_match": {
"request_transformation": {
"transformation_template": {
"headers": {
":path": {
"text": "/openai/deployments/gpt-4.1/chat/completions?api-version=2025-01-01-preview"
},
"api-key": {
"text": "<REDACTED>"
},
"content-type": {
"text": "application/json"
}
},
"passthrough": {}
}
}
}
}]
}
}
}
For every request hitting /openai, Gloo:
- Rewrites
:pathto the full Azure deployment URL includingapi-version - Injects
api-keyheader from the Kubernetes secret - Sets
content-typetoapplication/json - Rewrites
Hostheader to the Azure hostname (viahostRewrite) - Passes through the request body unchanged
The microservice sends a plain JSON body. Gloo handles all the Azure-specific plumbing.
Step 7: Test Gloo → Azure OpenAI Directly
curl -s -X POST \
"http://172.18.0.2:32475/openai" \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Say hello in one sentence."}
],
"max_tokens": 50
}' | python3 -m json.tool
{
"choices": [{
"content_filter_results": {
"hate": {"filtered": false, "severity": "safe"},
"self_harm": {"filtered": false, "severity": "safe"},
"sexual": {"filtered": false, "severity": "safe"},
"violence": {"filtered": false, "severity": "safe"}
},
"finish_reason": "stop",
"message": {
"content": "Hello! I hope you're having a wonderful day.",
"role": "assistant"
}
}],
"model": "gpt-4.1-2025-04-14",
"usage": {
"prompt_tokens": 23,
"completion_tokens": 11,
"total_tokens": 34
}
}
Gloo Gateway is routing to Azure OpenAI. No credentials in the request.
Step 8: Update the FastAPI Microservice
Now we strip the Azure SDK out of the microservice entirely. It becomes a thin HTTP client that knows nothing about Azure.
You can check sample python code to connect to Azure Open AI here.
As you can see the code is absolutely simple. It talks to LLM for answers. What’s interesting to note is code has no mention of Azure Open AI URL or API keys. All of those are managed by gateway.
Step 9: Docker Networking — The Final Hurdle
This was the issue I hit on as soon as I tested, have shared the fix for it as well.
Error: Timeout when FastAPI calls Gloo
curl -s -X POST http://localhost:8000/chat \
-H "Content-Type: application/json" \
-d '{"query": "Say hello in one sentence."}'
{"detail": "Error: timed out"}
The FastAPI container runs on the hworld_default Docker network (172.19.0.x). The vCluster (and Gloo Gateway) runs on the vcluster.solo-vcluster network (172.18.0.x). These two networks are isolated — the FastAPI container couldn't reach 172.18.0.2:32475.
Checking the container’s network:
docker inspect hworld-chat-api-1 | grep -A 10 '"Networks"'
"Networks": {
"hworld_default": {
"Gateway": "172.19.0.1",
"IPAddress": "172.19.0.2"
}
}
Fix: Connect the FastAPI container to the vCluster network
docker network connect vcluster.solo-vcluster hworld-chat-api-1
Verify connectivity from inside the container:
docker exec hworld-chat-api-1 python3 -c "
import urllib.request
req = urllib.request.urlopen('http://172.18.0.2:32475/api/pets')
print(req.read().decode())
"
[{"id":1,"name":"Dog","status":"available"},{"id":2,"name":"Cat","status":"pending"}]
Container can now reach the gateway.
Make it permanent in docker-compose.yml
A docker network connect is lost when the container restarts. Add the network permanently to docker-compose.yml:
services:
chat-api:
build: .
ports:
- "8000:8000"
env_file:
- .env
restart: unless-stopped
networks:
- default
- vcluster
networks:
default:
name: hworld_default
vcluster:
external: true
name: vcluster.solo-vcluster
Step 10: Test the Full Stack
curl -s -X POST http://localhost:8000/chat \
-H "Content-Type: application/json" \
-d '{"query": "Say hello in one sentence."}' | python3 -m json.tool
{
"query": "Say hello in one sentence.",
"response": "Hello! I hope you're having a wonderful day.",
"model": "gpt-4.1-2025-04-14",
"usage": {
"prompt_tokens": 23,
"completion_tokens": 11,
"total_tokens": 34
}
}
Photo by Kid Circus on Unsplash
🎉 The full chain is working:
curl → FastAPI (localhost:8000)
→ Gloo Gateway (172.18.0.2:32475)
→ Azure OpenAI (gpt-4.1)
→ response back through the chain
This is what we have implemented

Finally we have all pieces to work together.
Solo.io Gloo Gateway as an LLM Gateway — Architecture Summary
What we built is a centralized LLM gateway pattern where Solo.io Gloo Gateway Enterprise sits as the single intermediary between all microservices and every AI provider. Running inside a vCluster on Fedora Linux in this case in your enterprise world you may run it in your K8s or AKS or EKS or GKS. Gloo acts as an intelligent reverse proxy powered by Envoy — every request from every service flows through it before reaching Azure OpenAI (or any future LLM backend). No microservice holds credentials, knows provider endpoints, or carries any provider-specific SDK. They simply POST JSON to a gateway URL. Gloo handles credential injection via Kubernetes secrets, path and host rewriting, request transformation, rate limiting, and content filtering — all configured once, applied universally. The FastAPI service we refactored is a concrete example of this: we stripped out the entire Azure OpenAI SDK, removed four environment variables, and replaced them with a single GATEWAY_URL. The result is a clean separation of concerns where infrastructure policy lives in the gateway layer and application code stays focused purely on business logic.
Advantages
Single credential store. API keys for every LLM provider live exclusively as Kubernetes secrets in gloo-system. Rotating a key means updating one secret and restarting one pod — not hunting through dozens of microservice .env files or CI/CD pipelines.
Unified observability. Every token consumed, every request made, every latency spike across all services and all providers flows through one point. You get a single dashboard for cost attribution, usage reporting, and anomaly detection rather than stitching together per-service logs from different SDKs.
Provider portability. Switching from Azure OpenAI to AWS Bedrock, Google Vertex, or a local Ollama instance requires changing an Upstream and a VirtualService route — zero changes to any microservice. Teams writing application code never need to know which model they’re actually talking to.
Policy enforcement at scale. Rate limiting, prompt guardrails, content filtering, and per-service authentication are configured once in the gateway and apply automatically to every existing and future microservice. A new team onboarding gets all policies for free — they just get a route.
Zero SDK sprawl. Without a gateway, every team independently manages the Azure OpenAI SDK, handles retries, deals with rate limit errors, and keeps up with API version changes. With the gateway, that complexity is owned once, centrally.
Challenges
Single point of failure. The gateway is now in the critical path of every LLM call across the entire organization. If it goes down, everything stops. Mitigating this requires running multiple gateway-proxy replicas, a robust health check strategy, and a well-tested failover plan — complexity that needs to be designed upfront.
Latency overhead. Every request makes an extra network hop through Envoy. For most LLM calls where the model itself takes hundreds of milliseconds, this is negligible — but for high-frequency, low-latency use cases it adds up and needs to be measured, not assumed away.
Secret rotation at apply time. Our current approach bakes the API key as a static string into the VirtualService at kubectl apply time, read from the secret. This means rotating the key requires re-applying the VirtualService — it's not fully dynamic. A more robust solution (using Gloo's native AI Gateway CRDs in a newer license tier, or a secret store integration like Vault) would inject the key dynamically without a re-apply.
vCluster networking complexity. As we saw firsthand, running Gloo inside vCluster inside Docker creates non-obvious networking layers. NodePort doesn’t bind to localhost, Docker networks are isolated from each other, and connecting services across those networks requires manual intervention. In production this should be replaced with a proper Kubernetes cluster (EKS, AKS, GKE) where networking is straightforward.
Transformation debugging is opaque. When routing breaks, the error chain spans the microservice, Gloo’s control plane, Envoy’s xDS config, and the upstream provider — all with different log formats and failure modes. As we experienced, a single missing hostRewrite caused an hours-long debugging session. Investing in Gloo's observability stack (Prometheus, Grafana, and access logging) early pays dividends later.
License and version alignment. Gloo Enterprise features are license-gated and version-sensitive. The glooctl binary panic we hit was a version mismatch, the AI Gateway CRDs we wanted weren’t in our tier, and %VAR% syntax we tried was unsupported in Edge mode. Understanding exactly what your license covers before designing the implementation avoids expensive architectural pivots mid-project.
As always thanks for stopping by and taking time. Keep reading and stay safe!!
메타데이터
- post_id
- ab64f2ea138f
- slug
- solo-io-as-gateway-for-azure-open-ai-2-ab64f2ea138f
- url
- https://medium.com/@krishnan.srm/solo-io-as-gateway-for-azure-open-ai-2-ab64f2ea138f
- canonical_url
- https://medium.com/@krishnan.srm/solo-io-as-gateway-for-azure-open-ai-2-ab64f2ea138f
- author_url
- https://medium.com/@krishnan.srm
- status
- ok
- fetched_at
- 2026-07-14 07:32:35