Block Runaway LLM Bills

How I Built a Kubernetes Sidecar That Blocks Runaway LLM Bills — Automatically

AI agents are cheap to build and expensive to run. You will discover this eventually — usually at an inconvenient hour, via a message from finance that is politely worded but unmistakably urgent.

A single misconfigured agent session can exhaust thousands of dollars of API quota before any human notices. The conventional response is to add a budget check to the application code. This works until it doesn’t — until someone writes another LLM client without the check, until it gets disabled “temporarily” for a debugging session and never re-enabled, until there is an off-by-one at the token boundary that only manifests under load.

Cost controls implemented in application code are an honour system. Honour systems fail at scale.

The right place for enforcement is the network layer: transparent, automatic, and impossible to bypass from application code. LLM Budget Enforcer implements this as a sidecar proxy injected automatically into every pod via a Kubernetes mutating admission webhook. Your application code does not change. The budget cannot be accidentally omitted. Enforcement is uniform by construction.

The Pattern: Sidecar Injection

The sidecar proxy is a well-established Kubernetes pattern, popularised by service meshes like Istio and Envoy. The mechanics are straightforward: a proxy container runs alongside the application container in the same pod, sharing the network namespace and therefore localhost. The application routes LLM API calls to http://localhost:8080; the proxy checks the budget and forwards to the upstream — or terminates the call with a 429.

The harder problem is injection at scale: getting the sidecar into every pod without touching every deployment manifest. That is precisely what a mutating admission webhook is designed for.

The application container is never aware of the proxy. It points its LLM client at localhost:8080 in place of the upstream API — a one-line configuration change made once per team, not once per service. Budget enforcement becomes a platform property, not a recurring implementation detail.

Architecture Deep Dive

1. The Mutating Admission Webhook

Every time a pod is created in a labelled namespace, the Kubernetes API server sends an AdmissionReview to the webhook before the pod is scheduled. The webhook inspects the spec, constructs a JSON Patch that adds the sidecar container and injects the required environment variables, and returns Allowed. Kubernetes applies the patch atomically — the pod starts with the sidecar already present, with no post-start manipulation required.

Injection is opt-in, scoped to namespaces that carry an explicit label. Unlabelled namespaces are never touched, which keeps the blast radius narrow during rollout and gives teams a clean migration path.

# Enable injection for a namespace
kubectl label namespace my-ai-apps llm-budget-enforcer/inject=enabled

# Any new pod in this namespace will automatically get the sidecar

Key Design Decision The webhook uses failurePolicy: Ignore. If the webhook is unavailable, pod creation proceeds normally without the sidecar. This is not laziness — it is a deliberate safety property. An LLM cost guard should degrade gracefully, not take down production. The worst failure mode here is an unmonitored pod; the alternative is an unschedulable cluster. That calculus is not difficult.

Team identity is resolved through a priority chain: the pod annotation llm-budget-enforcer/team-id, then the app label, then the generateName prefix. In practice, most workloads already carry a meaningful app label. New deployments pick up the correct budget record without any additional annotation — the friction of adoption is deliberately low.

2. The Proxy Sidecar

The sidecar is a FastAPI application that binds to localhost:8080 inside the pod. It exposes three enforcement endpoints:

• **POST /v1/{path}** — OpenAI-compatible endpoint. Forwards to any OpenAI-compatible upstream (configurable via ENFORCER_UPSTREAM_URL).
• **POST /bedrock/invoke/{model_id}** — AWS Bedrock InvokeModel API. Signs requests using Pod Identity credentials.
• **POST /bedrock/converse/{model_id}** — AWS Bedrock Converse API. Unified cross-provider format.

The enforcement decision is made before the request is forwarded. The sequence:

The two-pass token count is worth understanding. tiktoken estimates the prompt tokens pre-call so that over-budget requests are rejected before they ever reach the upstream. Post-call, the actual token counts are read from the response’s usage field and used to update the state file. This produces both proactive blocking and accurate accounting — the running total reflects what the model actually billed, not a local approximation.

3. Budget State: Local + DynamoDB

The proxy maintains two layers of state:

Pod-local:/var/enforcer/state.json on a memory-backed emptyDir volume. Holds cumulative tokens used, call count, status (active / terminated)

Budget config: DynamoDB (keyed on team_id). Holds token budget ceiling, alert threshold percentage

The state file is intentionally pod-scoped and ephemeral: it disappears when the pod does. This models a session budget — each pod run receives its own allotment, with no bleed-over from previous runs. The tradeoff is deliberate: simplicity and isolation over cross-session accounting. Cumulative tracking across sessions is an obvious extension — write the running total to DynamoDB after each call — but it introduces coordination complexity that the v1 deliberately avoids.

# terraform/terraform.tfvars — manage budgets as code
team_budgets = {
  ml-research  = { token_budget = 500000, alert_at_pct = 80 }
  prod-agents  = { token_budget = 100000, alert_at_pct = 90 }
  experiments  = { token_budget = 50000,  alert_at_pct = 70 }
}

4. Observability: SNS + CloudWatch

On breach — when the budget is exhausted or the pre-call estimate would push the session over the limit — the proxy publishes to SNS and emits a BudgetBreach metric to CloudWatch. A TokensConsumed metric is emitted on every request, dimensioned by team ID. This gives you both event-based alerting (SNS → email or PagerDuty) and time-series visibility (CloudWatch) without any additional instrumentation.

CloudWatch QuerySELECT SUM(TokensConsumed) FROM LlmBudgetEnforcer GROUP BY TeamId surfaces per-team token burn rate across your entire fleet. Add a CloudWatch Alarm on this metric to fire before budgets are exhausted, not after.

AWS Authentication: Pod Identity

The proxy makes AWS API calls — DynamoDB reads, SNS publishes, CloudWatch metric puts, Bedrock inference — using EKS Pod Identity. No static credentials, no node-level instance profiles, no secrets mounted as environment variables. The EKS Pod Identity agent injects short-lived, automatically rotated credentials via a projected service account token volume; boto3 picks these up through the standard credential chain without any code changes.

Every injected pod receives AWS access scoped exactly to the enforcer IAM role — and nothing beyond it. The role policy is defined in Terraform, reviewed and versioned alongside the application code, and not escalatable from inside the pod. If the proxy is compromised, the blast radius is limited to what the token budget monitor was authorised to do.

Running It

The full stack — EKS cluster, AWS infrastructure, ECR repositories, Kubernetes manifests, webhook TLS certificates — provisions with a single idempotent command:

uv run scripts/provision.py

The end-to-end test validates the complete pipeline for under a dollar:

# After provisioning, run the E2E test
kubectl delete job llm-budget-enforcer-e2e -n llm-budget-enforcer-test --ignore-not-found
REGISTRY=<ecr-uri> uv run scripts/deploy.py --test-pods
kubectl wait --for=condition=complete job/llm-budget-enforcer-e2e \
  -n llm-budget-enforcer-test --timeout=300s
kubectl logs job/llm-budget-enforcer-e2e -n llm-budget-enforcer-test -c test-client

Expected output:

llm-budget-enforcer e2e — team=e2e-test  model=us.amazon.nova-micro-v1:0  max_tokens=10
[a] Direct Bedrock call ...
[a] PASS — Bedrock responded: 'ok'
[b] Proxy call via sidecar ...
[b] PASS — proxy responded: 'ok'
[c] Budget exhaustion ...
[c]   attempt 1: 200 OK
...
[c]   attempt 25: 200 OK
[c] PASS — 429 received on attempt 26

All phases PASSED.

Three phases, three guarantees: AWS credentials are working and Bedrock is reachable; the webhook successfully injected the sidecar and the proxy endpoint is live; budget enforcement fires correctly at the token boundary. The entire test costs under a cent, which means it can run in CI without budget anxiety — a deliberate design constraint from the start.

Pointing an existing LLM client at the proxy is one line:

import openai

# Before
client = openai.OpenAI()

# After - one change, full enforcement
client = openai.OpenAI(base_url="http://localhost:8080/v1")

Component Overview

Lessons Learned

The TLS bootstrapping problem

Admission webhooks require TLS. On EKS, the managed control plane connects to webhook servers via VPC-native pod IPs — the cross-account ENIs that AWS injects into your VPC are not subject to kube-proxy’s iptables rules, so ClusterIP service routing is not available from the API server’s perspective. This means the TLS certificate must include the pod IP as an IP SAN, not just the Kubernetes service DNS names.

That creates a bootstrap ordering problem: the pod cannot start without a TLS secret containing its own IP, but the IP is not assigned until after the pod is scheduled. The resolution: mount the TLS volume as optional: true. The webhook pod starts and crash-loops, but it is assigned an IP before the containers run. The cert generation script reads that IP, issues a self-signed server certificate with the correct IP SAN, writes the Kubernetes Secret, waits for the kubelet to sync the volume mount, then kills PID 1 inside the container — restarting only the container, not the pod, which preserves the network namespace and therefore the IP address the cert was issued for. Bootstrap complete.

failurePolicy: Ignore is not optional

During early testing I set failurePolicy: Fail, reasoning that it would surface injection errors quickly. Within minutes, an unrelated issue with the webhook deployment caused every pod creation in the cluster to fail — including the webhook pod itself, creating a self-reinforcing failure loop that required direct API server access to unwind. The lesson is not subtle: failurePolicy: Fail is appropriate only for webhooks that are themselves critical infrastructure, deployed with the same reliability guarantees as the API server. For everything else, Ignore is the only defensible default.

Bedrock token counting quirks

Different APIs surface token counts with different field names. The Bedrock InvokeModel API returns usage.inputTokens / usage.outputTokens (camelCase). The Bedrock Converse API is consistent with that convention. OpenAI uses usage.prompt_tokens, usage.completion_tokens, and usage.total_tokens (snake_case). In practice this means response-parsing logic cannot be shared across providers without format detection. The _count_response_tokens function handles all three conventions and falls back to the pre-call tiktoken estimate when none match — deliberately over-counting on ambiguous responses rather than under-counting, which would silently permit over-budget calls to succeed.

Roadmap

The v1 covers the core enforcement primitive. Several capabilities are in scope for near-term work:

• Streaming responses — The hardest open problem. Chunked token counting requires accumulating SSE frames and counting the full response retroactively. The pre-call estimate still enforces budget, but the post-call count is deferred.
• Multi-provider pricing table — Track dollar cost alongside token count. Each model has different input/output prices; a pricing table maps model IDs to cost-per-token.
• Soft budget warnings — The alert_at_pct DynamoDB field is already stored; wiring it to a pre-exhaustion SNS alert is the next step.
• Helm chart — One-command installation for clusters that already have Helm.
• Grafana dashboard — Pre-built CloudWatch dashboard for the LlmBudgetEnforcer metric namespace.

Conclusion

LLM cost control belongs in the infrastructure layer, not the application layer. Delegating it to individual developers — each implementing their own budget check in each service — produces the same fragility as asking each service to implement its own rate limiting. Some will do it correctly. Some will get it wrong. Some will skip it entirely under delivery pressure. The failure modes are invisible until they are expensive.

The sidecar pattern relocates enforcement to the platform layer, where it can be applied uniformly without depending on developer discipline. One webhook server, one proxy image, one DynamoDB table: every pod in every labelled namespace gets automatic, consistent, auditable budget enforcement. The application author does not need to know it exists.

The full source — proxy, webhook, Terraform, and E2E tests — is on GitHub under MIT. The test suite costs under a cent to run and exercises the complete path from admission to enforcement.

Get started Clone the repo, fill in terraform.tfvars, and run uv run scripts/provision.py. The E2E test validates the complete pipeline — namespace labelling, webhook injection, proxy forwarding, and budget exhaustion — in under five minutes for less than a cent.