Building a Sovereign, Cost-Optimised Intelligent Inference Gateway

How We Routed AI Queries Intelligently Across Local and Remote Models — Without Sending a Single Token to the Cloud

TL;DR: We built a production-grade AI inference gateway from scratch that classifies every user query using a small LLM, routes it to the right model based on complexity, enforces guardrails, tracks cost per user, and runs entirely on your own infrastructure — no cloud AI API required.

The Problem Nobody Talks About

When organisations start adopting AI, they typically do one of two things:

Option A — Route everything to the most powerful model. Every query, whether it is “What is Python?” or “Design a multi-region disaster recovery architecture”, goes to the same large, expensive model. The result is high cost, unnecessary latency, and wasted GPU compute.

Option B — Route everything to the cheapest model. Simple queries get answered well, but complex reasoning tasks return poor or incomplete responses. Users lose trust. The AI initiative stalls.

Neither option is right. And there is a third problem that neither addresses at all: data sovereignty.

Most teams point their applications directly at OpenAI, Anthropic, or Google. Every prompt, every document, every user query leaves the organisation’s infrastructure and travels to a third-party cloud. For enterprises in healthcare, finance, legal, defence, or any regulated industry — this is a non-starter.

What We Set Out to Build

We wanted an AI infrastructure that answered three hard questions simultaneously:

1. How do we run AI completely on our own infrastructure — no data leaving our network?
2. How do we avoid paying for a powerful model when a cheap one would do the job?
3. How do we route intelligently — not by guesswork, but by actually understanding the query?

The answer is an Intelligent Inference Gateway.

What Is an Inference Gateway?

Think of it as a smart traffic controller for AI requests.

Instead of your application talking directly to a single AI model, every request passes through the gateway. The gateway decides — based on rules, costs, or intelligence — which model should handle that specific request.

Your Application
       ↓
Inference Gateway  ←— the brain
       ↓
┌──────────────────────────────────────┐
│  Simple query   →  Small fast model  │
│  Complex query  →  Powerful model    │
│  SQL query      →  Specialist model  │
│  Policy breach  →  Blocked           │
└──────────────────────────────────────┘

This is not a new idea in software engineering. Load balancers, API gateways, and service meshes do the same thing for web traffic. We are applying the same pattern to AI inference.

Our Production Architecture

Here is what we built, end to end:

┌─────────────────────────────────────────────────────┐
│                   USER INTERFACE                    │
│              Open WebUI  (port 3000)                │
└─────────────────────┬───────────────────────────────┘
                      │
┌─────────────────────▼───────────────────────────────┐
│              INTELLIGENT ROUTER                     │
│                  (port 5000)                        │
│                                                     │
│  1. Detect internal UI tasks → route cheap          │
│  2. LLM Classifier (small model, ~200ms)            │
│     → complexity 1 / 2 / 3                          │
│  3. Keyword fallback if classifier unavailable      │
│  4. Guardrail error handler                         │
└──────┬──────────────┬──────────────┬────────────────┘
       │              │              │
   Tier 1         Tier 2         Tier 3
  Simple         Moderate        Complex
       │              │              │
┌─────────────────────▼───────────────────────────────┐
│                LITELLM GATEWAY                      │
│                  (port 4000)                        │
│                                                     │
│  Cost tracking · Token logging · Guardrails         │
│  Virtual keys · Spend limits · Admin UI             │
│  Fallback routing · Rate limiting                   │
└──────┬──────────────┬──────────────┬────────────────┘
       │              │              │
┌──────▼──────┐ ┌─────▼──────┐ ┌────▼───────────────┐
│  vLLM       │ │  vLLM      │ │  vLLM              │
│  Small LLM  │ │  Mid LLM   │ │  Large LLM         │
│  (Tier 1)   │ │  (Tier 2)  │ │  (Tier 3)          │
│  Edge/CPU   │ │  GPU VM    │ │  GPU VM            │
└─────────────┘ └────────────┘ └────────────────────┘
                      │
              ┌───────▼──────┐
              │  PostgreSQL  │
              │  (logging,   │
              │   cost data) │
              └──────────────┘

Every component runs on your own servers. Zero data leaves your network.

The Three Tiers Explained

We designed three model tiers, each serving a specific role:

Tier 1 — The Fast Lane

Model: Small LLM (1B–3B parameters) Engine: vLLM Handles: Simple factual questions, definitions, translations, short lookups Latency: Under 500ms Cost: Near zero

Examples of queries routed here:

• “What is the capital of Japan?”
• “Define microservices”
• “Translate hello to Tamil”

Tier 2 — The Reasoning Lane

Model: Mid-size LLM (7B–8B parameters, e.g. DeepSeek-R1, Mistral 7B) Engine: vLLM Handles: Explanations, comparisons, how/why questions, summaries Latency: 1–3 seconds Cost: Moderate

Examples of queries routed here:

• “Explain the difference between REST and GraphQL”
• “How does JWT authentication work?”
• “Compare SQL and NoSQL databases”

Tier 3 — The Power Lane

Model: Large LLM (20B+ parameters, e.g. LLaMA 3 70B, GPT-OSS 20B) Engine: vLLM Handles: System design, code generation, long-form writing, deep analysis Latency: 5–30 seconds Cost: Full GPU compute

Examples of queries routed here:

• “Design a microservices architecture for a fintech platform”
• “Write a production-grade authentication system in Python”
• “Analyse the CAP theorem and its implications for distributed databases”

The Intelligence Layer — LLM as a Classifier

This is the most important part of the system and what makes it genuinely production grade.

Early attempts at routing used keyword matching. If the query contained words like “design” or “architect”, it was routed to the powerful model. If it started with “what is”, it went to the cheap model.

This works — until it doesn’t. Keyword matching is brittle. It breaks on paraphrasing. It fails in languages other than English. It cannot understand intent.

The production-grade solution is to use a small, fast LLM as the classifier itself.

User sends query
       ↓
Small LLM receives the query + a classification prompt
       ↓ takes ~150–300ms, uses ~50–80 tokens
       ↓
Returns structured JSON:
{
  "complexity": 2,
  "reason": "Requires explanation and comparison of two concepts",
  "confidence": 0.95
}
       ↓
Router selects the appropriate worker model
       ↓
Worker model generates the full response

The classifier prompt instructs the small model to think about the query and return a complexity score of 1, 2, or 3 with a reason and a confidence score. Because it is an actual language model, it understands context, intent, phrasing, and domain — regardless of how the user words their question.

We also built a keyword-based fallback that activates automatically if the classifier model is unavailable for any reason. The system degrades gracefully rather than failing.

Why vLLM Instead of Ollama for Production

During development, we used Ollama to run models locally. Ollama is excellent for development and experimentation. However, for production deployments, vLLM is the right choice.

Feature Ollama vLLM Purpose Development / local use Production inference Throughput Single request at a time Thousands of concurrent requests Batching No Continuous batching GPU utilisation Basic Optimised (PagedAttention) OpenAI-compatible API Yes Yes Quantisation support Basic Advanced (GPTQ, AWQ, FP8) Multi-GPU support Limited Full tensor parallelism Latency at scale Degrades under load Consistent under high concurrency

vLLM uses an algorithm called PagedAttention that manages GPU memory the same way an operating system manages RAM — efficiently and with almost no waste. At scale this translates to 2–4x higher throughput compared to naive inference engines.

To run a model with vLLM in production:

docker run -d \
  --name vllm-tier1 \
  --gpus all \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.2-1B-Instruct \
  --host 0.0.0.0 \
  --max-model-len 4096 \
  --tensor-parallel-size 1

LiteLLM then points to this vLLM instance exactly as it would to any OpenAI-compatible endpoint.

LiteLLM — The Gateway Layer

Between the smart router and the model servers sits LiteLLM, an open-source AI gateway that provides everything you need for production operations.

We used it for:

Cost tracking — Every request is logged with input tokens, output tokens, and cost calculated from the pricing you configure. You see a real-time spend breakdown per model, per user, and per team from the Admin UI.

Virtual keys — Each team or application gets its own API key. The gateway tracks usage independently per key. When a team hits their monthly budget limit, their key is automatically blocked without any manual intervention.

Guardrails — We configured content filters that block specific categories of queries before they ever reach a model. In our example, queries asking for stock or investment recommendations are intercepted, blocked with a 403 response, and our router translates that into a friendly user-facing message.

Fallback routing — If the primary model is unavailable, LiteLLM automatically tries the next model in the fallback chain. The user never sees an error.

Admin UI — A full web dashboard at port 4000 shows real-time request logs, spend analytics, model health, team budgets, and guardrail activity — all without touching a config file.

Sovereign and Private by Design

Every component in this architecture runs on infrastructure you control:

Open WebUI         →  Your server
Smart Router       →  Your server
LiteLLM Gateway    →  Your server
vLLM instances     →  Your GPU servers
PostgreSQL         →  Your server
Model weights      →  Your disk

No query, no prompt, no response, and no user data ever leaves your network. There are no third-party API calls to OpenAI, Anthropic, or any cloud provider. The only external dependencies are the open-source model weights downloaded once from Hugging Face and stored locally.

This makes the architecture fully compliant with:

• GDPR — personal data never leaves your jurisdiction
• HIPAA — patient data stays within your controlled environment
• SOC 2 — full audit trail with no third-party data processors
• Government and defence — air-gapped deployment possible

For organisations that operate in regulated industries, this is the only responsible way to deploy AI at scale.

Real Cost Savings

The financial case for intelligent routing is straightforward.

Consider a real-world deployment serving 100,000 queries per day. Without routing, every query goes to the large Tier 3 model. With routing, query distribution typically looks like this:

Tier % of queries Model size Relative cost Tier 1 — Simple 60% 1B–3B 1x Tier 2 — Moderate 30% 7B–8B 3x Tier 3 — Complex 10% 20B–70B 12x

Without routing (all queries to Tier 3):

100,000 queries × 12x cost = 1,200,000 cost units/day

With intelligent routing:

60,000 × 1x  =    60,000
30,000 × 3x  =    90,000
10,000 × 12x =   120,000
Total        =   270,000 cost units/day

Saving: 77.5% reduction in compute cost.

In practice this translates directly to GPU hours, electricity, and hardware cost. For a mid-size deployment running on owned hardware, this can represent tens of thousands of dollars saved annually — while simultaneously improving response times for the majority of queries.

Guardrails — Policy Enforcement at the Gateway

One of the most important features for enterprise deployments is the ability to enforce policies at the infrastructure level, not the application level.

In our setup, we configured a guardrail in LiteLLM that detects queries asking for financial or investment advice. When triggered:

1. LiteLLM intercepts the request before it reaches any model
2. Returns a 403 Forbidden response to the router
3. Our router translates the 403 into a friendly, contextual message for the user
4. The entire event is logged with full details for audit purposes

The user sees:

“I’m sorry, I’m not able to provide personalised stock or investment recommendations. Please consult a licensed financial advisor.”

The compliance team sees a full audit log including the original query, the guardrail that fired, the category matched, the severity level, and the timestamp.

This pattern can be extended to any policy — PII detection, toxic content filtering, competitor mentions, off-topic queries, prompt injection detection, and more.

The Full Request Journey

Here is exactly what happens from the moment a user sends a message to the moment they see a response:

1. User types query in Open WebUI
          ↓
2. Open WebUI sends POST /v1/chat/completions to Smart Router

3. Smart Router checks: is this an internal UI task?
   (follow-up suggestions, title generation, etc.)
   → Yes: skip classifier, route to Tier 1 directly
   → No: continue

4. Smart Router calls LLM Classifier (Tier 1 model)
   with the query + classification system prompt
   → Returns {complexity: 1|2|3, reason, confidence}
   → Takes ~150–300ms

5. Router selects worker model based on complexity:
   1 → Small LLM  (Tier 1)
   2 → Mid LLM    (Tier 2)
   3 → Large LLM  (Tier 3)

6. Router forwards request to LiteLLM Gateway
   with the selected model name

7. LiteLLM runs pre-call guardrail checks
   → Blocked: return 403 → Router returns friendly message
   → Allowed: continue

8. LiteLLM forwards to the appropriate vLLM instance

9. vLLM runs inference on the selected model
   → Streams tokens back to LiteLLM

10. LiteLLM logs the request:
    - model used
    - tokens in / tokens out
    - cost calculated
    - user key recorded
    - guardrail status

11. Response streamed back through Router → Open WebUI → User

Total overhead from routing + classification: ~200–350ms

What We Used — The Full Stack

Component Technology Purpose User interface Open WebUI Chat interface for all users Intelligent router FastAPI (Python) Query classification and routing LLM classifier Small LLM via vLLM Complexity scoring Gateway LiteLLM Cost tracking, keys, guardrails Tier 1 model Small LLM (1B–3B) Simple queries Tier 2 model Mid LLM (7B–8B) Reasoning queries Tier 3 model Large LLM (20B–70B) Complex queries Inference engine vLLM Production-grade model serving Database PostgreSQL Persistent logging and cost data Containerisation Docker All components containerised

All components are open source. Total external licensing cost: zero.

What This Solves in the Real World

For a hospital: Clinical staff ask questions ranging from “what is the dosage of ibuprofen” to “analyse this patient’s complex medication interactions across 12 conditions”. Simple queries answer in under a second. Complex clinical reasoning gets the full power of a large model. Patient data never leaves the hospital’s own servers.

For a bank: Customer service queries (“what is my account balance”, “what are your mortgage rates”) go to a fast cheap model. Fraud analysis, risk assessment, and regulatory report generation go to the powerful model. Financial queries that should go to a human advisor are blocked at the gateway with a compliant response.

For a law firm: Simple document lookups route to the small model. Contract analysis and legal strategy queries route to the large model. Client data stays within the firm’s own infrastructure, satisfying attorney-client privilege requirements.

For a government agency: The entire stack runs air-gapped — no internet connection required after initial setup. Sensitive queries never leave the secure facility. Query complexity routing ensures expensive GPU resources are reserved for tasks that genuinely require them.

What Comes Next

The foundation we built is extensible. The natural next layers are:

Semantic caching — Cache responses to frequent identical queries in Redis. If ten users ask the same question, only the first one hits the model. Everyone else gets an instant response from cache.

Feedback-driven routing — If a user rates a response poorly, automatically retry with the next tier model and log which tier produced the better answer over time.

RAG integration — Connect the gateway to a vector database containing your organisation’s private documents. Before routing to a model, retrieve relevant context and inject it into the prompt. The model answers from your knowledge base, not just its training data.

Observability dashboard — Real-time visualisation of cost savings, tier distribution, classifier confidence scores, and guardrail activity across the organisation.

Multi-region deployment — Run model servers in multiple data centres. The gateway routes to the closest healthy instance, providing both low latency and geographic redundancy.

Conclusion

The conventional wisdom in enterprise AI is that you must choose between capability and control. Use the powerful cloud APIs and accept that your data leaves your network — or run local models and accept inferior results.

This architecture proves that is a false choice.

With an intelligent inference gateway, you get:

• Sovereignty — every byte stays on your infrastructure
• Intelligence — queries are understood and routed appropriately
• Cost efficiency — 60–80% reduction in compute cost through smart routing
• Compliance — guardrails and audit logs built into the infrastructure layer
• Scalability — vLLM handles production concurrency that development tools cannot

The technology to do this is entirely open source and available today. The barrier is not capability — it is knowing how to assemble the pieces.

We hope this architecture serves as a practical blueprint for teams who need AI that is powerful, private, and cost-efficient by design.

Built with: FastAPI, LiteLLM, vLLM, Open WebUI, PostgreSQL, Docker, and open-source LLMs.

All components self-hosted. Zero cloud AI API dependencies.