Part 2 — The Router

Series: Building an LLM Playground Farm. See Part 0 for the architecture and Part 1 for the inference backend.

At the end of Part 1 we had a single expert: a gemma3 snap, operated by a charm, publishing its OpenAI endpoint over the inference-api relation. Useful, but lonely. A farm needs many experts — and clients shouldn't have to know which one to call, or where it lives, or how many replicas are running.

That’s the router’s job. In this part we introduce **model-router: a charm that runs a LiteLLM proxy, discovers every inference backend you relate to it, and presents one** OpenAI-compatible endpoint in front of all of them.

The core idea: consume the same interface Part 1 provides

The inference-snap charm provides the inference_openai interface. The router requires it and also provides its own copy of it to clients:

# model-router charmcraft.yaml — github.com/ktsakalozos-canonical/model-router-operator
requires:
  inference-api:
    interface: inference_openai   # consumes inference-snap backends
provides:
  openai-api:
    interface: inference_openai   # presents one endpoint to clients

The router is therefore a transparent middle layer speaking the same protocol on both sides. From below it looks like a client of many models; from above it looks like a single model. Anything that could talk to one inference-snap can talk to the router unchanged — including, as we’ll see much later, another router.

What the charm actually installs

model-router is a machine charm that installs litellm[proxy] into a virtualenv and runs it under systemd. The on-disk layout is worth knowing because Part 3 builds directly on it:

/opt/model-router/venv/            # the litellm virtualenv
/etc/model-router/config.yaml      # rendered LiteLLM proxy config
/etc/model-router/catalog.json     # model capability catalogue (Part 3)
/etc/model-router/router_hook.py   # LLM-routing pre-call hook (Part 3)
/etc/systemd/system/model-router.service

Like the inference charm, it follows the reconcile pattern: config changes and relation events all funnel into one idempotent _reconcile that renders config, (re)starts the service, waits for health, and reports status.

def _reconcile(self, event):
    if not VENV_LITELLM.exists():
        self.unit.status = ops.MaintenanceStatus("Waiting for LiteLLM install")
        return
    collected = self._collect_backends()        # only *ready* backends
    backends = collected.backends
    if not backends:
        if collected.pending:                    # related, but still loading
            # Keep any last-good config and the running proxy in place; the
            # periodic update-status reconcile re-checks.
            self.unit.status = ops.WaitingStatus(
                "Waiting for backend(s) to finish loading")
            return
        self._stop_service()                     # nothing related at all        
        self.unit.status = ops.BlockedStatus(
            f"Waiting for an inference backend on '{API_RELATION}'")
        return

    changed = self._write_config(backends)
    changed = self._write_catalog(backends) or changed
    changed = self._write_hook() or changed
    changed = self._write_systemd_unit() or changed
    self._start_service(restart=changed)         # only restart on real change
    if not self._wait_for_proxy():               # poll /health/liveliness
        self.unit.status = ops.WaitingStatus("Waiting for the proxy to become healthy")
        return

    self._reconcile_ports()
    self._publish_endpoint()
    self.unit.status = ops.ActiveStatus(...)

With no backend related, the router sits blocked. But there is a subtler third state. Once a backend is related but is still loading its model, the router reports waiting and keeps its last-good config in place. This way an existing healthy fleet is never disrupted just because a freshly-added backend has not finished loading yet.

Backend discovery: relations, then /models(with readiness gating)

When you relate an inference app, the router reads its endpoint from the relation data bags. Recall from Part 1 that every unit publishes its own url, so the router iterates over units, not just applications. Discovery runs in two stages.

First it collects every databag that advertises a url (per unit, with app-level data as a fallback for older providers). At this point we know where each backend is, but not what it serves:

def _endpoint_from_data(self, source, data):
    url = data.get("url")
    if not url:
        return None
    # `snap` is used only to look up capabilities in MODEL_CATALOG (Part 3),
    # never as the served model id.
    snap = data.get("snap") or data.get("model") or self._default_alias()
    return Endpoint(source=source, url=url, snap=snap)

Second, for each endpoint the router asks the backend directly which model it is actually serving:

def _discover_model(self, url):
    with urllib.request.urlopen(f"{url.rstrip('/')}/models", timeout=10) as r:
        payload = json.loads(r.read())
    # returns payload["data"][0]["id"], else None

This is where readiness gating comes in. A backend enters the routing table only once /models returns a real id. An endpoint that advertises a URL but is not ready yet (eg the model is still loading and the server returns 503) is treated as pending and is never advertised.

Building the routing table

Now the clever bit. For every backend, the router emits two entries into LiteLLM’s model_list: one under the backend's real model name, and one under the shared alias.

for backend in backends:
    params = {"model": f"openai/{backend.model}",
              "api_base": backend.url, "api_key": NOAUTH_KEY}
    model_list.append({"model_name": backend.model, "litellm_params": dict(params)})
    model_list.append({"model_name": alias,          "litellm_params": dict(params)})

LiteLLM groups deployments by model_name, and that single fact gives us two behaviours for free:

• The alias (default) is one big group containing every backend, calling default round-robins across all models.
• Each real model name is its own group, calling gemma3 hits only the gemma3 backend(s); if you scaled gemma3 to three units, those three form one group and round-robin among themselves.

So scaling and multi-model routing fall out of the same mechanism. Add a unit, get another round-robin target. Relate a new model, get a new directly-callable name and another member of the alias group.

The router settings carry the load-balancing strategy and retry count:

"router_settings": {
    "routing_strategy": self._routing_strategy(),   # simple-shuffle by default
    "num_retries": self._num_retries(),
}

Configuration

[embed]

(Four more options — classifier-model, classifier-timeout, classifier-prompt — only matter in llm mode, so we save them for Part 3.)

This part stays in the default **shuffle** mode: round-robin. That's the baseline Part 3 upgrades.

Hands-on: put the router in front of two models

Building on the deploy from Part 1:

# Clone, pack and deploy the router
git clone https://github.com/ktsakalozos-canonical/model-router-operator.git
cd model-router-operator
charmcraft pack
juju deploy ./model-router_ubuntu-24.04-amd64.charm
# Deploy two inference backends (from Part 1)
juju deploy ./inference-snap_ubuntu-24.04-amd64.charm gemma --config inference-snap=gemma3
juju deploy ./inference-snap_ubuntu-24.04-amd64.charm coder --config inference-snap=qwen3-coder
# Wire the backends into the router
juju integrate model-router:inference-api gemma:inference-api
juju integrate model-router:inference-api coder:inference-api
juju status --watch 5s

Once healthy, the router reports something like:

model-router  active   routing 2 backend(s) on :4000
                        (alias 'default'; shuffle ('simple-shuffle');
                         models: gemma-3-4b-it-ov-int4-fq, qwen3-coder-30b-a3b-q4-k-m)

Call the farm through one endpoint

ROUTER=$(juju status --format=json | jq -r '.applications["model-router"].units[]["public-address"]' | head -1)
# See what the proxy advertises: the alias + every real model
curl http://$ROUTER:4000/v1/models
# Round-robin across everything via the alias
curl http://$ROUTER:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "default", "messages": [{"role":"user","content":"hi"}]}'
# Or target one model by name
curl http://$ROUTER:4000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model": "qwen3-coder-30b-a3b-q4-k-m",
       "messages": [{"role":"user","content":"Write a bash for-loop"}]}'

Scale a backend and watch the router adapt

juju add-unit gemma        # add a second gemma3 replica

No router reconfiguration needed. The new unit publishes its own endpoint (Part 1’s per-unit design), _collect_backends picks it up on the relation-changed event, and the gemma3 group now round-robins across two replicas.

Exposing the router to clients

Because the router provides openai-api (also inference_openai), other charms can consume it exactly like they would consume a raw inference-snap:

juju integrate model-router:openai-api your-client-app

It publishes url (http://<addr>:4000), port, and model (the alias) — a single stable endpoint that hides the entire fleet behind it.

Where we are on the map

We now have a real farm: many experts, one front door, automatic scale-out. But there’s an obvious weakness. In shuffle mode, a coding question might land on a general chat model and an image question on a text-only one — the router spreads load blindly, ignoring what each model is good at.

Fixing that is where it gets interesting. In Part 3 we flip routing-mode=llm and let a small local model read each query and pick the right expert.