Right-Sizing the Frontier: A Guide to LLM Routing, Workload-to-Model Matching, and Token-per-Dollar Economics

Why the model router has become a first-class architecture decision, why most of the savings come from far simpler levers, and how to build the whole stack without burning trust or budget.

tl;dr — Enterprise AI spend is climbing even as token prices fall, and the fashionable fix, an intelligent model router, is real but oversold. The evidence says price spreads of 10x to 30x inside model families make workload-to-model matching worth doing, yet the largest and safest savings (50 to 70 percent) come from static tiering, prompt caching, reasoning-effort caps, and batch APIs before any learned router fires. This guide covers the July 2026 pricing landscape, a six-dimension workload taxonomy with worked cost examples from healthcare to COBOL modernization, the routing ladder from config files to reinforcement learning, the gateway and router tooling market, and a staged build specification. The one rule that survives every case study, including OpenAI’s GPT-5 router rollback: a wrong cheap answer costs more than an unnecessary frontier call, so bias every threshold toward quality until production evidence says otherwise.

Complimentary Reading Link

There is a paradox at the center of enterprise AI economics in 2026. The price per token of large language models has fallen dramatically since 2023, yet total inference spend keeps climbing. The culprits are reasoning models that burn thousands of hidden chain-of-thought tokens per request, context windows that have ballooned past a million tokens, and agentic loops that make twenty model calls where an application once made one. Enterprises running millions of requests per month in regulated industries are routinely paying frontier prices for work a model one-tenth the cost could do with identical accuracy.

The fashionable answer is the router: an automated layer that inspects each request and assigns it to the cheapest model capable of handling it. The thesis, repeated in vendor decks and conference keynotes, is that a well-designed routing layer cuts inference spend by 40 to 85 percent while holding task quality within a few points of a frontier-only baseline [8].

That thesis is directionally right and oversold at the top end. This guide walks through the evidence: the current model tiering landscape, a workload taxonomy you can actually apply, the ladder of routing architectures from a config file to reinforcement learning, the tooling market, the efficiency levers that matter more than routing, and a staged build specification. Along the way I will flag where the headline numbers come from, what they were measured on, and where they quietly fall apart.

Three findings frame everything that follows.

First, price spreads inside model families are now large enough that allocation matters before any algorithmic cleverness. Frontier-to-small output-price ratios run 10x to 30x within major families, and past 100x if you reach across families to open-weight hosts [1], [2], [3], [5].

Second, the celebrated research numbers need qualification. RouteLLM, the reference peer-reviewed work, reported cost reductions of over 85 percent on MT-Bench at 95 percent of GPT-4 quality, but only 45 percent on MMLU and 35 percent on GSM8K, and it was measured on a GPT-4-to-Mixtral pair whose roughly 100x cost ratio exceeds any within-family spread you can buy today [8]. Benchmark quality is a weak proxy for enterprise task success.

Third, the safest and largest savings come from unglamorous controls: static workload-to-model mapping, prompt caching, reasoning-effort caps, and batch APIs. These routinely capture 50 to 70 percent of frontier-only spend before a learned router ever fires [1], [8], [28]. A learned router adds an incremental 10 to 25 points, and only on genuinely mixed-complexity traffic, and only if it does not defeat your prompt cache.

Let me define the core term before going further. LLM routing is the practice of selecting, per request or per role, which model (and at what reasoning budget) will serve a given task, with the objective of minimizing cost per successful task at held quality. The definition matters because the naive objective, cost per token, leads teams to decisions that increase retries, review burden, and churn.

Section 1: The Model Tiering Landscape

As of mid-2026, the commercial and open-weight market has stratified into distinct capability tiers with extreme price dispersion. Output tokens universally cost more than input tokens, which means verbose generation and hidden reasoning tokens carry the highest financial risk [2], [24].

A warning before the numbers: every price below was verified against provider pricing pages on July, 2026, and every one of them is a quarterly-decaying snapshot. Providers reprice, rename, and deprecate monthly. Anthropic’s Sonnet 5 introductory pricing expires August 31, 2026; DeepSeek deprecated legacy aliases on July 24, 2026. Re-verify against the provider’s own page before you commit a budget [1], [2], [3], [5].

The capability-and-price matrix

Family    Model            In $/M  Out $/M  Context  Cache-read  Batch  Effort dial
--------  ---------------  ------  -------  -------  ----------  -----  --------------
Anthropic Haiku 4.5        1.00    5.00     200K     0.10x       -50%   thinking
Anthropic Sonnet 5         3.00    15.00    1M       0.10x       -50%   thinking
Anthropic Opus 4.8         5.00    25.00    1M       0.10x       -50%   low..max
Anthropic Fable 5          10.00   50.00    1M       0.10x       -50%   adaptive
OpenAI    GPT-4.1 nano     0.10    0.40     1M       -75%        -50%   none
OpenAI    GPT-5.4-nano     0.20    1.25     400K     -90%        -50%   reasoning
OpenAI    GPT-5.6 Luna     1.00    6.00     ~400K    -90%        -50%   none..xhigh
OpenAI    GPT-5.6 Terra    2.50    15.00    ~1M      -90%        -50%   none..xhigh
OpenAI    GPT-5.6 Sol      5.00    30.00    ~1M      -90%        -50%   none..xhigh
OpenAI    GPT-5.5 Pro      30.00   180.00   ~1M      none        -50%   high
Google    Gemini 2.5 F-Lt  0.10    0.40     1M       0.10x       -50%   budget
Google    Gemini 3.5 Flash 1.50    9.00     1M       0.10x       -50%   budget
Google    Gemini 3.1 Pro   2.00    12.00    1M+      0.10x       -50%   budget
DeepSeek  V4-Flash         0.14    0.28     1M       0.02x       n/a    thinking
DeepSeek  V4-Pro (promo)   0.435   0.87     1M       0.03x       n/a    thinking
Mistral   Medium 3.5       1.50    7.50     256K     0.10x       -50%   effort
Mistral   Small 4          0.15    0.60     256K     0.10x       -50%   effort
Mistral   Ministral 3 8B   0.15    0.15     256K     0.10x       -50%   none
OSS host  Llama 3.3 70B    0.59    0.79     128K     none        n/a    none

Sources: provider pricing pages and tracking aggregators, verified July, 2026 [1], [2], [3], [4], [5], [6], [7].

For open-weight families, the deciding constraint is usually licensing and jurisdiction rather than price. Qwen models under 35B parameters are Apache 2.0; Llama 4 requires a Meta license above 700 million monthly active users; DeepSeek weights are MIT. Hosted on neutral infrastructure, the floor drops further: Groq serves Llama 3.3 70B at $0.59/$0.79 at 250+ tokens per second on LPU silicon; Together AI runs GPT-OSS 20B at $0.05/$0.20 [6], [7].

The ceiling on routing savings is a price ratio

The fastest way to estimate whether routing is worth building is to ignore routing sophistication entirely and look at the tier ratio inside your chosen family.

Family     Frontier -> small pair       Output ratio  Implication
---------  ---------------------------  ------------  --------------------------------
Anthropic  Fable 5 -> Haiku 4.5         10x           80%+ ceiling on easy-heavy mixes
Anthropic  Opus 4.8 -> Haiku 4.5        5x            moderate ceiling, low router risk
OpenAI     GPT-5.5 Pro -> GPT-4.1 nano  450x          extreme, spans subfamilies
OpenAI     GPT-5.6 Sol -> Luna          5x            same-family ceiling is moderate
Google     Gemini 3.1 Pro -> 2.5 F-Lt   30x           output-heavy work strongly favors
DeepSeek   V4-Pro -> V4-Flash           ~3x           routing must be very clean
Mistral    Medium 3.5 -> Ministral 8B   50x           very high ceiling, templated gen

The arithmetic is simple. If the weak tier costs one-tenth of the frontier tier, perfectly routing everything downward yields a theoretical maximum of 90 percent savings. Real systems never get there, because some fraction of requests must stay on the strong tier and the router itself adds overhead. But the ratio tells you whether the problem is worth solving at all. Routing across a 3x spread can work; the tolerance for router error is razor thin. Routing across a 10x to 30x spread is often worth building even in its crudest static form [1], [3], [5].

This ratio is also why the RouteLLM headline needs context. The paper’s own appendix estimates GPT-4 at roughly $24.70 per million tokens against Mixtral 8x7B at $0.24, a spread near 100x [8]. No within-family spread available today is that wide, so the 85 percent figure is a ceiling from a vanished pricing regime, and it applies to MT-Bench-style chat quality rather than your domain’s task success.

Intra-model dials: routing without switching models

Reasoning effort (also called thinking budget) is a per-request parameter that controls how many hidden chain-of-thought tokens a reasoning model may generate before answering. Those tokens bill at the full output rate and are invisible in most product logs, which makes them the most under-audited line in the inference budget.

Every major provider now ships this dial. OpenAI’s GPT-5.x exposes reasoning_effort from none through xhigh, plus text.verbosity [2], [14]. Anthropic's Opus 4.8 exposes low/high/xhigh/max effort levels and positions effort tuning as frequently a better lever than switching models; Fable 5 and Sonnet 5 use adaptive thinking [1], [23]. Gemini exposes a literal thinkingBudget parameter and bills reasoning tokens explicitly [3].

The evidence for capping effort is strong. A peer-reviewed benchmark on basic math reasoning found GPT-5 achieved identical 97 percent accuracy at medium and high effort while burning progressively more tokens, and o3 held 97 percent across low, medium, and high budgets [24]. The TALE line of research reports a 67 percent average reduction in token usage with under 3 percent accuracy loss under budget-aware reasoning [21]. OpenAI’s own internal coding-agent evaluations documented leaner prompts and constrained reasoning improving scores by up to 15 percent while cutting total tokens 41 to 66 percent [18]. The scale of the waste is easy to model: 100,000 requests per month generating 3,000 hidden reasoning tokens each on a $30-per-million-output model is roughly $9,000 per month of reasoning spend alone.

The practical policy: low effort for classification, extraction, and triage; medium as the default for professional work; high or xhigh only where an eval proves it changes the outcome. This substitutes for a great deal of inter-model routing, operates more cheaply, and, crucially, does not fragment your prompt cache.

Native routing: what the platforms actually offer

AWS Bedrock Intelligent Prompt Routing (GA April 2025) offers a single serverless endpoint that predicts per-model response quality and routes within a family (Claude, Llama, Nova pairs). AWS’s product page claims up to 30 percent cost reduction without compromising accuracy; its GA blog reported a stronger 63.6 percent average saving on RAG datasets, with 87 percent of prompts routed to Haiku while holding the Sonnet baseline [16].

Azure AI Foundry model router (version 2025–11–18) is a trained ML classifier deployed as a single Foundry model with Balanced, Cost, and Quality modes, now spanning a cross-vendor pool including GPT-5.2, the Claude family, Grok-4, DeepSeek, and Llama-4. A Microsoft community benchmark on a small prompt set measured 4.5 percent savings in Balanced mode, 4.7 percent in Cost, and 14.2 percent in Quality, a useful reminder that router savings scale with workload diversity and are modest on homogeneous traffic [17].

OpenRouter Auto routes via a meta-model powered by NotDiamond, exposes a 0–10 cost-quality dial, charges the routed model’s rate with no auto-router surcharge, and, tellingly, pins the model per conversation via a session fingerprint to preserve prompt-cache hits [18].

Two important absences. OpenAI’s much-discussed GPT-5 real-time router is a ChatGPT product feature, absent from the raw developer API. And Anthropic’s first-party API does not auto-route at all: you specify a model string per request, full stop [1], [2]. If you are building on the raw APIs of either, the routing layer is yours to build or buy.

Practitioner takeaway. The internal price ratio of your chosen family is the hard ceiling on routing ROI, and it is almost always narrower than the RouteLLM headline implies. Before you route across models, exhaust the reasoning-effort dial inside a single model. Verify every price against the provider’s own page before you build a business case.

Section 2: A Workload Taxonomy for Model Selection

Assigning workloads to tiers by intuition produces severe over-provisioning. “Easy versus hard” is too coarse. A defensible taxonomy evaluates every workload across six dimensions:

1. Reasoning depth: lookup, extraction, and tagging at one end; multi-hop synthesis in the middle; open-ended planning and policy interpretation at the other.
2. Output verifiability: machine-checkable outputs (schemas, unit tests, code databases) versus outputs only a human can judge.
3. Error-cost asymmetry: the penalty of a wrong answer relative to the cost of the query. A hallucinated FAQ answer is cheap; a hallucinated regulatory filing is a liability event.
4. Latency sensitivity: real-time user experience versus asynchronous back-office tolerance.
5. Context volume: short prompts, document-scale inputs, or repository- and corpus-scale context.
6. Regulatory and audit exposure: whether the decision must be explainable, reproducible, and logged.

The governing principle: send verifiable, low-error-cost, high-volume, shallow-reasoning work to the smallest tier that clears your evaluation bar, and reserve the frontier tier for high-error-cost, audit-exposed, multi-hop work where a wrong cheap answer triggers retries, escalations, or liability that dwarf the token savings. Both “frontier by default” and “cheap by default” fail in enterprise settings; the failure modes are just differently shaped.

Eight worked examples

Token volumes below are stated assumptions so you can rerun the math; prices are the verified July 2026 rates from Section 1. The frontier baseline uses Opus 4.8 ($5/$25 per million input/output).

1. Healthcare: coding lookup versus claims adjudication. ICD-10 and CPT code lookup from clinical notes looks like a language task and is actually a retrieval task. The NEJM AI benchmark found GPT-4, the best model tested, achieved only 33.9 percent exact-match on ICD-10-CM and 49.8 percent on CPT, with agreement near zero against a human coder [19]. The fix was architectural rather than a bigger model: a two-stage retrieve-and-rank system hit 100 percent on single-term conditions where vanilla GPT-3.5 managed 6 percent, and a GPT-4o validation system reached 93.6 percent accuracy validating existing codes [20], [21]. Fine-tuned small models show the same pattern, with domain-tuned GPT-4o mini and Llama variants reaching high exact-match rates on constrained generation [20]. So: small model plus retrieval for lookup and normalization; frontier tier only for end-to-end claims adjudication with medical-necessity reasoning, policy interpretation, and appeal drafting, which is multi-hop, high-error-cost, and audit-exposed.

Healthcare, per 1,000 requests (1,200 in / 300 out tokens)
Strategy         Model split                          Cost/1k   vs frontier
---------------  -----------------------------------  --------  -----------
Frontier-only    Opus 4.8 (100%)                      $13.50    baseline
Routed           Haiku 4.5 lookup 90% + retrieval,    $2.66     -80.3%
                 Opus 4.8 adjudication 10%

2. Insurance and reinsurance. First-notice-of-loss triage (structured intake, intent classification, departmental routing) is high-volume, verifiable, and cheap to get wrong per item: small or mid tier with escalation. Treaty language interpretation and exposure analysis is multi-hop synthesis over long, legally binding documents with catastrophic error asymmetry: frontier tier, long context, quality mode, full logging.

3. Banking and payments. Transaction categorization and dispute classification are machine-checkable, extremely high-volume, and latency-sensitive: small tier, or eventually a distilled classifier. Anti-Money Laundering (AML) narrative generation and Suspicious Activity Report (SAR) drafting combine narrative synthesis, evidence selection, and legal exposure: frontier tier with audit logging of the routing decision itself, because a regulator may one day ask why this SAR was drafted by which model at which effort level.

4. Legacy modernization. This one is close to home for anyone running COBOL modernization programs. Line- or paragraph-level COBOL-to-Java syntax translation is a bounded transformation a mid tier handles well. Whole-program dependency analysis, business-rule extraction, and refactoring plans are a different species. IBM’s ICSE 2026 work on enterprise-scale COBOL-to-Java translation is the strongest evidence: rather than throwing a bigger model at whole programs, it combines static program analysis (backward and forward slicing, dependency mapping) with a fine-tuned Granite LLM in a two-phase, semantically driven pipeline, precisely because naive line-by-line translation produces unmaintainable “JOBOL” [22]. The lesson generalizes: repository-scale reasoning needs a program-analysis substrate plus a strong planner; line-level transformation does not.

Legacy modernization, per 1,000 requests
Workload                          Split                       Savings vs frontier
--------------------------------  --------------------------  -------------------
Paragraph-level COBOL -> Java     90% Haiku, 10% Sonnet       ~91%
Repo-scale analysis + refactor    70% Sonnet, 30% frontier    ~60%

5. Customer support. FAQ deflection and intent classification are the canonical small-tier, high-throughput, latency-sensitive case. Multi-system agentic resolution (reading account state, invoking tools, composing remediation) compounds errors across state transitions; the objective shifts from answer quality to safe workflow completion: mid-to-frontier, with cascade-and-verify on tool outputs.

Support, per 1,000 requests (800 in / 250 out tokens)
Strategy         Model split                          Cost/1k   vs frontier
---------------  -----------------------------------  --------  -----------
Frontier-only    Opus 4.8 (100%)                      $10.25    baseline
Routed           Haiku 4.5 deflect 80% +              $2.98     -70.9%
                 Sonnet 5 resolve 20%

6. Software engineering agents: per-role routing. This is the clearest production pattern in the industry today. Per-role routing assigns different models to different roles inside one agentic session: the planner or orchestrator (which decides decomposition, tool choice, and stopping logic) runs on a frontier model, while worker subagents (file summarization, test generation, lint fixing) run on a small model. Claude Code routes its Explore subagent to Haiku by default while the orchestrator runs on Sonnet or Opus; one developer’s telemetry showed roughly 36 percent of subagent calls on Haiku accounting for only 2.3 percent of total cost [23]. Anthropic documents Sonnet decomposing a problem and orchestrating parallel Haiku 4.5 subagents, with Haiku 4.5 achieving 90 percent of Sonnet 4.5’s performance in Augment’s agentic coding evaluation [23]. OpenAI’s reasoning best-practices guidance recommends the same split: strong reasoning models to plan, cost-efficient models to execute [14].

SWE agent, per 1,000 subagent calls (4,000 in / 800 out tokens)
Strategy         Model split                          Cost/1k   vs frontier
---------------  -----------------------------------  --------  -----------
Frontier-only    Opus 4.8 (100%)                      $40.00    baseline
Per-role routed  Haiku 4.5 workers 70% +              $18.16    -54.6%
                 Opus 4.8 planner 30%

7. Document processing. Structured field extraction from forms is verifiable, schema-guarded, and small-tier until proven otherwise. Contract risk analysis interprets obligations, indemnification, and subtle legal intent; it is human-judged, high-error-cost synthesis: frontier tier.

8. Marketing and content. Templated product descriptions are the stable, high-volume cluster that should leave the frontier tier fastest; savings above 90 percent are routine, and an open-weight hosted model often suffices. Brand-voice long-form strategy is human-judged and reputation-sensitive: mid-to-frontier. Note that this is a case where cascades lose, because the first draft is user-visible; more on that in Section 3.

Practitioner takeaway. Route by task shape, never by department name. In every industry pair above, one side was structured, reviewable, or rule-bound and belonged on a cheap tier, while its sibling was ambiguous, long-context, or audit-exposed and belonged on a strong tier. The taxonomy compresses to one question per workload: is the output machine-checkable, and is a wrong answer cheap? Yes to both means push to the smallest tier that passes your eval and verify programmatically. A wrong answer that is expensive or audit-exposed means pay for the frontier tier and log the decision. And remember the medical-coding cautionary tale: the failure was architectural (no retrieval), and no tier upgrade fixed it. Fix the architecture before you fix the model.

Section 3: Routing Architectures, the Ladder from Static to Learned

Most enterprises should climb this ladder in order, earning each rung with evidence, because every step up buys flexibility while adding latency, failure modes, and governance burden.

Rung 1: Static per-endpoint and per-role routing

Static routing is a design-time mapping of tasks to models: a config file, effectively. Intent classification is hardcoded to a fast model; final generation to a stronger one; agent workers to Haiku via an environment variable (the pattern formerly called ANTHROPIC_SMALL_FAST_MODEL, now ANTHROPIC_DEFAULT_HAIKU_MODEL in Claude Code) [23]. Implementation cost is near zero, latency overhead is zero, and it captures the majority of available savings because enterprise traffic is rarely uniformly random. Billing classification stays billing classification. Its other underrated virtue is legibility: auditors and platform teams can read the cost-quality trade-off straight off the config. It fails only when a single endpoint genuinely receives mixed-complexity traffic that cannot be separated at design time.

Rung 2: Heuristic routing

Heuristic routing evaluates explicit structural signals: input length, code presence, tool count, file count, user tier, escalation history, conversation stage. Cheap to compute, transparent, auditable, and often surprisingly informative. Its weakness is brittleness: signals drift when prompts and behavior change, and a short prompt can be very hard.

Rung 3: Learned classifier routing

RouteLLM (Ong et al., ICLR 2025) is the reference work [8]. LMSYS trained four router types on Chatbot Arena preference data: similarity-weighted ranking, matrix factorization, a BERT classifier, and a causal-LLM classifier. The published results, precisely stated: cost reductions of over 85 percent on MT-Bench, 45 percent on MMLU, and 35 percent on GSM8K versus GPT-4-only, while achieving 95 percent of GPT-4’s performance, routing between GPT-4 and Mixtral 8x7B. The matrix-factorization router reached the 95 percent quality point using only 26 percent GPT-4 calls, and with judge-augmented training data that fell to 14 percent. A tunable cost-threshold parameter dials the trade-off, and the routers generalized to an unseen Claude 3 Opus and Llama 3 8B pair without retraining. One instructive detail: the simpler trained routers often beat the causal-LLM router at far lower serving cost. Routing overhead is small relative to generation; converted from the paper’s throughput figures, matrix factorization adds roughly 6 ms per request, BERT roughly 14 ms [8].

Successor work extends the field in four directions. Hybrid LLM routes on predicted query difficulty with a tunable quality target, achieving up to 40 percent fewer large-model calls with no quality drop [9]. RouterDC applies dual contrastive learning and reports gains over the best individual model both in-distribution and out [10]. GraphRouter casts routing as link prediction on a task-query-model graph and reports at least 12.3 percent improvement over prior routers, with gains generalizing to unseen models [11]. AgentRouter maps each step of a multi-step agentic trajectory to one of four tiers with a lightweight 12M-parameter classifier, reporting 72 percent cost reduction versus frontier-only by distinguishing planning steps from formatting steps [31].

Rung 4: LLM-as-router

A cheap model triages difficulty before dispatch. Flexible, needs no training data, but adds a full model call of latency (roughly 430 ms and $0.001 per classification in one commercial implementation) and can itself misjudge [30]. Wrong choice under tight latency budgets or when a trained classifier already covers your distribution.

Rung 5: Cascading with verification

Cascading inverts the prediction problem: instead of guessing the right model before generation, try the cheap model first and escalate only when verification fails. FrugalGPT pioneered the pattern and reported matching GPT-4 with up to 98 percent cost reduction on its benchmarks [12]. AutoMix added few-shot self-verification and POMDP-based escalation, cutting cost by over 50 percent [13]. Recent production-grade work (RLM-Cascade) reports 45.8 percent cost reduction on agentic coding traffic with an 88.8 percent draft-use rate [32].

The verification signal is everything. Schema validation, unit tests, execution traces, and self-consistency checks work when outputs are machine-checkable; a verifier model works when it costs less than the escalations it prevents. Cascades beat predictive routers on checkable outputs because you escalate only on a real failure signal rather than a prediction. They lose in two situations: when the first draft is user-visible (the user already saw the bad cheap answer) and when latency is tight (escalation means two sequential model calls).

Rung 6: Online and adaptive routing

The furthest rung uses contextual bandits and continuous retraining from production signals. OpenAI described the GPT-5 router as continuously trained on model-switch behavior, preference rates, and measured correctness [25]. Router-R1 pushes the research frontier by training the router itself with reinforcement learning, interleaving think and route actions under a cost-aware reward [12b].

And here the industry has its definitive cautionary tale. The GPT-5 launch in August 2025 shipped a faulty router that defaulted too many complex queries to the fast model. Users perceived the new flagship as dumber than its predecessor, the backlash was immediate, Sam Altman conceded publicly that “I think we totally screwed up some things on the rollout,” GPT-4o was reinstated for paid subscribers within days, and by December 2025 OpenAI removed the router for free and low-tier users entirely [25].

The structural lesson is false-negative asymmetry. A false positive (a simple query sent to an expensive model) wastes margin quietly. A false negative (a complex query sent to a weak model) hallucinates, breaks the workflow, triggers retries and escalations, and destroys user trust immediately. The two errors are priced in different currencies, and trust is the expensive one. Every routing threshold you set should be biased toward overspending on ambiguous requests until production evidence licenses tightening.

Failure modes and governance

Routers rot. A threshold tuned for Sonnet 4.5 is silently wrong for Sonnet 5. Routing can defeat prompt-cache hit rates by switching models mid-session (the reason OpenRouter Auto pins models per conversation). Regulated workloads need an audit log of every routing decision. The mitigations are unglamorous and mandatory: an offline eval harness that regression-tests the router on every model or threshold change, canary deployment of routing-policy changes, per-tenant budget enforcement at the gateway, and capture of response.model on every request for attribution [16], [17], [18].

Practitioner takeaway. Climb in order: static, heuristic, then learned or LLM-triage, and only if traffic diversity justifies it. Expect 35 to 50 percent from learned routing on reasoning-heavy or domain traffic, well short of the 85 percent benchmark headline. Whatever you build, bias the boundary toward the frontier model on ambiguity. The GPT-5 rollback is the proof that false negatives cost more than false positives.

Section 4: The Tooling Landscape

The market’s most common architecture error is conflating two product categories. A gateway is plumbing: one unified API across providers, failover, retries, virtual keys, spend tracking, budgets, logs, and caching. A router is intelligence: it classifies requests and selects models. Several tools market both; most do one well. Enterprises almost always need the plumbing before the intelligence.

Tool                  Type        License        Deploy      Classifies?   Overhead    Notes
--------------------  ----------  -------------  ----------  ------------  ----------  ------------------------------
LiteLLM               Gateway     MIT (OSS)      Self-host   No (proxy)    ~2-20 ms    Python; strain past ~500 RPS
Portkey               Gateway+    Apache-2.0     Both        Guardrails    <1-20 ms    PII/jailbreak, semantic cache
OpenRouter            Gateway     SaaS           Managed     Via Auto      40-55 ms    400+ models, 5.5% credit fee
OpenRouter Auto       Router      SaaS           Managed     Yes (NotD.)   40-55 ms    No routing surcharge
RouteLLM              Router      Apache-2.0     Self-host   Yes           ~6-24 ms    ICLR 2025 trained routers
NotDiamond            Router      SaaS           API         Yes           model call  Powers OpenRouter Auto
Martian               Router      SaaS           API         Yes           model call  Underlies Accenture Switchboard
vLLM Semantic Router  Router      Apache-2.0     K8s/Envoy   Yes           low (Rust)  ModernBERT, PII, sem. cache
aurelio sem-router    Router lib  MIT (OSS)      Self-host   Yes (embed)   low         Embedding kNN routing
LLMRouter (UIUC)      Router lab  OSS            Self-host   Yes (16+)     varies      Research library, not turnkey
Bedrock Intel. Route  Router      AWS service    Managed     Yes           low         Within-family only
Azure Foundry router  Router      Azure service  Managed     Yes (ML)      negligible  Balanced/Cost/Quality modes
Vercel AI Gateway     Gateway     SaaS           Managed     No            edge hop    Unified API + failover + rules

A few tools deserve elaboration.

LiteLLM is the archetypal open-source gateway: unified OpenAI-compatible access to over 100 providers, virtual keys, budget limits, fallback chains. The operational fine print matters: you run the PostgreSQL and Redis underneath it, LiteLLM self-reports roughly 2 ms median overhead on a tuned deployment while independent testing puts it at 10–20 ms in practice, and one 5,000-RPS stress test found its Python architecture became unreliable past roughly 500 RPS [27].

Portkey leans enterprise: 360-degree observability, guardrail integrations for PII redaction and prompt-injection defense, configuration-as-code, and semantic caching [28].

OpenRouter is the clearest hybrid, a marketplace broker with a real auto-router. Its economics deserve attention at scale: a 5.5 percent credit-purchase fee (5 percent BYOK) that at $50K per month of spend exceeds the cost of running a self-hosted proxy, which is the standard trigger to self-host [18], [27]. Its session-stickiness documentation is unusually honest about the cache-fragmentation problem most routing abstractions hide.

vLLM Semantic Router (Red Hat, Apache-2.0) is the most complete open-source router for self-hosters: a Go/Rust Envoy ExtProc filter, Kubernetes-native, using fine-tuned ModernBERT for intent classification, with semantic caching, PII detection, and jailbreak detection in one pipeline. Red Hat reports up to 48.5 percent latency and token reductions via automatic reasoning-mode adjustment, and a fine-tuned variant reaching 98.53 percent routing accuracy from 805 training examples; treat these as project-reported until reproduced [26].

RouteLLM and UIUC’s LLMRouter are the research-to-practice bridges. RouteLLM ships trained routers as a drop-in OpenAI client replacement; LLMRouter offers sixteen-plus strategies spanning kNN, matrix factorization, BERT, graph, Elo, and multi-round approaches. Right tools for teams building routing as a capability; wrong tools for teams that need unified billing next quarter [8].

Vendor claims versus the one independent benchmark

Treat every vendor savings percentage as directional until reproduced on your own traffic. Martian’s figures (52.4 percent error-rate reduction, 92 percent cost reduction in a help-chat scenario) are vendor-reported and unverified [30]. NotDiamond’s homepage cites a customer seeing 39 percent accuracy gains, yet the one peer-reviewed cross-router benchmark, RouterArena (ICLR 2026), ranked NotDiamond twelfth of twelve routers evaluated, stating that it “frequently selects expensive models,” and ranked GPT-5’s own router seventh [13b]. RouterArena’s broader findings across roughly 8,000 queries, nine domains, and five metrics: commercial routers do not necessarily beat open-source ones, no router tops all metrics, robustness under query perturbation is a common weak point, and all routers fall short of the oracle mainly by failing to recognize when a cheap model suffices. The field calls this the routing plateau [13b].

The honest costs

Every hop adds latency. The realistic floor is 10–20 ms for a self-hosted gateway hop and 40–55 ms for a managed multi-provider router; across a twenty-call agent chain, a 40 ms floor is nearly a second of added latency. Managed aggregators impose percentage fees that compound at volume. Self-hosting removes intermediary margin and adds provider drift, auth maintenance, logging pipelines, on-call responsibility, and configuration churn as providers rename model IDs monthly. None of these costs appear in benchmark papers, and almost none appear in vendor savings claims [27], [28], [18].

Practitioner takeaway. Buy or build a gateway before you buy or build a router; you need spend tracking and failover regardless of routing sophistication. Self-host once monthly spend passes roughly $50K or when data residency is a hard requirement. And weigh the one independent benchmark more heavily than any vendor deck: it ranked the most-marketed commercial router last.

Section 5: Token-per-Dollar Efficiency Beyond Routing

Routing is one lever among six, and it is usually neither the first nor the highest-ROI one to pull. Ranked by effort-to-savings ratio, highest first:

1. Prompt caching

Prompt caching lets the provider store the processed prefix of a prompt (system instructions, tool schemas, retained documents) and serve subsequent requests against it at a steep discount. Anthropic cuts cache-read input cost by 90 percent (reads at 0.1x base, writes at 1.25x for the 5-minute TTL, break-even after a single hit) with explicit cache_control breakpoints, up to four, that let you cache a retrieved document block separately from the system prompt, a real advantage for RAG workloads. OpenAI's automatic caching discounts cached input 50 to 90 percent depending on model for prefixes of 1,024-plus tokens. Google cuts reads to 10 percent of base plus a per-hour storage fee. DeepSeek's automatic disk cache drops V4-Flash input from $0.14 to $0.0028 per million on hits [1], [2], [3], [5], [28b].

The production evidence is unusually concrete. ProjectDiscovery’s engineering team documented moving dynamic content out of the cacheable prefix and lifting their hit rate from 7 percent to 74 percent in a single deployment, then to 84 percent and beyond, with one task running 67.5 million input tokens across 1,225 steps at a 91.8 percent cache rate, cutting total LLM spend by roughly 59 percent [28b]. On a 50,000-token stable prefix reused across an agent session, caching is the difference between paying full input price on every step and paying 10 percent.

2. Reasoning-budget caps

Covered in Section 1. Set an explicit per-request-type effort policy: low for classification, extraction, and triage; medium by default; high only where evals prove it matters. The diminishing-returns evidence shows most tasks plateau far below maximum effort [24], [21]. This lever defeats no cache and adds no latency, which no router can claim.

3. Batch APIs

Anthropic, OpenAI, and Google all offer a flat 50 percent discount on both input and output for asynchronous processing with results inside 24 hours [1], [2], [3]. Nightly summarization, bulk extraction, evals, and migration prep should assume batch before assuming routing. Zero quality change, near-zero engineering effort, and it stacks with caching.

4. Semantic caching at the gateway

Semantic caching embeds incoming queries and returns a stored answer when a sufficiently similar past query exists, skipping inference entirely. Portkey and the vLLM Semantic Router both provide it; hit rates of 40 to 60 percent are reported on high-similarity traffic like support and FAQ, returning in under 100 ms versus seconds of inference [28], [26]. It is also more dangerous than prompt caching, because it can return a semantically similar but business-wrong prior answer. Scope it to low-stakes, high-similarity workloads and keep it away from policy-sensitive or regulated reasoning.

5. Context pruning and compaction

Long agent loops inflate context with conversational history and tool outputs, and every inflated token is billed on every subsequent call. Aggressive pruning, summarization of past actions, and compaction of tool schemas produce compounding savings; one team reported a 38 percent token reduction simply by capping agentic tool-call iterations at six hops [18b]. An underappreciated corollary: routing a bloated prompt to a cheaper model is often worse than shrinking the prompt and keeping the model.

6. Distillation and fine-tuning

Distillation uses a frontier “teacher” model to generate training traces, fine-tunes a small “student” on them, and migrates a stable task cluster permanently off the expensive tier. Reported cost reductions run 5x to 50x depending on student size; a peer-reviewed study lifted Llama-3.2–3B from under 1 percent GSM8K accuracy to 51–55 percent using frontier traces [29]. The economics only close at volume: a 100M-tokens-per-month workload at frontier prices runs roughly $50K monthly, and a distilled student amortizes within a year only above roughly 50M tokens per month on a stable task. The safe deployment is the cache-and-router pattern: a confidence-aware router sends 85 to 95 percent of inputs to the student and the residual to the teacher, with a clean rollback path and a defensible operating band of two to three points of eval degradation. The decision rule: repetitive, stable, measured workloads get distilled; heterogeneous, rare, changing workloads get routed [29], [20].

The interaction effect nobody models: routing defeats caching

This is the most important and least-discussed trade-off in the entire domain. Prompt caching rewards staying on the same model across a session; a router that switches models mid-session throws the cache away and forces a fresh, often more expensive, cache write on the new model.

Model it explicitly. Take an agent session of ten calls with a 20,000-token stable prefix on Sonnet 5 ($3/M input, cache reads at $0.30/M). With a sticky model and warm cache, calls two through ten pay $0.006 of input each instead of $0.06, a 90 percent saving. If a router switches the session to Opus 4.8 at call five, calls five through ten lose the Sonnet cache and pay a fresh Opus cache write ($0.125 for the prefix alone) plus higher per-token rates, wiping out the routing saving and then some. This is exactly why OpenRouter Auto pins the model per conversation [18].

The operating rule: route at session boundaries, cache within them, and never let a per-turn router run inside an agent loop that depends on a warm cache. For long-lived sessions, a single-model strategy with effort control and prompt caching frequently beats a multi-model strategy even when the cheap model is much cheaper on paper.

Practitioner takeaway. Prompt caching, reasoning caps, and batching are higher-ROI and lower-risk than any router and should be exhausted first; together they routinely deliver 50 to 70 percent savings with no quality risk. Distillation has the highest ceiling and the highest floor on volume. If your platform team is building a classifier before it has enabled caching, it is solving the wrong problem first.

Section 6: Build Specification and Decision Framework

A production routing system is a policy engine with observability rather than a clever prompt trick. Five layers, each constraining a different failure mode: ingress and budgets prevent teams from quietly burning frontier spend; routing policy allocates compute; observability turns cost and quality into signals; offline evaluation catches regressions when models or thresholds change; a lifecycle process keeps policy synchronized with production.

Reference architecture

+-------------------------------+
  App / Agents  -------> |  GATEWAY INGRESS              |
                         |  - unified OpenAI-compat API  |
                         |  - per-team virtual keys      |
                         |  - per-tenant budget caps     |
                         |  - failover / retries         |
                         +---------------+---------------+
                                         |
                         +---------------v---------------+
                         |  ROUTING LAYER                |
                         |  Stage 1: static per-endpoint |
                         |  Stage 2: heuristic signals   |
                         |  Stage 3: classifier / triage |
                         |  (bias boundary to frontier   |
                         |   on ambiguity)               |
                         +---------------+---------------+
                                         |
             +-------------+-------------+-------------+
             v             v             v             v
          Small tier   Mid tier    Frontier tier   Cascade+verify
          (Haiku/      (Sonnet/    (Opus/Fable/    (cheap first,
           Flash/       GPT-5.6     GPT-5.5 Pro)    escalate on
           nano)        Terra)                      failed check)
                                         |
                         +---------------v---------------+
                         |  OBSERVABILITY                |
                         |  - cost/latency/quality/route |
                         |  - capture response.model     |
                         |  - cache hit rate per route   |
                         +---------------+---------------+
                                         |
                         +---------------v----------------+
                         |  OFFLINE EVAL HARNESS          |
                         |  - regression-test router on   |
                         |    every model/threshold change|
                         |  - LLM-as-judge vs baseline    |
                         |  - monthly retrain from traces |
                         +--------------------------------+

At low maturity the routing layer is a rules table in configuration. At high maturity it is a service with a feature extractor, classifier, and policy thresholds. The threshold is the key control: in the research literature it is a cost dial, and in practice it is your organization’s current willingness to overspend to avoid false negatives. That choice belongs to architecture leadership, never to an individual application team.

Staged adoption roadmap

At roughly 100K requests per month. Stand up the gateway, spend attribution, endpoint-to-tier static mapping, prompt caching on stable prefixes, an explicit reasoning-effort policy, and batch for offline jobs. No learned router. This stage typically captures the large majority of available savings; AWS’s own GA data showed 63.6 percent from within-family routing on RAG datasets, and the RouteLLM matrix-factorization results show simple selection recovering most of the gap [16], [8].

At roughly 1M requests per month. Add heuristic routing on strong signals, per-agent-role routing inside agent loops, semantic caching at the gateway, and the offline eval harness. Introduce a learned classifier only on the one or two endpoints with genuinely mixed traffic. Cheap-first cascades belong here wherever outputs are machine-checkable. Expected cumulative savings sit in the 50 to 60 percent band on the right workload mix [8], [9], [12].

At roughly 100M requests per month. Routing becomes platform infrastructure: distill stable clusters above the 50M-tokens-per-month amortization floor onto fine-tuned students, run a trained classifier with monthly threshold retraining from production traces, enforce per-tenant budgets, canary all routing-policy changes, and audit-log routing decisions for regulated workloads. At this scale small threshold improvements are real money, provided the platform team can detect router regressions quickly [29], [26].

Scoring rubric for any new workload

Assign 0 to 2 on each dimension; a higher total pushes toward the frontier tier.

Dimension                      0 (small tier)      1 (mid)        2 (frontier)
-----------------------------  ------------------  -------------  -------------------
Reasoning depth                lookup/extract      single-hop     multi-hop/planning
Output verifiability           machine-checkable   partial        human-judged
Error-cost asymmetry           wrong answer cheap  moderate       wrong answer costly
Latency sensitivity            can wait            interactive    can wait (quality)
Context volume                 <8K                 8K-200K        >200K
Regulatory/audit exposure      none                internal       regulated/audited

Total 0–3: small tier, verify programmatically, consider distillation. Total 4–7: mid tier, cascade-and-verify. Total 8–12: frontier tier, quality mode, full audit logging.

Metrics that matter

Hold the organization to cost per successful task rather than cost per token. A cheap wrong answer that triggers three retries costs more than one frontier call, and cost-per-token dashboards hide that entirely. Supporting metrics: escalation rate (share of cascade calls escalating), router false-negative rate (complex queries sent to weak models, the trust-killer), quality delta versus a frontier baseline via LLM-as-judge on a held-out set, cache hit rate per route, and average routing latency overhead. If leadership wants one north-star number, make it cost per accepted outcome or cost per case closed.

Thresholds that should change your plan mid-flight: if router false-negative rate exceeds a few percent on complex traffic, widen the boundary toward the frontier model immediately. If cache hit rate on an agent endpoint drops below 50 percent, your router is probably switching models mid-session; pin the model per session. If a workload is below 50M tokens per month, skip distillation; it will not amortize. If a learned router’s measured advantage over static tiering is under roughly 15 points on your traffic, keep the static router; the operational complexity is not worth the residual.

Limitations and evidence quality

Three structural caveats apply to everything above. Benchmark contamination and the benchmark-to-domain gap: MT-Bench-style quality does not predict adjudication accuracy, policy compliance, or downstream retry rates, and RouterArena found robustness under query perturbation to be a common router weakness [13b]. Pricing instability: every number here decays; re-verify quarterly at minimum. Directional claims: vendor savings figures (Martian, NotDiamond customer results, the Red Hat benchmarks, the AWS single-scenario numbers) are reported rather than independently reproduced, and the RouteLLM millisecond latencies are derived from published throughput rather than published directly. This report covers text LLM routing; multimodal, embedding-model, and voice routing follow related but distinct economics.

The CFO Summary

AI inference is now one of the fastest-growing and least-governed lines in the engineering budget. The core inefficiency is paying frontier prices for work a mid or small tier handles at one-fifth to one-thirtieth the cost. The right frame is an allocation layer for a fixed intelligence budget rather than another AI component.

The durable, low-risk savings on realistic enterprise traffic are 50 to 70 percent, achieved primarily through three levers: static assignment of each workload to the cheapest model that passes a quality bar, prompt caching (verified in production cutting total spend roughly 59 percent at one engineering organization), and caps on reasoning tokens that bill at premium rates and are invisible in product logs. A staged program captures the bulk of this in the first six weeks with minimal engineering risk. Learned routing extends savings further, but only where an evaluation harness proves quality stays within a business-approved delta of the frontier baseline.

The key financial risk is false economy, and the industry has already run the experiment. When OpenAI’s GPT-5 launch routed complex questions to a weak model to save cost, the backlash forced a public apology and a rollback within four months. Cheap wrong answers create retries, analyst reviews, escalations, and churn that exceed the cost of a frontier call. The router is an optimization engine with asymmetric downside: its job is to cut spend where quality is safely substitutable and to protect spend where failure is expensive. Bias it toward overspending on ambiguity, and hold it to cost per successful task.

Expected outcome: a 50 to 70 percent reduction in inference spend within one quarter at held quality, with a further step-down available at very high volume through distillation, which pays back in weeks to months above roughly 50 million tokens per month.

The Platform Team Build Checklist

FOUNDATION (weeks 1-2)
[ ] Deploy gateway: unified OpenAI-compatible API, failover, retries
[ ] Per-team virtual keys + per-tenant budget caps enforced at ingress
[ ] Capture per request: cost, latency, response.model, cache hit rate
[ ] Establish frontier-only cost baseline for the business case

STATIC TIERING + CACHING (weeks 3-6, captures most savings)
[ ] Map every endpoint to the cheapest tier that passes its eval
[ ] Per-agent-role routing: workers on small model, planner on frontier
[ ] Enable prompt caching on every stable system prompt / tool block
[ ] Set explicit reasoning-effort policy per request type
[ ] Move all latency-tolerant jobs to Batch API (-50%)
EVAL + SELECTIVE ROUTING (weeks 7-12)
[ ] Offline eval harness: LLM-as-judge vs frontier baseline, held-out set
[ ] Regression-test router on EVERY model or threshold change
[ ] Add heuristic routing on strong signals (length, tools, code, tier)
[ ] Learned classifier ONLY on endpoints with mixed-complexity traffic
[ ] Semantic caching at the gateway for high-similarity traffic
[ ] Cascade-and-verify where outputs are machine-checkable
SCALE + GOVERNANCE (quarter 2+, at high volume)
[ ] Distill stable clusters >50M tok/mo onto fine-tuned small models
[ ] Deploy distilled students behind confidence-aware cache-and-router
[ ] Monthly threshold retraining from production traces
[ ] Canary all routing-policy changes; audit-log decisions (regulated)
GUARDRAILS (continuous)
[ ] Alert if router false-negative rate exceeds a few percent
[ ] Alert if agent-endpoint cache hit rate < 50% -> pin model per session
[ ] Never run a per-turn router inside a cached agent loop
[ ] Track cost per SUCCESSFUL task, not cost per token
[ ] Re-verify all provider prices quarterly

If you cannot replay production traces against a frontier baseline and explain the route decision for a regulated case, you do not yet have a production-grade routing layer. Build the governance and the measurement first; the clever classifier can wait.

References & Further Readings

[1] Anthropic, “Pricing,” Claude Platform Docs, verified Jul. 14, 2026. https://platform.claude.com/docs/en/about-claude/pricing

[2] OpenAI, “Pricing,” OpenAI API Docs, verified Jul. 14, 2026. https://developers.openai.com/api/docs/pricing

[3] Google, “Gemini Developer API Pricing,” Google AI for Developers, verified Jul. 14, 2026. https://cloud.google.com/gemini-enterprise-agent-platform/generative-ai/pricing

[4] Google Cloud, “Agent Platform (Vertex AI) Pricing,” verified Jul. 14, 2026.

[5] DeepSeek, “Models & Pricing,” DeepSeek API Docs, verified Jul. 2026. https://api-docs.deepseek.com/quick_start/pricing

[6] Together AI model pricing, via AI Pricing Guru, synced Jul. 9, 2026.

[7] Groq pricing, via AI Pricing Guru, Jun. 2026.

[8] I. Ong et al., “RouteLLM: Learning to Route LLMs with Preference Data,” arXiv:2406.18665, ICLR 2025. https://arxiv.org/abs/2406.18665

[9] D. Ding et al., “Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing,” arXiv:2404.14618, ICLR 2024. https://arxiv.org/html/2404.14618v1

[10] S. Chen et al., “RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models,” arXiv:2409.19886, NeurIPS 2024.

[11] T. Feng et al., “GraphRouter: A Graph-based Router for LLM Selections,” arXiv:2410.03834, ICLR 2025.

[12] L. Chen, M. Zaharia, and J. Zou, “FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance,” arXiv:2305.05176, TMLR 2024.

[12b] H. Zhang, T. Feng, and J. You, “Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning,” arXiv:2506.09033, 2025.

[13] P. Aggarwal et al., “AutoMix: Automatically Mixing Language Models,” NeurIPS 2024. https://arxiv.org/abs/2310.12963

[13b] Y. Lu et al., “RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers,” arXiv:2510.00202, ICLR 2026 poster; and “The Routing Plateau: Understanding and Breaking the Accuracy Limits of LLM Routers,” arXiv:2606.07587.

[14] OpenAI, “Reasoning Models” and “Reasoning Best Practices,” OpenAI API Docs, 2025–2026. https://developers.openai.com/api/docs/guides/reasoning

[16] AWS, “Route Prompts Between Models: Amazon Bedrock Intelligent Prompt Routing,” product page and GA announcement, Apr. 2025. https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html

[17] Microsoft, “Model Router for Microsoft Foundry: Concepts and How It Works,” Microsoft Learn, version 2025–11–18; and “Optimising AI Costs with Microsoft Foundry Model Router,” Microsoft Community Hub, 2026.

[18] OpenRouter, “Auto Router: Intelligent Model Selection” and “Prompt Caching,” OpenRouter Docs, 2026. https://openrouter.ai/docs/guides/routing/routers/auto-router

[18b] OpenAI, “Latency Optimization” and “Model Guidance,” OpenAI API Docs, 2025–2026; Anthropic, “Context Windows,” Claude Platform Docs, 2026.

[19] A. Soroush et al., “Large Language Models Are Poor Medical Coders: Benchmarking of Medical Code Querying,” NEJM AI, 2024. https://ai.nejm.org/doi/abs/10.1056/AIdbp2300040

[20] Z. Hou et al., “Enhancing Medical Coding Efficiency Through Domain-Specific Fine-Tuned Large Language Models,” npj Health Systems, 2025; and “Large Language Models Are Good Medical Coders, If Provided with Tools,” arXiv:2407.12849, 2024.

[21] “Validation of 13,102 ICD-10-CM Codes Using an LLM-Based System,” medRxiv, 2025; T. Han et al., “Token-Budget-Aware LLM Reasoning” (TALE), Findings of ACL, 2025.

[22] V. Chakravarthy et al., “Enterprise-Scale COBOL-to-Java Translation: LLMs Augmented with Program Analysis,” IBM Research, ICSE 2026; “Quality Evaluation of COBOL to Java Code Transformation,” arXiv:2507.23356; Thoughtworks, “Legacy Modernization Meets GenAI,” 2024.

[23] Anthropic, “Introducing Claude Haiku 4.5,” Oct. 15, 2025; “Model Configuration,” Claude Code Docs; “Effort” and “Choosing the Right Model,” Claude Platform Docs, 2026.

[24] “Do LLMs Overthink Basic Math Reasoning?,” arXiv:2507.04023, 2025.

[25] OpenAI, “Introducing GPT-5,” Aug. 7, 2025; Fortune, “GPT-5’s Model Router Ignited a User Backlash,” Aug. 12, 2025; The Decoder, “GPT-5 Router Rollback,” Dec. 2025; The Verge, Altman remarks, Aug. 2025.

[26] Red Hat, “vLLM Semantic Router,” blog and Red Hat Developer articles, 2025–2026; “When to Reason: Semantic Router for vLLM,” arXiv:2510.08731; “98x Faster LLM Routing Without a Dedicated GPU,” arXiv:2603.12646.

[27] BerriAI, “LiteLLM,” GitHub and docs, 2026; “We Benchmarked 5 LLM Gateways at 5,000 RPS,” DEV Community, 2026; OpenRouter pricing documentation.

[28] Portkey, “AI Gateway,” docs and gateway repository, 2026.

[28b] Artificial Analysis, “Prompt Caching: Cost & Performance Analysis Across Providers”; ProjectDiscovery Engineering, “How We Cut LLM Costs With Prompt Caching,” 2026.

[29] “The Distillation Game,” arXiv:2605.22737, 2026; TensorZero and Snorkel AI distillation cost-quality evidence.

[30] NotDiamond, homepage, docs, and Rootly case study, 2026; Martian, product announcements and Accenture investment announcement, 2024–2026; MorphLLM router documentation.

[31] “AgentRouter: Heterogeneous Model Routing for Cost-Optimal Multi-Step Agentic Workflows,” OpenReview, 2026.

[32] “RLM-Cascade: Response-Level Speculative Decoding for Cost-Efficient LLM API Serving,” arXiv:2606.22840, 2026.