Most organisations route 100% of their LLM requests to their most capable — and most expensive — model. The cost case for intelligent routing is compelling: the same output quality at 15% of the price.
The Express Courier Problem
Imagine a logistics company that routes every package — a birthday card, a legal contract, a same-day medical sample — via overnight express courier. The reasoning is straightforward: express is the best option, and we cannot afford for anything to go wrong. The result is a cost structure that bears no relationship to the actual value of what is being shipped.
Most organisations’ LLM usage looks exactly like this. Every request — whether it is a complex legal document analysis, a simple FAQ response, or an internal status update — goes to the same frontier model. The model handles all of them equally well. The invoice reflects this equally.
The analogy breaks down only in one direction: with packages, there is at least a reason to worry about what happens if the birthday card goes by standard post. With LLM requests, the vast majority do not need the most capable model — and the output quality difference, when routing is done carefully, is effectively undetectable.
The Price Gap Is Large
To understand why routing matters, it helps to look at the actual numbers. GPT-4o, a frontier model, is priced at $2.50 per million input tokens and $10.00 per million output tokens. GPT-4o-mini, the smaller member of the same model family, is priced at $0.15 per million input tokens — a 16-fold difference.
The gap widens further when EU-based cloud models (such as Mistral’s offerings, hosted in EU data centres) are considered for general internal queries, and widens to near-zero when on-premises locally hosted models handle requests that must not leave the corporate network at all.
These are not marginal differences. A company spending €5,000 per month on LLM APIs today — a realistic figure for a mid-sized organisation with moderate AI usage — could reach the same output for approximately €750 per month with appropriate routing. The €4,250 monthly saving represents not a compromise on quality but a correction of a pricing mismatch. This estimate is based on the RouteLLM study.
How Routing Works
Intelligent model routing is a classification problem applied before a request reaches any LLM. Each incoming request is evaluated on two axes: its complexity (does it require nuanced reasoning, or is it a well-scoped, predictable task?) and its data sensitivity (does it contain personally identifiable information, confidential business data, or content that must not leave the corporate perimeter?).
Based on this classification, the request is directed to the appropriate tier:
| Request class | Target model | Cost per 1M tokens | Typical share |
|---|---|---|---|
| Confidential / PII | Local LLM (on-premises) | €0 | ~40% |
| Internal / general queries | EU Cloud (e.g. Mistral) | €0.30–1.00 | ~40% |
| Simple / public tasks | US mini model | €0.10–0.20 | ~20% |
The confidential tier deserves particular attention. Requests containing customer data, personal identifiers, or proprietary business information should not traverse external networks at all — regardless of cost. Routing them to an on-premises model simultaneously eliminates cost (no API charges) and eliminates the data transfer risk that GDPR and the EU AI Act are both concerned with. It is the rare case where the cheaper option is also the more compliant one.
The general internal tier covers the largest volume of typical enterprise LLM usage: drafting documents, answering internal queries, summarising meeting notes, generating code. These requests do not contain sensitive data, but they also do not require frontier-model reasoning. EU-based cloud models at a fraction of the cost handle them with output quality that is functionally indistinguishable for these use cases.
The simple public tier — product descriptions, FAQ responses, content reformatting — is where the cheapest available models are entirely appropriate. The tasks are well-defined and low-stakes; the only argument for routing them to a frontier model is habit.
The Evidence Base
This is not a theoretical claim. RouteLLM, a research project from UC Berkeley and LMSYS published in 2024, demonstrated empirically that intelligent routing between a strong and a weak model can achieve 85% cost savings while maintaining 95% of frontier-model output quality across standard benchmarks. The key finding is that the majority of queries in real enterprise workloads are not at the capability frontier of current models — they are well within the reach of smaller, cheaper alternatives.
The 85% figure cited here corresponds to a routing policy calibrated for quality preservation. Organisations willing to accept a modest quality trade-off on genuinely simple tasks can achieve higher savings; those with quality-critical workflows can be more conservative in their routing thresholds and still achieve savings of 60–70%.
Return on Investment
The investment required for intelligent routing is the gateway layer that sits between the organisation’s applications and the LLM APIs. This gateway must classify incoming requests, enforce routing decisions, and (for the confidential tier) maintain the on-premises inference infrastructure.
For an organisation spending €5,000/month on LLM APIs, the routing-adjusted cost is approximately €750/month — a saving of €4,250/month or €51,000 per year. At typical infrastructure and implementation costs, the investment pays back within four to eight weeks from cost savings alone, before any security or compliance benefit is counted.
This is an unusual characteristic for a security infrastructure investment. Most security controls are costs — necessary ones, but costs nonetheless. A routing gateway is one of the few infrastructure components that generates a measurable financial return while simultaneously reducing risk exposure.
The Natural Fit with Security
There is a structural reason why cost routing and security enforcement belong in the same infrastructure layer. Both require a gateway that sees every request before it reaches the LLM. Both require request classification — security enforcement needs to know what the request contains for threat detection, and routing needs to know what it contains for model selection. Both benefit from comprehensive logging of every request and response.
A gateway that classifies requests for routing purposes already has the information needed to apply content filtering, PII detection, prompt injection screening, and output sanitisation. The security controls do not add a separate inspection step — they run on the same classification pass that routing already requires.
This means that an organisation implementing routing for cost reasons gets security enforcement as a structural by-product, and an organisation implementing a security gateway gets intelligent routing as a structural capability. The two goals, which might appear to require separate investments, are in practice served by the same infrastructure component.