LLM routing and model selection

TL;DR

• LLM routing sends each query to the cheapest model capable of handling it, instead of routing everything to the most expensive model.
• A simple cascade (try GPT-4o-mini first, escalate to GPT-4o if confidence is low) can reduce API cost on heterogeneous workloads, but savings and quality loss depend on the escalation rate and threshold.
• Routing can be rule-based (query length, keyword triggers), classifier-based (trained on historical quality data), or LLM-judged (a small model decides which large model to use).
• Martian, Unify, and open-source routers like RouteLLM provide automated routing across providers. Many teams build custom routers tuned to their specific quality/cost tradeoff.
• The key metric is the quality-cost Pareto curve: plot quality (eval score) against cost per query, and find the configuration that maximizes quality per dollar.

30-Second Explanation

Mental model: a router is a policy that chooses a model—or declines or escalates—using a query’s expected quality, cost, latency, and availability. The right threshold comes from evals and production feedback; a cascade is a trade-off, not a free quality win.

The problem it solves

Suppose an AI-powered document analysis feature processes 500K queries per month. In an illustrative pricing scenario, routing everything to GPT-4o at $5.00 per million input tokens puts the monthly bill at roughly $15,000. Query analysis shows that 75% are simple lookups ("what is the sender's name on this invoice?") that GPT-4o-mini handles with equivalent accuracy on the measured eval. GPT-4o-mini is modeled here at $0.15 per million tokens, 33x cheaper; replace these figures with current provider rates.

If you routed 75% of queries to GPT-4o-mini, the illustrative calculation cuts costs from $15K to roughly $4K per month. The result is useful only if the quality comparison holds on representative traffic and includes router, retry, and escalation costs. Teams can pay a substantial premium for frontier models when they do not route by query complexity, but the size of that premium varies.

Here is the before state: every query flows to the most expensive model.

After routing, simple and complex queries each reach the right model tier.

The routed system costs roughly $4K per month instead of $15K under these assumptions, a 73% reduction if the simple-majority quality result and all other cost assumptions hold.

What is it?

LLM routing is the practice of dispatching each query to the least costly model expected to meet the quality bar for that specific query. Instead of using one model for everything, a router sits in front of a pool of models and makes a per-request model selection decision, sometimes including refusal or escalation.

Think of triage in an emergency room. A triage nurse assesses each patient's severity and routes them to the appropriate care level: the general practitioner, urgent care, or the emergency trauma team. Not every patient needs the trauma team, and sending everyone there wastes resources and slows care for genuine emergencies. The nurse makes a fast, imperfect assessment that optimizes resource usage without sacrificing quality for those who truly need the highest level of care.

In LLM routing, the "nurse" is a lightweight decision layer (a classifier, a confidence check, or an embedding comparison) that assesses query complexity and selects the appropriate model tier.

How it works

Cascade routing (small-to-large)

Cascade routing tries the cheapest model first, then assesses the response quality. If the quality meets the threshold, the response is returned immediately. If it does not, the router escalates to the next tier.

Quality assessment options:

• Logprob-based confidence: compute the mean log probability of generated tokens, convert to a probability. Low confidence signals a hard query.
• Separate quality classifier: a small model trained to judge output quality without generating a full response.
• Format checks: if the model refuses to answer or produces a generic non-response, escalate.

The cascade terminates as soon as a satisfactory response is found. Typical cascade: GPT-4o-mini then GPT-4o, or Claude Haiku then Claude Sonnet then Claude Opus.

Latency may improve on the majority path. If 75% of queries resolve at tier 1 and GPT-4o-mini is faster, those queries can respond sooner. The escalated 25% pay tier-1 plus tier-2 latency, so P99 can worsen even while P50 improves. Monitor both percentiles and the escalated subset.

# Simplified confidence-based cascade router
import math

async def cascade_router(query: str) -> str:
    # Try cheap model first
    response = await call_model("gpt-4o-mini", query)

    # Check confidence via logprob sum
    avg_logprob = sum(response.logprobs) / len(response.logprobs)
    confidence = math.exp(avg_logprob)  # convert to probability

    if confidence >= 0.85:
        return response.content

    # Escalate to expensive model
    return await call_model("gpt-4o", query)

Classifier-based routing

Classifier-based routing trains a lightweight model (a fine-tuned BERT-scale classifier, roughly 110M parameters) to predict which tier should handle a given query. The classifier runs before any LLM inference, adding roughly 5ms of overhead with no first-pass inference cost wasted.

Training data comes from historical production logs: annotated queries where you know which model tier produced acceptable quality. Build a golden evaluation set through human review, then use those labels to train the classifier. Features that work well include query length, presence of code, numerical reasoning signals, domain-specific keywords, and entity count.

Classifier routing can beat a cascade when the escalation rate is high. If data shows 60% of queries would escalate, a classifier may avoid paying for tier-1 inference on those requests upfront, but classifier training, inference, and errors add their own costs.

Semantic routing (content-based)

Semantic routing embeds the query using a cheap embedding model and compares it to category centroid vectors derived from labeled training examples. The router selects the most similar category centroid and dispatches to the specialized model for that domain.

This is most valuable when domain-specific models perform well on the relevant evals and cost less per token. A medical-query fine-tuned model may deliver better accuracy on some clinical questions than a general model while running on smaller, faster hardware, but high-stakes use still needs domain review and safety controls.

The overhead is 2-10ms for the embedding call plus cosine similarity computation. Categories with high embedding-space separation (medical vs. legal vs. code) get clean routing. For ambiguous queries near category boundaries, keep a fallback to a general-purpose model using a minimum-similarity threshold.

Multi-provider routing

Multi-provider routing can dispatch queries across OpenAI, Anthropic, Google, and self-hosted models based on live signals such as current cost, latency, availability, and capability match. A configuration might use a long-context model for long documents, a model with strong structured-output support for extraction, and another model for nuanced reasoning. These are hypotheses to validate rather than fixed provider capabilities.

Provider circuit breakers and rate-limit distribution are common reliability components. When a provider's error rate exceeds a threshold, a circuit breaker can open and reroute traffic to an approved fallback. Rate-limit distribution reduces the chance of hitting one provider's per-minute cap, but failover must preserve policy, data-handling, and quality requirements.