Design an LLM gateway

30-second answer

• The gateway is a thin reverse proxy (sub-50ms overhead) between all internal services and LLM providers. It centralizes model routing, rate limiting, cost tracking, and failover so individual teams do not manage provider integrations themselves.
• Intelligent routing with a lightweight classifier saves 40-60% on LLM spend by sending simple lookups to GPT-4o-mini ($0.15/1M input) and reserving GPT-4o ($2.50/1M input) or Claude Opus ($15/1M input) for complex reasoning tasks.
• Token-budget rate limiting beats request-count rate limiting. One request with 100K tokens costs 1,000x more than one request with 100 tokens. The rate limiter must track both request count (to protect the gateway) and token budget (to protect the budget).
• Circuit breaker failover with model equivalency mapping (GPT-4o to Claude 3.5 Sonnet, GPT-4o-mini to Claude Haiku) keeps traffic flowing during provider outages. Each provider averages 2-4 incidents per month lasting 5-30 minutes.
• A key operational detail is that streaming complicates accounting. You do not know the output token count until the stream finishes, so rate limiting, cost tracking, and timeout handling all need post-stream reconciliation.

Requirements and assumptions

Functional requirements

• The gateway proxies all LLM API calls from internal services to external providers (OpenAI, Anthropic, Google) through a unified API interface.
• The gateway routes each request to the cheapest capable model based on query complexity classification (simple, medium, hard).
• Each team has a configurable token budget (daily and monthly caps) with automatic enforcement and alerting at 80% usage.
• The gateway retries failed requests against an alternate provider when the primary provider returns 5xx errors or times out.
• The gateway logs every request with latency, input/output tokens, cost, model used, and team ID for observability and audit.
• The gateway supports a semantic cache that returns cached responses for identical or near-identical prompts.

Non-functional requirements

• Gateway routing overhead: P99 under 50ms (excluding LLM inference latency).
• Throughput: 10,000 LLM requests per minute from all teams combined.
• Availability: 99.95% uptime. A gateway outage blocks all AI features across the company.
• Cost reduction: at least 40% savings compared to every team using GPT-4o for all requests.
• Failover time: under 2 seconds to detect a provider failure and route to the backup.
• Cache hit rate: 30-50% for support and FAQ workloads, reducing both latency and cost.

The hardest engineering problem here: streaming breaks your accounting. When a request uses server-sent events, you know the input tokens up front but the output tokens trickle in one chunk at a time. Your rate limiter must reserve a pessimistic token budget on entry, then reconcile the actual usage after the stream completes. Get this wrong and teams either blow past their budgets or get blocked unnecessarily.

Assumptions

• Internal callers use one stable request and streaming contract; provider-specific formats remain behind adapters.
• Team budgets are policy inputs, not a substitute for infrastructure protection. Request-rate limits, token reservations, spend caps, and provider quotas are separate controls.
• Provider credentials, routing rules, cache keys, and audit records are managed by the gateway; application teams do not need direct provider access for normal traffic.

5-minute approach

Design the gateway as a small, observable control plane on the request path: normalize the request, authenticate and rate-limit it, choose a capable provider/model, reserve budget, call through an adapter, and normalize the response.

• Keep the synchronous path short: cache lookup, routing, token reservation, provider call, stream normalization, and usage reconciliation.
• Keep health polling, model equivalence, budget reports, prompt analysis, and routing-policy updates off the hot path.
• Make retries provider-aware and idempotent; do not duplicate a non-idempotent tool call merely because a stream ended ambiguously.
• Treat cache policy, model fallback, and downgrade behavior as per-team configuration with audit trails.

The entities and endpoints below define the gateway contract before the provider-specific details.

Core entities

GatewayRequest

• request_id, team_id, model_requested, model_routed, prompt_hash, input_tokens, estimated_output_tokens, priority, received_at

GatewayResponse

• response_id, request_id, provider_used, model_used, output_tokens, latency_ms, cost_usd, cache_hit (boolean), completed_at, status (success/failed/fallback)

TeamBudget

• team_id, daily_limit_tokens, monthly_limit_tokens, daily_used_tokens, monthly_used_tokens, burst_allowance, auto_downgrade_enabled, alert_threshold_pct

ProviderHealth

• provider_id, endpoint, error_rate_1m, error_rate_5m, p50_latency_ms, p99_latency_ms, circuit_state (closed/open/half-open), last_failure_at, last_success_at

ModelEquivalency

• model_a, model_b, task_compatibility_score, format_adapter (openai-to-anthropic, etc.), prompt_rewrite_required (boolean)

CacheEntry

• cache_key (prompt_hash + model + temperature), embedding_vector, response_text, token_count, created_at, ttl, hit_count

API design

POST /v1/chat/completions - unified LLM inference endpoint (OpenAI-compatible)

Request: {
  "model": "auto",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "What is the capital of France?" }
  ],
  "temperature": 0.0,
  "stream": true,
  "metadata": {
    "team_id": "team-search",
    "priority": "high",
    "bypass_cache": false
  }
}
Response (non-streaming): {
  "id": "gw_req_abc123",
  "model": "gpt-4o-mini",
  "routed_from": "auto",
  "provider": "openai",
  "choices": [{ "message": { "content": "The capital of France is Paris." } }],
  "usage": { "input_tokens": 28, "output_tokens": 9, "cost_usd": 0.0000096 },
  "gateway_metadata": {
    "routing_decision": "simple_query",
    "cache_hit": false,
    "gateway_latency_ms": 12,
    "total_latency_ms": 340
  }
}

The unified endpoint keeps the OpenAI chat completions format so teams can switch to the gateway by changing one base URL. The model: "auto" option triggers intelligent routing. Teams can still pin a specific model if needed.

GET /v1/teams/{team_id}/usage - query budget usage and remaining allocation

Response: {
  "team_id": "team-search",
  "daily": { "used_tokens": 4200000, "limit_tokens": 10000000, "pct_used": 42 },
  "monthly": { "used_tokens": 98000000, "limit_tokens": 500000000, "pct_used": 19.6 },
  "top_models": [
    { "model": "gpt-4o-mini", "requests": 12400, "tokens": 3100000, "cost_usd": 1.86 },
    { "model": "gpt-4o", "requests": 340, "tokens": 1100000, "cost_usd": 13.75 }
  ],
  "auto_downgrade_active": false
}

Teams need visibility into their own usage. This endpoint powers dashboards and budget alerts. Finance uses the aggregated version across all teams for forecasting.

GET /v1/providers/health - provider health status for ops dashboards

Response: {
  "providers": [
    { "provider": "openai", "status": "healthy", "error_rate_1m": 0.2, "p50_latency_ms": 280, "circuit": "closed" },
    { "provider": "anthropic", "status": "healthy", "error_rate_1m": 0.1, "p50_latency_ms": 310, "circuit": "closed" },
    { "provider": "google", "status": "degraded", "error_rate_1m": 8.5, "p50_latency_ms": 1200, "circuit": "half-open" }
  ]
}

Used by the on-call team to see provider status at a glance and understand routing decisions during incidents.

DELETE /v1/cache - invalidate cache entries (admin only)

Request: {
  "scope": "team",
  "team_id": "team-search",
  "older_than": "2026-04-01T00:00:00Z"
}
Response: { "entries_deleted": 1247 }

Cache invalidation for when a team changes their system prompt or when a provider updates model behavior after a version bump.

45-minute interview approach

Keep the discussion about a gateway design question: centralize controls without turning the proxy into a second application platform.

• 0–5 min — Clarify scope: ask which providers and APIs are supported, whether streaming and tools are required, who owns budgets, and what failover behavior is acceptable.
• 5–10 min — Requirements and estimates: state request rate, token-rate distribution, routing overhead, availability, failover time, cache hit target, and savings goal.
• 10–18 min — Interfaces and data model: define the normalized request/response, provider adapter, team policy, budget ledger, health record, cache entry, and usage event.
• 18–28 min — High-level design: draw auth, request limits, token reservation, router, cache, provider adapters, circuit breakers, stream normalization, usage store, and health controller.
• 28–38 min — Deep dive: explain token counting, semantic-cache safety, provider equivalence, retries, streaming reconciliation, and 429 versus 5xx handling.
• 38–42 min — Scale and operations: cover regional gateway replicas, hot configuration, provider quotas, queue/backpressure, chargebacks, tracing, and secret rotation.
• 42–45 min — Trade-offs and close: compare central control with added latency and blast radius, state graceful degradation, and identify which policy is safest to fail closed.

High-level design and data flow

The gateway sits between every internal service and every LLM provider. Think of it as an API gateway specifically designed for LLM traffic, with three capabilities that a generic gateway (Kong, Envoy) cannot provide: intelligent model routing based on query complexity, token-aware rate limiting, and cross-provider failover with prompt format adaptation.

The architecture has two planes. The data plane handles the hot path: receive request, check cache, route to provider, stream response back. This must be fast (sub-50ms overhead). The control plane handles configuration, health monitoring, and budget enforcement. This runs asynchronously and updates the data plane through shared state in Redis.

Building this inside an existing API gateway with custom plugins can work initially, but streaming support, token counting, and model-equivalency mapping are LLM-specific concerns that often justify a dedicated service.

Draw this diagram early and name the capabilities that make this more than a generic API gateway: routing, token-aware controls, provider adaptation, and observability.

Here is the core request flow animated step by step. Notice how the gateway adds minimal overhead on the hot path (auth check, rate limit lookup, cache check, route decision) and handles the expensive work (provider call, logging) without blocking the response.