Design LLM inference serving

30-second answer

• Two request tiers require fundamentally different infrastructure: interactive (TTFT under 1s, streaming) and batch (maximize throughput, no streaming needed).
• Route requests to the cheapest capable model using a lightweight classifier. Complex queries go to Sonnet/Opus; simple lookups go to Haiku.
• KV cache management with PagedAttention (as in vLLM) is a major GPU-efficiency lever. Shared prefix pages across requests with long system prompts can cut memory use by 40-60%.
• Autoscale on GPU memory utilization plus queue depth, not CPU. GPU memory at 70% means scale up; at 20% means scale down.
• Cost allocation by team makes model usage visible. Without it, teams may default to the most expensive model because the trade-off is hidden.

Requirements and assumptions

Functional requirements

• 10 product teams can make LLM API calls to models including GPT-4o, Claude 3.5 Sonnet, and a self-hosted Llama 3 70B.
• Interactive requests stream responses (token-by-token) with TTFT under 1 second.
• Batch requests run in the background with no streaming requirement; throughput-maximized.
• Teams can specify model preference or allow automatic routing based on query complexity.
• Per-team usage (tokens in/out, cost) is tracked and reported daily.

Non-functional requirements

• 10K concurrent users across all teams, with individual team peaks up to 2K.
• Interactive tier: TTFT under 1s at P90, total response time under 10s.
• Batch tier: throughput maximized; 24-hour completion guarantee for queued jobs.
• 99.9% availability for the interactive tier. Batch tier allows degraded operation (slower queues) during incidents.
• Self-hosted models must cost at least 40% less than equivalent commercial API spend at scale.

Assumptions

• Interactive and batch workloads have different SLOs and run on separate capacity pools, even if they share model weights or a gateway contract.
• The serving layer owns scheduling, batching, GPU memory, and streaming; product teams do not manage individual GPU processes.
• Self-hosted inference is used only where its utilization, quality, and operational cost justify it. A commercial API remains an explicit fallback for interactive continuity.

5-minute approach

Start by splitting the workload into interactive and batch tiers. Route each request to a capable model, admit it through a queue, use continuous batching and paged KV-cache allocation on the GPU, stream interactive output, and measure both latency and utilization.

• Put authentication, quotas, model routing, and usage accounting at the gateway boundary.
• Use continuous batching so new sequences join between decode steps; use paged KV-cache management so memory follows actual sequence length.
• Scale on queue depth, time-to-first-token, tokens per second, and GPU memory—not CPU alone.
• Keep batch work interruptible and checkpointed so spot capacity can be used without risking the job's result.

The entities and API below establish the request, worker, batch, and usage boundaries.

Core entities

InferenceRequest

• request_id, team_id, model_preference, tier (interactive/batch), prompt_tokens, max_completion_tokens, stream, priority, created_at

ModelInstance

• instance_id, model_id, gpu_node_id, status (active/draining), queue_depth, memory_utilization_pct, requests_per_second

UsageRecord

• record_id, team_id, request_id, model_id, input_tokens, output_tokens, cost_usd, latency_ms, timestamp

ModelConfig

• model_id, provider (openai/anthropic/self-hosted), tier_eligibility[], cost_per_1k_input, cost_per_1k_output, max_context_tokens, routing_weight

API design

POST /v1/chat/completions (unified inference endpoint, penAI-compatible)

Request:  {
  "model": "auto",
  "messages": [{"role": "user", "content": "Summarize this document..."}],
  "stream": true,
  "x-tier": "interactive",
  "x-team-id": "team_payments"
}
Response: SSE stream of { "delta": { "content": "token" }, "model_used": "claude-haiku-3" }

POST /v1/batch (submit background batch job)

Request:  { "requests": [...], "callback_url": "https://...", "priority": "normal" }
Response: { "batch_id": "batch_abc", "estimated_completion": "2026-04-05T14:00:00Z" }

GET /v1/usage?team_id=team_payments&date=2026-04-05

Response: { "team_id": "team_payments", "input_tokens": 4200000, "output_tokens": 840000, "cost_usd": 12.34 }

45-minute interview approach

Treat this as an inference-serving design question. Establish the workload split and latency budget before discussing GPU optimizations.

• 0–5 min — Clarify scope: ask which models, context lengths, streaming requirements, workload mix, regions, and hardware constraints are in scope.
• 5–10 min — Requirements and estimates: estimate concurrent sequences, input/output token rates, TTFT and completion SLOs, batch deadlines, GPU memory, and cost targets.
• 10–18 min — Interfaces and data model: define completion requests, model tiers, queue jobs, worker leases, token usage, latency samples, and team chargeback records.
• 18–28 min — High-level design: draw gateway, router, interactive/batch queues, schedulers, GPU workers, KV cache, streaming path, usage store, and fallback provider.
• 28–38 min — Deep dive: explain continuous batching, PagedAttention, speculative decoding, topology-aware parallelism, and admission control under memory pressure.
• 38–42 min — Scale and operations: cover autoscaling signals, heterogeneous GPU pools, spot preemption, model rollout, queue fairness, and observability.
• 42–45 min — Trade-offs and close: compare self-hosted versus managed inference and throughput versus tail latency; state how a saturated or unhealthy pool degrades.

High-level design and data flow

All requests enter through an API gateway that handles authentication, rate limiting, and tier classification. The gateway stamps each request with the team ID and routes it to the model router. The model router makes two decisions: which model (based on complexity and cost), and which serving pool (interactive or batch) to send the request to.

Self-hosted models (Llama 3 70B) run on a dedicated GPU cluster managed by vLLM. Third-party models (GPT-4o, Claude) are proxied through their commercial APIs with retry logic and circuit breakers. The gateway presents a unified OpenAI-compatible interface regardless of the backend, so teams don't need to change code when the routing changes.

The serving layer is the part that changes fundamental GPU economics. vLLM's continuous batching replaces static batching: as soon as any sequence in a batch finishes generation, a new request is inserted into the freed slot. GPU utilization jumps from 40-60% (static batching) to 70-90% (continuous batching). Combined with PagedAttention's memory management, you can serve 4-6x more concurrent requests on the same GPU fleet.

Inference request lifecycle

This animation traces a single interactive request from arrival to final response. Watch how the KV cache hit on the shared system prompt prefix eliminates 40% of prefill work, and how continuous batching lets the request join a running batch mid-decode rather than waiting for a new batch to form.