Inference optimization

TL;DR

• Two useful latency metrics for streaming LLM UX are TTFT (Time to First Token) and TBT (Time Between Tokens). Targets such as 500ms TTFT and 50ms TBT are product-specific goals, not universal requirements.
• KV caching stores attention key-value tensors so each new token only computes against cached context, not the full sequence from scratch.
• Continuous batching can improve GPU utilization over naive static batching by inserting new requests as old ones finish; the gain depends on traffic and output-length variance.
• PagedAttention manages KV cache memory in pages, reducing fragmentation and pre-allocation waste; the size of the saving depends on workload and implementation.
• Speculative decoding uses a small draft model to propose tokens verified in parallel by the target model; speedups vary with draft-model acceptance and hardware.
• KV caching and continuous batching are common serving baselines for high-throughput workloads, but the right stack depends on traffic, latency goals, and hardware.

---

30-Second Explanation

Mental model: serving has a prefill phase that processes the prompt and a decode phase that generates tokens. KV caching, batching, memory paging, parallelism, and speculative decoding reduce repeated work or improve hardware utilization, but the best combination depends on traffic, model, hardware, and latency targets.

The problem it solves

Your chatbot works great in development. Single-user latency is acceptable, the model fits in VRAM, and responses stream smoothly. Then you launch to 1,000 concurrent users and everything breaks.

A 70B model generates tokens one at a time. Each token requires a full forward pass through all 80 layers, reading tens of gigabytes of weight data from GPU memory. If you serve requests sequentially, the GPU spends most of its time waiting for memory reads, not computing. Utilization sits at 5-10%. You're paying $3/hour for an A100 that's 90% idle.

This wall can appear quickly after an LLM-powered product launches. As an illustrative calculation, if a 70B model produces 30 tokens/second and each response uses 50 tokens, one worker handles about 0.6 requests/second; capacity for 100 concurrent generations then depends on batching, hardware, utilization, and queueing, so a simple GPU count is only a rough estimate.

The root cause isn't compute speed. Modern GPUs are enormously powerful. The problem is utilization: naive inference wastes most of that power on memory stalls and idle cycles. Inference optimization is the set of techniques that close this gap.

---

What is it?

Inference optimization is the collection of techniques that reduce latency, increase throughput, and lower cost when serving LLMs in production. The techniques compose: a modern serving stack like vLLM applies KV caching, continuous batching, PagedAttention, and prefix caching simultaneously. Understanding each one separately is the prerequisite to reasoning about tradeoffs.

Think of it like a restaurant kitchen. A raw LLM is a chef who cooks one dish start-to-finish before taking the next order (sequential inference). Inference optimization is everything that turns that into a real kitchen: prep stations (KV cache), taking new orders as plates go out (continuous batching), efficient shelf space for ingredients (PagedAttention), and a sous chef who preps dishes the head chef just has to approve (speculative decoding).

Two metrics anchor every optimization decision. If an interviewer asks about LLM serving, start with these.

---

How it works

Latency metrics: TTFT and TBT

Time to First Token (TTFT) measures how long after sending a request the user sees the first token appear. This is the "thinking" delay. For interactive chat, TTFT above 2 seconds feels broken. Competitive production targets are under 500ms.

Time Between Tokens (TBT) is the interval between each streamed token. Human reading speed is roughly 250 words per minute. At 1.3 tokens per word, that's about 5.4 tokens/second, or ~185ms between tokens. Anything under 50ms TBT looks instantaneous to users. Above 100ms, streaming starts to feel choppy.

These metrics create different optimization pressures. TTFT is dominated by the prefill phase (processing the entire input prompt at once). TBT is dominated by the decode phase (generating one token at a time). Some optimizations help one metric but hurt the other.

For your interview: TTFT and TBT are the first thing you name when discussing LLM serving. They show you understand the user experience side, not just the infrastructure side.

KV cache

LLMs generate tokens autoregressively: each new token depends on all previous tokens. The transformer's attention mechanism computes query, key, and value vectors for the current token, then attends over the keys and values of all prior tokens. Without caching, you'd recompute K and V for tokens 1 through N-1 every time you generate token N.

KV cache stores the key and value tensors from every prior attention computation. When generating token N, the model reads cached K/V for tokens 1 through N-1 and only computes new K/V for token N. The computation per decode step drops from processing the full sequence to processing a single token.

The tradeoff is memory. For a 70B model with 80 layers, each layer stores K and V tensors sized proportionally to the sequence length and hidden dimension. A single 128K-context request can consume 50-100GB of KV cache. This is why KV cache management is the central challenge of LLM serving, and why PagedAttention was such a breakthrough.

Continuous batching

Static batching groups multiple requests together and processes them as a single batch. Every request in the batch must wait for the longest one to finish before any results are returned. If request A generates 10 tokens and request B generates 500, request A's response is held until B completes.

Continuous batching (also called iteration-level batching) solves this by checking for completed requests at every decode step. When a request emits its end-of-sequence token, it exits the batch immediately, and a waiting request takes its slot.

The throughput increase is dramatic. Static batching on a 70B model might achieve 200-400 tokens/second total. With continuous batching, the same GPU can push 2,000-4,000 tokens/second because slots never sit idle waiting for the longest request.

Continuous batching can reduce the number of GPUs needed for variable-length traffic, sometimes substantially. Its impact should be measured against the existing scheduler and workload rather than assumed to be a fixed multiplier.

PagedAttention

Even with KV caching and continuous batching, there's a hidden waste problem: memory fragmentation. Traditional KV cache implementations pre-allocate a contiguous memory block for the maximum possible sequence length per request. If you set max_seq_len to 4096 but most responses are 200 tokens, you're wasting 95% of the allocated KV cache memory.

PagedAttention (introduced by vLLM) borrows the virtual memory concept from operating systems. Instead of pre-allocating contiguous blocks, it divides KV cache into small fixed-size pages (typically 16 tokens each). Pages are allocated on demand as the sequence grows, and freed immediately when the request finishes.

This addresses two problems at once. Pages do not need to be contiguous, and allocation can follow the sequence as it grows. The vLLM paper reported large reductions in KV-cache waste in its benchmark setup; the practical saving and concurrency increase vary with sequence lengths, allocator, and workload.

Speculative decoding

The decode phase is memory-bandwidth-bound: the GPU reads the entire model for each token but only does a tiny amount of compute. Speculative decoding exploits this by using a small, fast draft model (e.g., a 1B model) to generate several candidate tokens quickly. Then the large target model verifies all candidates in a single forward pass.

Here's the key insight: verification is parallelizable. The target model can check 5-10 draft tokens in roughly the same time it takes to generate one token, because the bottleneck is reading the model weights (which happens once regardless of how many tokens you verify). If the draft model's predictions match, you've generated 5-10 tokens for the cost of one target model forward pass plus one cheap draft model pass.

When the draft model's proposals are accepted, the target model can verify several tokens in one pass; when it disagrees, the target model supplies the next token and the process continues. Speedup depends on acceptance rate, output length, and hardware. Exact verification can preserve the target model's distribution, but implementation details still need testing.

The catch: speculative decoding helps decode latency (TBT) but doesn't help prefill (TTFT). It also adds complexity and requires maintaining two models in memory. For short outputs (under 20 tokens), the overhead may not be worth it.

Prefix caching

Most production LLM deployments use a system prompt: instructions, persona, tool definitions, few-shot examples. This prompt is identical across all requests. Without prefix caching, every request re-processes the system prompt from scratch during prefill.