Thundering Herd

TL;DR

• The thundering herd happens when a popular cache key expires and many concurrent requests simultaneously find a cache miss, all racing to recompute the same expensive value.
• Every one of those requests can hit your database directly, creating an illustrative traffic spike that is orders of magnitude above normal load within milliseconds.
• Three mitigations exist: probabilistic early expiry (evict slightly before TTL expires), mutex-on-miss (only one request recomputes, the rest wait), and request coalescing (deduplicate in-flight fetches at the cache layer).
• This is a common cause of cache-induced database overload. A cache can look healthy during normal traffic and still expose the database to a sudden burst when a hot key expires.

30-second explanation

A thundering herd is a synchronized cache miss: many requests discover that the same hot key expired and all recompute it at once. The cache has turned one read into a burst of database work. Add jitter, coordinate one refresh, or coalesce the in-flight requests so the database sees bounded work.

5-minute explanation

Start with the fixed-TTL timeline: a popular key expires, concurrent readers miss, and each reader runs the same expensive query. The important distinction is between preventing synchronized expiry and limiting work after a miss. Probabilistic early expiry spreads refreshes; a mutex or single-flight mechanism lets one request refresh while others wait or use stale data; request coalescing deduplicates the in-flight fetch. Whichever option you choose, set lock and request timeouts, protect the database with connection limits, and monitor cache misses together with query rate and latency.

The Problem

Consider this illustrative scenario: a most-read database query has a 5-minute TTL. At 9 a.m. on a Monday, when traffic spikes, that key expires. In the same 50-millisecond window, 4,000 concurrent requests check the cache, get a miss, and each independently fires a SELECT * FROM products WHERE category_id = ? against the primary database.

The database goes from 200 queries per second to 4,000 queries per second almost immediately. If the query takes 500ms under load, the connection pool can exhaust in under a second. The DB starts queueing connections. Cache-fill queries may time out, leaving the key unpopulated and causing more misses for the next 30 seconds. The site may start returning errors.

The painful thing is that this looks like a database problem. Your dashboards show DB CPU at 100%, connection timeouts, query latency at 30 seconds. But the root cause is a design choice in your cache layer: you gave every concurrent reader the same miss behaviour. A useful diagnostic clue is a query spike that repeats at the cache TTL interval; scaling the database alone does not remove the synchronized miss.

Why It Happens

The thundering herd emerges from three individually reasonable decisions that combine into a structural risk.

Fixed TTL with no jitter. You set EX 300 on every key. Every instance of the same key expires at the exact same millisecond. Under low traffic, this is fine. Under high traffic, it creates a synchronized cliff.

No coordination between readers. Each application instance checks the cache independently. There's no "someone is already fetching this" signal. When the key is gone, every reader acts as if it's the only one.

Recompute time exceeds arrival interval. If your DB query takes 500ms but new requests arrive every 0.25ms, you'll accumulate 2,000 duplicate queries before the first one finishes. The gap between "miss detected" and "cache repopulated" is the danger window.

The following timeline is illustrative; the request counts and timings depend on traffic and query latency:

Time T: Key expires
T+0ms:  1 request checks cache → MISS → goes to DB
T+1ms:  200 more requests check cache → all MISS → all go to DB
T+2ms:  2000 more requests ... all MISS ... all go to DB
T+500ms: First DB response comes back → key populated
T+500ms: 2000 duplicate DB queries still in flight

Every request independently checked the cache before any of them had a chance to repopulate it. Adding hardware alone does not address the synchronized miss. The structural issue is how cache misses are handled under concurrent load.

How to Detect It

Thundering herd has recognizable signals. Once you know the pattern, a dashboard can often point you toward it quickly.

A strong signal is a DB CPU graph with sawtooth spikes at regular intervals that match the cache TTL. That pattern points toward a thundering herd, but confirm it by correlating cache misses with duplicate queries and latency.

Here's a quick Redis diagnostic to check if your hot keys are expiring in sync:

# Watch keyspace events for expiry patterns
redis-cli --no-auth-warning MONITOR | grep -E "EXPIRE|DEL|GET" | head -1000

# Check if a specific key pattern has simultaneous misses
redis-cli INFO stats | grep keyspace
# keyspace_hits:4523987
# keyspace_misses:12  <-- normally low
# If misses spike to thousands in a burst, that's the herd

The Fix

Before the three main mitigations, there's a dead-simple first step that every caching system should implement.

Fix 0: TTL jitter (simple baseline)

Randomize your cache TTL by +/- 20%. For an illustrative base TTL of 300 seconds, keys expire between 240 and 360 seconds. This alone won't prevent a herd on a single hot key, but it prevents "all keys expiring at once" scenarios that amplify the problem.

function ttlWithJitter(baseTtl: number, jitterPercent = 0.2): number {
  const jitter = baseTtl * jitterPercent;
  return Math.floor(baseTtl + (Math.random() * 2 - 1) * jitter);
}

// Usage: instead of redis.set(key, value, "EX", 300)
await redis.set(key, value, "EX", ttlWithJitter(300));

This is not a complete solution. It reduces the herd size but doesn't eliminate it for individual hot keys. Treat it as baseline cache hygiene, like input validation: apply it in most systems, but don't rely on it alone.

Fix 1: Mutex-on-miss (lock-based refresh)

When a request gets a cache miss, it acquires a distributed lock (e.g., Redis SET NX PX 5000). Only the lock holder recomputes the value and writes it back. Every other request waits briefly, then reads from cache.

async function getWithMutex(key: string): Promise<Value> {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  const lockKey = `lock:${key}`;
  const acquired = await redis.set(lockKey, "1", "NX", "PX", 5000);

  if (acquired) {
    // We own the lock: recompute and populate
    const value = await db.query(key);
    await redis.set(key, JSON.stringify(value), "EX", 300);
    await redis.del(lockKey);
    return value;
  } else {
    // Someone else is recomputing, wait briefly and retry
    await sleep(50);
    return getWithMutex(key); // retry
  }
}

Trade-off: The lock introduces serialization. Under heavy load, hundreds of requests may pile up waiting. Keep the lock TTL short and the recomputation fast.

The code is illustrative. In production, release the lock in a finally block and cap retries so a failed recompute cannot leave callers retrying indefinitely.

Fix 2: Probabilistic early expiry (XFetch)

Instead of expiring at a fixed TTL, each reader independently decides to proactively refresh the value slightly before it expires. The closer the key is to expiring, the higher the probability of triggering a refresh.

function shouldEarlyExpire(ttlRemaining: number, delta: number, beta = 1.0): boolean {
  // XFetch algorithm: P(refresh) increases as TTL approaches zero
  return Date.now() / 1000 - delta * beta * Math.log(Math.random()) >= expireTime;
}

The key insight: instead of one expiry event causing a thunderstorm, expiry is spread across many small, gradual refreshes. No lock needed.

Trade-off: Probabilistic early expiry adds some unnecessary recomputes (a reader might refresh a key that still has 30 seconds left). You're trading a small amount of extra DB load during normal operation for reducing the largest spike at expiry. In practice, the extra load is often smaller than the herd spike it prevents.

When it shines: XFetch is ideal for systems with many hot keys and unpredictable traffic patterns. It requires no locks, no coordination infrastructure, and works across distributed application instances without any shared state.

Fix 3: Request coalescing (deduplication layer)

A dedicated layer (your cache client, a sidecar, or a middleware) deduplicates in-flight fetches. If 200 requests miss the same key simultaneously, the coalescing layer can issue one DB call and fan the result out to all 200 waiters within that coalescing scope.

const inFlight = new Map<string, Promise<Value>>();

async function getWithCoalescing(key: string): Promise<Value> {
  const cached = await redis.get(key);
  if (cached) return JSON.parse(cached);

  // Check if someone else is already fetching this key
  if (inFlight.has(key)) {
    return inFlight.get(key)!; // wait on the same promise
  }

  // We're the first: fetch and let everyone else piggyback
  const fetchPromise = db.query(key).then(async (value) => {
    await redis.set(key, JSON.stringify(value), "EX", 300);
    inFlight.delete(key);
    return value;
  });

  inFlight.set(key, fetchPromise);
  return fetchPromise;
}

Libraries like dataloader (Node.js) do this natively for GraphQL. The approach works well when all your cache clients run in the same process. For distributed coalescing across multiple application instances, you need a shared coalescing layer (like Nginx's proxy_cache_lock or a custom Redis-based dedup).

When to Use Each Fix

Here's how mutex-on-miss changes the flow. Instead of 4,000 DB queries, the illustrative burst produces one recompute:

Choosing the Right Fix

Not sure which fix to use? Walk through this decision tree:

Severity and Blast Radius

Thundering herd can be a high-severity anti-pattern when the cache protects a shared database. The impact depends on key popularity, recompute cost, and database headroom. The ranges below are illustrative, not universal.

• Blast radius: Services that share the database behind the cache. A hot key can affect more than the service that owns the key if those services share the same database capacity.
• Cascade risk: High when overload causes connection timeouts, retries, and more load. A hot key can make an entire product path unavailable for several minutes.
• Recovery time: As an illustrative range, 1–5 minutes if the DB absorbs the spike; 10–30 minutes if the DB crashes or the connection pool deadlocks.
• Detection to fix: Diagnosis may take hours, although adding mutex-on-miss to an existing cache layer can be a small code change. Prevention is easier when the mitigation is part of the initial design.

When the Risk Is Low

Not every cache expiry under load is a problem worth solving. The thresholds below are illustrative starting points; compare them with the capacity and latency of your own system.

• Low-traffic keys (under 10 QPS): The "herd" is 1-2 requests. Your DB won't notice.
• Fast recomputes (under 5ms): Even 100 duplicate queries finish before the pile-up matters.
• Read replicas with headroom: If your read replica can absorb 10x normal load for a few seconds, the herd may be a short-lived blip rather than a sustained outage.
• Development and staging environments: Don't add mutex complexity to systems that never see real concurrency.
• Cache-aside with short TTL as a performance optimization, not a reliability layer: If the system works fine without cache and you're just shaving latency, thundering herd is an annoyance, not a failure mode.

How to Explain the Pattern

When explaining a TTL-based cache that feeds a relational database, start with the expiry path: a hot key expires, concurrent readers miss, and each reader recomputes the same value. Then name the mitigation that fits the workload.

The key trade-off is between mutex serialization and the extra recomputes of probabilistic early expiry. Also account for the lock's own failure mode: if its TTL is shorter than the recompute, multiple refreshes can start and create a secondary herd.

Test Your Understanding

Recap

1. Thundering herd: all concurrent requests simultaneously miss the same cache key and race to the DB.
2. The trigger is a fixed TTL expiry under high concurrency, not a bug, but a structural design gap.
3. Mutex-on-miss serializes the recompute; probabilistic early expiry spreads the refresh; coalescing deduplicates in-flight fetches.
4. Layer your defences: probabilistic early expiry at the cache layer + connection pool limits at the DB layer.
5. Monitoring signal: a sharp spike in DB query rate that correlates with cache hit rate dipping to zero for a specific key prefix.
6. Don't confuse thundering herd (TTL-driven, periodic) with cache stampede (write-driven, irregular). The trigger is different, and so are the best fixes.
7. The simplest first step: add TTL jitter. Randomize your cache expiry by +/- 20% to spread expirations across a wider time window.