Caching

TL;DR

• A cache is a fast storage layer that can answer repeated reads before they reach the database. The cache hit rate is a useful first estimate of how much read traffic remains for the database.
• At a 95% hit rate, the database receives about 5% of cacheable reads—roughly 20× less than without that cache. The actual benefit depends on which requests are cacheable and how the hit rate is measured.
• The core trade-off is freshness vs. latency and cost: a cache stores a copy that may be older than the source of truth. TTLs, invalidation, and versioning bound that staleness.
• Cache-aside is a common default read pattern. Write-through, write-behind, write-around, and explicit invalidation are alternatives chosen for different write and freshness requirements.
• Cache invalidation—deciding when a copy is no longer safe to use—is one of the hardest parts of operating a cache. A stale value can look like a successful response, so the design needs observable freshness and failure behavior.

The Problem It Solves

Imagine an e-commerce platform with 500,000 concurrent users. Suppose 80% of them are looking at the same 100 product pages during a busy sale.

The PostgreSQL database has 12 million products, but those 100 rows are being read roughly 4,000 times per second each. For this illustrative calculation, assume each SELECT takes 5ms: 400,000 reads/second × 5ms implies about 2,000 concurrent units of work if all requests overlap. A connection pool capped at 200 would queue requests rather than process them all at once.

The query time climbs from 5ms to 50ms as connection contention sets in, then to 500ms as the connection pool queue fills. Your monitoring dashboard shows CPU at 95%, I/O wait climbing. At 10:03 a.m. the DB falls over.

The issue is not that the data is complex to retrieve; it is that the same 100 rows are being fetched repeatedly. Read replicas may add read capacity, but they do not remove the repeated work. A cache can reuse the result when the data is sufficiently stable and the staleness policy is acceptable.

The first candidate fix is to answer repeated requests from a faster layer instead of recomputing the same result for every request. Replicas or more app servers may still be useful for other bottlenecks.

What Is It?

A cache is a faster storage layer that sits between an application and its source of truth. It holds copies of recently or frequently accessed data so future requests can be served without reading the source every time.

Analogy: Think of a coffee shop that serves 300 customers a day, and 90% of them order the same three drinks. The barista could grind beans fresh for every single cup. Or, they could brew a large batch of the popular drip coffees at the start of the hour and pour from it.

Most customers get their coffee in 10 seconds instead of 3 minutes. The three popular coffees are the cache. The bean grinder is the database.

The batch brew is the cache population. Stale coffee that's sat for 4 hours is the invalidation problem — at some point, you throw out the old batch and brew a fresh one.

With a 95% hit rate in this illustrative workload, the database handles about 20,000 cacheable reads/second instead of 400,000, plus any writes and non-cacheable queries. Whether that is comfortable depends on the database's actual capacity, but the cache has removed most repeated read work.

Anchor a caching discussion in the hit rate and its consequence: “At a 95% hit rate, this cacheable read path sends about 5% of requests to the database.” That explains the mechanism without treating the ratio as a guarantee for every workload.

How It Works

Here is the request flow for cache-aside, a common default pattern:

1. Client sends a request — e.g., GET /products/7429. The app server needs product data.
2. Check the cache first — The app constructs the cache key (product:7429) and sends a Redis GET. The latency depends on the network and deployment; an in-region cache lookup is often much faster than a database query.
3. Cache hit → return immediately — If the key exists, deserialize the value and return it. The database is not involved on this path.
4. Cache miss → fetch from source — Redis returns nil. The app queries the database (SELECT * FROM products WHERE id = 7429).
5. Populate the cache — The app takes the database result, serializes it, and writes it to Redis with SET product:7429 <data> EX 300 (an illustrative 5-minute TTL). Future requests may skip the database until the entry expires or is invalidated.
6. Return the result — The first caller pays the source latency. Subsequent callers may get the cached value until the TTL expires or the key is invalidated.

async function getProduct(productId: string): Promise<Product> {
  const cacheKey = `product:${productId}`;

  // Step 1: Check cache (usually faster than the source on a hit)
  const cached = await cache.get(cacheKey);
  if (cached !== null) {
    return JSON.parse(cached) as Product; // Cache hit
  }

  // Step 2: Cache miss — fetch from the database
  const product = await db.queryOne<Product>(
    'SELECT id, name, price, description FROM products WHERE id = $1',
    [productId]
  );

  if (!product) throw new NotFoundError(`Product ${productId} not found`);

  // Step 3: Populate cache with an illustrative 5-minute TTL
  // Fire-and-forget is optional; a failed cache write must not hide a
  // successful source read, and the next request can retry population.
  cache.set(cacheKey, JSON.stringify(product), { EX: 300 }).catch(console.error);

  return product; // DB miss latency — only the first caller pays this cost
}

The first caller to request product 1337 after a miss (or after TTL expiry) pays database latency. Other callers may receive the cached value until the entry expires or is invalidated. This asymmetry is useful when reads repeat within the freshness window.

Walking through both paths makes the trade-offs visible: the hit path protects the source, while the miss path determines freshness, stampede behavior, and error handling.

Key Components

Cache Layers

Many production systems use multiple cache layers. Each layer can be closer to the user and faster than the one below it, but usually has smaller capacity and a separate freshness or invalidation policy.

The browser and CDN layers are covered in depth in the CDN article. The remainder of this article focuses on the application cache layer — the one you design, own, and debug.

For an interview, three layers are often enough: browser/CDN, application cache, and database. Add latency estimates only as illustrative assumptions, then spend time on the application-layer decisions you actually control.

Read Strategies

How the cache gets populated is a design decision. Cache-aside and read-through are common read patterns; the correct choice depends on who owns the source lookup and how much control the application needs.

Cache-Aside (Lazy Loading)

The application manages the cache directly. On a miss, the app fetches from the database and populates the cache itself. This is the pattern shown in the "How It Works" code above.

Characteristics: Only data that has actually been requested is cached, so cold data does not occupy cache space. The first request to a key usually pays source latency; subsequent requests can use the cache. Staleness is bounded by TTL only if the entry is allowed to expire, and explicit invalidation can shorten that window.

Cache-aside is often a good default because the source lookup and fallback behavior remain explicit in application code.

Cache-aside is a common default for read-heavy systems.

Read-Through

The cache itself is responsible for loading data from the source on a miss. The application talks only to the cache on the read path; the cache's loader queries the source.

Characteristics: Simpler application code — one data access layer handles everything. The cache must be configured with your DB schema and connection, often via libraries like Spring Cache or AWS DAX for DynamoDB. The first-request latency and thundering-herd exposure are identical to cache-aside.

Read-through is most useful when a framework or cache library naturally supports it and the team is comfortable debugging the data-access layer it introduces. Cache-aside can be easier to reason about because the miss path remains visible in the application.

Write Strategies

When data changes, what gets updated and in what order? Many caching bugs originate in this write path. Caching is not only a read concern; the design must specify how writes, invalidation, retries, and failures interact.

Write-Through

Data is written to the cache and the database in the same write path. A cache and a database do not provide an automatic cross-system transaction, so the design must state which system is authoritative and what happens if one write fails. Some systems wait for both writes; others acknowledge the durable database write and repair the cache asynchronously.

async function updateProductPrice(productId: string, newPrice: number): Promise<void> {
  // The database is authoritative. The cache is refreshed after the commit;
  // a cache failure should be observable and repaired, but must not make a
  // successful database write look like a failed price update to the caller.
  await db.query('UPDATE products SET price = $1 WHERE id = $2', [newPrice, productId]);
  try {
    await cache.set(`product:${productId}`, JSON.stringify({ price: newPrice }), { EX: 300 });
  } catch (error) {
    metrics.increment('cache_refresh_failed');
    repairQueue.enqueue({ productId, newPrice });
  }
}

Use when: Read-heavy data benefits from a refreshed cache value after a successful write, and the extra write latency and failure handling are acceptable. For inventory or account data, the authoritative database should still decide whether an operation is allowed.

Avoid when: Write-heavy workloads such as analytics events or counters, unless the cache is intentionally part of the write model and its durability is understood.

Write-Behind (Write-Back)

Data is written to the cache first and acknowledged to the caller immediately. The cache flushes changes to the database asynchronously via a background process or queue.

async function recordPageView(articleId: string): Promise<void> {
  // Increment in-memory counter — responds in < 1ms
  await cache.incr(`views:${articleId}`);

  // Enqueue DB flush — batched, happens out-of-band
  await queue.enqueue({ type: 'flush_views', articleId });
  // If the cache node crashes before flush, view counts since last flush are lost
}

Use when: Very high write throughput where a documented loss window is acceptable, such as view counters, click events, or some IoT telemetry. A payment or inventory record should not rely on a cache as its only durable copy.

Avoid when: Data represents money, inventory, or any state that must survive a cache failure without loss.

Write-Around

Data is written directly to the database, bypassing the cache entirely. If a corresponding cache entry exists, it must either be invalidated or be allowed to remain stale until its TTL expires; otherwise readers may see the old value.

async function archiveOldOrder(orderId: string): Promise<void> {
  // Direct DB write — this data is read rarely; caching it wastes memory
  await db.query('UPDATE orders SET archived = true WHERE id = $1', [orderId]);
  // No cache interaction — cache either holds stale data until TTL expires,
  // or this key was never cached in the first place
}

Use when: Data is written once and read rarely, such as archived records, audit logs, or cold historical data. Caching it may consume memory without a useful hit-rate benefit.

Cache Invalidation on Write

One common pattern is: write to the database, then delete the cache key. The next read miss can repopulate from the source of truth.

async function updateUserProfile(userId: string, profile: UserProfile): Promise<void> {
  // 1. Write to source of truth first
  await db.query(
    'UPDATE users SET name = $1, email = $2 WHERE id = $3',
    [profile.name, profile.email, userId]
  );

  // 2. Invalidate the key after the source write succeeds.
  // Updating the cache here can race with another writer; deletion keeps
  // the source of truth responsible for rebuilding the value.
  await cache.del(`user:${userId}`);
}

For cache-aside, write to the source first and invalidate afterward. Document what happens if invalidation fails, and make the next read safe to retry.

Cache Invalidation

The familiar saying about cache invalidation and naming is memorable because invalidation is difficult to reason about. The practical question is: given that the source has changed, when is the cached copy no longer safe to use?

Invalidation is the question: given that the database has changed, when does the cache know to discard its stale copy?

TTL-Based Invalidation

The simplest approach is a fixed expiry. After the TTL, Redis can evict the key and the next read repopulates it from the current source data.

If the data changes frequently enough for users to notice staleness, TTL alone may be insufficient; event-driven invalidation or a read path with stronger consistency may be needed.

Event-Driven Invalidation

On every write, proactively delete relevant cache keys. When invalidation succeeds, the stale window is roughly the delay between the source write and the invalidation call. If an event is lost or a consumer is down, the old value can remain until another repair or its TTL, so reliable delivery and a fallback policy matter.

A users row change might need to invalidate user:{id}, feed:{id}, profile:{id}, and recommendations:{id}. Missing one dependency allows stale data to persist silently, so the dependency graph should be documented and tested.

Version-Based Invalidation

Embed a version number in the cache key. When data changes, increment the version. Old keys become unreachable and eventually expire.

// Version-stamped key: product:7429:v3
const cacheKey = `product:$\{productId\}:v${product.version}`;

// After an update, the new version key is used; the old key simply expires via TTL
// No explicit invalidation needed — the old key is unreachable from new reads

Elegant for immutable-snapshot use cases. The downside: stale keys linger until TTL — memory overhead grows proportional to how often data is updated.

Invalidation is the part of caching that will bite you in production — TTL, events, or versioning, you need a strategy before you ship.

Eviction Policies

When Redis runs out of memory, the eviction policy determines which keys are dropped to accommodate new writes.

Trade-offs

The fundamental tension here is freshness vs. performance and cost. A cache is a copy of the source that may be stale. The performance gain comes from reusing that copy, so the design must say how much staleness is acceptable.

The engineering challenge is deciding how stale is acceptable for each type of data — and building the invalidation mechanics that enforce that bound. Get the staleness tolerance wrong and your users will notice.

When to Use It / When to Avoid It

Caching is most useful when the same data is read repeatedly, the source lookup is expensive enough to matter, and a bounded amount of staleness is acceptable. The full decision depends on the read/write mix and failure behavior.

Use caching when:

• Your read traffic is much higher than your write traffic for a given dataset.
• The same data is read repeatedly within a window shorter than your acceptable staleness window.
• Database query response time is a meaningful fraction of total request latency.
• You need to scale reads past what a single primary or read replica can handle.
• You store derived or computed values that are expensive to recompute (aggregations, rendered templates, ML inference results).

Avoid caching (or be very careful) when:

• Data is write-heavy and read-once — audit logs, financial ledger entries. Cache pollution with zero hit benefit.
• Data must be visible to every reader immediately after a write—for example, a payment acknowledgement or inventory decision. TTL-based caching is not sufficient by itself; use explicit invalidation or a strongly consistent read path.
• You're prototyping. Cache adds complexity that masks performance problems. Measure DB performance first, then cache proven bottlenecks.
• Cache failure would cause silently incorrect behavior. If your app serves wrong inventory or wrong prices when the cache breaks, verify your invalidation logic is complete before deploying.

If an application serves the same read-heavy data to many concurrent users, caching may reduce source pressure substantially. It is still a measured design choice: a cache adds memory, invalidation, observability, and failure handling.

The Thundering Herd Problem

One important cache failure mode occurs when a TTL expires on a popular key. It is called a thundering herd because many requests miss together and repeat the same source lookup.

The scenario: Your most popular product (product:viral-item) is cached with a 60-second TTL. At 09:00:00.000, the key expires. Within the next 50ms, 1,000 requests arrive for that product — every single one gets a cache miss.

Every single one fires a database query simultaneously. Your database spikes from 50 queries/second to 1,050 queries/second in 50ms. If your DB connection pool has 100 connections, 900 queries queue immediately.

Latency climbs. Cache population slows. The spike persists until repopulation finishes — which takes longer because the DB is already overloaded.

For a high-traffic key, an uncoordinated expiry can create a database spike. The size of that spike depends on request concurrency, source capacity, and how quickly the value can be repopulated.

Fixing the Thundering Herd

Option 1: Mutex lock — Only the first requester that observes a miss acquires a lock and fetches from DB. All others wait, then return the populated value.

function getWithMutex(key, fetchFn):
  cached = cache.get(key)
  if cached exists → return cached

  lockKey = "lock:" + key
  acquired = cache.set(lockKey, "1", NX=true, EX=10)
  // NX = "set only if key does not exist" — atomically claims the lock

  if acquired:
    try:
      value = fetchFn()             // only this one request hits the database
      cache.set(key, value, EX=300)
      return value
    finally:
      cache.del(lockKey)            // always release the lock, even on error

  // Another process is already fetching — wait and retry
  sleep(50ms)
  return getWithMutex(key, fetchFn) // lock is gone; value should be in cache now

Option 2: Probabilistic Early Expiration (PER) — Before TTL fully expires, randomly refresh the cache with probability rising as expiry approaches. No lock contention; stale data is briefly served while the background refreshes.

function getWithEarlyExpiration(key, ttl, fetchFn):
  value     = cache.get(key)
  remaining = cache.ttl(key)

  if value is null → return null   // complete miss — caller handles fallback

  // Randomly trigger a background refresh before the key fully expires.
  // Probability rises as remaining TTL shrinks toward zero.
  threshold = ttl × 0.20           // begin considering refresh at 20% TTL left
  if remaining < threshold AND random() < 1.5 × (1 - remaining / threshold):
    background: cache.set(key, fetchFn(), EX=ttl)  // async — does not block caller

  return value   // always return current cached value immediately

Option 3: TTL jitter — Instead of a fixed TTL across a batch of keys, add random variance at population time. Expiry events spread out across a window instead of firing simultaneously.

const BASE_TTL = 300;
const JITTER = 60; // ± 1 minute

const ttl = BASE_TTL + Math.floor(Math.random() * JITTER * 2) - JITTER;
await cache.set(cacheKey, value, { EX: ttl }); // TTL between 240–360 seconds

TTL jitter is inexpensive to implement and reduces synchronized expiry without lock contention. It is a useful option for batches of keys populated together, but it does not replace capacity limits or request coordination.

Applied Patterns

The patterns below are deliberately generic. They show how caching decisions fit a workload without making claims about a particular company's current implementation.

Two-level cache for a social feed

A social-feed service can use an in-process L1 cache for the hottest content and a distributed L2 cache such as Memcached or Redis for shared data. L1 avoids the network round trip for the most popular items; L2 lets different app instances share a larger working set. A feed can use fan-out-on-write for ordinary accounts, while a very high-fan-out account uses a pull path or a short-lived hot-post cache instead of writing to every follower's feed synchronously.

This illustrates that caching can be part of the read model and feed architecture, not just a small optimization around one database query. The design still needs a recovery plan and a freshness bound.

Lease-based cache fill

In a large distributed cache, a miss can be given a lease or request token to coordinate the first refill. If a concurrent write invalidates the key while the lease holder is reading from the source, the lease can be revoked and the reader must verify freshness before publishing the value. This reduces the chance that a slow read repopulates an older value after an invalidation.

Pre-warmed page cache

A page-heavy site can combine an in-process MemoryCache with a shared L2 cache and refresh popular values in background jobs before their TTL expires. That keeps user traffic away from the database for the hottest pages, while the database remains the source of truth and the refresh job remains observable.

The common lesson is that caching eventually becomes a consistency and invalidation problem as much as a latency problem.

How to Explain It in an Interview

The useful signal is not just naming Redis. Explain what data is cacheable, what hit rate you expect, how much staleness is acceptable, and what happens on a miss or cache failure.

30-second answer

“A cache stores frequently reused data closer to the request path so repeated reads do not reach the source every time. I would choose a cache key and TTL, measure hit rate and latency, define invalidation on writes, and decide whether stale data is acceptable. Cache-aside is a common default, but the source of truth and failure path must remain clear.”

5-minute explanation

Start with the read pattern and a rough estimate: how often the same data is requested, how expensive the source query is, and how much staleness the product allows. Show the cache-aside hit and miss paths, then choose a key namespace, serialization format, TTL, eviction policy, and invalidation strategy. Explain write-through, write-behind, or write-around only if the write workload needs them. Close with cache failure, thundering-herd protection, observability, and whether the system can degrade to the source without overwhelming it.

For a deeper discussion:

• Name your cache hit rate and calculate the DB impact: "95% hit rate → DB sees 5% of reads → 20× less traffic — that's the difference between needing 5 read replicas and needing none."
• Explain the write strategy choice. Write-through can suit data that benefits from a refreshed cache; write-behind can suit high-throughput counters with a documented loss window; explicit invalidation should account for concurrent readers and failed events.
• Know the thundering herd problem and two mitigations: mutex lock (serialize misses) or probabilistic early expiration (background refresh before TTL expires).
• Address the cache as a failure domain: replication or managed high availability, plus a graceful degradation path such as bounded fallback to the database with a circuit breaker.
• Distinguish eviction policy (Redis choosing what to drop when full) from invalidation strategy (you proactively evicting stale data after a write). These are separate mechanisms that work together.

Common follow-up questions and example answers:

Know these cold — cache invalidation, thundering herd, and Redis SPOF handling come up in nearly every system design interview at the senior level.

Deep Dive: Scenario Walkthroughs

Test Your Understanding

Quick Recap

1. A cache is a fast storage layer that can answer repeated reads before they reach the source. Hit rate, request mix, and source capacity together determine the benefit.
2. In cache-aside, the application checks the cache first, falls back to the source on a miss, and populates the cache on return. Write-through keeps cache and source updates coordinated; write-behind improves write latency only when its durability window is acceptable.
3. A TTL is useful for self-healing, but not every entry needs the same expiry policy; some values are explicitly invalidated. Short TTLs improve freshness at the cost of hit rate, while long TTLs improve reuse but risk stale data. Jitter TTL values across hot batches to reduce an avalanche.
4. For cache-aside, a common default is to write the source first and invalidate the key afterward. Directly refreshing a value can race with concurrent writers; stricter ordering may require versions, leases, or reliable events.
5. A thundering herd occurs when many requests miss a popular key together. A mutex can serialize the refill, while early refresh or TTL jitter can reduce synchronized misses when serving a slightly older value is acceptable.
6. A cache is a potential failure domain. Use the deployment's replication or clustering options where appropriate, and build a graceful degradation path that does not flood the source.
7. In an interview, state the expected hit rate and calculate its effect on the cacheable read path, then explain TTL, invalidation, miss coordination, and cache failure.

Related Concepts

• Load balancing — Load balancers route traffic across app servers; caches reduce the database traffic those app servers generate. They solve different bottlenecks and may be used together.
• CDN — A CDN is a geographically distributed cache for static and edge-cacheable content. Understanding browser cache headers and CDN invalidation is the natural extension of application-layer caching concepts.
• Databases — Caching exists to protect databases from read fan-out. Understanding database connection pooling, query cost, and replica lag helps calibrate the right TTL and hit-rate targets for any caching strategy.
• Replication — Database read replicas and Redis replication are two different answers to the same read-throughput problem. Read replicas suit complex queries; caches suit repeated simple reads. Knowing when to use each prevents over-engineering.
• Rate limiting — Redis is the canonical store for distributed rate limit counters. The atomic INCR and Lua scripting patterns from this article apply directly to building correct, race-condition-free rate limiters.