How TikTok avoids showing you the same video twice

The scenario

A short-video feed can produce far more candidates than a person can watch. If the service returns the same clips repeatedly, the feed feels broken; if it keeps an exact history for every viewer forever, storage and lookup cost grow without bound.

A practical design separates “do not show this again in the current session” from “avoid recent repeats across a longer window.” Exact state protects correctness for a small hot set; approximate state controls memory for a large history.

30-second mental model

Check a session-level exact set first, then an approximate recent-history structure such as a Bloom filter or compact bitset, and finally apply ranking constraints. After delivery, record impressions asynchronously and update the durable recent-history store with idempotent events. False positives cost variety; false negatives can cause a repeat, so the system should favor a bounded, recoverable mistake rather than silently dropping too much inventory.

TikTok’s private feed and storage implementation is not assumed here. The architecture and parameters are a scalable reference model; size them from measured traffic and product repeat-tolerance.

5-minute end-to-end flow

1. Request a candidate batch with a viewer ID, session ID, and the current filter/version.
2. Remove items already in the session’s exact seen set; use a recent-history approximation for older impressions.
3. Rank the survivors while preserving exploration and enough fallback inventory if the filter is too aggressive.
4. Return a page and update the session exact set immediately so rapid refreshes do not repeat it.
5. Emit impressions asynchronously, partitioned by viewer, and update durable history idempotently.
6. Rotate or rebuild approximate buckets, watch saturation and false-positive estimates, and fall back to a broader candidate pool when a user’s filter is overfull.

The Architecture

The key insight in this architecture is that impressions flow one way (from app to Kafka) and dedup reads flow the other (from dedup service to Bloom filter and Redis). The write path is async and the read path is sync inside the recommendation latency budget.

The Kafka pipeline does three jobs: updating the Bloom filter, updating the Redis sorted set, and archiving to the columnar store for ML training. These three consumers each do different things with the same event, which is exactly why the fan-out model is right here. Adding a fourth consumer (analytics, abuse detection, etc.) costs nothing on the producer side.

The recommendation engine generates 1,000 candidates. The dedup filter knocks that down to roughly 50 clean videos. The final ranker scores those 50 and returns the top 20 to the client. That funnel, 1000 to 50 to 20, is a common pattern in production recommendation systems and worth naming explicitly in an interview.

The request flow for a single For You Page fetch works like this:

1. The client sends a feed request with its session ID.
2. The feed service loads the session impression buffer from the session store. For a new session, this buffer starts empty.
3. The recommendation engine runs retrieval and generates 1,000 candidate video IDs using Approximate Nearest Neighbor search over user and video embeddings.
4. The dedup filter checks each candidate across three layers in priority order: session buffer first (in-memory hash set, no network call), then the 30-day Bloom filter (30 Redis BITFIELD GETs in one pipelined round trip), then the Redis sorted set (ZSCORE on the last 500 entries).
5. Any candidate that hits in any layer is removed. Roughly 950 candidates are filtered, leaving about 50 clean candidates.
6. The final ranker scores the 50 clean candidates and returns the top 20 to the client.
7. The 20 served video IDs are immediately added to the session impression buffer, before any watch events are confirmed by the client. This prevents the same video appearing twice in consecutive feed requests even if no impression event has been received yet.

Here is how the storage footprint breaks down at the stated scale:

The Bloom filter at 162 TB is sharded across a Redis cluster and entirely in DRAM. The Redis sorted set at 4 TB is also in DRAM. The columnar store is the only component that grows unboundedly, but it lives in cheap object storage and is accessed only by offline ML training jobs. The read path (feed generation) never touches the columnar store.

Deep Dive 1: Bloom Filters for Seen-Video Tracking

A Bloom filter is a probabilistic data structure that answers the question "have I seen this ID before?" in O(1) time and constant space. It can produce false positives (says "seen" when the video is actually new) but never false negatives (it never says "not seen" when the user actually watched it).

For TikTok's dedup problem, the error direction matters enormously. A false negative means re-serving a video the user has already watched. That is a bad experience and exactly what we are trying to prevent. A false positive means occasionally withholding a video the user has not seen. That is a minor loss of one video from a feed of thousands. The acceptable false negative rate is zero. A false positive rate of 0.1% is completely fine.

I will tune the Bloom filter to target a 0.1% false positive rate. At that rate, roughly 1 in 1,000 candidate videos gets incorrectly filtered. The user never notices.

The sizing math: if a user watches 100 videos per day for 30 days, that is 3,000 video IDs in the filter. To hit a 0.1% false positive rate with 3,000 elements, the optimal filter needs roughly 43,000 bits (about 5.4 KB) with 10 hash functions. Per user, that is one tiny bitfield. Across 1 billion users, that is approximately 5.4 TB of Bloom filter state, which fits comfortably in a Redis cluster.

The time-windowing trick is important. Rather than one monolithic Bloom filter per user, I keep 30 daily filter windows. Each day gets its own bitfield. When a day expires (older than 30 days), I delete that bitfield. This makes the rolling window automatic: you never need to "remove" individual video IDs from the filter (which Bloom filters cannot do). You just expire the whole day bucket.

Deep Dive 2: Impression Log Pipeline

The Bloom filter handles the dedup reads. The impression log pipeline handles the writes. Every time a user watches a video, that event needs to: update the Bloom filter, update the Redis sorted set, and land in long-term columnar storage for ML training. Three consumers. One producer. This is a textbook Kafka fan-out.

Partitioning Kafka by userID is the critical design choice here. All events for the same user land on the same partition, which means the Bloom Filter Writer processes a user's events in order. If you partitioned by videoID instead, events for the same user would scatter across 200 partitions, and the Bloom filter updates would be processed by different consumers concurrently, causing write races.

The Redis sorted set writer has a gating condition: only record impressions where watch_pct >= 0.5. Skips and brief flickers do not go into the sorted set. This keeps the set clean and aligned with the definition of "meaningfully seen." The Bloom filter is more aggressive, recording any view above a lower threshold, because false positives are cheap there.

Deep Dive 3: Session vs Lifetime Deduplication

The trickiest part of the dedup architecture is the gap between what the user has seen in the current session and what the persistent store knows about. The Bloom filter and Redis sorted set update asynchronously, minutes after the actual impression. But the user expects deduplication to be instantaneous.

If the user opens TikTok and watches video A at 2:00 PM, then closes and reopens at 2:01 PM before the impression has been persisted, the Bloom filter does not yet know about video A. Without session-scoped deduplication, the feed could serve video A immediately after reopening.

The session buffer lives on the application server for the duration of the request session, not on the client. When the recommendation pipeline processes a feed request, it passes the current-session impression set as part of the request context. The dedup service checks this in-memory set first (O(1) hash lookup), then the Redis sorted set, then the Bloom filters.

On session end, the application server receives the session-end event and merges the session buffer into the Redis sorted set. Only at that point do the persistent layers know about the current session's impressions. Until then, the session buffer is the source of truth for that user's active session.

Bottlenecks, failure modes, and operations

1. The write amplification problem: Every impression event needs to update multiple stores: Bloom filter bits, Redis sorted set entries, and the columnar archive. At 100B events per day, even a 1-second delay in any consumer can cause the consumer to fall minutes behind. Kafka consumer lag monitoring is critical. If the Bloom filter writer falls 5 minutes behind, the dedup accuracy degrades for those 5 minutes.

2. Cold start for new users: A new user has an empty Bloom filter and an empty Redis sorted set. Every video is a candidate. The dedup layer passes all 1,000 candidates through to ranking. That is fine for dedup (no false positives), but it also means the recommendation model has no negative signal (seen history) to work with, which affects the quality of embeddings used for retrieval. Cold start in recommendation systems always has this dual problem: no positive history and no negative history.

3. Video ID reuse and content recycling: TikTok occasionally re-uploads popular videos under new IDs (due to copyright takedowns, re-encodings, or creator re-posts). The same content appears under a new ID, so the dedup system treats it as a new video. This is actually correct behavior: the dedup system tracks impressions by ID, not by content hash. Content-level deduplication (detecting visually identical videos) is a separate, harder problem involving perceptual hashing or video embedding comparison.

4. Cross-device session state: Two active sessions on different devices for the same account share the persistent Bloom filter and Redis sorted set but have separate session buffers. A video watched on device A will not be excluded from device B's feed until the session buffer from A merges to the persistent store. This is a known best-effort situation. Building real-time cross-device session coordination is not worth the complexity for the marginal improvement.

5. The 30-day boundary effect: When videos exit the 30-day window, they become eligible for the feed again. For a prolific user who watches 100 videos/day, roughly 100 videos cycle back into eligibility every day. This is fine and intentional. But it creates a subtle anomaly where a user might see a video they watched exactly 30 days ago resurface. A tiered confidence decay (reducing exclusion probability as the window approaches expiry) makes this transition less jarring.

6. Impression event schema evolution: The impression payload (userID, videoID, watch_pct, timestamp) will grow over time. Adding a rewind_count field or an engagement_type enum without backward-compatible schema management breaks old consumers. Use a schema registry (Avro or Protobuf with schema evolution rules) rather than plain JSON so new fields are optional and old consumers handle them gracefully.

Failure Modes and Monitoring

This system fails silently. When the dedup layer degrades, it does not throw errors -- it just stops filtering correctly. Users start seeing already-watched videos. Understanding the specific failure signatures helps you both operate the system and discuss it in an interview.

Kafka consumer lag is the primary metric to watch. The three consumers (Bloom filter writer, Redis writer, columnar store writer) are independent. Bloom filter writer lag is the most user-visible: if it falls 5 minutes behind, users see videos they watched in the last 5 minutes reappear in the feed. Alert at 30 seconds of lag on the Bloom filter writer, 60 seconds for the Redis writer. The columnar store writer tolerates longer lag (up to 5 minutes) since it only affects ML training data freshness.

Bloom filter saturation is a quiet failure. A daily bitfield with more than 80% of its bits set has a dramatically elevated false positive rate. A user who watched 2,000 videos in a single day (a binge session or a bot) will overflow a filter sized for 100 videos/day. Track the fill ratio of each daily bitfield and cap writes at a reasonable maximum (say, 500 per day) to prevent a single anomalous session from corrupting the filter.

Redis sorted set cardinality anomalies indicate the ZREMRANGEBYRANK trim step is broken. If trimming stops, sorted sets grow unboundedly. Alert on any user's sorted set exceeding 600 entries and trigger a manual trim.

Recommended alert thresholds for the dedup monitoring dashboard:

Common mistakes and misconceptions

Practical checklist

• Define the repeat policy separately for the current session, recent history, and long-term inventory.
• Use exact state for the hot path and approximate state only where a bounded false-positive cost is acceptable.
• Size Bloom filters from measured impressions, desired false-positive rate, bucket duration, and heavy-user skew.
• Keep impression writes asynchronous but idempotent, partitioned by viewer, and observable for lag and loss.
• Protect the feed when a filter is saturated: rotate buckets, resize, or fall back to more candidates rather than returning an empty page.
• Distinguish filter false positives from pipeline lag, missing events, and client/server clock errors.
• Preserve exploration and content diversity; deduplication is a constraint on ranking, not the entire ranking objective.
• Measure repeat rate, filter rejection rate, candidate shortfall, impression lag, saturation, and user-level quality impact.

Test Your Understanding

Quick Recap

1. TikTok's dedup problem is a write-at-scale problem: 100 billion impression events per day makes per-user database rows impractical.
2. Bloom filters let you track 30 days of seen-video history per user in roughly 160 KB of space, with a tunable false positive rate of 0.1%.
3. Daily time-windowed Bloom filter buckets with Redis TTLs eliminate the need for any active cleanup job and implement the rolling 30-day window automatically.
4. The Redis sorted set provides exact deduplication for the most recent 500 video IDs, covering the high-frequency re-serve window.
5. A session-scoped in-memory buffer handles the async lag between impression and persistent-store update, preventing same-session re-serves.
6. Kafka impression events are partitioned by user ID so all per-user state updates are serialized through a single consumer instance, preventing write races.
7. The candidate funnel runs 1,000 generated candidates through three dedup layers to produce 50 clean candidates for final ranking.
8. Cross-device session dedup is best-effort. Periodic session flushes reduce the exposure window without requiring real-time cross-device coordination.
9. False positives in the Bloom filter are acceptable and tunable. The failure mode to prevent is false negatives (re-serving seen videos), not false positives (occasionally withholding a new video).
10. The Kafka partition key (userID) is not incidental. It guarantees all impression events for a given user are processed in sequence by a single consumer, preventing write races on the per-user Bloom filter and Redis sorted set.

Related Concepts

• Bloom Filters: The probabilistic data structure at the core of the dedup layer. Understanding bit array sizing, hash function selection, and false positive rate math is essential background for this system.
• Kafka Fan-Out Pattern: The impression pipeline uses a single Kafka topic with multiple consumers, each updating different state. This pattern appears in activity feeds, notification systems, and any real-time event pipeline.
• Redis Data Structures: Sorted sets with ZADD and ZREMRANGEBYRANK are the key Redis primitives for the exact-recent impression store. Understanding sorted set complexity and eviction is foundational.
• Recommendation Systems Architecture: The dedup filter is downstream from the candidate retrieval and upstream from the final ranker. Understanding where dedup fits in the two-stage retrieval pipeline puts this article in context.
• Probabilistic Data Structures: Bloom filters are the canonical example, but the broader category includes Count-Min Sketch (frequency estimation), HyperLogLog (cardinality estimation), and Cuckoo filters (a Bloom variant that supports deletion). Each trades some accuracy for dramatic space savings at scale.
• Write-Ahead and Async Pipelines: The impression pipeline uses a fire-and-forget write model via Kafka, with the read path (dedup checks) fully decoupled from the write path (impression logging). This async separation is a recurring pattern in large-scale systems wherever write throughput would otherwise block read performance.
• Time-Windowed State with TTL: Using Redis TTL for automatic expiry instead of active deletion is a broadly applicable pattern. It appears in rate-limiting windows, session expiry, fraud detection lookback windows, and any system that needs bounded history without a background sweeper job.
• Two-Stage Ranking Architecture: Most large-scale recommendation systems use retrieval (fast, approximate, returns 1,000 candidates) followed by scoring (slower, precise, returns the top 20). The dedup filter sits between these two stages. Understanding this pipeline positions you for any feed or recommendation design question.