Pastebin

What is Pastebin?

Pastebin stores arbitrary text or code snippets behind a short URL that can be shared with anyone. The apparent simplicity hides two real engineering problems: generating millions of unique short IDs without collisions, and serving a read workload that outpaces writes by 100:1 without turning the database into a bottleneck. The design combines ID generation, blob storage, TTL-based expiration, and caching in a compact end-to-end system.

TL;DR

Generate paste IDs from an atomic counter and encode the value in base62. Store small metadata in PostgreSQL, place paste content in object storage when size or CDN delivery justifies it, seed Redis on write, and serve hot or global reads from Redis and a CDN. Treat caches as derived state.

Enforce expiration on every read using expires_at and return 410 Gone for an expired paste. A periodic cleanup worker and an object-storage lifecycle rule reclaim space later. Rate-limit anonymous creation by IP and registered creation by account, escape paste content as text, and make metadata/object writes retryable and observable.

Scope and Assumptions

This design assumes:

• The core product is create-once, read-many text/code pastes with a short URL and an optional expiration time.
• Anonymous access is allowed; authentication and ownership controls are extension points. The gateway still rate-limits by IP or account before accepting a write.
• The illustrative workload is 100K pastes per day, 10M views per day, an average paste size of 10KB, and occasional 1–10MB outliers.
• A successful create must be durable until expiry, reads target under 50ms p99, and read availability is preferred over perfectly fresh cache state.
• Full-text search, collaborative editing, real-time analytics, abuse classification, and other product surfaces are out of scope for the core path.

Functional Requirements

Core Requirements

1. Users can create a paste with a block of text or code.
2. Each paste gets a unique short URL.
3. Pastes can optionally expire after a set duration.
4. Users can view a paste via its URL.

Below the Line

• Full-text search across all pastes.
• Real-time collaborative editing.

The hardest part in scope: Generating globally unique short IDs without collisions while keeping the write path fast is the central design challenge. We will dedicate Deep Dive 1 entirely to it.

Full-text search is below the line because it requires a separate search index (Elasticsearch or similar), a background ingestion pipeline, and query infrastructure that sits beside, not inside, the core read/write path. To add it, we would stream every new paste to a Kafka topic, have a consumer index the content, and build a search endpoint that queries the index rather than the paste database.

Real-time collaborative editing requires operational transforms or CRDTs, conflict resolution, cursor synchronization, and a WebSocket layer for broadcasting changes. This is effectively a separate product surface and does not interact with the batch paste storage model we are designing here.

Non-Functional Requirements

Core Requirements

• Low read latency: Paste content served in under 50ms p99.
• High availability: 99.99% uptime. Availability over consistency for paste reads.
• Scale: 10M DAU. 100K pastes created per day (~1.2 writes/second). 10M paste views per day (~116 reads/second).
• Durability: No silent data loss. A paste that was successfully created must be retrievable until it expires.
• Storage capacity: Average paste size 10KB. 100K pastes per day equals ~1GB of new content per day. Total storage requirement: ~1TB over 3 years.

Under 50ms latency for paste reads means a cache layer in front of the database is non-negotiable. A direct PostgreSQL read adds 10-30ms of query time before any network overhead.

99.99% uptime means we need at least one standby replica so a primary failure does not cause downtime. For paste reads, we prefer availability over consistency: a slightly stale cache hit is better than a failed request.

Below the Line

• Real-time view count analytics per paste.
• Abuse detection and spam filtering.

Real-time view count analytics is below the line because it requires a separate write path (incrementing a counter on every read) and an analytics pipeline that sits outside the core paste storage loop. To add it, capture a view event on every GET, publish it to a Kafka topic, and aggregate counts asynchronously in a time-series store like ClickHouse or TimescaleDB. The core read path stays untouched.

Abuse detection and spam filtering is below the line because it needs content classification infrastructure (ML models, regex rulesets, URL scanning) that would add latency to the write path. To build it, run an async content scanner that processes new pastes from a queue and flags or deletes ones that match abuse patterns, keeping the synchronous write path fast.

Read/write ratio: For every paste created, expect roughly 100 views. This 100:1 read/write skew shapes every downstream decision: the database read path is the likely bottleneck, a cache is non-negotiable, and read replicas matter more than write replicas. State this ratio early because it justifies the rest of the design.

30-Second Answer

1. Rate-limit the create request, validate the text and TTL, allocate a collision-free counter value, and base62-encode it into the paste_id.
2. Persist metadata durably and write content to the selected backend, then seed Redis with a TTL matching expires_at.
3. Serve reads from the CDN for large/public content or Redis for hot small content; fall back to object storage or a read replica on a miss.
4. Check expires_at inline on every read and return 410 Gone after expiry. A periodic worker and storage lifecycle rule perform eventual cleanup.
5. Keep PostgreSQL focused on indexed metadata and writes, and make object writes, cache fills, retries, and cleanup idempotent.

5-Minute Explanation

The durable record is a small PostgreSQL metadata row containing paste_id, timestamps, ownership information, and the content location. An atomic counter gives each paste a unique integer; base62 makes that integer URL-friendly without random-collision retries. The application can keep very small pastes inline, but object storage is a better fit for large content and CDN delivery.

On the write path, validate the size and expiration policy, allocate the ID, store the content and metadata, and seed Redis. The exact ordering of object and metadata writes should be explicit: if an object-store write is pending, keep the metadata in a non-readable state or reconcile it before exposing the paste. The client receives the URL only after the required durable writes succeed.

On the read path, check the CDN or Redis first, then object storage or a read replica on a miss. The application still checks expires_at so expiration is enforced even when a cleanup job is behind. Return 410 Gone for a known expired paste and 404 Not Found for an unknown ID. A 100:1 read/write ratio makes cache hit rate, CDN behavior, and viral-key protection more important than write throughput.

The operational design is deliberately simple: expiration is a correctness check plus eventual garbage collection, caches are rebuildable, rate limits protect the write path, and content is rendered as text rather than executable HTML.

45-Minute Interview Approach

Use this agenda to answer the design question and prioritize the two real bottlenecks—unique IDs and read amplification:

• 0–5 minutes — Clarify the contract: Confirm anonymous versus authenticated use, maximum paste size, expiration options, public/private behavior, content rendering, and whether search or collaboration is required.
• 5–10 minutes — Establish scale: Calculate creates, views, the 100:1 read/write ratio, average and tail paste sizes, storage growth, the 50ms p99 target, and the durability/availability requirements.
• 10–15 minutes — Define entities and APIs: Use Paste, optional User, POST /pastes, and GET /pastes/{paste_id}. Define 410 versus 404 and the pagination/ownership implications if listing is added.
• 15–22 minutes — Draw the write path: Show validation, rate limiting, atomic counter/base62 ID generation, metadata persistence, content storage, and the cache seed. Keep the counter decision explicit.
• 22–30 minutes — Draw the read path: Start with cache-aside or write-through Redis, then add object storage and a CDN for large/global reads. Explain why a viral paste must not overload PostgreSQL.
• 30–35 minutes — Explain expiration: Check TTL inline, return 410, run batched metadata cleanup, and use an object-store lifecycle rule or retryable deletion for content.
• 35–41 minutes — Cover reliability and security: Discuss dual-write reconciliation, cache loss, replica failover, rate limits, XSS-safe text rendering, access control, abuse controls, and metrics.
• 41–45 minutes — Close with trade-offs: Compare random IDs, hashes, and counters; inline database content versus object storage; CDN versus origin reads; then recap what remains out of scope.

Core Entities

• Paste: A short ID, raw content (or a reference to object storage), creation timestamp, expiration timestamp, and an optional owner ID.
• User: An account that owns pastes, relevant for rate limiting and distinguishing anonymous from registered users. Authentication is out of scope.

We will revisit schema details, including indexes and storage layout, in the deep dives. The entities above are enough to reason about the API and the data flow.

API Design

Start with one endpoint per functional requirement.

FR 1 - Create a paste:

# FR 1: Create a new paste
POST /pastes
Body: { content, expiration_seconds?, syntax_hint? }
Response: { paste_id, paste_url, expires_at? }

POST because we are creating a new resource and the server assigns the ID. The client does not supply the paste_id. Return the full paste_url so the client does not need to reconstruct it from the ID.

FR 2 - View a paste:

# FR 2: Retrieve paste content by ID
GET /pastes/{paste_id}
Response: { paste_id, content, created_at, expires_at? }

Unlike a URL shortener, Pastebin serves content directly rather than redirecting. Return 410 Gone for expired pastes rather than 404 Not Found, because 404 means "never existed" and 410 means "existed but is gone now." That distinction matters for debugging and for abuse detection.

FR 3 and FR 4 - Expiration is transparent. Any endpoint that retrieves a paste checks expires_at inline and returns 410 Gone if the paste is past its expiration. No separate expiration endpoint is needed. The background cleanup job covered in the HLD handles eventual hard deletion from storage.

High-Level Design

Critical flows

The critical flows are paste creation, hot and cold reads, and expiration/cleanup. The numbered designs below start with a simple relational path and add caching, object storage, and lifecycle management only where the workload requires them.

1. Users can create a paste

The write path: validate input, generate a short ID, and persist to the database.

For now, treat short ID generation as a black box. Deep Dive 1 walks through exactly three options and picks the right one.

Components:

• Client: Web or mobile interface sending POST /pastes requests.
• App Server: Validates input size (enforce a hard 10MB cap), generates a paste_id, and writes to the database.
• PostgreSQL: Stores paste_id, content, created_at, expires_at, and owner_id. The source of truth for all paste data.

Request walkthrough:

1. Client sends POST /pastes with content and an optional TTL in seconds.
2. App server validates the content size is under the limit and that the TTL is a positive integer.
3. App server generates a short paste_id (black box for now).
4. App server inserts the paste_id, content, and expires_at into PostgreSQL.
5. App server returns the paste_url to the client.

The write path at 1.2 writes/second is trivial for a single PostgreSQL instance. Keep this path concise and then pivot to the read side, where the 100:1 skew and viral keys create the main scaling pressure.

2. Users can view a paste

The read path carries 99% of the traffic. At 100:1 read/write, every design decision here matters more than anything on the write side.

First, the naive approach: every GET request to /pastes/:paste_id reads directly from PostgreSQL. At 116 reads/second in steady state, PostgreSQL handles this comfortably. But a single viral paste changes everything: hundreds of concurrent requests for the same paste_id hit the same database row and can exhaust connections needed by all other pastes.

The fix is a Redis cache seeded on write. When a paste is created, the app server writes the content into Redis immediately with a TTL matching the paste's expires_at. On a cache hit, the request never touches the database.

Separate reads from writes at the database layer by adding a read replica so the primary only ever receives INSERTs and DELETEs.

Components added:

• Redis Cache: In-memory store, sub-1ms per lookup. Stores paste content keyed by paste_id. TTL equals the paste's expiration so content auto-evicts correctly without explicit invalidation.
• Read Replica: Async replica of the PostgreSQL primary. Serves cache-miss reads so the primary is never touched by a GET request.

Request walkthrough (read path):

1. Client sends a GET request to /pastes/:paste_id.
2. App server checks Redis for paste_id.
3. Cache hit: return content directly in under 1ms.
4. Cache miss: query the read replica for the paste row.
5. Check expires_at. If expired, return 410 Gone.
6. Write content into Redis with TTL equal to expires_at - NOW().
7. Return content to the client.

At a 90%+ cache hit rate, the read replica handles fewer than 12 reads/second in steady state. The cache absorbs everything else.

3. Pastes expire

Expiration has two layers: an inline check to block stale reads immediately, and a background worker for periodic hard deletes.

The inline check is already present in the read path above: every GET checks expires_at before returning content, returning 410 Gone if expired. Redis TTL handles cache-side eviction automatically when the TTL reaches zero. Together, these two mechanisms ensure a client can never receive expired content.

The background cleanup worker removes expired rows from the database so storage does not grow without bound. Run it as a scheduled task every 60 seconds. It does a batched DELETE: query for all pastes with expires_at < NOW() and delete them in batches of 500 rows per iteration.

Component added:

• Expiry Worker: Scheduled job running every 60 seconds. Batch-deletes expired rows from PostgreSQL. Runs on a dedicated connection pool so it does not compete with application traffic.

The Expiry Worker is deliberately simple. If it falls behind during a heavy expiration burst, increase the batch size or raise the run frequency. Expired content is already blocked at the read layer, so falling behind on cleanup is a storage efficiency problem, not a correctness problem.

Deep Dives

Pastebin is an easy-difficulty question, so we cover two deep dives: ID generation and large paste storage with CDN delivery.

1. How do we generate unique paste IDs?

Every paste needs a short, URL-safe ID. The constraints are strict: IDs must never collide globally, must be generated without a bottleneck serializing all writes, and should be short enough to fit cleanly in a URL (6–8 characters). A common mistake is to jump straight to UUIDs: a full UUID is 36 characters, which is longer than needed for a paste URL and expands database indexes.

2. How do we handle large paste sizes and CDN delivery?

The average paste is 10KB, but real workloads include full log files, configuration dumps, and threaded stack traces that regularly hit 1–10MB. Storing all content inline in PostgreSQL creates backup and serving problems. Ask what happens when a user pastes a 5MB server log; tail-size behavior matters more than the average alone.

Final Architecture

The central insight is the layer split: PostgreSQL holds only 200-byte metadata rows, GCS holds the content, and the CDN absorbs read traffic before it ever reaches your servers. The 100:1 read/write ratio is handled by the CDN and Redis operating as a two-tier absorber, leaving the database to do what it does best: small, indexed ACID writes and metadata lookups.

Reliability, Security, and Operations

Reliability. PostgreSQL metadata and the object store are the durable sources of truth; Redis and the CDN are rebuildable or expirable projections. Make the metadata/content write sequence explicit, record an incomplete state when needed, and run a reconciliation job for orphaned metadata or objects. Use replicated database and object storage, retryable cleanup, a dead-letter path for repeated failures, and cache-miss fallback that does not make expired content visible.

Security. Treat paste content as untrusted text: escape it on display, use a restrictive content security policy, validate content type and size, and never execute pasted HTML or scripts by default. Apply IP/account rate limits, protect private objects with short-lived signed URLs, enforce ownership checks when private pastes are added, and keep storage credentials, tokens, and raw private content out of logs. Restrict the cleanup worker and object-store permissions to the prefixes it owns.

Operations. Monitor create success and latency, ID allocation, metadata/content reconciliation gaps, Redis and CDN hit rates, origin fetch latency, read-replica lag, object-store errors, expiration backlog, orphan count, rate-limit rejects, and 4xx/5xx rates by paste size. Alert when cleanup or lifecycle deletion falls behind, when a hot key creates origin pressure, or when cache misses exceed the capacity of the fallback path. Test cache loss, database failover, object-store outage, duplicate cleanup, viral reads, and restore from backup.

Trade-offs and Alternatives

• Counter/base62 vs. random IDs: A counter gives collision-free, compact IDs and easy capacity planning, but reveals creation order. Random IDs hide order but need collision retries and a uniqueness constraint. Hashes are deterministic but can leak content relationships and still need collision handling.
• Inline PostgreSQL content vs. object storage: Inline content is simple for small pastes and transactional with metadata. Object storage isolates large blobs, supports lifecycle rules, and works with a CDN, but introduces a dual-write/reconciliation problem.
• Redis vs. CDN: Redis is useful for API responses and hot keys close to the application. A CDN is better for globally repeated public content. They solve different latency and bandwidth problems and can be layered.
• Write-through vs. cache-aside: Write-through makes the first read fast but couples creation to cache availability. Cache-aside keeps the durable write path simpler, at the cost of a cold-read miss; either way, the cache must be disposable.
• Inline expiry vs. background deletion: Inline checks enforce the product rule immediately. Background deletion reclaims storage eventually and should never be the only expiration mechanism.

Follow-Up Questions

• How would private pastes work? Add authentication and ownership checks, keep objects private, and issue short-lived signed URLs or stream content through an authorized service.
• What if PostgreSQL succeeds but object storage fails? Do not expose the paste as complete; retry through an outbox or reconciliation workflow and clean up any orphaned row or object.
• How do you handle a 100MB paste or a viral paste? Enforce a hard size limit or a separate large-object path, use direct object upload and CDN caching, and protect the origin with request collapsing and rate limits.
• How would search and analytics be added? Publish create/view events asynchronously to a search or analytics pipeline; keep those consumers off the synchronous read and write paths.
• What happens if Redis is lost? Rebuild or refill it from PostgreSQL/object storage; correctness comes from the durable record and inline expiry, not from the cache.
• Can users choose custom aliases? Treat an alias as a separate namespace with a unique constraint, reserved words, abuse checks, and a clear collision/error policy.

Common Mistakes

• Generating random short IDs without a uniqueness constraint or collision-retry strategy.
• Claiming that a six-character base62 space lasts millions of years at the stated write rate; 62^6 is about 56.8 billion values, or roughly 1,500 years at 100K creations per day.
• Storing every large paste inline in the primary database without considering backups, row size, origin bandwidth, or CDN delivery.
• Relying only on a cleanup job for expiration, or returning 404 when the product needs to distinguish an expired paste with 410 Gone.
• Rendering arbitrary paste content as HTML, omitting output escaping, or allowing pasted scripts to run in a viewer.
• Making view counters synchronous, ignoring viral cache keys, or treating Redis/CDN state as durable source data.
• Omitting rate limits, object-store cleanup failures, backup restores, and metadata/content reconciliation.

Interview Cheat Sheet

• Lock down 3-4 core features first, then explicitly name what is out of scope (full-text search, collaborative editing) so the scope is deliberate.
• State the 100:1 read/write ratio within the first few minutes. It explains every caching and scaling decision you make for the rest of the interview.
• The hardest sub-problem is unique ID generation: use an atomic Redis counter with base62 encoding, not random strings or content hashes.
• A 6-character base62 code covers 62^6 = 56 billion values, enough for roughly 1,500 years at 100K pastes per day.
• Redis INCR is atomic: no distributed lock, no collision, no retry logic needed in the write path.
• Use INCRBY 1000 (counter batching) under load to reduce Redis round-trips by 1,000x. Each App Server instance holds a local batch until exhausted.
• Seed the Redis cache on write so the first read of any paste is a cache hit, not a database round-trip.
• Expiration has two layers: inline expires_at check on every read for immediate enforcement, plus a background worker for periodic hard deletes so storage does not grow without bound.
• Return 410 Gone for expired pastes, not 404. The distinction tells operators whether a paste never existed (404) or was removed after a TTL (410).
• Store paste content in object storage (GCS or S3) and only metadata in PostgreSQL. At 1TB over 3 years, keeping content inline in the database creates multi-hour backup windows and blocks CDN delivery.
• A CDN (CloudFront or Fastly) in front of GCS serves content globally under 50ms p99 without touching your application servers on subsequent requests.
• Handle anonymous vs. registered users with a Redis rate-limit counter keyed by IP for anonymous clients and by user_id for registered ones, using a sliding window to prevent paste flooding.
• For private pastes: generate S3 presigned URLs with a 5-minute expiry instead of using the CDN, so access control is enforced on every request without changing CDN configuration.
• The Expiry Worker runs every 60 seconds and deletes in 500-row batches from PostgreSQL only. Use an S3 lifecycle rule to auto-delete content objects so the worker never touches GCS directly.

Test Your Understanding

1. Why use an atomic counter for paste IDs?

   It allocates each integer once, so base62 encoding produces compact IDs without random collision retries.

2. Why use both Redis and a CDN?

   Redis accelerates API and hot-key lookups near the application; the CDN serves repeated public content from global edges and protects the origin.

3. Why return 410 Gone for an expired paste?

   404 means the ID is unknown, while 410 communicates that a known resource existed but expired or was removed.

4. What enforces expiration if the cleanup worker is late?

   The read path checks expires_at inline; cleanup is only eventual storage reclamation.

5. What is the source of truth after Redis loss?

   PostgreSQL metadata and the content backend. Redis can be repopulated from those durable records.

Recap

Use an atomic counter plus base62 for compact unique IDs, keep durable metadata separate from large content, and absorb the 100:1 read skew with Redis and CDN caching. Enforce TTLs on reads, clean up asynchronously, render content safely, and monitor the boundary between metadata, object storage, and derived caches.

Related Concepts

Base62 encoding; ID generation and uniqueness constraints; object storage; CDN caching; cache-aside and write-through caching; TTL and lifecycle deletion; rate limiting; content security policy; asynchronous reconciliation; read replicas; hot-key protection.