YouTube

What is YouTube?

YouTube is a platform where users upload, store, and stream video globally. The engineering challenge is not storage; it is converting 500 hours of raw video per minute into multiple adaptive bitrate formats and delivering each stream at the right resolution to viewers in 200+ countries within minutes of upload. The problem is best framed as a pipeline design first and a storage problem second: getting the transcoding architecture right unlocks the remaining decisions.

TL;DR

Create a video record, return a scoped pre-signed multipart upload URL, and let the client upload bytes directly to durable object storage. An upload-complete event starts an idempotent, chunk-parallel transcoding workflow that writes adaptive-bitrate segments and a manifest back to object storage. Publish readiness to Kafka so search indexing, CDN warming, and other consumers remain asynchronous.

Serve playback through a CDN using HLS or DASH, read metadata from a partitioned store, and keep view-count increments in a fast derived counter with periodic durable flushes. The key contract is fast upload acknowledgement and playback start, not synchronous transcoding completion.

Scope and assumptions

These are illustrative planning assumptions for the design; codec, region, and retention choices should be validated with real media workloads:

• The platform receives about 500 hours of uploads per minute and serves roughly 1 billion hours of playback per day, with a target playback start under 2 seconds.
• Each accepted video produces several resolution/codec variants, split into independently processable segments. Upload acknowledgment is under 500 ms; readiness is asynchronous and may take minutes.
• Raw uploads and encoded segments are durable across multiple regions or failure domains. Public playback is CDN-backed; private playback requires authorization and signed delivery URLs.
• Video metadata, playback manifests, search documents, and view counters are separate access patterns. At-least-once workflow delivery and eventual search/index freshness are acceptable.
• This article covers VOD upload, processing, metadata, search, and adaptive playback. Comments, recommendations, live streaming, ads, subscriptions, DRM, and full moderation workflows are outside the primary design.

Functional Requirements

Core Requirements

1. Users can upload a video file.
2. After upload, the video becomes available to watch (transcoded into multiple resolutions).
3. Users can stream a video at a resolution appropriate for their device and connection.
4. Users can search for videos by title and description.

Below the Line (out of scope)

• Comments, likes, and subscriptions
• Recommendations and personalized home feed
• Live streaming
• Monetization and ads

The hardest part in scope: Video transcoding. A raw uploaded file must be converted into 6+ resolution variants (360p, 480p, 720p, 1080p, 4K, HDR) before the upload is considered complete. At 500 uploads per minute, the transcoding pipeline is the highest-throughput compute subsystem in the architecture.

Comments, likes, and subscriptions are below the line because they do not affect the upload or streaming paths. To add them, store a video_comments table keyed by (video_id, comment_id) and a video_reactions table keyed by (video_id, user_id). Like counts can be cached in Redis and reconciled to a database asynchronously.

Recommendations are below the line because they form a completely separate offline ML pipeline. To add them, emit watch events to a Kafka topic and train a collaborative filtering model offline, serving recommendations via a low-latency feature store.

Live streaming is below the line because it replaces the upload-then-transcode model with a real-time ingest and segment delivery model (HLS or DASH live). The architecture diverges significantly from the stored video path.

Monetization is below the line because ad serving is a separate system with its own auction, targeting, and reporting infrastructure that does not touch the core upload or playback path.

Non-Functional Requirements

Core Requirements

• Availability: 99.99% uptime for video playback. Availability over consistency: a viewer watching a video should never see a playback error due to backend failures.
• Latency: Video playback begins within 2 seconds of pressing play. Upload acknowledgment completes within 500ms (the actual processing continues asynchronously).
• Throughput: 500 hours of video uploaded per minute. 1B hours of video watched daily (roughly 41.7M concurrent streams at any moment, calculated as 1B hours × 3,600 s/hr ÷ 86,400 s/day).
• Durability: Uploaded video must never be lost. Stored across at least 3 geographic regions.
• Search latency: Search results return within 500ms p99.

Below the Line

• Sub-100ms time-to-first-byte via edge PoPs in every major city
• Real-time view count consistency

Read/write ratio: Video streaming traffic dwarfs upload traffic by a factor of roughly 1,400:1. For every 500 hours of video uploaded per minute, roughly 694,000 hours of video are consumed per minute (1B hours per day ÷ 1,440 minutes). This asymmetry shapes every decision in this article: the entire write path (upload, transcode, storage) can be slow and asynchronous because the read path (streaming) must be fast and globally distributed.

The 2-second playback start target rules out serving video files directly from a central origin server. Network round-trip time alone from Asia to a US datacenter is 150-200ms, and streaming a 1080p file at 8 Mbps from a single origin saturates bandwidth quickly. CDN edge delivery is mandatory, not optional.

Call out the 2-second playback target early: it rules out a single-origin setup and makes CDN delivery a non-negotiable component before the rest of the design.

30-second answer / outline

1. Create a Video record and return a scoped, expiring pre-signed multipart upload URL so large bytes bypass the API tier.
2. After object-storage completion, publish UploadComplete to Kafka and enqueue a workflow that validates, splits, and transcodes chunks into several resolutions/codecs in parallel.
3. Commit the manifest only after required variants are complete, update metadata to ready, and publish VideoReady for search indexing and CDN warming.
4. Serve the manifest and segments through a CDN using HLS/DASH; the player adapts bitrate based on bandwidth and buffer state, with object storage as origin.
5. Make each upload, chunk, manifest, index update, and view-count flush idempotent; retry failed jobs, isolate bad inputs, and surface processing status instead of blocking the upload request.

5-minute explanation

Start with the asymmetry: uploads are large, relatively infrequent writes, while playback is a globally distributed, bandwidth-heavy read path. Upload bytes should go directly from the client to object storage with multipart checksums and resumability. The Upload Service handles authorization and metadata, creates a processing row, and returns immediately.

Object-storage completion triggers a durable workflow. A coordinator splits the source into independently processable chunks; workers transcode each chunk for each required resolution and codec. Completion markers make retries safe, and the workflow publishes a manifest only when the required set is complete. A failed codec or malformed source goes to a retry/dead-letter path without blocking unrelated videos. VideoReady then drives asynchronous Elasticsearch indexing and optional CDN prewarming.

Playback requests fetch a small HLS or DASH manifest and then request short segments from a nearby CDN edge. The CDN absorbs repeated traffic; object storage is the origin and durable source. The Video Service serves metadata and may increment a Redis view counter, but it does not proxy segment bytes or synchronously update a relational row for every view. Metadata is denormalized for watch and uploader access, while search is a separate eventually consistent index.

The design’s reliability boundary is staged readiness: a visible video should reference only an atomically published manifest whose required variants exist. Upload, transcode, index, CDN, and counter operations can be replayed independently. Playback can continue from cached segments during a metadata or search outage, and clients should distinguish processing, ready, and failed states.

45-minute interview approach

Prioritize the upload-to-ready workflow and the playback delivery path; defer social features and monetization unless they change those flows.

• 0–5 minutes — Clarify the contract: Confirm VOD versus live, file limits, codecs/resolutions, upload resumability, readiness SLA, playback start/rebuffer targets, privacy, search, retention, and deletion.
• 5–10 minutes — Establish scale: Calculate upload bytes, concurrent playback/egress, segment size and request rate, transcoding CPU/GPU work, object-store capacity, CDN hit ratio, metadata/search volume, and cost drivers.
• 10–15 minutes — Define entities and APIs: Walk through Video, VideoVariant, User, SearchIndex, upload sessions, status polling, manifest retrieval, search, checksums, and signed URLs.
• 15–22 minutes — Draw upload and processing: Show the gateway, Upload Service, metadata store, pre-signed object upload, completion event, workflow queue, chunking, worker fan-out, manifest publication, and VideoReady.
• 22–30 minutes — Draw playback: Show the player, manifest, CDN, segment cache, object-storage origin, adaptive bitrate selection, origin failover, and metadata lookup. Keep binary data off application servers.
• 30–35 minutes — Deep dive on transcoding and storage: Cover chunk granularity, stragglers, codec/variant policy, idempotency, Cassandra access patterns, view counters, and search indexing.
• 35–41 minutes — Reliability, security, and operations: Cover resumable uploads, corrupt inputs, retries, deletion propagation, CDN/origin failure, signed URLs, privacy, queue lag, rebuffering, and cost/egress monitoring.
• 41–45 minutes — Trade-offs and close: Compare proxy uploads, whole-file versus chunk transcoding, HLS versus DASH, CDN strategies, metadata stores, and search engines; recap why object storage plus async processing plus CDN is the core.

Core Entities

• Video: The uploaded content. Carries a video_id, uploader_id, title, description, status (processing, ready, failed), and created_at. The status field tracks where the video is in the transcoding pipeline.
• VideoVariant: A single transcoded output for a specific resolution and codec. Links back to video_id and stores the CDN URL for the variant file. A single video produces 6-8 variants.
• User: An account. Carries a user_id, display name, and channel metadata.
• SearchIndex (derived): An inverted index over video titles and descriptions. Not a stored table; populated asynchronously from Video records and served by a dedicated search service.

The full schema, indexes, and partition keys are deferred to the data model deep dive. The four entities above are sufficient to drive the API design and High-Level Design.

Treat SearchIndex as a derived entity rather than a first-class stored table; introduce it when functional requirement 4, search, is discussed.

API Design

Upload a video:

POST /videos/upload
Body: multipart or a pre-signed S3 URL response
Response: { video_id, upload_url }

Get video metadata and playback manifest:

GET /videos/{video_id}
Response: { video_id, title, description, status, manifest_url }

Stream a video (adaptive bitrate manifest):

GET /videos/{video_id}/manifest.m3u8
Response: HLS manifest listing all resolution variants

Search for videos:

GET /search?q={query}&cursor?
Response: { videos: [...], next_cursor }

Pre-signed upload URL: Rather than accepting binary file data through the API server, the Upload Service generates a pre-signed S3 URL and returns it to the client. The client uploads directly to S3 bypassing the application tier entirely. This keeps large binary payloads off the API servers, removes an entire network hop, and lets S3 handle multipart resumable uploads natively. The API server only deals with metadata.

HLS vs raw file URL: The manifest_url points to an HLS (.m3u8) or DASH manifest, not a direct video file URL. The manifest lists all available resolution variants and segment URLs. The video player selects segments adaptively based on available bandwidth. This is how YouTube, Netflix, and every major streaming platform delivers video today.

Cursor-based pagination applies to search results. Offset pagination breaks when new videos are indexed between pages. A cursor encoding the last-seen video_id ensures stable pagination.

My recommendation for the upload flow is to return a video_id immediately with status=processing and have the client poll for status=ready. Blocking the upload API response on transcoding completion would mean the client waits 5-15 minutes for a 201.

High-Level Design

The critical flows are: create and complete an upload, process it into a ready manifest, serve metadata/search independently, and deliver segments from the CDN with object storage as the durable origin.

1. Users can upload a video file

The write path: client requests an upload URL, uploads directly to object storage, the server records the video metadata and begins transcoding.

Components:

• Client: Web or mobile app sending the initial upload request.
• Upload Service: Validates the request, generates a pre-signed upload URL, and creates a Video record with status = processing.
• Object Storage (S3): Stores the raw uploaded file. Durable, replicated, designed for large binary objects.
• Video DB: Stores video metadata and tracks processing status.

Request walkthrough:

1. Client sends POST /videos/upload with the video title and optional description.
2. Upload Service creates a Video record in the Video DB with status = processing.
3. Upload Service generates a pre-signed S3 URL (valid for 1 hour) and returns it with the video_id.
4. Client uploads the raw video file directly to S3 using the pre-signed URL.
5. S3 triggers a storage event when the upload completes.

The client uploads the raw file directly to S3, bypassing the Upload Service entirely. The API tier only handles metadata. Transcoding is deferred to the next requirement.

2. After upload, the video becomes available to watch

Transcoding pipeline: when the raw upload lands in S3, an async worker picks it up, converts it into multiple resolution variants, stores them back in S3, and marks the video ready.

Components: