File Downloader

TL;DR

Split the service into a control plane and a data plane. The API authenticates the request, creates a download session, and returns a short-lived presigned CDN/object-storage URL. The CDN handles byte delivery and HTTP Range requests, so application servers do not proxy large files. The client persists chunk-aligned bytes_confirmed, resumes with Range requests, and uses a dynamic work queue for parallel downloads. Normalize auth parameters out of the CDN cache key, but keep them available for edge validation, and verify chunks plus the assembled file before completion.

The design assumes files are immutable for a session or identified by a stable version/ETag. The key invariants are that the server records bytes the client has actually persisted, a changed file cannot be silently appended to an old download, expired sessions and URLs have a recovery path, and integrity checks distinguish accidental corruption from adversarial tampering.

Scope and assumptions

• The service covers authenticated downloads, pause/resume, progress, arbitrary byte ranges, parallel multipart transfer, CDN caching, and integrity validation. Uploads, DRM, per-user throttling, scheduling, and analytics remain below the line.
• The capacity figures are illustrative planning assumptions: files from 1KB to 50GB, 10M concurrent downloads, a 50Mbps client target, 200ms cached first byte, and a 72-hour session lifetime.
• A CDN or equivalent edge layer is assumed to support signed-request validation and Range responses. Provider-specific cache-key, partial-object, shield, and token settings must be verified before production use.
• File metadata and sessions are control-plane state; file bytes and per-file manifests live in storage/data-plane systems. A session must bind to a file version, checksum, and ETag so replacement is detected.
• Progress checkpoints are advisory resume state, not proof of delivery to a client. The client confirms only bytes that were fully written and verified locally.

What is a file download service?

A file download service delivers files from server storage to clients reliably and efficiently. The simplicity is deceptive: the real engineering challenge is serving arbitrary byte ranges from multi-gigabyte files concurrently across millions of connections, without flooding your origin storage or losing a 5 GB resume state when a mobile client drops off the network for 30 minutes. This question tests CDN architecture, HTTP Range requests, connection management, distributed session state, and how to separate the control plane (session management) from the data plane (moving bytes).

Functional Requirements

Core Requirements

1. Users can initiate file downloads from web or mobile clients.
2. Downloads can be paused and resumed; state persists across client restarts.
3. The client reports real-time download progress (bytes received out of total).
4. The system handles files from 1 KB to 50 GB.

Below the Line (out of scope)

• File upload pipeline
• Per-file digital rights management (DRM)
• Per-user download speed throttling
• Download scheduling and queue prioritization

The hardest part in scope: Serving byte-range requests for 50 GB files across 10 M concurrent connections without proxying bytes through application servers. The challenge sits at the intersection of CDN edge caching, Range request semantics, and distributed session state for pause-and-resume.

File upload is below the line because it is an independent write path with different reliability constraints (chunked upload, deduplication, virus scanning) that do not influence the download architecture.

DRM is below the line because it requires license server integration, per-device key management, and encrypted segment delivery. A later extension could generate a time-limited DRM token alongside the pre-signed download URL and configure the CDN to enforce token validity before serving each segment.

Speed throttling is below the line because it adds significant CDN configuration complexity. A later extension could configure per-client token-bucket rate limits at the CDN edge. That configuration layer sits above the download path in this design.

Non-Functional Requirements

Core Requirements

• Availability target: 99.99% uptime. Availability over strict consistency; a slightly stale progress count is far better than an unavailable download endpoint.
• Latency target: First byte under 200 ms p99 for CDN-cached files; under 1 s for origin-fetched cold files.
• Throughput assumption: 10 M concurrent downloads sustained; p99 single-client transfer speed at or above 50 Mbps.
• Integrity requirement: Every delivered file must SHA-256-match the server-recorded checksum. Partial delivery must also be verifiable per chunk.
• Range requirement: The data plane must respond correctly to any Range: bytes=X-Y request within a file, enabling pause-resume and parallel multi-part download.

Below the Line

• Geographic IP-based content restrictions
• Per-user bandwidth billing and metered egress quotas
• Real-time download analytics (top files, geographic heat maps)

Read/write ratio: For popular software packages or media files, the read-to-write ratio reaches 1,000,000:1 or more. A single OS release uploaded once can be downloaded by tens of millions of clients over days. The entire design is a read optimization problem. Every architectural decision traces back to this skew: how do we move bytes from origin to client without the origin seeing most of the traffic?

The concurrency, throughput, latency, hit-rate, memory, and cost figures in this article are illustrative targets or workload assumptions, not provider guarantees. Validate them with the CDN, file-size distribution, client network mix, and origin contract selected for deployment.

Under 200 ms first-byte latency for cached files demands CDN edge placement within one network hop of the client. The 10 M concurrent download target rules out proxying bytes through application servers entirely: even ignoring bandwidth, each open TCP connection parks a goroutine or thread, and 10 M goroutines is not a fleet you want to maintain.

30-second answer / outline

“I would make the API a control-plane service and keep file bytes on the CDN/object-storage data plane. POST /downloads authorizes a file, creates a session, and returns a short-lived signed URL plus size, checksum, ETag, and manifest metadata. The client uses HTTP Range requests, persists a chunk-aligned bytes_confirmed, and downloads large files through a bounded work queue that retries failed slices. The CDN strips signature parameters from the cache key while validating them at the edge. Per-chunk checksums and a final full-file hash protect accidental corruption; a signed Merkle manifest is the stronger integrity option.”

5-minute explanation

1. Start with the bottleneck: streaming every byte through application servers makes concurrent connection count, bandwidth, and origin egress scale with clients.
2. Separate planes: the API creates sessions and signs access; the CDN serves bytes and handles Range requests; object storage is the origin. The session binds the URL to a file version and checksum.
3. Make downloads resumable: the client writes completed ranges to temporary segments, periodically confirms only contiguous persisted bytes, refreshes expired URLs, and sends If-Match to detect replacement.
4. Make large downloads efficient: use small range work units with bounded parallel workers, requeue failures, assemble in order, and checkpoint the lowest contiguous completed offset.
5. Protect cache and integrity: normalize cache keys without bypassing edge authorization, use edge/shield policies for popular files, validate per-chunk hashes, and perform a final full-file check before marking complete.

45-minute interview approach

This is an interview plan for the design question, not a claim that the article should take 45 minutes to read.

• 0-5 minutes — Clarify scope: confirm authenticated versus public files, immutable versions, maximum size, resume across devices, client platforms, CDN capabilities, and integrity/threat requirements.
• 5-10 minutes — Requirements and capacity: state the illustrative concurrency, throughput, first-byte, availability, Range, checksum, session-TTL, and read/write assumptions. Calculate aggregate egress to motivate CDN offloading.
• 10-17 minutes — APIs and entities: sketch session creation, session state/progress, file metadata/manifest, Range delivery, and the File/DownloadSession/FileChunk relationships.
• 17-25 minutes — Baseline and evolution: draw the naive proxy path, quantify its connection and bandwidth cost, then evolve to presigned URLs, CDN delivery, and origin shielding.
• 25-33 minutes — Critical flows: walk initial download, pause, crash recovery, URL refresh, Range 416/ETag mismatch, parallel parts, assembly, and completion checkpointing.
• 33-40 minutes — Deep dives: compare checkpoint locations, fixed versus adaptive versus work-stealing ranges, cache-key policies, edge/shield behavior, and full-file versus per-chunk/Merkle integrity.
• 40-45 minutes — Reliability, security, and operations: cover bearer-URL leakage, key rotation, DDoS/rate limits, session cleanup, retries, origin protection, observability, and what changes for DRM or mutable files.

Core Entities

• File: The downloadable artifact. Carries file_id, name, size_bytes, content_type, storage_key (S3 object key), checksum (SHA-256 of full file), and created_at.
• DownloadSession: Tracks pause-and-resume state. Carries session_id, file_id, client_id, bytes_confirmed (last checkpointed byte offset), total_bytes, status (active, paused, completed, expired), download_url, and expires_at.
• FileChunk: Addressable sub-range of a large file (introduced in the integrity deep dive). Carries chunk_index, file_id, start_byte, end_byte, and a per-chunk checksum.

Full schema with indexes and access patterns is deferred to the data model section of the deep dives. These three entities are sufficient to drive the API design and High-Level Design from here.

API Design

FR 1 - Initiate a download:

The naive instinct is a direct GET that streams the full file:

GET /files/{file_id}
Response: 200 OK, Content-Type: application/octet-stream, binary body

This breaks immediately at scale: the API server proxies all bytes, consuming one thread or goroutine per active connection. At 10 M concurrent downloads, that is 10 M goroutines before accounting for bandwidth.

The evolved shape separates intent from delivery. The client tells the API what it wants; the API returns a URL pointing directly to the CDN:

POST /downloads
Body: { file_id, client_id? }
Response: { session_id, download_url, file_size, checksum, content_type, expires_at }

The download_url is a pre-signed URL pointing to the CDN or S3 directly. The client fetches from that URL without touching the application server again:

GET {download_url}
Headers: Range: bytes=0-  (optional; omit for full file)

Response: 200 OK
  Accept-Ranges: bytes
  Content-Disposition: attachment; filename="large-file.zip"
  Content-Length: 10737418240

  206 Partial Content  (when Range header present)
  Content-Range: bytes 104857600-10737418239/10737418240
  Accept-Ranges: bytes

POST over GET for session creation: creating a DownloadSession is a write with a persistent side effect. POST is correct. The pre-signed URL encodes the authorization; the CDN validates it at the edge on each range request without calling back to the application server.

FR 2 - Pause and resume:

PATCH /downloads/{session_id}
Body: { status: "paused" | "resumed", bytes_confirmed: 104857600 }
Response: { session_id, status, bytes_confirmed, download_url }

bytes_confirmed is the last byte offset the client has fully received and verified. On resume, the client issues a Range: bytes={bytes_confirmed}- request against the download_url.

GET /downloads/{session_id}
Response: { session_id, file_id, status, bytes_confirmed, total_bytes, download_url, expires_at }

GET fetches current session state when a client restarts and needs to know where it left off.

FR 3 - Progress:

Progress is primarily tracked client-side: the download client counts bytes received in memory and renders a progress bar locally. Server-side progress is useful for server-to-server transfers or multi-device syncing:

GET /downloads/{session_id}/progress
Response: { bytes_confirmed, total_bytes, percentage, status, transfer_rate_bps }

Authorization note: All endpoints assume a client token in the Authorization header. The pre-signed download URL embeds authorization in its HMAC signature; the CDN validates it without a round-trip to the API server. If authentication is in scope, associate each session with a user_id from the session token and enforce it on POST /downloads.

High-Level Design

1. Initiating a download - the naive proxy path

The simplest possible design streams bytes through the API server itself.

Components:

• Client: Web or mobile app sending POST /downloads, then fetching the file.
• API Server: Creates a session, fetches the file from S3, and streams bytes back to the client.
• Sessions DB: Stores download session rows (session_id, file_id, bytes_confirmed, status).
• S3 / Object Storage: Stores the raw file bytes.

Request walkthrough:

1. Client POSTs to /downloads with file_id.
2. API Server creates a session row in the Sessions DB.
3. API Server opens a GET to S3 for the full file.
4. API Server streams S3 response bytes directly back to the client.
5. Client accumulates bytes to disk.

This is the write path and byte delivery path combined. The API server here carries all download bandwidth.

What breaks: At 10 M concurrent downloads at 50 Mbps each, the API tier must sustain 500 Tbps of egress. No application server fleet handles that at reasonable cost. Each connection parks a goroutine for the duration of a multi-hour large file download. S3 charges per API call and per-GB egress; every client download hits S3 directly.

2. Evolved - pre-signed URL and CDN offloading