Data Migration

TL;DR

Design the migration as a durable, chunked batch job. A coordinator creates an immutable manifest, transfer workers read source chunks under a global rate limit, and workers write each chunk to a stable target key. An independent verifier recomputes the target checksum before the chunk is marked complete. A durable task queue, leases, idempotent writes, and checkpoints let another worker resume after a crash. The migration is complete only when every chunk is verified and the final integrity report is written.

Scope and assumptions

• This is a bulk migration of a bounded dataset from an on-premises file system or database to cloud object storage or a managed database. Schema transformation and post-copy change data capture (CDC) are separate phases described as extensions.
• The baseline numbers are illustrative design assumptions: 1 PB over 30 days is about 1.4 TB/hour (roughly 0.39 GB/s using decimal units), with 64 MB chunks and up to 50 workers. A worker that reads at 200 MB/s gives about 10 GB/s aggregate read capacity across 50 workers, while a 150 MB/s write rate gives about 7.5 GB/s aggregate write capacity. Actual limits come from the source, network, and target.
• Source records or byte ranges have stable identities during the bulk phase. If source data changes, the migration needs a snapshot, CDC, or a cutover protocol; the bulk-copy path alone cannot make a moving source current.
• The target supports atomic object replacement, conditional writes, or resumable multipart writes. A byte-offset checkpoint is safe only when the target can append or replace the corresponding range without leaving an object with a missing prefix.
• SHA-256 is the illustrative integrity algorithm. A deployment should choose the algorithm and storage format with its threat model, compliance requirements, and hardware support in mind.

Numbers such as chunk size, worker count, thresholds, and timeouts are tuning inputs rather than universal guarantees. Keep the units explicit when estimating capacity.

What is a large-scale data migration system?

A data migration system copies data from one infrastructure (typically on-premises databases) to another (typically cloud object storage or a managed database), verifying that nothing was lost or corrupted in transit. The key engineering challenge is not the copying itself. It is verifying integrity at petabyte scale, limiting the read load placed on production source systems, and recovering from machine failures that happen days into a multi-week job without restarting from scratch. The design therefore combines distributed coordination, idempotency, checkpointing, integrity verification, and throughput control.

Functional Requirements

Core Requirements

1. Copy petabytes of data from source systems to the target cloud storage.
2. Verify that every byte was transferred without corruption.
3. Tolerate and resume from machine failures without re-transferring data already migrated.
4. Minimize impact on the production systems being read from.

Below the Line (out of scope)

• Schema transformation / ETL. We assume source and target schemas are compatible. Real migrations often require field remapping, type coercion, or format changes. To add ETL, you'd insert a transformation worker between the reader and the writer, applying a schema map that's versioned alongside the migration job. Keeping that transformation logic correct across billions of rows is an article in itself.
• Change data capture (keeping source and target in sync after cutover). Once the bulk migration completes, new writes to the source create drift. CDC pipelines like Debezium or DMS handle this, but they belong to a post-migration sync phase, not the bulk transfer system we're designing.

Call out both scope boundaries explicitly. CDC is important for a live source, but it belongs to a post-bulk synchronization phase rather than the core byte-transfer path.

The hardest part in scope: Verifying that every byte arrived correctly is the central integrity challenge. Counting rows is not enough. Computing checksums per chunk and comparing them end-to-end, at petabyte scale, without an unnecessary extra source pass, deserves a full deep dive.

Non-Functional Requirements

Core Requirements

• Scale: 1 PB of data migrated over 30 days. That averages roughly 1.4 TB/hour (about 0.39 GB/s) using decimal units.
• Throughput per worker: Each transfer worker is assumed to sustain 200 MB/s read from source and 150 MB/s write to target (roughly 720 GB/hour read and 540 GB/hour write per worker). At 50 parallel workers, aggregate read throughput reaches about 10 GB/s, providing roughly 25x headroom above the average read rate required by the 30-day illustrative target.
• Data integrity: Zero bytes lost or corrupted. Every chunk is verified with a SHA-256 checksum computed on the source before transfer and recomputed on the target after write.
• Production impact: Total read rate on the source is capped at 10% of peak production query load. Exceeding this would starve live traffic and is a hard constraint, not a nice-to-have.
• Resumability: After any failure, the coordinator detects the lost worker within 30 seconds and reassigns the task within 1 minute. No chunk that was successfully written and verified is re-transferred.

Below the Line

• Sub-second migration latency (this is a batch system, not a streaming one)
• Multi-region active-active replication during migration
• Automatic schema compatibility validation

Read/write ratio: A migration is fundamentally read-heavy on the source and write-heavy on the target. For every 1 byte read from source, exactly 1 byte is written to target, plus roughly 1 additional read on the target for verification. This 1:1:1 read/write/verify pattern means source capacity is the binding constraint for throughput, not target write bandwidth. That shapes the entire architecture: the coordinator's job is to minimize how many times we touch the source, not the target.

30-Second Answer / Outline

1. Clarify whether the source is a stable snapshot, the target format, the cutover requirement, and the source read budget.
2. Split the dataset into independently addressable chunks and persist the manifest before dispatching work.
3. Use a coordinator, durable queue, leased tasks, and stateless workers to copy chunks in parallel under a global source token bucket.
4. Compute and persist a source checksum, write each chunk idempotently, and have an independent verifier read the target and compare checksums.
5. Checkpoint verified progress, retry only failed or expired tasks, and publish a job-level integrity report after all chunks pass.
6. If the source is changing, add a CDC and cutover phase rather than pretending the bulk copy alone provides freshness.

5-Minute Explanation

The client submits a migration job with source and target locations plus a throughput limit. The coordinator discovers the source layout, creates a manifest of stable 64 MB chunks, and records each chunk's checksum and status in durable metadata. Workers claim leased tasks from a queue, acquire source-read tokens, read their assigned ranges, and write to stable target keys such as {job_id}/{chunk_id}.

A transfer acknowledgment is not a completion acknowledgment. After a write, an independent Integrity Verifier reads the target chunk, recomputes SHA-256, and compares it with the source checksum. Only a matching chunk becomes verified; a mismatch is re-enqueued for that chunk. The coordinator can then build a Merkle-root summary and an auditable report from the verified chunk checksums.

Worker heartbeats and queue visibility timeouts handle machine loss. A replacement worker reads the durable checkpoint and resumes a resumable write or retries the current chunk. Stable keys make duplicate delivery safe. The coordinator protects the production source with a global token bucket and adapts the refill rate to source latency and CPU, while the target can scale independently. A live mutable source requires a later CDC drain and cutover protocol.

45-Minute Interview Approach

This is a discussion plan for the design question, not a claim that the article should take 45 minutes to read.

• 0-5 minutes — Clarify the contract: Confirm source type, target type, dataset size, whether the source is static, acceptable downtime, integrity definition, cutover needs, and source read budget.
• 5-10 minutes — Estimate capacity: Convert the data volume and deadline into an average rate, choose an illustrative chunk size, estimate worker throughput, and identify whether source, network, or target is the binding constraint.
• 10-18 minutes — Establish the happy path: Draw client, coordinator, manifest store, task queue, transfer workers, source, and target. Walk through submit, claim, read, write, and status transitions.
• 18-25 minutes — Add integrity: Explain source checksums, target read-back verification, idempotent keys, the distinction between transferred and verified, and the final report.
• 25-32 minutes — Add failure handling: Cover leases, heartbeats, checkpoint durability, duplicate delivery, partial writes, retry limits, dead-letter tasks, and coordinator recovery.
• 32-38 minutes — Protect the source: Compare fixed per-worker limits with a coordinator-level token bucket and feedback-based AIMD. Discuss chunk size and verification I/O as source-load trade-offs.
• 38-43 minutes — Address mutability and operations: Explain snapshot versus CDC, cutover, encryption and access control, observability, reconciliation, and audit retention.
• 43-45 minutes — Close with trade-offs: Re-state the completion invariant—every chunk is verified—and name the main alternative designs and their costs.

Core Entities

• MigrationJob: The top-level unit representing a single migration run. Tracks source location, target location, throughput limit, overall status (pending, running, paused, complete, failed), and job-level metadata like start time and estimated completion.
• MigrationChunk: A contiguous slice of data within the job, described by a byte range or row range. Each chunk carries its SHA-256 source checksum, transfer status (pending, in-progress, verified, failed), and the worker ID that last attempted it.
• WorkerTask: A unit of work dispatched to a specific transfer worker. Maps one chunk to one worker, with a lease expiry time. If the worker does not heartbeat before expiry, the task is reassigned.
• IntegrityReport: The result of comparing source and target checksums for a chunk or the full job. Stores both checksums, comparison result, and timestamp. This is the audit trail that proves the migration completed correctly.
• MigrationCheckpoint: The last durably confirmed progress marker for a job. Written after each chunk is verified. Enables the coordinator to reconstruct in-flight state after a crash without re-scanning the entire job manifest.

Schema details belong in the data model deep dive. These five entities are enough to reason about the full design.

API Design

One endpoint per core functional requirement. The naive shapes work for small migrations, and we'll show where they break at scale.

FR 1: Submit a migration job:

Naive shape:

POST /migrations
Body: { source_path, target_path }
Response: { job_id, status: "pending" }

This works but gives the coordinator no signal about how aggressively to read from the source. A 50-worker migration targeting a production OLTP database would immediately saturate it. The evolved shape adds explicit throttle parameters: