Database Control Plane

TL;DR

Separate the database control plane from the data plane. The control plane owns table metadata, schema, partition placement, access policies, and recovery decisions; data nodes serve reads and writes. Store authoritative metadata in a replicated Raft-backed Metadata Store, drive long-running operations through idempotent coordinators, and publish versioned routing updates atomically. Use health monitoring and replica promotion for failures, background copy plus WAL delta replay for live rebalancing and shard splits, and short-lived credentials verified locally on data nodes.

Scope and assumptions

• The design covers one cluster with table lifecycle, partition placement, node health, capacity changes, live shard splitting, and per-table access control. Storage-engine internals, query execution, backups, and multi-region federation are below the line.
• The baseline is illustrative: 1,000 tables, 100 data plane nodes, and 10,000 partitions. Thresholds such as heartbeat intervals, cache TTLs, replica counts, and split lag must be tuned to the data plane and failure model.
• Authoritative topology and schema writes are linearizable through the Metadata Store. Data plane clients may cache routing entries for a short TTL for performance; a stale cache must yield a version mismatch or MOVED response and refresh rather than silently accepting a misrouted write.
• Partitions have replicas and the recovery path promotes only a healthy, sufficiently current replica. Rebuilding a replacement replica is a separate background step after routing is restored.
• Credentials are short-lived signed tokens. Immediate revocation, per-row authorization, and multi-region identity-provider behavior are explicit extensions with additional hot-path or propagation costs.

The numbers and named technologies below are design choices or examples, not guarantees of any particular database product.

What is a database control plane?

A distributed database is two systems stacked on top of each other: the data plane handles reads and writes, while the control plane decides where data lives, who can access it, and what to do when nodes fail or capacity changes. The engineering challenge is that you are designing a distributed system whose job is to manage another distributed system. Every metadata decision must be strongly consistent, because a stale routing entry silently directs a write to the wrong shard.

This question tests control-plane architecture, distributed metadata management, failure detection protocols, live data rebalancing, and the consistency trade-off in a context where stale authoritative metadata is not acceptable.

Functional Requirements

Core Requirements

1. Create, configure, and delete tables: schema definition, partition key selection, and capacity provisioning.
2. Monitor all data plane nodes and automatically recover from node failures via partition reassignment.
3. Scale capacity up or down by adding nodes, splitting shards, and rebalancing partitions without downtime.
4. Authenticate database clients and enforce per-table access control policies.

Below the Line (out of scope)

• Data plane internals (storage engine, WAL management, compaction, query execution)
• Point-in-time restore and application-level backup restore workflows
• Cross-cluster federation and global table multi-region replication
• Query planner and cost-based optimization

The hardest part in scope: Live shard splitting without downtime. The control plane must coordinate a multi-step partition migration while the data plane continues serving read and write traffic on the exact shards being split. The design gives this problem a full deep dive.

Data plane internals are below the line because the control plane and data plane communicate over a narrow API: the control plane tells data nodes which partitions they own and data nodes report health metrics back. What the data node does with its partitions internally is a separate bounded context.

Point-in-time restore is below the line because it does not change the control plane topology logic. A possible extension would periodically snapshot partition data to object storage and add a RestoreTable API that provisions a new table with data rolled back from a chosen snapshot plus WAL replay.

Cross-cluster federation is too large for a single interview. A possible extension is a global routing tier with a cross-region control plane; its schema convergence and write-routing protocol would need to be designed for the chosen consistency model.

Query planner optimization is below the line because it lives inside the data plane's execution layer, not in the control plane's routing or topology logic. A possible extension would add column statistics (cardinality, null rate, value histograms) to the schema registry and expose them to a dedicated query planner service before execution is handed off to the data plane.

Non-Functional Requirements

Core Requirements

• Consistency: Committed metadata writes (routing table, schema, topology) must be linearizable (RPO = 0 for committed metadata writes). A stale cached routing entry must not silently direct a write to the wrong shard; the data plane must reject it or refresh it.
• Availability: 99.99% uptime for the control plane service. A control plane outage means no new tables can be created and node failures cannot be automatically recovered.
• Provisioning latency: Table creation completes within 10 seconds for tables with up to 10 partitions. Clients receive a CREATING status immediately and poll for ACTIVE.
• Failure detection: Node failures are detected within 30 seconds. Partition reassignment (recovery) completes within 120 seconds of detection.
• Scale: Support 1,000 tables per cluster, 100 data plane nodes per cluster, and 10,000 partitions per cluster.

Below the Line

• Sub-second metadata propagation across all nodes (5-second propagation delay is acceptable)
• Multi-region control plane replication (single-region in this design)

Read/write ratio: The control plane sees two entirely different traffic patterns. Provisioning operations (CreateTable, ModifyCapacity, UpdatePolicy) are rare; 1,000 per day cluster-wide is an illustrative estimate. Routing lookups happen on every data plane request. At 100 nodes processing 10,000 requests per second each, an uncached design would face 1,000,000 routing lookups per second. The central tension is therefore clear: authoritative writes must be strongly consistent, while routine lookups should use fast local caches and version checks so the Metadata Store is not a hot-path bottleneck.

Authoritative metadata must be strongly consistent because a stale routing entry can become a silent write-to-wrong-shard error. The 99.99% availability target means the control plane itself must run with Raft-based redundancy, not a single process.

30-Second Answer / Outline

1. Separate the control plane from the data plane and define the Metadata Store as the authoritative source for tables, schemas, partitions, nodes, and policies.
2. Put the Metadata Store behind a replicated Raft group; make table creation, scaling, failure recovery, and shard splitting asynchronous, idempotent workflows.
3. Use health monitoring plus Raft-gated recovery to promote a sufficiently current replica and then rebuild replica capacity in the background.
4. Rebalance or split partitions with background copy and WAL delta replay, followed by a short write pause and one atomic versioned routing-table switch.
5. Let data nodes serve the hot path from local routing caches and short-lived signed credentials; stale entries must produce a refreshable routing error.
6. Close with metrics for heartbeat staleness, partition load, replica lag, and Metadata Store health.

5-Minute Explanation

The client-facing CP API validates a request, writes intent and a job record to the strongly consistent Metadata Store, and returns an asynchronous status such as CREATING or SCALING. Background coordinators read those jobs, assign partitions to healthy nodes, notify the data plane, and update the table only after the required acknowledgments arrive.

For failures, nodes send heartbeats and the Health Monitor moves them through HEALTHY, SUSPECTED, and FAILED. The Recovery Coordinator reads the affected partitions, promotes a healthy current replica, commits the new routing map atomically, and starts a background replica rebuild. For scale-out or a hot shard, the Scaler copies data while the old primary serves traffic, streams WAL deltas, pauses writes briefly, and commits a new routing map in one Raft operation.

Clients and data nodes cache routing metadata for speed, but the cache is not the authority. Each entry carries a version; a stale owner returns MOVED or an equivalent retryable error, and the client refreshes. Access is handled similarly off the data hot path: the Auth Service checks policy once and issues a short-lived signed token that data nodes verify locally.

45-Minute Interview Approach

This is a discussion plan for the design question, not a claim that the article should take 45 minutes to read.

• 0-5 minutes — Set the boundary: Clarify single cluster versus multi-region, data plane responsibilities, table operations, capacity model, replica assumptions, and the meaning of consistency during cache staleness.
• 5-10 minutes — Estimate the control-plane workload: Use the illustrative table, node, partition, and request counts. Separate rare metadata writes from high-volume cached routing lookups.
• 10-16 minutes — Draw the baseline: Show CP API, Raft-backed Metadata Store, Partition Assigner, data nodes, and the asynchronous table-provisioning flow.
• 16-23 minutes — Add failure recovery: Define heartbeat intervals, suspicion windows, authoritative failure decisions, replica promotion, routing-cache refresh, and replica rebuild.
• 23-31 minutes — Add scale and split flows: Explain vnode/range assignment, background copy, WAL delta replay, bounded write pause, and atomic routing replacement.
• 31-36 minutes — Add access control: Compare long-lived keys, inline AuthZ, and locally verified short-lived tokens; discuss revocation and key rotation.
• 36-41 minutes — Cover consistency and operations: State metadata invariants, version checks, idempotency, locks, metrics, alert thresholds, and control-plane quorum loss.
• 41-45 minutes — Close with trade-offs: Re-state what is strongly consistent, what is eventually refreshed, and which extensions—multi-region, backups, query planning—are out of scope.

Core Entities

• Table: A named logical table with a schema, partition key definition, and current status (CREATING, ACTIVE, DELETING, SCALING). The table record is the anchor for all provisioning operations.
• Partition: A shard of a table covering a contiguous key range. Carries the key range boundaries, the primary node assignment, replica node assignments, a version counter, and current status (HEALTHY, SPLITTING, MIGRATING).
• Node: A physical data plane server. Carries its endpoint address, health status (HEALTHY, SUSPECTED, FAILED), last heartbeat timestamp, and the list of partition IDs it owns.
• AccessPolicy: A binding of a principal (IAM role or user ARN) to a set of allowed operations (read, write, admin) on a specific table with an allow or deny effect.
• SchemaVersion: An immutable snapshot of a table's column definitions at a given version number. Used for schema evolution and compatibility validation when modifying a live table.
• Credential: A short-lived signed token issued by the control plane that embeds the principal's allowed actions on a specific table with an expiry timestamp. Verified by data nodes without a round-trip to the control plane.

Key fields are shown below for the most important entities; full index and storage optimization details are deferred to a schema deep dive if pursued in the interview.

API Design

FR 1 - Create and manage a table:

POST /tables
Body: {
  table_name, partition_key, sort_key?,
  capacity: { mode: "PROVISIONED"|"ON_DEMAND", read_units?, write_units? },
  schema: { attributes: [{ name, type }] }
}
Response: { table_arn, status: "CREATING" }

The API returns CREATING immediately rather than blocking until ACTIVE. Partition assignment and data node initialization take several seconds. Clients poll GET /tables/{name} for status. This asynchronous pattern is common for provisioning workflows.

GET /tables/{table_name}
Response: { table_arn, status, partition_count, capacity, created_at }

DELETE /tables/{table_name}
Response: HTTP 202 Accepted

DELETE returns 202 rather than 204 because partition deallocation and data cleanup are async operations. A 204 would imply the deletion is complete.

FR 2 - Inspect cluster health (operator-facing):

GET /clusters/{cluster_id}/nodes
Response: {
  nodes: [{ node_id, endpoint, status, partitions_owned,
            metrics: { cpu_pct, disk_pct, ops_per_sec, replication_lag_ms } }]
}

FR 3 - Modify table capacity:

PUT /tables/{table_name}/capacity
Body: { mode: "PROVISIONED", read_units: 5000, write_units: 1000 }
Response: { status: "SCALING", estimated_completion_seconds: 45 }

Capacity changes are also async. A SCALING status means the Partition Assigner may be splitting or merging shards in the background. The client polls the table status until it returns to ACTIVE.

FR 4 - Manage access policies and issue credentials:

PUT /tables/{table_name}/access-policies/{principal_arn}
Body: { actions: ["read", "write"], effect: "allow" }
Response: { policy_version }

POST /tables/{table_name}/credentials
Body: { principal_arn, ttl_seconds: 3600 }
Response: { token, expires_at }

POST /credentials issues short-lived signed tokens, not long-lived API keys. A leaked long-lived key gives an attacker indefinite access. A 1-hour token limits blast radius to a narrow window and rotates automatically without any client credential management.

High-Level Design

1. Creating and configuring a table

The write path for table provisioning is a multi-step workflow, not a single synchronous call.

The naive approach is a single HTTP endpoint that registers a schema and returns 200. The problem is that table creation actually involves partition math (how many shards for this capacity?), node selection (which healthy nodes should own them?), and data node initialization (allocate storage, notify the node). None of that fits in a synchronous HTTP response. The correct pattern is: write intent to Metadata Store, return CREATING, let a background coordinator drive the workflow to completion.

Components:

• CP API Service: Stateless HTTP service. Validates the CreateTable request, writes the table record with status CREATING to the Metadata Store, and returns immediately.
• Metadata Store: A Raft-based strongly consistent KV store (like etcd). The single source of truth for the routing table, schema registry, and topology. Every control plane decision writes here first.
• Partition Assigner: Reads pending CreateTable jobs. Computes the initial partition count from the requested capacity. Assigns each partition to a healthy data node. Writes the partition map to the Metadata Store.
• Data Plane Nodes: Receive partition ownership notifications. Allocate storage for the new partition. Reply to the CP API with an ACK.

Request walkthrough: