Health Monitor

TL;DR

• Poll the cluster from a partitioned pool of regional checkers, and treat every check result as an event that can be replayed.
• Keep the latest status in a fast cache for dashboards, while retaining raw checks and hourly aggregates in a time-series store.
• Drive alerts with a durable state machine: tolerate transient failures, deduplicate an ongoing incident, retry notification delivery, and escalate when there is no acknowledgement.
• Keep remediation behind an explicit policy, an idempotent action queue, and a circuit breaker. The monitoring system should fail safe when its own dependencies are unhealthy.
• For the illustrative scenario, 10,000 nodes checked every 30 seconds produce about 10,000 / 30 = 333.3 check results per second.

Scope and assumptions

This article designs a cluster health monitoring system that checks machines or service instances, exposes current and historical status, alerts on sustained failures, and can run a small set of approved remediation actions. It tests time-series storage, distributed polling, alert delivery, and the trade-off between false positives and missed incidents.

The interview scenario uses these illustrative assumptions:

• 10,000 nodes, each checked every 30 seconds, with an HTTP health endpoint and a bounded request timeout.
• Binary liveness plus measured latency are in scope; richer CPU, memory, and disk metrics are optional extensions rather than the primary alert signal.
• Operators need current status, alert acknowledgement, historical status, and an audit trail for remediation.
• A node can be unreachable because it is down or because the network path is broken. The system reports the observation and can correlate results from more than one checker region; it does not claim to identify the physical root cause with certainty.
• The numeric scale, latency, retention, and alert thresholds below are interview requirements for this design, not production measurements or provider guarantees.

Functional Requirements

Core Requirements

1. Periodically check the health of every service and machine in the cluster.
2. Surface the current status of each component via an API and a dashboard.
3. Fire an alert when a health check fails for a configurable number of consecutive intervals.
4. Support automated remediation actions (restart a process, remove a node from load balancer rotation).

Below the Line (out of scope)

• Log aggregation (covered in Design a Distributed Logging System).
• Full metrics pipeline: CPU graphs over time, percentile reporting, SLO burn rates (covered in Design a Metrics Collection System).
• Multi-tenant monitoring of external customer clusters.

The hardest part in scope: Reliable alert delivery with as few missed alerts and false positives as practical. A missed sustained failure is worse than a temporarily stale dashboard, so the alert pipeline gets a full deep dive: durable triggering, deduplication, flap detection, and escalation policy.

Log aggregation is below the line because it is a separate write-heavy pipeline with different storage semantics. A later extension could stream log lines to a Kafka topic and feed them into a search cluster beside the health-check path, not inside it.

The full metrics pipeline is below the line because this article focuses on binary health checks (up/down) and threshold-based alerting rather than time-series analytics and SLO math. A later extension could expose a metrics endpoint and let a separate metrics service handle aggregation and percentiles.

Multi-tenant monitoring is below the line because it introduces auth isolation, billing, and noisy-neighbor concerns that are separate from the core design.

Non-Functional Requirements

Core Requirements

• Scale: 10,000 nodes in the cluster. Each node checked every 30 seconds. Peak ingest rate: approximately 333 check results per second.
• Alert latency: An alert fires within 90 seconds of a sustained failure being detected (3 consecutive failed checks at a 30-second interval).
• Availability: 99.9% uptime for the monitoring system itself. The monitoring system must survive the failures it is designed to detect.
• Data retention: Raw check results retained for 30 days. Hourly status aggregates retained for 1 year.
• Read latency: Dashboard current-status queries return in under 200ms. Historical status queries return in under 2 seconds.

Below the Line

• Sub-second alert latency (acceptable to trade for simpler architecture)
• Custom integrations beyond PagerDuty/Slack/webhook

Write-to-read ratio: This system writes 333 check results per second and receives far more reads from dashboards, alert queries, and history lookups. The dominant design constraint here is not read throughput but write correctness. Missing one alert when a node is genuinely down is far more damaging than a brief dashboard staleness.

The 99.9% availability target for the monitoring system is an explicit scenario trade-off, not a statement that monitoring outages are harmless. A simpler replicated design may be preferable to adding coordination that increases the failure surface, provided the monitoring stack itself has independent checks and recovery procedures.

30-second answer

Use regional Health Checkers assigned by a Coordinator. Each checker actively probes its nodes, records the result in a durable time-series store, and updates a latest-status cache. A durable check-result stream feeds an Alert Manager, which applies a consecutive-failure state machine before creating one alert per ongoing incident. A retrying Notification Dispatcher handles external delivery, while a guarded Remediation Engine consumes approved actions from a per-node queue. Operators read current state from the cache and history from the time-series store. The checkers, alert path, and monitoring-of-the-monitoring path must all tolerate individual component failure.

5-minute explanation

The central split is between observation, current state, history, and action. The Node Registry tells checkers what to probe. Checkers use bounded pull requests so the system can observe endpoint reachability and latency, then write each result durably. Redis (or an equivalent cache) holds the latest result and alert state for fast dashboard reads; it is not the historical source of truth.

The Alert Manager consumes durable result events rather than relying only on best-effort cache notifications. It moves a node through HEALTHY, PENDING, FIRING, and RESOLVED, creates an alert once the configured failure threshold is reached, and hands delivery to a retrying dispatcher. Remediation is a separate, audited workflow with runbook policy, stale-action checks, and a circuit breaker.

The design scales by partitioning node ownership across regional checkers and reassigning a shard when a checker heartbeat expires. The read path stays cheap because current status is cached; the history path uses time-aware storage and downsampling. The important caveat is that a health observation is not the same as a diagnosis: a failed probe can mean a dead node, a failed dependency, or a network partition.

The entities, APIs, and concrete design anchors below make that explanation precise before the request flows and deep dives add detail.

Core entities

• Node: A single machine or service instance being monitored. Carries a node ID, hostname, cluster region, service type, and registration timestamp.
• HealthCheck: One check result for one node at one point in time. Carries the node ID, check timestamp, status (healthy/unhealthy/unknown), HTTP response code, latency in milliseconds, and an optional error message.
• Alert: A record of a sustained failure and its notification state. Carries the node ID, onset time, resolved time, severity, and the recipient list that was notified.
• RemediationAction: A record of an automated action taken in response to an alert. Carries the alert ID, action type (restart/drain), execution time, and outcome.

The full schema is deferred to the data model deep dive. These entities are sufficient to drive the API and high-level architecture.

API design

FR 1 and FR 2 -- check health and surface status:

GET /nodes/{node_id}/status
Response: { node_id, status, last_check_at, consecutive_failures, latency_ms }

A GET because this is a pure read. The consecutive_failures field does work in the response: the dashboard can show "3 consecutive failures" alongside "UNHEALTHY" instead of the raw binary, which is more useful to an on-call engineer.

GET /nodes?region=us-east-1&status=unhealthy
Response: { nodes: [...], next_cursor: "..." }

Cursor-based pagination because the caller is a dashboard rendering potentially thousands of nodes. Filtering by status and region up front means the dashboard loads the degraded nodes first, which is what operators care about in an incident.

FR 3 -- list and acknowledge alerts:

GET /alerts?status=firing&severity=critical
Response: { alerts: [...], next_cursor: "..." }

PATCH /alerts/{alert_id}
Body: { status: "acknowledged", acknowledged_by: "user@example.com" }
Response: { alert_id, status, acknowledged_by, acknowledged_at }

PATCH (not PUT) because we are updating a subset of the alert fields. The acknowledgement endpoint exists to close the loop: once an on-call engineer has seen the alert, the system stops re-notifying.

FR 4 -- trigger remediation:

POST /nodes/{node_id}/remediate
Body: { action: "restart" | "drain", initiated_by: "system" | "user@example.com" }
Response: { action_id, status: "queued", estimated_completion_ms: 5000 }

POST because remediation is an action, not a resource creation or update. The initiated_by field distinguishes automated remediation from human-triggered remediation in the audit log, which matters for post-incident review.

Make acknowledgement an explicit API action rather than a UI-only toggle; the acknowledged_by field creates an audit trail for post-incident review.

45-minute interview approach

Use this section only to pace a design interview; keep the implementation detail in the architecture and deep-dive sections.

• 0-5 minutes — clarify the prompt: Confirm that the system monitors an internal cluster, the health endpoint and timeout, who receives alerts, and which remediation actions are allowed.
• 5-10 minutes — requirements and estimates: State the illustrative 10,000-node, 30-second cadence, calculate about 333.3 checks per second, and agree on alert latency, retention, and dashboard latency.
• 10-15 minutes — entities and APIs: Identify nodes, check results, alerts, and remediation records. Sketch current-status, alert acknowledgement, and remediation endpoints.
• 15-25 minutes — baseline architecture and flows: Draw the checker pool, registry, status store, latest-status cache, Query API, and the write, read, and alert flows. Explain why the monitor cannot depend on a single checker.
• 25-35 minutes — choose deep dives: Let the interviewer choose between checker assignment, health-result storage, or alert delivery. Use the corresponding options below to compare a naive design with the selected design.
• 35-41 minutes — reliability, security, and operations: Cover checker failover, durable alert triggering, notification retries, remediation limits, authentication, audit logs, and monitoring the monitor.
• 41-45 minutes — trade-offs and close: State what is eventually consistent, what is authoritative, which assumptions would change the design, and answer follow-ups.

High-level architecture and critical flows

The design has three critical flows. The write flow probes nodes and records each observation; the read flow serves current status from the cache and history from the time-series store; the action flow turns sustained failure observations into durable alerts, notifications, and guarded remediation. The sections below evolve those flows from a simple polling loop to the distributed design.

1. Periodically check the health of every node

The simplest health check system is a single polling loop: one service walks the list of all registered nodes and sends an HTTP GET to each node's /healthz endpoint every 30 seconds.

Check taxonomy (what we actually measure):

• Liveness: HTTP 200 from /healthz within the 5-second timeout. The binary up/down signal that drives alerts.
• Response latency: The round-trip time for the /healthz request, stored as latency_ms per check. A node can return 200 but be degraded if latency spikes.
• Saturation metrics (optional, agent-reported): CPU%, memory%, disk%. Stored alongside the check result in the Status Store but scoped out of the core alert path per FR. Useful for capacity planning dashboards.

Components:

• Node Registry: A database table listing every node, its address, and the check interval. The Checker queries this to know who to poll.
• Health Checker: The polling service. For each node, it opens an HTTP connection, sends GET /healthz, waits for a 200 response, and writes the result to the Status Store.
• Status Store: A time-series-friendly database that holds every check result.

Request walkthrough:

1. Health Checker queries Node Registry: "give me all nodes due for a check."
2. For each node, Health Checker sends GET /healthz with a 5-second timeout.
3. Node responds with HTTP 200 (healthy) or connection timeout (unhealthy).
4. Health Checker writes { node_id, timestamp, status, latency_ms } to Status Store.
5. Status Store acknowledges the write.

This covers the basic write path. The read path (surfacing status to operators) and the alert logic come in the next two requirements.

The issue in this illustrative 10,000-node scenario: At a 30-second interval, the checker must initiate approximately 10,000 / 30 = 333.3 checks per second. A sequential loop at an illustrative 20ms average latency would take 10,000 x 20ms = 200 seconds per cycle, so it would fall about 170 seconds behind before the next cycle starts. Async I/O can handle many concurrent connections, but a single process is still a failure domain and must be bounded by connection, CPU, and file-descriptor limits.