Metrics Collector

TL;DR

• Scrape metrics close to the targets: use local Prometheus-style collectors within each cluster, then remote-write batches to a central aggregation layer.
• Buffer centrally with Kafka so synchronized agent restarts and TSDB slowdowns create lag rather than immediate sample loss; add jitter to avoid a thundering herd.
• Store samples in a purpose-built TSDB with label indexing, delta-of-delta timestamp compression, and Gorilla-style float compression.
• Keep raw high-resolution data briefly, downsample into minute and hour tiers, and route range queries to the tier that matches the requested time span.
• Evaluate alerts on fresh local data in a separate path from dashboard queries, persist alert state, and deduplicate notifications before paging.
• Treat label cardinality, retention, consumer lag, and alert freshness as first-class operating constraints; a fast query engine cannot rescue an unbounded series set.

Scope and assumptions

This article designs a server metrics collection system for infrastructure and service-level numeric measurements. It covers scraping, batching, buffering, TSDB storage, range queries, alert evaluation, notification, and long-term downsampling. Log search, tracing, and ML anomaly detection are separate consumers or products.

The illustrative interview scenario assumes:

• 10,000 servers expose about 100 metrics each. A 10-second scrape cadence produces roughly 100,000 samples per second, with synchronized-fleet bursts up to about 300,000 samples per second.
• Collectors can reach targets inside their cluster or data center. Local collectors remote-write batches to a central service, so a central scraper does not need direct access to every host.
• Operators need recent high-resolution dashboards, historical trend queries, and threshold alerts that fire within 30 seconds of a sustained breach.
• Raw samples are retained for 15 days, one-minute aggregates for 1 year, and one-hour aggregates for 3 years. Each series has a bounded label set; arbitrary per-request labels are not acceptable.
• At-least-once ingestion is acceptable if samples or aggregate writes are idempotent. The design does not promise perfect continuity across an outage longer than the configured local and Kafka buffers.
• All rates, retention windows, and latency targets below are scenario requirements to validate with capacity tests, not guarantees of any specific TSDB or collector.

What is a server metrics collection system?

A server metrics collection system scrapes CPU, memory, disk, and network statistics from every machine in a fleet, stores those measurements as time-series data, and lets engineers query and alert on them in near real time. The interesting engineering challenge is not the scraping; it is handling the write volume at scale (thousands of servers emitting metrics every 10 seconds means hundreds of thousands of data points per minute), choosing a storage engine that evaluates range queries over months of data in under a second, and deciding whether each server should push metrics out or wait to be pulled.

Open with the write volume math because it changes every downstream choice. The scenario tests write-heavy system design, time-series database internals, alerting pipeline architecture, and the push-versus-pull trade-off.

Functional Requirements

Core Requirements

1. Collect CPU, memory, disk, and network metrics from thousands of servers every 10 to 60 seconds.
2. Store time-series data with enough resolution to detect and alert on short-duration spikes.
3. Support dashboard queries returning aggregate metrics over arbitrary time ranges (last 5 minutes to last 6 months).
4. Trigger alerts when a metric crosses a configurable threshold for a sustained duration.

Below the Line (out of scope)

• Log collection and log-based alerting (covered in Design a Distributed Logging System)
• Distributed tracing and service-level APM
• ML-based anomaly detection on metric streams
• Multi-tenant metric isolation with per-customer encryption

The hardest part in scope: Storing time-series data efficiently. Naive row-per-sample storage in a relational database quickly reaches tens of billions of rows. The correct answer is a purpose-built time-series database (TSDB) that compresses samples using delta-of-delta encoding and gorilla float compression, enabling 10-40x storage reduction while keeping range queries under 100ms.

Log collection is below the line because it requires separate indexing infrastructure (inverted index for full-text search) that is architecturally distinct from the append-only numeric time-series store described here. To add it, build a Log Ingestion Service that writes to Elasticsearch or a ClickHouse full-text index in parallel with the metrics pipeline.

Distributed tracing is below the line because it requires a causal graph store keyed by trace ID, not a time-series store. To add it, run an OpenTelemetry Collector that fans spans to a Jaeger backend (Cassandra or Elasticsearch) alongside the metrics pipeline.

ML-based anomaly detection is below the line because it is a consumer of the metrics pipeline, not a change to the pipeline itself. To add it, subscribe a streaming job (Flink) to the metrics ingestion topic and emit anomaly events to an alert bus when a model scores a metric as anomalous.

Non-Functional Requirements

Core Requirements

• Write throughput: 10,000 servers, each emitting 100 metrics every 10 seconds, produces 100,000 metric samples per second sustained. Peak (all servers flushing simultaneously) can reach 300,000 samples/second.
• Query latency: Dashboard range queries must return in under 1 second at p95 for time ranges up to 7 days. Longer ranges (up to 6 months) must complete in under 5 seconds.
• Retention: 15 days of raw data (10-second resolution). 1 year of downsampled data (1-minute resolution). 3 years of heavily downsampled data (1-hour resolution).
• Alert latency: Threshold alerts must fire within 30 seconds of the metric crossing the threshold.
• Durability: 99.9% of metric samples must be stored. Brief gaps during a collector restart are acceptable; sustained loss is not.
• Availability: 99.9% uptime for the query and alert path. The collection path can have momentary gaps during deployments.

Below the Line

• Sub-100ms alert latency (requires streaming evaluation rather than batch rule evaluation)
• Per-metric access control (who can query which server's metrics)
• Cardinality explosion protection at ingestion time

Read/write ratio: This system is heavily write-skewed. 100,000 samples/second sustained means roughly 8.6 billion samples per day on the write side. Dashboard queries are sporadic bursts from on-call engineers; alert evaluations happen every 10 seconds per rule but touch only aggregated data. The write-to-interactive-read ratio is roughly 1,000:1. Every major design decision in this article is driven by that number: make writes cheap, make reads pre-computed wherever possible.

The 1-second query SLA on 7-day ranges is the forcing constraint for storage format. Any database that requires a full table scan to answer that query is architecturally wrong for this system.

30-second answer

Use a hybrid collection model: Prometheus-style collectors pull metrics within each cluster, then remote-write batches to a central aggregator. Buffer the central path with Kafka, partition by host or series, add scrape jitter, and rate-limit TSDB writes. Store samples in a compressed, label-indexed TSDB with 15 days of raw retention; hourly downsampling creates one-minute and one-hour tiers for longer ranges. A separate Query Service routes requests by time span and caches dashboard results. A dedicated Alert Evaluator reads fresh local data, persists a pending/firing/resolved state, and sends deduplicated notifications. Guard label cardinality and monitor lag, freshness, and buffer health.

5-minute explanation

The key scale calculation is 10,000 hosts times 100 metrics every 10 seconds: about 100,000 samples per second, with synchronized restarts producing a larger burst. Scraping locally avoids a single fleet-wide poller and works across network boundaries. Agents or local Prometheus instances batch samples, add jitter, and remote-write them toward the central tier. Kafka absorbs the burst; a rate-limited TSDB writer controls the storage side.

The TSDB is built around metric series and time chunks rather than one relational row per sample. A label index resolves a selector to series IDs, compressed chunks store timestamps and values efficiently, and retention tiers keep recent data precise while older data is aggregated. The Query Service chooses raw, minute, or hour data based on the requested range and uses a short-lived cache for repeated dashboard queries.

Alerting must not share a saturated dashboard path. The Alert Evaluator reads fresh local data, applies threshold duration and state transitions, and hands notification intents to a retrying, deduplicating service. Reliability depends on bounded label cardinality, local/central buffering, idempotent downsampling, and alerts on Kafka lag and data freshness. The system can tolerate temporary delay, but it should make stale dashboards and late alerts visible.

45-minute interview approach

Use this agenda to make the write rate, freshness boundary, and cardinality model explicit:

• 0-5 minutes — clarify the prompt: Confirm metric types, scrape cadence, target discovery, push versus pull constraints, dashboard ranges, alert semantics, retention, and whether the system is single-cluster or multi-region.
• 5-10 minutes — requirements and estimates: Calculate about 100,000 samples/sec sustained and 300,000 samples/sec burst, then state the 1-second recent-query target, 30-second alert target, retention tiers, durability, and label-cardinality assumptions.
• 10-15 minutes — entities and APIs: Define MetricSample, MetricSeries, AlertRule, AlertEvent, and Dashboard. Sketch batch ingest, PromQL-style range query, rule management, and alert history APIs.
• 15-25 minutes — baseline architecture and critical flows: Draw target agents, local Prometheus, remote-write aggregator, Kafka, TSDB writer, TSDB tiers, Query Service/cache, Alert Evaluator, state store, and Notification Service. Walk through ingest, query, alert, and downsample flows.
• 25-35 minutes — choose the deep dive: Prioritize hybrid push/pull, jitter plus Kafka backpressure, TSDB internals, or cardinality control. Compare the simple option with the design that meets the stated load and freshness target.
• 35-41 minutes — reliability, security, and operations: Cover local buffers, replay, duplicate samples, out-of-order timestamps, downsampler idempotency, label limits, auth, tenant isolation, query limits, lag, and monitoring the monitoring system.
• 41-45 minutes — trade-offs and close: State which path is authoritative, where data can be stale, why alerts use fresh local data, how retention affects resolution, and what changes for 10x series or sub-second alerting.

Core Entities

• MetricSample: A single measurement: metric name, host ID, tags (region, service, environment), timestamp, and float64 value. The atomic unit of storage.
• MetricSeries: A unique combination of metric name + tag set. All samples for one series share the same labels. The unit of indexing in a TSDB.
• AlertRule: A threshold condition on a metric query: expression, evaluation interval, duration (how long the condition must hold before firing), and notification channel.
• AlertEvent: A fired or resolved alert instance tied to a rule, with the triggering value, timestamp, and affected host or service.
• Dashboard: A saved collection of metric queries rendered as time-series charts or gauges. Backed by the query service.

Schema details and TSDB internals are deferred to the deep dives. These five entities drive the API and High-Level Design.

API Design

Start with one endpoint per functional requirement, then evolve where the naive shape breaks.

FR 1 - Ingest metric samples:

The naive push shape:

POST /metrics
Body: { metric_name, host_id, tags, timestamp, value }
Response: HTTP 204

This breaks at 100,000 samples/second: one HTTP request per sample is absurd overhead. The evolved shape pushes batches from each server's local agent:

POST /metrics/batch
Body: {
  host_id: "server-1234",
  timestamp: 1743417600,
  samples: [
    { name: "cpu.usage", tags: { core: "0" }, value: 72.4 },
    { name: "mem.used_bytes", tags: {}, value: 12884901888 }
  ]
}
Response: HTTP 204

Each batch covers one scrape interval (10 to 60 seconds of samples from one host). The agent keeps unsent batches in local memory for up to 5 minutes to handle transient collector downtime. Without a local buffer, a 2-minute collector restart creates permanent gaps in every affected series, so five minutes is a useful minimum for this scenario.

FR 2 - Query metrics:

GET /query_range
Query params:
  expr=cpu.usage{host="server-1234",env="prod"}
  start=2026-03-29T00:00:00Z
  end=2026-03-29T06:00:00Z
  step=60s
Response: {
  data: [ { timestamp, value } ],
  resolution: "1m",
  downsampled: false
}