Log Aggregator

TL;DR

• Run a lightweight agent on every host with a bounded local disk buffer so application logging stays decoupled from network and Kafka outages.
• Use Kafka as the durable fan-in buffer, with separate topics for unstructured logs and numeric metrics. Commit downstream offsets only after the corresponding writes succeed.
• Keep hot logs in hourly, inverted-indexed Elasticsearch/OpenSearch indexes and archive compressed batches to object storage for cheap long-term retention.
• Serve historical search through a cursor-based scatter-gather Query API, while serving tail -f through a Kafka-backed SSE path that bypasses the search index.
• Send metrics through a separate time-series pipeline into Prometheus-compatible storage; evaluate alerts there so a log-indexing incident cannot suppress alerts.
• Treat the stated rates as an interview scenario: about 1 TB/day at baseline, with a much larger short-lived burst used to test buffering and horizontal scale.

Scope and assumptions

This article designs an internal observability backbone for logs and numeric metrics emitted by a large server fleet. It covers collection, buffering, indexing, historical queries, live tailing, dashboards, alert evaluation, and hot-to-cold retention. It does not attempt to build a full tracing or SIEM platform.

The illustrative interview scenario assumes:

• 10,000 hosts emit structured and unstructured logs, with a baseline around 1 TB/day and short peak bursts that can reach 10 million events per second.
• Operators search recent logs by time range, service, host, level, and message text; results are paginated and bounded rather than returning an unbounded export.
• Logs stay indexed for 30 days and are archived in compressed columnar batches for up to 365 days. Cold queries are slower and use a separate API or query engine.
• At-least-once delivery is acceptable. Duplicate log events are possible after retries, so events should carry stable IDs or a deduplication key where exact counts matter.
• Metrics are numeric time series with bounded label cardinality and have different retention, query, and alerting needs from full-text logs.
• All rates, latency targets, and retention windows below are design requirements for the scenario, not guarantees about any particular vendor.

What is a distributed logging and metrics system?

A distributed logging system collects, stores, and makes searchable every log line emitted by every server in a fleet. At 10,000 application servers, the challenge is collecting events without losing them during traffic spikes, indexing about 1 TB/day so queries return in under 5 seconds, and keeping 30 days of hot data queryable while archiving a full year cheaply. The scenario tests pipeline architecture, write-heavy system design, inverted-index fundamentals, and the trade-off between storage cost and query latency. The metrics path is included because numeric time series need a different storage and alerting model.

Functional Requirements

Core Requirements

1. Collect logs and metrics from thousands of servers in near real time.
2. Store logs such that they can be searched and queried by time range, service name, and log level.
3. Export aggregated metrics for dashboarding and threshold-based alerting.

Below the Line (out of scope)

• Distributed tracing and APM (Application Performance Monitoring)
• Log-based security intrusion detection (SIEM)
• Log-based billing or audit trails with tamper-proof guarantees

The hardest part in scope: Indexing logs fast enough to query. Writing 1TB/day at 12 MB/sec is manageable. The trap is that naive storage (one log line = one document) makes full-text search across billions of rows take minutes, not seconds. Time-partitioned columnar segments with an inverted index are the correct answer, and explaining why is the heart of this design.

Distributed tracing is below the line because it requires correlating spans across services using a trace context header (W3C TraceContext or Zipkin format). It is architecturally distinct from log aggregation: traces need a causal graph store, not a full-text index. To add it, build a Trace Ingestion Service that accepts OTLP spans from instrumented services, fans them into a separate Kafka topic, and writes to a trace backend keyed by trace_id. The log pipeline described here is unchanged.

SIEM is below the line because it requires real-time pattern matching against threat signatures, which demands a streaming analytics engine (Flink or Spark Streaming) on top of the log pipeline. This is a consumer of logs, not a change to the pipeline. To add it: attach a Flink job to the Kafka log topic that evaluates each log event against a rule set and emits alerts to a Security Incident topic.

Log-based billing with tamper-proof guarantees is below the line because it requires append-only immutable storage with cryptographic chaining. The log pipeline described here does not guarantee immutability. To add tamper evidence, attach a Write-Once Object Store (S3 Object Lock with Compliance mode) and write signed log batches there in parallel with the primary index.

Non-Functional Requirements

Core Requirements

• Durability: 99.9% of log messages must be delivered. Some loss is acceptable (a handful of debug logs dropped during a Kafka restart is tolerable; error logs must not be lost).
• Write throughput: The baseline scenario is 1 TB/day, or about 12 MB/sec averaged over a day. A short-lived stress burst can reach 10 million events/second; at 1.2 KB/event that burst is about 12 GB/sec and requires a horizontally scaled buffer rather than a single ingest host.
• Query latency: Time-range search across all logs (e.g., all ERROR events for service payments in the last 10 minutes) must return in under 5 seconds at p95.
• Retention: 30 days hot (indexed, fast query via Elasticsearch or equivalent). 365 days cold (archival in S3/object storage, query via Athena or batch scan).
• Scale: 1 TB of log data ingested per day. After 30 days, roughly 30 TB of hot storage. After 365 days, roughly 120 TB of cold archival storage after 3x compression (365 TB before compression).

Below the Line

• Sub-second query latency (requires pre-aggregated materialized views, changes the indexing model)
• Multi-tenant log isolation with per-customer encryption-at-rest
• Log anomaly detection using ML (consumer of logs, not a pipeline change)

Read/write ratio: This is a write-heavy system. Thousands of servers emit logs continuously while queries arrive in sporadic bursts from on-call engineers or dashboards. A rough ratio is 100:1 writes to reads during normal operations, inverted briefly during incidents when many engineers are querying simultaneously. This asymmetry drives the architecture: the write path needs cheap buffering and bounded loss, while the read path needs fast on-demand search.

The 5-second p95 query SLA is the key number. It is strict enough to require a proper inverted index rather than a full table scan, but lenient enough that every query does not need a pre-materialized per-service aggregate. That constraint drives the storage-tier decision.

30-second answer

Put a local agent on each host, buffer batches durably on disk, and publish them to a replicated Kafka cluster. Split the stream into log and metrics topics. A Log Indexer consumes log batches and writes hourly Elasticsearch indexes plus compressed object-storage archives; a separate Metrics Consumer writes numeric points to Prometheus-compatible TSDB storage. The Query API fans bounded searches across only the relevant time partitions and returns cursor-paginated results. A Kafka-backed Streaming Service serves live tails over SSE, and Alertmanager evaluates rules on the metrics path. At-least-once delivery, idempotent consumers, retention policies, and independent failure domains keep ingestion, search, and alerting operable.

5-minute explanation

The central design choice is to separate collection from every downstream consumer. A host-local agent protects the application from network stalls and keeps a disk-backed queue during Kafka or collector outages. Kafka absorbs bursts and allows the log indexer, cold-archive writer, metrics consumer, and live-tail service to progress independently. The producer acknowledges only after the replicated buffer accepts the batch; downstream offsets advance only after successful writes.

Logs and metrics then follow different storage paths. Logs are variable-schema text, so the indexer writes exact-match fields such as service and level plus a full-text message field into hourly indexes. It archives the same batches in compressed columnar form for long retention. Numeric metrics use a TSDB with label indexing, compression, downsampling, and PromQL-style evaluation. This avoids forcing dashboards and alerts through a text-search engine.

The read path is also split by freshness. Historical log search uses a bounded scatter-gather query over relevant shards, with per-shard timeouts and cursor pagination. Live tail reads the newest Kafka records and pushes matching events over SSE, so it does not wait for index refreshes. Alerting reads the metrics store on its own schedule. This isolates an Elasticsearch outage from live tailing and metrics alerts, while Kafka retention gives consumers a replay window for recovery.

45-minute interview approach

Use this agenda to pace the log-aggregator design and spend time on the constraints that change the architecture:

• 0-5 minutes — clarify the prompt: Confirm whether logs are internal or multi-tenant, which fields must be searchable, whether live tailing and metrics are both in scope, the acceptable loss model, and hot/cold retention.
• 5-10 minutes — requirements and estimates: State the baseline 1 TB/day rate, the short burst scenario, the 5-second p95 search target, the 30-day hot window, the one-year archive, and the read/write skew. Distinguish sustained volume from burst capacity.
• 10-15 minutes — entities and APIs: Define LogEvent, MetricPoint, IndexSegment, and AlertRule. Sketch batched ingestion, bounded cursor search, metrics range query, alert creation, and SSE tail APIs.
• 15-25 minutes — baseline architecture and critical flows: Draw host agents, local disk buffers, Kafka, the log indexer, Elasticsearch, object storage, the metrics consumer, TSDB, Query API, and Alertmanager. Walk through ingest, indexed search, live tail, and alert evaluation.
• 25-35 minutes — choose the deep dive: Prioritize the bottleneck the interviewer selects: disk-backed collection and at-least-once delivery, time-partitioned indexing and scatter-gather, or Kafka-backed live streaming. Explain the naive option, its failure mode, and the chosen evolution.
• 35-41 minutes — reliability, security, and operations: Cover consumer lag, disk-buffer overflow, duplicate events, index rebuilds, retention deletion, query timeouts, authentication, redaction, encryption, and monitoring the observability system itself.
• 41-45 minutes — trade-offs and close: State what is authoritative, where eventual consistency is acceptable, how a failed consumer catches up, which guarantees are not promised, and how the design changes for multi-tenancy or a higher query SLA.

Core Entities

• LogEvent: A single log line with timestamp, service name, host, log level, message body, and optional structured fields (request ID, user ID, error code).
• MetricPoint: A single numeric measurement at a point in time: metric name, value, tags (host, service, region), and timestamp. Stored separately from logs.
• Index Segment: A time-bounded, immutable chunk of the log index. One segment covers a fixed time window (e.g., one hour of data). The unit of querying.
• Alert Rule: A threshold condition on a metric (e.g., error_rate > 1% for 5min). Evaluated periodically against the metrics pipeline.
• Dashboard: A saved collection of metric queries rendered as charts. Backed by the metrics query service.

Schema design and partition strategies are deferred to the deep dives. The five entities above are sufficient to drive the API and High-Level Design.

API Design

FR 1 - Ingest logs from a server:

POST /ingest/logs
Body: { events: [{ timestamp, service, host, level, message, fields? }] }
Response: 202 Accepted

202 (not 200) because the log pipeline is asynchronous. The request is accepted and queued; it is not yet durable. Clients that need durability acknowledgment should use the Kafka SDK directly.

Batching in the request body (the events array) is mandatory: single-event ingestion at 10,000 events/sec per server would create catastrophic per-request overhead on the HTTP layer.

FR 2 - Query logs by time range, service, and level:

Naive shape:

GET /logs/search?service=payments&level=ERROR&from=2026-03-29T10:00Z&to=2026-03-29T10:10Z
Response: { logs: [...], next_cursor }

This naive shape breaks at scale: returning all matching logs in one shot at 1TB/day means a 10-minute window can contain millions of matching rows. The evolved shape adds cursor-based pagination and a result limit.

Evolved shape:

GET /logs/search?service=payments&level=ERROR&from=2026-03-29T10:00Z&to=2026-03-29T10:10Z&limit=100&cursor={opaque_cursor}
Response: { logs: [...], next_cursor, total_matched }

Cursor-based pagination is required here. Offset-based pagination (skip N) requires the query engine to scan and discard N rows on every page request, which is prohibitively expensive against a time-series index. The cursor encodes the last-seen segment ID and offset, allowing the query engine to resume without re-scanning.

FR 3 - Export metrics for dashboarding and alerting:

GET /metrics/query?metric=http_error_rate&from=2026-03-29T09:00Z&to=2026-03-29T10:00Z&step=60s&tags=service:payments
Response: { datapoints: [{ timestamp, value }] }

POST /alerts
Body: { metric, condition, threshold, duration_s, notification_channel }
Response: { alert_id }

The metrics query API is modelled on Prometheus's range query API (/query_range). The step parameter controls downsampling resolution. Dashboards use this endpoint to render time-series charts.

FR 4 - Stream live logs (tail -f equivalent):

GET /logs/stream?service=payments&level=ERROR
Response: text/event-stream (Server-Sent Events)

Server-Sent Events over HTTP is the right choice here. WebSockets are bidirectional and add unnecessary complexity when the client only reads. SSE is a standard HTTP connection that the server pushes events on; proxies and load balancers handle it well. The client receives data: events as new log lines match the filter.

High-Level Design and Critical Flows

1. Collect logs from thousands of EC2 servers

The collection path: a lightweight agent on each server buffers log lines locally and ships batches to Kafka. The agent absorbs local write spikes without creating backpressure on downstream consumers.

The naive approach is to have each server POST logs directly to an ingestion API. That breaks immediately: 10,000 servers each making HTTP calls to a single ingestion service creates millions of concurrent connections and eliminates any buffering. One slow ingest service stalls the entire fleet. A common failure mode is that the ingest service falls over during deployments or traffic spikes because the application and collection paths are coupled.

The key insight is that collection and ingestion must be decoupled by a durable buffer (Kafka). The agent on each server is responsible for one thing: getting bytes off disk and into Kafka reliably. Everything downstream can fail and restart without losing a log line.

Components:

• Log Agent (Fluent Bit): A lightweight sidecar process on every EC2 instance. Tails log files or consumes from journald. Batches events into 1-second windows and ships to Kafka. Writes a local disk buffer if Kafka is unreachable.
• Kafka (Log Topic): Central durable buffer. Partitioned by service_name so that all logs for a given service go to the same partition set. Replication factor 3 for durability.
• Kafka (Metrics Topic): Separate topic for numeric metric points. Partitioned by metric_name.

Request walkthrough:

1. Application writes a log line to stdout or a log file on the EC2 instance.
2. Fluent Bit agent tails the file (or reads from stdout pipe), parses the log line, and enriches it with host, service, and region metadata.
3. Fluent Bit batches events in a 1-second window and writes the batch to the Kafka logs topic, partitioned by service_name.
4. If Kafka is unreachable, Fluent Bit writes to a local disk buffer (up to 512 MB) and retries with exponential backoff. This is what gives us the 99.9% durability guarantee: the agent survives transient Kafka outages without dropping events.
5. Kafka brokers replicate the batch to 2 additional brokers before acknowledging. acks=all is set on the producer.