Tracing Pipeline

What is a distributed tracing system?

A distributed tracing system tracks a single request as it flows through multiple microservices, recording timing and metadata at each hop. Visit an e-commerce site and your checkout request might touch an API gateway, a cart service, an inventory service, a payment service, and a notifications service before completing.

Without tracing, a 2-second slowdown is a mystery. With tracing, you see the inventory service added 1.8 of those seconds.

The apparent problem is observability. The hard engineering problem is doing it without adding measurable latency to production requests while ingesting millions of spans per second, and assembling them into a coherent call graph on demand. The latency constraint separates this from a simple logging problem and should anchor the design.

This question tests your knowledge of context propagation, async buffering patterns, write-heavy storage design, and the tradeoffs between head-based and tail-based sampling.

TL;DR

Instrument each service with W3C traceparent, buffer spans in-process, batch them to stateless collectors, and acknowledge ingestion after the collector durably publishes to Kafka. Independent consumers write full spans to Cassandra, searchable summaries to Elasticsearch, and service metrics to Redis. The query service assembles a trace by trace_id, searches summaries by indexed fields, and marks incomplete traces as partial.

Keep all network and storage work off the production request thread. Use bounded buffers, at-least-once replay, retention policies, and tail-sampling rules that retain errors and latency outliers while limiting healthy-trace cost.

Scope and assumptions

These are illustrative planning assumptions for the design; deployments should validate them with representative traces:

• About 500 instrumented services produce up to 1 million spans per second. A typical request creates roughly 10 spans, but fan-out and batch workloads can be much larger.
• Span collection adds less than 1 ms to application request latency, trace lookup completes within 2 seconds, spans are retained for 30 days, and search summaries for 7 days.
• HTTP, gRPC, and asynchronous message boundaries carry a standard trace context. At-least-once delivery is acceptable; a trace may be incomplete when a service crashes or an SDK buffer is lost.
• Full span payloads, searchable summaries, and pre-aggregated service metrics have different access patterns and may use different stores. Sampling is a cost/retention policy, not a reason to block production traffic.
• This article covers collection, propagation, storage, querying, sampling, and service-level summaries. Log aggregation, continuous profiling, real-time alerting, and guaranteed cross-region delivery are outside the primary design.

Functional Requirements

Core Requirements

1. Every request gets a globally unique trace ID that propagates through all downstream services.
2. Each service emits a span (start time, duration, operation name, metadata) linked to the parent trace.
3. Engineers can search by trace ID, service name, or error status and see the full call graph.
4. The system surfaces tail latency and error hot spots across the service graph.

Below the Line (out of scope)

• Log aggregation and log-to-trace correlation
• Continuous profiling (CPU flamegraphs, heap allocation tracking)
• Real-time alerting on trace data

The hardest part in scope: Collecting spans without impacting production latency. Every instrumented service is a potential victim of a slow or unavailable collector. The SDK design and the collector pipeline together must make span emission invisible to request throughput.

Log aggregation is below the line because it has a different ingestion pipeline and storage model. To add correlation, embed the trace ID into every log line and store it as a searchable field in the log aggregation system. The trace ID becomes a join key between logs and traces.

Continuous profiling is below the line because it requires a separate sampling profiler in each service process and a flame graph rendering pipeline. It does not share the span collection pipeline.

Real-time alerting is below the line because it requires a stream processing layer on top of the span pipeline. To add it, consume from the same Kafka topic, compute error rate and p99 latency per service in a sliding window, and trigger alerts on threshold breaches.

Non-Functional Requirements

Core Requirements

• Low overhead: Span emission adds less than 1ms to production request latency. The instrumentation must be invisible to throughput.
• Ingestion scale: Handle 1M spans per second from 500 or more services.
• Query latency: Retrieving all spans for a single trace ID completes in under 2 seconds.
• Retention: Spans retained for 30 days; search indexes retained for 7 days.
• Availability: 99.9% for the collector pipeline. A small fraction of dropped spans under extreme load is acceptable; blocking production services is not.

Below the Line

• Sub-second trace assembly
• Multi-region span collection with guaranteed cross-region delivery
• Guaranteed exactly-once delivery (at-least-once is sufficient)

Read/write ratio: This system is extremely write-heavy. A deployment handling 50K requests/second where each request touches 10 services generates 500K spans/second. Engineers query traces during incidents, not continuously. Expect a 1000:1 write-to-read ratio. Every architectural decision in the storage and ingestion tiers traces back to this write dominance.

Call out this ratio early because it rules out treating PostgreSQL as the primary high-volume span store and points toward an append-oriented, horizontally scalable pipeline.

The less-than-1ms overhead constraint rules out any synchronous network call on the hot path. Span data must be buffered in-process and flushed asynchronously on a background thread. The collector pipeline must be fully decoupled from production request latency.

The 2-second trace assembly target shapes the storage schema: fast trace lookup requires partitioning span storage by trace ID, not by time. A time-partitioned schema would scan every time bucket to find spans for a single trace, which is too slow at our query target.

30-second answer / outline

1. Create a trace ID at the ingress boundary, create a span ID at each service, and propagate both in W3C traceparent across HTTP, gRPC, and message headers.
2. Buffer spans in a bounded SDK queue and send batches asynchronously to stateless collectors; the collector validates and publishes them to Kafka without blocking application requests.
3. Use separate Kafka consumer groups for full-span storage in Cassandra, searchable summaries in Elasticsearch, and service-level latency/error aggregates in Redis.
4. Serve trace-ID lookups from Cassandra, search from Elasticsearch, and dashboards from Redis. Assemble the parent/child graph and return partial when expected spans are missing.
5. Apply tail-based retention after error/outlier flags are captured, with replayable consumers, TTLs, redaction, and explicit drop/backpressure behavior.

5-minute explanation

Start with the hot-path constraint. If a service waits for a collector or database, the tracing system can increase the latency of the production system it is meant to observe. The SDK therefore creates lightweight span records, puts them into a bounded in-process buffer, and flushes batches on a background worker. When the buffer is full, the policy should drop or sample according to priority rather than block the request thread.

The propagation carrier contains the trace ID, the current span ID as the downstream parent, and sampling flags; it is not the full span payload. Every transport adapter must extract and inject that context, including asynchronous message headers. Collectors validate size and schema, publish to Kafka, and return a bounded acknowledgement. Kafka then lets storage, indexing, aggregation, and sampling scale independently and replay after a downstream outage.

The storage design follows access patterns. Cassandra partitions full spans by trace ID for a fast call-graph lookup. Elasticsearch stores a smaller summary with service, operation, status, and time fields for incident searches. Redis holds short-lived service metrics for dashboards. The Query Service reads the appropriate store, builds the tree from parent_span_id, and makes gaps visible instead of pretending the trace is complete.

Sampling is safest after the system has enough signal to identify errors and outliers. Preserve error traces and latency outliers, retain a representative baseline of healthy traces, and delete or expire lower-value data according to policy. The main correctness contract is honest observability: do not claim exactly-once collection or complete traces when SDK drops, service crashes, or sampling have removed data.

45-minute interview approach

This agenda keeps the production hot path and the trace-assembly access pattern ahead of optional observability features.

• 0–5 minutes — Clarify the contract: Confirm supported transports, trace and span semantics, search filters, maximum trace size, overhead budget, freshness, retention, sampling, PII, and partial-trace behavior.
• 5–10 minutes — Establish scale: Estimate requests per second, spans per request, span size, daily volume, retention storage, query volume, and the difference between incident-time reads and continuous writes.
• 10–15 minutes — Define entities and APIs: Walk through Trace, Span, SpanContext, Service, batch ingest, trace lookup, search, and service-summary endpoints. State that 202 means accepted by the collector, not stored everywhere.
• 15–22 minutes — Draw propagation and collection: Show SDK buffers, transport adapters, collectors, validation, Kafka, backpressure, and why no synchronous network call is allowed on the request path.
• 22–30 minutes — Draw storage and query flows: Partition full spans by trace ID in Cassandra, index summaries in Elasticsearch, aggregate metrics in Redis, then assemble a tree and mark orphaned spans.
• 30–35 minutes — Deep dive on sampling and failure semantics: Prioritize head versus tail sampling, error/outlier retention, bounded buffers, duplicate spans, long-running traces, and replay.
• 35–41 minutes — Reliability, security, and operations: Cover collector/Kafka/Cassandra/Elasticsearch failure, TTLs, PII redaction, authorization, tag limits, lag, drops, and partial-result visibility.
• 41–45 minutes — Trade-offs and close: Compare storage engines, agent versus sidecar collection, sampling policies, and synchronous versus asynchronous designs; recap the zero-hot-path-latency invariant and invite follow-ups.

Core Entities

• Trace: The end-to-end record of one request journey. Identified by a 128-bit trace ID. Contains a root span and zero or more child spans. Has a computed status (ok, error, or partial) based on its constituent spans.
• Span: One unit of work within a trace. Fields: trace_id, span_id, parent_span_id (null for the root span), service_name, operation_name, start_time_unix_ns, duration_ms, status (ok or error), and a tags map for arbitrary key-value metadata such as http.status_code, user_id, or an exception message.
• SpanContext: The lightweight propagation carrier passed between services at every RPC boundary. Contains trace_id, span_id, and a sampling_flag. This is what travels inside the W3C traceparent header, not the full Span record.
• Service: A named instrumented service in the dependency graph. Used for building the service topology map and for search facets (filter all traces involving payment-service).

The full schema, column types, and Cassandra partition key design are deferred to the storage deep dive. These four entities are sufficient to drive the API and High-Level Design.

API Design

There are two distinct API surfaces: the ingest API used by SDK instrumentation libraries running inside each service, and the query API used by the UI and on-call engineers.

Ingest API (SDK to Collector):

POST /v1/spans
Body: [
  {
    trace_id, span_id, parent_span_id?,
    service_name, operation_name,
    start_time_unix_ns, duration_ms,
    status, tags: { key: value }
  },
  ...
]
Response: 202 Accepted

The endpoint accepts batches, not individual spans. The SDK buffers spans in-process and flushes in batches of 100-500 spans every 5 seconds. Batch ingest reduces network overhead by roughly 100x compared to one-span-per-request.

The 202 response is fire-and-forget: the SDK does not wait for storage confirmation.

Query API:

GET /api/traces/{trace_id}
Response: { trace_id, root_span, spans: [...], status, duration_ms }

GET /api/traces?service=&operation=&error=&min_duration_ms=&start=&end=&limit=50&cursor=
Response: { traces: [...summary...], next_cursor: "..." }

GET /api/services
Response: { services: ["api-gateway", "cart-service", "payment-service", ...] }

GET /api/services/{service}/operations
Response: { operations: ["POST /checkout", "GET /cart", ...] }

The search endpoint uses cursor-based pagination because time-range queries over trace metadata can span millions of records. Offset-based pagination is unstable when new spans arrive during a query window.

GET /api/traces/{trace_id} is the most latency-sensitive query. On-call engineers arrive at a trace ID from a log line or an alert. This lookup must return the full call graph in under 2 seconds.

302 vs search result distinction: The ingest and query APIs run on separate services with independent scaling. The ingest path handles 1M writes/second; the query path handles a few hundred reads per second. They share no code path and no server pool.

High-Level Design

The critical flows are: propagate context through every boundary, collect spans without blocking the application, fan out durable events to purpose-built consumers, and assemble honest complete-or-partial traces for engineers.

1. Trace ID propagation across service boundaries

The fundamental problem: when Service A calls Service B, Service B needs to know the trace ID so it can link its span to the same trace. Without an explicit mechanism, every service creates an isolated span with no parent relationship.

Naive approach: carry only a custom X-Request-ID header with the trace ID. Service B can group its span with Service A's, but it does not know which of Service A's spans was the parent. This baseline is useful because many systems already have a request ID, but it cannot express hierarchy.

Components:

• SDK (client agent): A library embedded in each service process. Creates spans, records start and end times, and reads or writes the trace header on all outbound calls.
• Service A / Service B: Production services, unchanged except for the SDK running inside them.

What breaks: A plain request ID tells you two spans belong to the same trace, but not their parent-child relationship. If Service A calls three downstream services in parallel, you cannot tell which call was the direct parent of a given error span. The hierarchical call tree is lost.

Service B knows the trace ID but cannot record a parent_span_id. You can group spans by trace, but you cannot reconstruct which service called which.

Evolved approach: W3C traceparent header carrying both trace ID and parent span ID.

The key insight is that the propagation carrier must encode the parent span ID, not just the trace ID. The W3C traceparent standard defines a single compact header: traceparent: 00-{trace_id}-{parent_span_id}-{flags}. The caller's span ID becomes the parent_span_id for the callee.

Components:

• SDK (updated): Injects traceparent on all outbound calls (HTTP headers, gRPC metadata, message headers). Extracts it on all inbound requests.
• Services A, B, C: Each extracts trace_id and parent_span_id from the header, creates a child span with parent_span_id set to the caller's span ID, then injects its own span ID on any further outbound calls.

Request walkthrough:

1. User hits Service A. No traceparent header present: this is the trace root.
2. Service A generates a 128-bit trace_id and a 64-bit span_id_A.
3. Service A calls Service B with header traceparent: 00-{trace_id}-{span_id_A}-01.
4. Service B extracts trace_id and sets parent_span_id = span_id_A. Generates its own span_id_B.
5. Service B calls Service C with header traceparent: 00-{trace_id}-{span_id_B}-01.
6. Each service emits its completed span to the collector asynchronously.