Data pipelines

Introduction

Data pipelines turn operational data into dependable inputs for analytics, search, monitoring, and machine learning. Their design is mostly about making movement and transformation observable, replayable, and appropriate for the required freshness and correctness. Mental model: source → durable landing or log → validated transformation → independently consumable destinations, with checkpoints at each boundary.

TL;DR

• A data pipeline moves data from sources to destinations, transforming it along the way. The hard part is doing this reliably at scale without losing records or violating latency guarantees.
• Batch processing trades latency for simplicity (hourly/nightly runs). Stream processing trades complexity for freshness (sub-second). Many production systems use both.
• ELT (load raw data first, transform in-warehouse) is common in cloud data warehouses. The medallion architecture (bronze/silver/gold) organizes the transformation layers.
• Pipeline failure handling requires three capabilities: dead-letter queues for poison records, checkpointing for crash recovery, and replay from source for fixing transformation bugs.
• Stream processing often adds windowing (tumbling, sliding, session), watermarks, state, and exactly-once semantics as complexity dimensions that batch jobs handle differently.

The Problem It Solves

Your e-commerce company starts with a simple setup: the product team queries the production database directly for analytics. SELECT COUNT(*) FROM orders WHERE created_at > NOW() - INTERVAL '24 hours' runs fine when you have 10,000 orders per day.

At 500,000 orders per day, that query takes 45 seconds and locks rows that the checkout service needs. Your DBA adds a read replica for analytics queries. That buys you 6 months.

At 2 million orders per day, the analytics team needs joins across orders, customers, inventory, and clickstream data. These joins require denormalized tables that don't exist in the OLTP schema. Someone writes a Python script that runs on a cron job at 3 AM, pulls data from four tables, transforms it, and loads it into a separate analytics database. It works until the script crashes silently one night and nobody notices for three days. Three days of revenue dashboards show zeros, and the CFO thinks the company lost all its revenue.

At 10 million events per day (orders plus page views, clicks, searches, ad impressions), the 3 AM cron job takes 6 hours to run. It fails halfway through on out-of-memory errors. The analytics database has stale data. The fraud detection team needs real-time signals, not yesterday's data. The recommendation engine needs feature vectors computed from the last hour of clickstream data, not last night's.

As a system grows, the cron script that "works fine for now" can become a fragile piece of infrastructure. Silent failures are particularly costly when a dashboard continues to show cached data from the last successful run.

The answer is not a better cron script. It's a data pipeline: a system designed from the ground up for reliable, scalable, monitored data movement and transformation.

What Is It?

A data pipeline is a system that moves data from sources to destinations through a series of processing stages, each of which extracts, transforms, enriches, or aggregates the data. The "pipeline" metaphor is literal: data flows in one end, gets processed at each stage, and arrives at the destination in a different shape than it started.

Think of it like a water treatment plant. Raw water (data) flows in from rivers and reservoirs (sources). It passes through filtration, chemical treatment, and quality testing stages (transformations). Clean water (processed data) flows out to homes and businesses (destinations). The plant runs 24/7, monitors pressure and quality at every stage, and has bypass systems for when one stage fails. A data pipeline does the same thing for information.

The key insight: a data pipeline is not just a script that moves data. It's a system with monitoring, failure handling, replay capabilities, and schema management. The difference between "a cron job that runs SQL" and "a data pipeline" is the difference between a garden hose and a water treatment plant.

How It Works

Let's trace a single record through a production pipeline: an order placed on an e-commerce platform that needs to reach the analytics warehouse, the search index, and the fraud detection system.

1. Event production. The checkout service publishes an order.created event to Kafka topic orders. The event includes order ID, customer ID, items, total, payment method, and timestamp.

2. Ingestion. A Flink consumer reads from the orders topic. It validates the schema (all required fields present, types correct) and drops malformed records to a dead-letter topic for investigation.

3. Enrichment. The Flink job looks up the customer's profile from a Redis cache (country, account age, lifetime spend). It joins this context onto the order event, producing an enriched record.

4. Transformation. Business rules are applied: currency conversion to USD, tax calculation, fraud risk score from a sidecar ML model. The enriched, transformed record is written to a downstream Kafka topic orders.enriched.

5. Fan-out to destinations. Three independent consumers read from orders.enriched: one writes to Snowflake (analytics), one updates Elasticsearch (search), one feeds the real-time fraud dashboard (alerting). Each consumer checkpoints its Kafka offset independently.

6. Quality verification. An hourly batch job compares record counts between source (Kafka) and destination (Snowflake). If the counts diverge by more than 0.1%, it triggers an alert.

// Simplified Flink-style pipeline stage (enrichment + transformation)
async function processOrderEvent(event: OrderEvent): Promise<EnrichedOrder> {
  // Step 1: Validate schema
  if (!event.orderId || !event.customerId || !event.total) {
    await deadLetterQueue.send(event, "missing required fields");
    return null;
  }

  // Step 2: Enrich with customer context
  let customer = await redis.get(`customer:${event.customerId}`);
  if (!customer) {
    customer = await customerDB.findById(event.customerId);
    if (!customer) {
      await deadLetterQueue.send(event, "customer not found");
      return null;
    }
    await redis.set(`customer:${event.customerId}`, customer, { ttl: 3600 });
  }

  // Step 3: Transform
  const enrichedOrder: EnrichedOrder = {
    ...event,
    customerCountry: customer.country,
    totalUSD: convertToUSD(event.total, event.currency),
    fraudScore: await fraudModel.score(event, customer),
    processedAt: new Date().toISOString(),
  };

  // Step 4: Emit to downstream topic
  await kafka.produce("orders.enriched", enrichedOrder);
  return enrichedOrder;
}

Every stage is independently monitorable. If enrichment latency spikes, you see it in the stage metrics. If the fraud model slows down, that stage's processing time increases while others are unaffected. This isolation distinguishes a pipeline from a monolithic script.

To reason about a pipeline, trace one record end to end: an order event hits Kafka, gets enriched with customer data from Redis, is transformed with business rules, and is fanned out to Snowflake, Elasticsearch, and the fraud dashboard. A concrete record makes dependencies, checkpoints, and failure boundaries easier to see.

Key Vocabulary and Components

Types / Variations

Batch vs. Stream Processing