Batch vs. stream processing

TL;DR

Default instinct: start with batch. Batch is simpler to build, test, debug, and recover from failure. Move to streaming only when latency requirements demand it. The majority of data pipelines in the world are batch, and that is fine.

30-Second Explanation

Batch processing collects data and handles it on a schedule or in bounded chunks. It is simpler, easier to replay, and a good fit when hours or minutes of delay are acceptable. Stream processing handles events continuously and is justified when the product needs fresh results or event-time reactions. Streaming adds state, ordering, late-data, checkpoint, and recovery concerns, so start with batch and move only when the freshness requirement is real.

5-Minute Explanation

1. Set the freshness target: overnight, hourly, minute-level, or sub-second. This is the first decision, not the technology name.
2. Compare the input and output shape. Batch can scan a bounded dataset and recompute; streaming must maintain state as events arrive and may see duplicates, gaps, or late events.
3. Design correctness for the chosen model. Define event time versus processing time, watermarks, windows, checkpoints, replay, and exactly-once or idempotent effects for a stream.
4. Price operations: backfills and deterministic reruns are usually simpler in batch; streaming needs lag monitoring, state recovery, rebalancing, and schema evolution.
5. Consider a hybrid. Keep the stream for fresh incremental results and run periodic batch jobs to reconcile, backfill, or rebuild state.

The Framing

Your analytics team runs a nightly batch job. At midnight, a Spark job reads the day's events from S3, aggregates revenue by product category, and writes the result to the data warehouse. The marketing team sees yesterday's numbers at 8 AM. This worked for years.

Then the business launches a flash sale. The CEO asks: "How are flash sale numbers looking?" The answer: "We will know tomorrow morning." This is not acceptable. The business wants to see revenue accumulating in real time, not 8-18 hours later.

A streaming pipeline would solve this: Kafka ingests every purchase event, Flink computes a running sum, and the dashboard updates every second. But now you need to handle late-arriving events, out-of-order data, exactly-once semantics, stateful processing, and checkpointing. The batch job was 200 lines of Spark. The streaming pipeline is 2,000 lines of Flink with a separate state management layer.

A common example is a team rewriting a working batch pipeline as a streaming pipeline, only to discover that the operational complexity was not worth the freshness improvement. The question is not "can we stream this?" but "does the business need sub-minute freshness for this data?"

The honest answer: most data pipelines should be batch. Streaming is for when freshness is a hard business or operational requirement, not a nice-to-have.

How Each Works

Batch: process a bounded dataset

Batch processing operates on a fixed, bounded dataset. You know the start and end of the data before processing begins. The job reads all the data, transforms it, and writes the result.

Typical batch pipeline:
  1. Schedule trigger (cron, Airflow DAG) fires at midnight
  2. Read input: SELECT * FROM events WHERE date = '2024-03-15'
     (or scan S3 partition s3://lake/events/2024/03/15/)
  3. Transform: filter, aggregate, join, enrich
  4. Write output: INSERT INTO warehouse.daily_revenue ...
  5. Job completes. Success or failure is binary.

Tools: Apache Spark, Hadoop MapReduce, dbt, Airflow, AWS Glue

Fault tolerance is simple: if the job fails, re-run it on the same input. The input is immutable (it is yesterday's data sitting in S3). Re-running produces the same output. This is idempotent by nature.

Resource usage is bursty. The cluster spins up, processes the data, and shuts down. You pay for compute only during the job window. Spark on-demand clusters cost a fraction of always-on streaming infrastructure.

The limitation is freshness. The output is only as fresh as the last completed batch. If you run hourly batches, the data is up to 1 hour stale. If daily, up to 24 hours stale.

Stream: process an unbounded dataset

Stream processing operates on a continuous, unbounded flow of events. There is no "end" to the data. Events arrive indefinitely, and the processor maintains running state.

Typical stream pipeline:
  1. Consumer reads from Kafka topic (continuous)
  2. For each event: deserialize, validate, transform
  3. Update running state (counters, windows, aggregations)
  4. Emit result downstream (another Kafka topic, database, dashboard)
  5. Checkpoint state periodically for fault tolerance

Tools: Apache Flink, Kafka Streams, Spark Structured Streaming, AWS Kinesis

Fault tolerance is complex. The processor maintains in-memory state (running counts, window aggregations). If it crashes, that state is lost unless checkpointed. Flink uses periodic snapshots to a durable store (S3, HDFS). Kafka Streams uses a local RocksDB state store backed by a changelog topic in Kafka. Recovery means restoring from the last checkpoint and replaying events since then.

Resource usage is continuous. The streaming job runs 24/7. You pay for always-on compute. For low-volume streams, this is wasteful. For high-volume streams (100K+ events/sec), the continuous processing model is more efficient than repeated batch startups.

Windowing: how streams group time

Streams process infinite data, but aggregations need boundaries. Windowing defines those boundaries.

Tumbling window: fixed-size, non-overlapping intervals.
  [00:00-01:00]: 142 events → emit count=142
  [01:00-02:00]: 189 events → emit count=189
  Use for: per-minute metrics, hourly aggregates

Sliding window: fixed-size, overlapping intervals.
  [00:00-01:00]: 142 events
  [00:30-01:30]: 167 events (overlaps by 30 min)
  Use for: moving averages, rate limiting (100 req per 60s sliding)

Session window: dynamic size, gap-based.
  User clicks at 10:05, 10:06, 10:08, then silence.
  Session = [10:05-10:08] (30-min inactivity gap closes it)
  Use for: user session analytics, engagement tracking

Watermarks solve the late-arrival problem. A watermark is a timestamp assertion: "I believe all events with timestamp before W have arrived." Events arriving after their watermark are either dropped or routed to a side output for late processing. Flink allows configurable watermark strategies (bounded out-of-orderness is the most common: "events can be up to 5 seconds late").

Head-to-Head Comparison