Weather Service

What is a weather data service?

A weather data service ingests atmospheric readings from distributed sensor networks and third-party data providers, stores time-stamped measurements, and serves current conditions plus short-term forecasts by geographic coordinate. The engineering challenge is not the meteorology: every user query must translate an arbitrary lat/lng into readings from nearby stations and aggregate them in under 100ms while the ingestion pipeline handles thousands of sensor writes per second in parallel. It tests IoT ingestion, time-series storage, geospatial indexing, and read-heavy caching.

TL;DR

Normalize sensor and provider inputs into canonical observation events, acknowledge them after durable queue publication, and let separate consumers write the historical time-series store and the latest-reading cache. Resolve user coordinates through a cached PostGIS lookup, fetch the nearest stations' snapshots from Redis, and aggregate them without scanning history.

Serve current conditions with a freshness-aware cache policy and short-term forecasts through a rounded-coordinate proxy cache. Use CDN-cached polling for ordinary clients; add SSE only for displays that need sub-minute updates. Retain raw observations for history, pre-aggregate common trends, and make stale data, upstream failures, and ingestion lag visible to clients and operators.

Scope and assumptions

The following are illustrative interview planning assumptions, not meteorological guarantees or commitments from a named provider:

• About 100,000 stations, one reading per minute, or roughly 1,667 observations/second sustained; design for about 3x burst headroom.
• Approximately 10 million daily active users polling every 5 minutes: about 33,000 reads/second on an even average, with an illustrative 5x peak of 165,000 reads/second.
• Current conditions target under 100ms p99 and no more than 5 minutes of data age. The response includes observed_at and a freshness indicator so the client can show when a value is stale.
• At least 2 years of observations are retained. Current conditions, nearest-station resolution, and a short-term provider forecast are in scope; numerical weather prediction, alerting, bulk export, and personalization are separate consumers or extensions.
• Coordinates are normalized for cache keys. The default lookup radius is 25 km, but sparse areas may request a larger radius; the radius must be part of any coordinate-to-station cache key.
• Kafka, Redis, PostgreSQL/PostGIS, TimescaleDB, and a CDN are illustrative implementation choices. Provider cadence, cache hit rates, spatial-query latency, and compression ratios require measurement.

Functional Requirements

Core Requirements

1. Ingest weather readings from sensor networks or third-party weather APIs at regular intervals.
2. Store current and historical observations per geographic location.
3. Serve current conditions and short-term forecasts by latitude/longitude.
4. Refresh the client UI with updated data at a configurable interval.

Below the Line (out of scope)

• Building numerical weather prediction (NWP) forecasting models.
• Severe weather push alerting.
• Historical data bulk export and analytics.
• User authentication and personalization.

Forecasting model training is out of scope because NWP algorithms (ensemble methods, physics simulations) require specialized infrastructure separate from the data serving layer. The service can call an external model API and cache the returned forecast. The serving logic is a cache-aside proxy, not a modeling pipeline.

Severe weather alerting could sit beside the write path as a separate consumer: read each new Observation from the Kafka topic, evaluate it against geo-fenced alert cells, and fire a notification if a threshold is crossed. It is deferred because it does not change the ingestion or read path here.

Historical export belongs in a data warehouse fed by a separate Kafka consumer from the same ingestion topic. The export consumer writes to the warehouse; the serving path does not need to touch it.

User authentication would add a user_id context to the read API and allow personalization such as saved locations. It layers on top of the existing API design without changing the storage or geo-resolution logic.

The hardest part in scope: Translating an arbitrary user coordinate into up-to-date conditions from nearby stations requires a geospatial index that stays fast as the station count grows, and a cache invalidation policy that knows when a reading is stale. Both problems sit in tension: a tighter cache TTL means fresher data but more database reads at scale.

Non-Functional Requirements

Core Requirements

• Scale (writes): 100K weather stations each ingesting one reading every 60 seconds = approximately 1,667 writes/sec sustained. Design for 3x burst headroom to handle thunderstorm events when stations report more frequently.
• Scale (reads): 10M DAU each polling every 5 minutes = approximately 33K reads/sec at even distribution. Expect 5x peak spikes = up to 165K reads/sec during morning weather checks.
• Read latency: Current conditions delivered in under 100ms p99 end-to-end.
• Data freshness: Current conditions stale by no more than 5 minutes. Older than 5 minutes, the app should indicate the last-updated timestamp rather than silently displaying stale data.
• Availability: 99.9% uptime for the read path (current conditions and forecasts). Brief ingestion lag during a partial outage is acceptable; dark screens are not.
• Durability: Historical observations retained for at least 2 years for trend analysis and display.

Below the Line

• Sub-second real-time streaming of sensor readings (pub/sub dashboard use cases).
• Per-sensor raw data export with millisecond timestamps (industrial IoT).

Read/write ratio: Roughly 33K reads vs 1,700 writes peak = approximately 20:1. The system is read-dominant but not as skewed as a URL shortener (1,000:1). The interesting design tension here is that the write path must be durable and throughput-consistent (sensors cannot block waiting for slow writes), while the read path must be fast (geo-query plus aggregation under 100ms). These two paths have conflicting needs that push toward a clean separation via a message queue.

30-second answer / outline

1. Normalize MQTT, HTTP, CoAP, and provider payloads into canonical Observation events and publish them to Kafka before acknowledging ingest.
2. Consume the topic twice: batch-write durable observations to a time-partitioned TimescaleDB hypertable and update one Redis snapshot per station.
3. Resolve (lat, lng, radius) through a rounded-coordinate cache backed by PostGIS ST_DWithin and a GIST index, then MGET the nearest stations' snapshots.
4. Cache current responses at the CDN and forecasts at rounded coordinates; expose observation age and upstream/model timestamps.
5. Use continuous aggregates for historical trends, monitor consumer lag and stale stations, and degrade to a labeled last-known value when a cache or provider is unavailable.

5-minute explanation

Start with the illustrative read/write split: about 1,700 observations/second versus 33K average reads/second, with burstier traffic on both sides. Sensors need a fast acknowledgement and durable buffering, so the Adapter publishes to Kafka and consumers handle storage at their own pace.

One consumer writes the raw event stream to a time-partitioned time-series store; another maintains the latest reading per station in Redis. Current-condition reads first resolve a rounded coordinate to a small list of nearby station IDs, using Redis before PostGIS, then fetch those snapshots with MGET. The historical store is reserved for trends, backfill, and a controlled fallback rather than normal current reads.

Forecasts are a separate cache-aside path to an external model provider. Rounding coordinates improves cache reuse, but the response must retain the provider's generation time. Ordinary clients use CDN-cached polling; sub-minute operator displays can use SSE with explicit connection and reconnect limits.

The detailed high-level architecture and critical flows below show ingestion, geo-resolution, current conditions, forecasts, refresh behavior, time-series storage, and failure handling. The data model deep dives explain chunking, aggregates, geospatial boundaries, and retention.

45-minute interview approach

This is a time-boxed plan for answering the design question in an interview, not a claim that the article should be read in 45 minutes.

• 0–5 minutes — Clarify the contract: Confirm station sources, observation cadence, forecast horizon, coordinate accuracy, radius, freshness, retention, alerts, and whether historical analytics is in scope.
• 5–10 minutes — Establish scale: Use the illustrative station, write-rate, read-rate, burst, retention, and p99 assumptions. Separate sensor ingestion from user queries and provider calls.
• 10–15 minutes — Define APIs and schema: Walk through batch ingest, current conditions, forecast, observed_at, generated_at, station IDs, coordinate rounding, radius, and stale-data behavior.
• 15–22 minutes — Draw ingestion: Show adapters, Kafka, durable consumer, snapshot consumer, TimescaleDB, Redis, offset commits, replay, and the 202 acknowledgement semantics.
• 22–30 minutes — Draw the current-read path: Show CDN, Query Service, coordinate cache, PostGIS ST_DWithin, station snapshots, aggregation, and historical fallback.
• 30–35 minutes — Deep dive on storage and geo: Compare an unpartitioned table, time partitions, and TimescaleDB; then compare bounding boxes, geohash neighbors, and PostGIS.
• 35–41 minutes — Reliability, security, and operations: Cover malformed readings, out-of-order events, stale stations, cache stampedes, upstream forecast failure, rate limits, location privacy, consumer lag, and retention.
• 41–45 minutes — Trade-offs and close: Compare polling versus SSE, CDN versus origin caching, PostGIS versus geohash, and raw scans versus continuous aggregates. Recap freshness and failure behavior, then invite follow-ups.

Core Entities

• Station: A geographic measurement source (lat, lng, altitude, station_type). Can be a physical IoT sensor or a virtual aggregation point from a third-party provider. Changes rarely; the table is small and cache-friendly.
• Observation: A single time-stamped reading from one station. Captures temperature_c, humidity_pct, pressure_hpa, wind_speed_kph, wind_direction_deg, and precipitation_mm. The primary hot data; billions of rows accumulate over months.
• Forecast: A set of predicted conditions for a future time window at a location. Sourced from an external NWP API and cached locally. Read-only to this system; the weather service does not generate forecasts.
• WeatherSnapshot: A pre-computed "latest reading" record per station, held in Redis. Avoids live aggregation on every user query. Rebuilt from new Observations as they arrive.

The primary data flow is: raw Observations feed both the durable historical store (TimescaleDB) and the live snapshot cache (Redis). Every user read for current conditions hits the snapshot cache, not the historical store. Schema and indexing decisions are deferred to the deep dives.

API Design

One endpoint per functional requirement, grouped by the requirement it satisfies.

FR 1 (ingest observations):

POST /v1/observations
Content-Type: application/json
Body: {
  station_id: "KNYC",
  readings: [
    {
      observed_at: "2026-04-03T12:00:00Z",
      temperature_c: 18.5,
      humidity_pct: 72,
      pressure_hpa: 1013,
      wind_speed_kph: 14,
      wind_direction_deg: 270,
      precipitation_mm: 0
    }
  ]
}
Response 202: { ingested_count: 1 }

Batch array rather than single-record: sensors often buffer 5-10 readings locally during connectivity gaps and flush in a burst. A batch endpoint handles this without the client making one HTTP call per reading. The 202 (Accepted) response confirms the payload was received and queued; it does not guarantee durable storage, which is handled asynchronously by the Kafka consumer.

FR 2 and FR 4 (current conditions with configurable refresh):

GET /v1/weather/current?lat=40.7128&lng=-74.0060&radius_km=25
Response 200: {
  location: { lat: 40.7128, lng: -74.0060 },
  observed_at: "2026-04-03T12:00:00Z",
  temperature_c: 18.5,
  humidity_pct: 72,
  pressure_hpa: 1013,
  wind_speed_kph: 14,
  wind_direction_deg: 270,
  precipitation_mm: 0,
  nearest_station_id: "KNYC",
  nearest_station_distance_km: 2.4
}
Cache-Control: max-age=300

The Cache-Control: max-age=300 header drives the UI refresh interval without a separate parameter. CDN edges and browsers respect this header and serve cached responses for 5 minutes before re-requesting. The optional radius_km parameter (default 25 km) controls how wide a net to cast for nearby stations; clients in rural areas with sparse sensor coverage can increase this to 100 km without changing anything else in the pipeline.

FR 3 (short-term forecast):

GET /v1/weather/forecast?lat=40.7128&lng=-74.0060&hours=24
Response 200: {
  location: { lat: 40.7128, lng: -74.0060 },
  generated_at: "2026-04-03T12:00:00Z",
  hourly: [
    { hour: "2026-04-03T13:00:00Z", temperature_c: 19.0, precipitation_prob: 0.1, wind_speed_kph: 12 }
  ]
}
Cache-Control: max-age=1800

Forecast provider cadence varies by product and model. A max-age=1800 TTL is an illustrative choice that limits repeat upstream calls while keeping the cache window visible; the generated_at field tells the client when the underlying model ran, independent of when the cached copy was served.

High-Level Design

1. Ingest weather data from sensor networks and third-party APIs

The write path receives batches of sensor readings from two source types: direct sensor adapters using MQTT or HTTP push, and scheduled pollers that call third-party APIs (NOAA, Tomorrow.io) every few minutes.

Components: