Notification Service

What is a notification service?

A notification service delivers messages from your product to users across push, email, and SMS. Sending one notification is trivial. Sending a billion without losing any, without blowing APNs rate limits during a viral campaign, and without delivering the same notification twice because a worker crashed mid-send is where the real engineering lives.

This system design brings producer-consumer boundaries, external provider behavior, and idempotency into one flow. The important questions are how to absorb fan-out, isolate provider backpressure, and define delivery semantics that remain honest when a provider times out.

TL;DR

Accept notification intents durably, publish them through a durable event log, route them to independent per-channel queues, and let channel-specific workers call push, email, or SMS providers. Keep the database as the source of truth for accepted work, use retries and a delivery ledger for at-least-once processing, and treat provider delivery receipts as asynchronous evidence rather than an end-to-end guarantee.

Bulk campaigns need their own fan-out capacity and provider-aware rate limits so they cannot starve transactional traffic. Preferences and quiet hours must be checked close to dispatch, with an explicit freshness policy for opt-outs. The detailed sections below retain the data model, APIs, queue topology, bulk calculations, retry state machine, and preference-cache alternatives.

Scope and assumptions

The design covers push, email, and SMS notification orchestration, including immediate and scheduled sends, user preferences, bulk fan-out, retries, and provider callbacks. In-app inboxes, OTP-specific latency, content experimentation, and rich media delivery are outside the main path. The throughput, latency, user-count, campaign-size, and provider-capacity figures below are illustrative interview assumptions; validate them against the actual traffic distribution, provider account limits, and load tests.

Functional Requirements

Core Requirements

1. Send notifications through mobile push (APNs/FCM), email, and SMS channels.
2. Support immediate and scheduled sends.
3. Guarantee at-least-once delivery; deduplicate at the client where possible.
4. Allow users to manage preferences and opt out of specific channels.

Below the Line (out of scope)

• In-app notification bell and inbox (badge counts, read/unread state)
• Transactional OTP flows with tight sub-3-second delivery requirements
• A/B testing of notification content and send-time optimization
• Rich push notifications with images and deep-link action buttons

The hardest part in scope: Fan-out at scale. When a platform pushes a new post to 50 million followers, those 50 million push notifications must land within minutes without saturating APNs, causing other notifications to queue behind the campaign for hours, or delivering duplicates if the fan-out worker crashes halfway through.

An in-app notification bell is out of scope because it requires a separate storage model (an inbox per user with read/unread state) and a real-time delivery mechanism. A future extension could write notification records to a dedicated inbox table after successful delivery, expose a GET /users/{id}/notifications paginated endpoint, and push badge-count updates through Server-Sent Events or WebSocket.

Transactional OTP flows share the SMS delivery path but require a much tighter latency budget than general notifications. A future extension could route OTP events to a separate high-priority SMS queue with reserved capacity, while still keeping bounded admission and retry controls so an outage cannot create an unbounded backlog.

A/B testing is a product layer on top of delivery. A future extension could resolve the template variant at enqueue time, assign users to experiment arms through a feature-flag service, and publish the selected variant to Kafka.

Rich push is out of scope because push providers impose payload limits and separate media-attachment flows. A future extension could store media URLs in the notification payload and let the device SDK download assets asynchronously on receipt rather than bundling the assets in the push payload.

Non-Functional Requirements

Core Requirements

• Throughput: Handle 1M notifications per second at peak across all channels combined.
• Latency: Real-time notifications queued within 1 second of trigger; delivered within 10 seconds end to end for push.
• Availability: 99.99% uptime for the ingestion API. Delivery workers tolerate brief restarts as long as the queue persists.
• Durability: No notification is lost once accepted by the ingestion API.
• Scale: 1 billion registered devices, 500M DAU.

Below the Line

• Sub-second push delivery end to end (APNs and FCM add their own tail latency beyond our control)
• Real-time delivery receipts and per-user read confirmations

Write-heavy reality: This system is almost entirely writes. Every inbound event produces at least one dispatch per channel, and bulk campaigns produce millions. There is no hot read path comparable to a URL shortener; the challenge is absorbing enormous write throughput without data loss and without blowing through external provider rate limits every time a marketing team sends a campaign. Every design decision in this article traces back to that 1M peak writes per second constraint.

The 99.99% availability target applies to the ingestion API only, not end-to-end delivery. Each external provider (APNs, FCM, Twilio, SES) has its own availability and latency behavior, and the service cannot exceed those dependencies. Design for at-least-once processing and idempotent workers rather than promising real-time delivery that depends on third-party uptime.

30-second answer / outline

1. Persist an accepted notification or scheduled job before acknowledging the caller; use a transactional outbox or a replayable sweeper to bridge the database and event log.
2. Route through a durable pending topic, resolve preferences, render channel-specific payloads, and publish to independent push, email, and SMS topics.
3. Give each channel a separate worker fleet, provider client, rate limiter, retry policy, and dead-letter path so one provider outage does not block the others.
4. Track delivery attempts with a unique ledger key, classify provider errors, and use provider idempotency features where they exist; otherwise expose the remaining duplicate-delivery risk honestly.
5. Isolate bulk fan-out from real-time traffic and monitor queue age, provider throttling, retry volume, preference freshness, and permanent token/address failures.

5-minute explanation

The ingestion API owns acceptance, validation, scheduling, and the durable notification record. A caller receives an ID only after the service has a recoverable record; publishing the event can happen in the same transaction through an outbox or be repaired by a sweeper. The pending topic is a hand-off boundary, not the business source of truth.

The Router Worker resolves the recipient's current eligible channels, applies quiet hours and suppression rules, renders a channel-specific payload, and fans out one event per channel. Independent channel topics absorb different provider limits. Workers perform the external calls asynchronously, write delivery outcomes, retry transient failures with delay scheduling, and send poison messages to a dead-letter queue.

At-least-once processing means a crash between a provider call and the local ledger write can create a duplicate unless the provider accepts an idempotency key or the application uses a provider-specific reconciliation protocol. Preferences therefore need a freshness and fail-closed policy for legally required opt-outs. Large campaigns use cursor-based segmentation, sharded fan-out, separate connection pools, and reserved capacity so the normal notification path remains usable.

45-minute interview approach

This is a time-boxed interview plan, not a promise that the article should be read in 45 minutes.

• 0–5 minutes — Clarify the contract: Confirm channels, immediate versus scheduled sends, fallback semantics, user opt-outs, delivery receipts, retention, and whether “delivered” means accepted by a provider or confirmed by a device.
• 5–10 minutes — Establish scale: Use the illustrative device, DAU, peak-throughput, campaign-size, latency, and provider-limit assumptions; ask how much traffic is bulk versus transactional.
• 10–15 minutes — Define APIs and invariants: Walk through single send, bulk send, preference replacement, idempotency keys, accepted versus scheduled status, and the invariant that accepted work is recoverable.
• 15–22 minutes — Draw the high-level path: Show ingestion, durable state, pending topic, router, preference cache, per-channel topics, workers, provider gateways, and delivery ledger.
• 22–30 minutes — Deep dive on fan-out: Explain cursor scans, sharded campaign topics, bulk/real-time isolation, token-bucket controls, partition keys, and backpressure.
• 30–35 minutes — Deep dive on delivery correctness: Cover retries, delay topics, delivery-attempt uniqueness, provider timeouts, duplicate risk, callbacks, and dead-letter handling.
• 35–41 minutes — Reliability, security, and operations: Discuss outbox recovery, provider outages, opt-out freshness, credential isolation, payload validation, PII protection, queue age, and reconciliation.
• 41–45 minutes — Trade-offs and close: Compare push/email/SMS worker designs, Bloom-filter preference checks, scheduled polling, multi-region ownership, and the boundary between provider acceptance and actual user delivery.

Core Entities

• Notification: A single delivery event carrying channel, recipient identifier, template reference, rendered payload, status, and optional scheduled delivery time.
• User: The recipient account, tied to a device token (push), email address, and phone number per channel.
• UserPreference: A per-user, per-channel opt-in flag with optional quiet-hours window configuration.
• NotificationTemplate: A reusable payload template with variable slots for personalization (order ID, username, amount, etc.).
• DeliveryLog: An append-only record of each delivery attempt: timestamp, outcome (success, transient failure, permanent failure), and provider response code.

Full schema, indexes, and column types are deferred to the data model deep dive. The entities above are enough to drive the API design and High-Level Design.

API Design

FR 1 and FR 2: Send a notification:

# Accept a single notification and queue it for delivery
POST /v1/notifications
Body: {
  user_id: "u_123",
  channels: ["push", "email"],
  template_id: "order_confirmed",
  template_vars: { "order_id": "o_456" },
  scheduled_at?: "2026-03-29T15:00:00Z"
}
Response: { notification_id: "n_789", status: "queued" }

Accepting channels as an array rather than a scalar lets the caller specify a primary channel with fallbacks in one request. Using template_id rather than a raw body prevents XSS and keeps payloads auditable. scheduled_at defaults to "now" when absent, covering both immediate and scheduled sends in the same endpoint.

FR 1: Bulk send to a user segment:

# Schedule a notification campaign to an entire user segment
POST /v1/notifications/bulk
Body: {
  segment_id: "new_users_march",
  channels: ["push"],
  template_id: "onboarding_day1",
  scheduled_at?: "2026-03-29T09:00:00Z"
}
Response: { batch_id: "b_999", estimated_recipients: 4200000, status: "scheduled" }

Do not accept a user_ids array in the request body. A 50M element array creates a request body that is impossible to parse and a timeout bomb for the ingestion service. Segment-based sends resolve the recipient list asynchronously inside the fan-out pipeline, returning immediately with a batch_id for status polling.

FR 4: Manage user preferences:

# Read and replace the full channel opt-in/out preference set for a user
GET  /v1/users/{user_id}/preferences
PUT  /v1/users/{user_id}/preferences
Body: { push: true, email: false, sms: true }
Response: { user_id: "u_123", push: true, email: false, sms: true, updated_at: "..." }

Use PUT over PATCH because the preference object is small and always replaces the full set of channel flags. PATCH with partial updates adds merge-conflict complexity for no benefit at this schema size.

High-Level Design

1. Source systems submit a notification via the ingestion API

The ingestion path: source system calls REST API, notification service validates and persists, event published to Kafka for async processing.

Components:

• Source System: Any internal service (product, payments, auth) that needs to trigger a notification. Calls POST /v1/notifications with a template reference and recipient.
• Notification Service: Validates the payload, writes a notification record to the database with status pending, then publishes the event to Kafka.
• Notification DB: PostgreSQL. Stores the notification record as the source of truth for status tracking. Insert on receipt, update on delivery outcome.
• Kafka (ingestion topic): Receives the notification event after the successful DB write. All downstream processing happens from this topic, never from the source system directly.

Request walkthrough:

1. Source system sends POST /v1/notifications with user_id, template_id, and optional scheduled_at.
2. Notification Service validates: verifies the template exists, user_id is non-null, channel list is non-empty.
3. Notification Service inserts a record into Notification DB. If scheduled_at is in the future, status is scheduled; otherwise pending.
4. Notification Service publishes the event to the notifications.pending Kafka topic, keyed by user_id for ordered per-user processing.
5. Notification Service returns { notification_id, status: "queued" } to the source system.

The write to Notification DB happens before the Kafka publish. If Kafka is temporarily unavailable, the record persists in the DB with status pending and a background sweeper re-publishes it. A transactional outbox is the stronger implementation because it commits the notification and its publish intent together; a sweeper is still useful for recovery and reconciliation.