Job Scheduler

TL;DR

• Store a durable job definition and create one execution record per scheduled occurrence.
• Separate the scheduler, which decides what is due, from workers, which invoke handlers. Use database claims or a Redis timing index only as an optimization over durable state.
• The transport is at least once. Exactly-once effects require the handler to accept a stable execution or idempotency key.
• Recover scheduler claims with leases and an outbox, and recover crashed workers with a stale-running watchdog plus durable retry state.
• For the illustrative scenario, 1 million active jobs and 10,000 executions per second require indexed due-work scans, bounded polling, backpressure, and independent worker scaling.

Scope and assumptions

This article designs a distributed scheduler for cron-like and one-off jobs that invoke a handler endpoint, retry failures, and expose execution status. The core problems are time-based selection, duplicate scheduling, worker crash recovery, retry durability, and throughput under bursty due work.

The interview scenario uses these illustrative assumptions:

• Up to 1 million active job definitions and a peak of 10,000 execution attempts per second.
• A job is scheduled once per occurrence; recurring jobs compute their next occurrence after a successful claim.
• The scheduler provides at-least-once dispatch. End-to-end exactly-once effects are possible only when the handler and its side effects are idempotent.
• A handler has a timeout and retry policy. Workflow DAGs, continuous streaming jobs, and geo-distributed scheduling are out of scope.
• The scale, timing, retry, and availability values below are interview requirements, not production measurements or guarantees.

Functional Requirements

Core Requirements

1. Register jobs with a cron expression or a specific execution time.
2. Dispatch each scheduled occurrence at approximately its scheduled time; exactly-once effects require an idempotent handler.
3. Retry failed jobs with configurable backoff.
4. Provide job status visibility (pending, running, succeeded, failed).

Below the Line

• Workflow DAGs with job dependencies (systems like Airflow or Temporal)
• Real-time streaming jobs

The hardest part in scope: Preventing duplicate scheduling and silent loss across a distributed scheduler and worker fleet. Multiple scheduler instances can find the same due rows, while crashes can happen between claiming, publishing, execution, and status updates. The design therefore uses durable claims, an outbox, leases, and idempotent handlers; the infrastructure contract remains at least once.

Workflow DAGs are below the line because dependency tracking is a separate layer that sits on top of a scheduler rather than inside it. A later extension could introduce a Workflow entity holding a directed acyclic graph of job nodes, with a DAG runner that fires each node only after all upstream dependencies complete. That runner would submit individual leaf jobs to this scheduler rather than replacing it.

Real-time streaming jobs are below the line because they require continuous, stateful, low-latency processing that is fundamentally different from the discrete "fire once" model here. A later extension could integrate a stream processor as a separate pipeline alongside this system, not modify the scheduler.

Non-Functional Requirements

Core Requirements

• Scale: Illustrative scenario of 1M active jobs in the system at any time and 10K job executions per second at peak throughput.
• Scheduling precision: Each job should be dispatched within the illustrative 1-second target of its scheduled time.
• Availability: Illustrative target of 99.99% uptime. If a scheduler node dies, another should take over within seconds, not minutes.
• Delivery semantics: At-least-once delivery is acceptable at the infrastructure layer. Exactly-once is enforced at the application layer via idempotency keys in the handler.

Below the Line

• Sub-100ms scheduling precision
• Geo-distributed scheduling across regions

Sub-100ms scheduling precision is below the line because it forces a fundamentally different architecture: you cannot poll a database fast enough. To achieve it, you would replace the PostgreSQL polling loop with an in-memory timer wheel (like Kafka's or Netty's HashedWheelTimer) running on each scheduler node, pre-loading job fire times into memory and triggering off OS-level timers. That is a specialized real-time system, not a general-purpose scheduler.

Geo-distributed scheduling is below the line because it introduces cross-region clock synchronization, partition-tolerant consensus for job ownership, and the question of whether a job should fire in the region closest to the handler or the region that registered it. A later extension could assign each job a "home region" and run independent scheduler fleets per region, with a global control plane for cross-region job migration on failover.

Read/write ratio: Jobs are write-heavy at creation time, then execution state becomes write-heavy as workers claim and finish attempts. A single cron job with a 1-second interval produces 86,400 execution records per day. If all 1M active jobs ran every second, the upper bound would be 86.4 billion records per day, not tens of millions; real workloads must be modeled from their frequency distribution. Due-work scans and state transitions determine the index strategy on job_executions, the caching approach, and why a naive polling loop breaks at scale.

The illustrative 1-second precision target constrains the scheduler's polling interval to 500ms or less. A 5-second sleep between polls means a job due at T=1 does not fire until T=5, a 4-second slip that violates the scenario SLA. Polling every 500ms leaves some headroom for query time and network jitter, but the actual interval should be measured against batch size and load.

The 99.99% availability target means a single scheduler node is not acceptable. That is at most 52 minutes of downtime per year, and a single process restart takes 10-30 seconds. A multi-node scheduler with automatic failover is required.

30-second answer

Persist job definitions and one durable execution record per scheduled occurrence. Scheduler instances claim due records with a lease and publish them through an outbox to a durable queue or timing index; workers only execute and report state. A stale-running watchdog recovers worker crashes, and retry records store backoff durably. At-least-once delivery is the infrastructure contract; exactly-once effects require the handler to deduplicate by a stable execution ID.

5-minute explanation

Registration validates the cron or one-off time, stores the handler and retry policy, and computes next_run_at. The scheduler is deliberately separate from execution: it finds due occurrences, claims each one once for a lease, and publishes an execution ID. Workers pull the ID, mark the execution running, invoke the handler with an idempotency key, and record success or failure.

The durable database is the source of truth. Database row claims are the simple correctness design; at higher throughput, a Redis sorted set can act as a pre-loaded timing index, but it must be filled from durable state and repaired through an outbox. A queue absorbs bursts between scheduling and execution, while workers scale independently.

Retries are new durable attempts with exponential backoff. If a worker dies before recording its outcome, a watchdog identifies stale-running executions and applies the same retry policy. If a scheduler dies after claiming but before publishing, lease expiry plus outbox replay makes the occurrence visible again. Duplicate delivery remains possible, so handlers and downstream writes must be idempotent.

The entities and APIs below anchor those flows; the high-level architecture then shows registration, scheduling, execution, retry, and status reads as separate responsibilities.

Core entities

• Job: The static definition of work to be done. Carries the schedule (cron expression or one-off timestamp), the handler endpoint to invoke, retry policy, and lifecycle status.
• JobExecution: A single invocation of a Job. Tracks which attempt this is, when it ran, which worker claimed it, and whether it succeeded or failed. Each scheduled fire of a Job produces one new JobExecution.

We will revisit schema details, including indexes and partitioning, in the scaling and failure deep dives below. The two entities above are sufficient to drive the API and high-level architecture.

API design

FR 1 - Register a job:

POST /jobs
Body: { schedule, handler_endpoint, max_retries, timeout_seconds }
Response: { job_id }

FR 2 - List executions for a job:

GET /jobs/{job_id}/executions?cursor=<execution_id>&limit=50
Response: { executions: [...], next_cursor }

FR 3 - Manually trigger a retry:

POST /jobs/{job_id}/retry
Response: { execution_id }

FR 4 - Get current job status:

GET /jobs/{job_id}
Response: { job_id, status, last_execution_at, next_run_at }

Use POST /jobs because job creation is not idempotent by default: two identical POST requests for the same cron expression create two separate scheduled jobs with distinct job_id values. GET /jobs/{job_id}/executions uses cursor pagination rather than offset pagination because the job_executions table grows unboundedly; OFFSET 100000 requires scanning 100,000 rows before returning anything, which degrades as the table grows. POST /jobs/{job_id}/retry creates a new JobExecution immediately and publishes it to the queue, bypassing the scheduler's polling cycle for fast manual intervention.

45-minute interview approach

Use this section only as the pacing plan for a scheduler design prompt; keep correctness and throughput detail in the architecture and deep dives.

• 0-5 minutes — clarify the contract: Confirm cron versus one-off jobs, handler timeouts, retry policy, cancellation, time-zone semantics, and whether "exactly once" means dispatch or side effects.
• 5-10 minutes — requirements and estimates: State the illustrative 1M active jobs, 10K executions per second, 1-second dispatch target, availability target, and expected execution-history retention.
• 10-15 minutes — entities and APIs: Identify Job and JobExecution. Sketch registration, status, execution-history, manual-retry, and any cancellation endpoint; define the stable execution ID.
• 15-25 minutes — baseline architecture and flows: Draw the App Server, durable database, Scheduler, queue or timing index, Worker Pool, and watchdog. Walk through registration, due-work claim, handler execution, and status reads.
• 25-35 minutes — choose deep dives: Let the interviewer select distributed claims, worker failure and retries, or throughput scaling. Compare a single scheduler, leader election, database claims, and a pre-loaded timing index.
• 35-41 minutes — reliability, security, and operations: Cover outbox recovery, lease expiry, idempotent handlers, stale-running detection, queue backpressure, handler authentication, time skew, and observability.
• 41-45 minutes — trade-offs and close: Explain the at-least-once boundary, precision versus polling cost, execution-history retention, failure windows, and what would change for workflows or multi-region scheduling.

High-level architecture and critical flows

The scheduler has four critical flows: registration persists a job definition, scheduling creates or claims a due execution, execution invokes the handler and records the outcome, and recovery reclaims work after scheduler or worker failure. The database is durable state; queues, leases, and timing indexes are derived coordination mechanisms.

FR 1 and FR 2 - Register and store jobs

The simplest starting point: client registers a job, the App Server validates the cron expression, computes the first next_run_at timestamp, and stores the definition in PostgreSQL.

Request walkthrough:

1. Client sends POST /jobs with a cron expression and handler endpoint.
2. App Server validates the cron syntax and computes the initial next_run_at.
3. App Server inserts a Job row into PostgreSQL with status=active.
4. App Server returns job_id to the client.