Start practising

SYSTEM DESIGN · BUILDING BLOCKS · 13 OF 14

Job Scheduler and Workers: Delayed, Recurring and Reliable

How to run background work reliably at scale: why cron on one box fails, the jobs table, how a scheduler finds due jobs and why only one should act on each, claiming work with leases and SKIP LOCKED, heartbeats and reclaiming jobs from dead workers, retries and dead letters, recurring schedules and missed runs, priorities and fairness, scaling the scheduler, metrics, and sizing.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
1 / 15

Press play to hear the narration, or step through at your own pace.

Notes

The problem

The naive version is a crontab on one server. If that server dies, jobs silently stop running, and nobody notices until invoices are missing. Put the same crontab on three servers for safety and every job runs three times. There are no retries, no history, no way to ask whether last night's job ran. And one machine's CPU caps how much work you can do.

Why cron on a box fails
  • the box dies → jobs silently
  • never run
  • run cron on 3 boxes for
  • safety → every job runs 3x
  • no retries, no history,
  • no "did the 2 a.m. job run?"
  • one machine's CPU limits
  • how much work you can do

Data model

Start with the data. Every job is a durable row: its type and payload, when it should run, its status, how many attempts so far, which worker holds it and until when, a unique dedupe key so the same job can't be submitted twice, and a cron expression if it repeats. Index run at, because the scheduler's main question is 'what is due now?'

jobs table
id            bigint pk
type          'send_reminder'
payload       jsonb
run_at        timestamptz (index)
status        queued|running|done
              |failed
attempts      int, max 5
lease_until   timestamptz
worker_id     text
dedupe_key    unique
schedule      cron expr or null

Architecture

The architecture has four parts. An API to submit, cancel and inspect jobs. A scheduler that, every second or so, finds jobs whose run time has passed, marks them queued and pushes them onto a ready queue. The queue buffers work, often one per priority. And a pool of workers pulls jobs, runs them and reports back, autoscaled on queue depth. Separating 'when' from 'who runs it' lets each scale on its own.

Four parts
  • API: submit, cancel, query
  • scheduler: every second,
  • find due jobs, mark them
  • queued, push to the queue
  • queue: buffers ready work,
  • one per priority
  • workers: pull, run, report;
  • scale on queue depth

One scheduler acts

Run three scheduler instances for availability, but only one should act at a time, or due jobs get enqueued three times. Use leader election from Concepts episode thirty-one: the active scheduler holds a lease in etcd or a database row and renews it every few seconds. If it dies, the lease expires and another takes over within about fifteen seconds. Still make enqueueing idempotent, checking the status as you update it, because two leaders can briefly overlap.

Avoiding double scheduling
  • run 3 scheduler instances
  • only the lease holder scans
  • (etcd lease, or a DB row
  • with an expiry)
  • leader dies → lease expires
  • → another takes over in ~15 s
  • still: make the enqueue step
  • idempotent (status check)
  • fencing token on writes

Claiming

At moderate scale you can skip the separate queue and let workers claim jobs straight from the table. The key is SELECT FOR UPDATE SKIP LOCKED in Postgres or MySQL. Each worker locks up to ten due rows, skipping rows another worker has already locked, then marks them running with a sixty-second lease. No two workers ever get the same job, and nobody waits on anybody else's lock.

Claiming work atomically
UPDATE jobs SET status='running',
  worker_id=$me,
  lease_until=now()+'60 s'
WHERE id IN (
  SELECT id FROM jobs
  WHERE status='queued'
    AND run_at <= now()
  ORDER BY priority, run_at
  LIMIT 10
  FOR UPDATE SKIP LOCKED)
RETURNING *;

Dead workers

Workers die mid-job. While a job runs, its worker extends the lease every twenty seconds or so. If worker two crashes, the heartbeats stop, the lease expires, and a reaper puts the job back to queued for someone else. The subtle case: a worker that was just paused, not dead, may wake up and finish too. So the job ran twice, and handlers must be idempotent, for example by checking whether the reminder was already sent.

Leases and heartbeats
  • worker extends lease_until
  • every ~20 s while running
  • worker crashes → lease
  • expires → reaper sets job
  • back to queued
  • slow, not dead? it may still
  • finish → job ran twice
  • → handlers must be idempotent

Retries

Retries are just rescheduling. On a failure, increment attempts and set the next run time with growing delays and some jitter: thirty seconds, two minutes, ten minutes, an hour. Distinguish errors: a 503 from a provider is worth retrying, an invalid payload is not. After the last attempt, mark the job failed, alert, and give operators a retry button. And give every job type a timeout, so a hung job doesn't hold a worker forever.

Retry policy
  • on error: attempts += 1,
  • run_at = now + backoff
  • 30 s, 2 m, 10 m, 1 h, 6 h
  • (+ jitter)
  • retryable vs not: 503 yes,
  • invalid payload no
  • max attempts → status failed,
  • alert, manual retry button
  • timeout per job type

Recurring jobs

Recurring jobs store a cron expression with a time zone. When an occurrence is scheduled, also create the next one, with a dedupe key of job plus scheduled time so it can never be created twice. After an outage, decide per job what to do with missed runs: catch up every one, run once, or skip. A billing job should catch up; a cache warmer should skip. And watch daylight saving time: two thirty in the morning may not exist, or may happen twice.

Recurring jobs
  • store cron expr + timezone
  • on run: insert the NEXT
  • occurrence (dedupe key =
  • job + scheduled time)
  • missed runs (outage):
  • catch up all, run once, or
  • skip: choose per job
  • DST: 2:30 a.m. may not exist
  • or happen twice

Priorities

Not all jobs are equal. A password reset email must not wait behind a million bulk exports. Use separate queues for critical, default and bulk work, with dedicated workers for critical. Poll with weights, so bulk work is slow but never starves completely. And cap concurrency per tenant, so one customer exporting a million reports can't monopolise every worker.

Priorities and fairness
  • separate queues: critical,
  • default, bulk; dedicated
  • worker pools for critical
  • weighted polling so bulk
  • is slow but never starves
  • per-tenant concurrency cap:
  • one customer's 1M exports
  • cannot block everyone
  • rate limit calls to APIs

Scaling

When one scheduler can't scan fast enough, partition the jobs into, say, sixty-four buckets by hashing the id, and let each scheduler instance lease a set of buckets, so they divide the work without overlap. For jobs far in the future, store them in time buckets, or use a Redis sorted set keyed by run time, where fetching what's due is a cheap range query. Workers stay stateless and scale on queue depth.

Scaling past one scheduler
  • partition jobs by hash(id)
  • into 64 buckets
  • each scheduler instance
  • leases a set of buckets
  • far-future jobs: time-bucket
  • them (hour tables, or Redis
  • sorted set by run_at)
  • workers: stateless, scale on
  • queue depth and job lag

Monitoring

What would you monitor? The headline is schedule lag: how long after its run time a job actually starts, at p ninety-nine. Then queue depth per priority, failure rate per job type, retry and dead-letter counts, and jobs stuck running past their timeout. Add a dead man's switch: an alert that fires when an important job did not report success, because silence is how scheduled jobs usually fail.

What to watch
  • schedule lag: start time −
  • run_at (p99 under 5 s?)
  • queue depth per priority
  • failure rate per job type
  • retries and dead-letter count
  • stuck jobs: running past
  • their timeout
  • "dead man's switch": alert
  • if the nightly job did NOT run

Sizing

Sizing. Fifty million jobs a day averages about five hundred eighty a second, but schedules cluster, so midnight might see twenty times that, twelve thousand jobs a second. If a job takes two seconds, that's twenty-four thousand running at once, around four hundred eighty workers with fifty slots each. Or add a few minutes of jitter to scheduled times and flatten the spike. Job rows add fifty gigabytes a day, so archive finished jobs.

Sizing
50M jobs/day → avg ~580/s,
  peak (midnight cron) 20x
  ≈ 12,000 due per second

job takes 2 s on average
  → 12,000 × 2 = 24,000
  concurrent at peak
  50 slots per worker →
  ~480 workers (or smooth
  the spike with jitter)
rows: 50M × 1 KB = 50 GB/day

Say it like this

In the interview: jobs are durable rows with a run time, status, lease, attempts and a dedupe key. One active scheduler per partition, chosen by a lease, moves due jobs to queues. Workers claim jobs with SKIP LOCKED or from the queue, heartbeat to keep their lease, and lose it if they die. Delivery is at least once, so handlers are idempotent. Retries back off into a failed state. Priorities and tenant caps keep it fair, and schedule lag is the metric.

In the interview
  • jobs table: run_at, status,
  • lease, attempts, dedupe key
  • scheduler: leader lease,
  • partitioned buckets
  • claim: SKIP LOCKED or queue
  • leases + heartbeats → reclaim
  • at-least-once → idempotent
  • retries w/ backoff → failed
  • priorities + tenant caps
  • watch: schedule lag

Interview questions

Why is running cron on a single server not enough?

It is a single point of failure, so jobs silently stop if the box dies; running it on several boxes runs every job several times; and it offers no retries, history or scaling beyond one machine.

How do you ensure a scheduled job is not triggered twice by multiple schedulers?

Have only one active scheduler per partition, chosen by a renewable lease in etcd, ZooKeeper or a database row, and make the enqueue step idempotent with a status check or dedupe key in case two leaders briefly overlap.

How do multiple workers pick jobs without conflicts?

Claim jobs atomically, for example with SELECT FOR UPDATE SKIP LOCKED to lock a batch of due rows that no other worker holds and mark them running with a lease, or pull from a queue that hands each message to one consumer.

What happens if a worker crashes mid-job?

Its lease stops being renewed by heartbeats and expires, and a reaper returns the job to the queued state so another worker picks it up; because a slow worker might still finish, handlers must be idempotent.

How do you implement retries for failed jobs?

On a retryable error increment attempts and reschedule with exponential backoff plus jitter; after the maximum attempts mark the job failed, send it to a dead-letter list and alert, and do not retry permanent errors like invalid input.

How do recurring jobs work, and what about missed runs?

Store a cron expression and time zone and create each next occurrence with a dedupe key of job plus scheduled time; after downtime apply a per-job policy to either catch up all missed runs, run once, or skip.

How do you stop one tenant from starving everyone else?

Use separate queues by priority with dedicated workers for critical jobs, weighted polling so low priority still progresses, and per-tenant concurrency limits.

What is the most important metric for a job scheduler?

Schedule lag, the delay between a job's scheduled time and when it actually starts, along with queue depth, failure rates and alerts for important jobs that did not run at all.