Start practising

SYSTEM DESIGN · PATTERNS · 7 OF 7

Long-Running Tasks: Job Queues, Workers, Leases and Status Updates

What to do when work takes seconds to hours, like transcoding a video, generating a report or judging code. Why doing it inside the request fails, then the standard moves: an async job API that returns a job ID, a queue and worker pool, leases and heartbeats, retries with a dead-letter queue, status updates, autoscaling on queue depth, fairness, and cancellation.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
1 / 16

Press play to hear the narration, or step through at your own pace.

Notes

The signals

Listen for work that takes more than a few seconds: transcoding, PDF exports, code judging, CI builds, bulk imports, heavy model inference. Load balancers and proxies typically cut idle connections after about sixty seconds, and phones drop connections far sooner. When a task can outlive the request that started it, this is your pattern.

Signals in the question
  • video or audio transcoding
  • report, export, invoice PDFs
  • code judging, CI builds
  • ML inference on big inputs
  • bulk email, data imports
  • takes 5 s to hours
  • load balancer idle timeout:
  • often 60 s
  • users can wait, but not
  • with a spinner

The naive design

The naive design does the work inside the request handler. At sixty seconds the load balancer returns a timeout, though the server may still be working. The user hits retry, and now the job runs twice. A deploy or crash loses the job silently. And heavy jobs eat the CPU that fast API requests need, so the whole site slows down. You cannot scale the two kinds of work separately.

Why it breaks
  • request times out at 60 s,
  • the work may continue
  • user retries → the job runs
  • twice
  • deploy or crash → job lost,
  • nobody knows
  • heavy jobs starve the fast
  • API requests on the box
  • cannot scale the two apart

Move 1: async job API

Move one: change the API. The client posts the job, the API records it with status queued and returns two hundred two Accepted with a job ID, in milliseconds. A separate endpoint reports status and progress, and another cancels. Send an idempotency key on submit, so a retried submit returns the same job instead of creating a second one.

The job API
POST /exports
  Idempotency-Key: abc
→ 202 Accepted
  { job_id: 77, status: queued }

GET /jobs/77
→ { status: running,
    progress: 40 }

DELETE /jobs/77  (cancel)

Move 2: queue + workers

Move two: a queue and a worker pool. The API writes the job row and puts the job ID on a queue. Workers on a separate fleet, possibly with big CPUs or GPUs, pull jobs, run them, write results and update the status. Now the API stays small and fast, and you scale workers independently. Use an outbox so the job row and the queue message cannot disagree. Concepts episode twenty-eight covers queues.

Queue and worker pool
  • API: write job row, enqueue
  • the id (or use an outbox)
  • workers: pull, run, write
  • results, update status
  • separate fleet: big CPU or GPU
  • boxes, scaled on their own
  • API stays fast and small
  • see the Job scheduler block
  • for timed jobs

Leases

The key mechanism is the lease. When a worker pulls a job, the queue hides it for, say, five minutes rather than deleting it. The worker acknowledges when done, and only then is it removed. If the worker crashes, the lease expires and the job reappears for someone else. That gives at-least-once execution, so jobs must be idempotent: write output to a fixed key, and use upserts.

Leases and acks
  • pull = lease, the job is hidden
  • (SQS visibility timeout)
  • worker acks when finished
  • → deleted
  • worker dies → lease expires
  • → job reappears for another
  • = at-least-once execution
  • so jobs must be idempotent:
  • same output key, upserts

Heartbeats

What about a job that runs for an hour? The worker sends a heartbeat every thirty seconds that extends its lease and records progress. If heartbeats stop, the job is presumed dead and re-queued. For big jobs, checkpoint: if segments one to forty of a video are already transcoded and saved, a restart begins at forty-one. Even better, split the job into many small ones, one per segment, and run them in parallel.

Jobs longer than the lease
  • heartbeat every 30 s:
  • extend lease, save progress
  • no heartbeat for 2 min
  • → presume dead, re-queue
  • checkpoint big jobs:
  • transcoded segments 1-40
  • saved → restart at 41
  • or split into many small
  • jobs (one per segment)

Retries

Retries. Transient failures, like a timeout or a busy dependency, get retried with exponential backoff and jitter. Permanent failures, like a corrupt input file, should fail fast and tell the user. After about five attempts, move the job to a dead-letter queue so one poison job cannot loop forever and block a worker. Alert when the dead-letter queue grows.

Retries and dead letters
attempt 1 fails → retry in 10 s
attempt 2 fails → retry in 40 s
attempt 3 fails → retry in 160 s
attempt 5 fails → dead-letter

transient (timeout, 503):
  retry with jitter
permanent (bad input):
  fail fast, tell the user
alert on dead-letter depth

Telling the user

How does the user find out? The simplest answer is polling the status endpoint every few seconds, which is perfectly fine for most products. For a snappier experience, push the update over SSE or a WebSocket, see the real-time updates pattern. API customers get a signed, retried webhook. For hour-long jobs, send an email. The result itself is usually a file in object storage behind a presigned link.

Getting the result back
  • poll GET /jobs/77 every 2 to
  • 5 s (simple, fine for most)
  • push over SSE or WebSocket
  • (see Real-time updates)
  • webhook for API customers:
  • POST to their URL, signed,
  • retried
  • email for jobs of hours
  • result: presigned link to S3

Scaling workers

How many workers do you need? Use Little's law: concurrent workers equals jobs arriving per second times seconds per job. Fifty jobs a second at thirty seconds each means fifteen hundred jobs running at once. Autoscale on queue depth, or better, on the age of the oldest waiting job, which maps directly to what users feel. Keep separate queues for quick and slow jobs, so one long job type cannot block the short ones.

How many workers?
Little’s law:
busy workers = arrivals/s
              × seconds per job

50 jobs/s × 30 s = 1,500
  concurrent workers

autoscale on queue depth or
  age of the oldest job
separate queues: fast vs slow,
  paid vs free

Fairness

Fairness. If one customer submits a hundred thousand jobs into a single first-in, first-out queue, everyone behind them waits hours. Give tenants their own queues or quotas, schedule across them fairly, cap how many jobs one tenant can run at once, and keep a priority lane so interactive jobs jump ahead of overnight batches. Rate limit submissions at the API.

Fairness and limits
  • one customer submits 100k
  • jobs: everyone else waits
  • per-tenant queues or quotas
  • weighted fair scheduling
  • priority lanes: interactive
  • above batch
  • cap concurrency per tenant
  • rate limit submissions
  • (see the Rate limiter block)

Cancellation

Give every job a maximum runtime, for example twice its normal ninety-ninth percentile, and kill it beyond that. Cancellation is cooperative: the API marks the job as cancelling, and the worker checks that flag at each checkpoint and stops at the next safe point, cleaning up partial outputs. Always record the final status and a reason, so support can answer what happened to my export.

Timeouts and cancellation
  • every job has a max runtime
  • (kill at 2x the p99)
  • cancel: set status=cancelling
  • worker checks the flag at each
  • checkpoint and stops
  • already running on a GPU? stop
  • at the next safe point
  • clean up partial outputs
  • record final status + reason

Trade-offs

The trade-offs. Going async buys reliability and independent scaling, at the cost of more moving parts and a user experience that has to show progress honestly. At-least-once delivery forces idempotent jobs. Polling is simple but wasteful; push is instant but stateful. Splitting into small jobs adds parallelism and resumability, but also scheduling overhead. Monitor queue depth, oldest job age, failure rate and dead letters.

Trade-offs
  • async: robust and scalable,
  • but more moving parts
  • and a harder UX
  • at-least-once: jobs must be
  • idempotent
  • polling: simple, wasteful
  • push: instant, stateful
  • many small jobs: parallel and
  • resumable, more overhead
  • monitor: depth, age, failures

Where it shows up

Where you will use it. The online code judge runs submissions in sandboxed workers and reports a verdict. YouTube's upload pipeline transcodes in the background. The job scheduler adds timing on top of this pattern. CI pipelines run builds as jobs. The notification system fans out bulk sends, and a web crawler is a giant queue of fetch jobs.

Problems that use it
  • Online code judge: run code
  • in sandboxed workers
  • YouTube: transcode pipeline
  • Job scheduler: timed and
  • recurring jobs
  • CI/CD pipeline: builds
  • Notification system: bulk
  • sends
  • Web crawler: fetch queue

Say it like this

The phrasing. Submitting returns two hundred two with a job ID. The API stores the job and enqueues it. Workers lease jobs, send heartbeats and checkpoint, and jobs are idempotent because delivery is at least once. Failures back off and land in a dead-letter queue. The client polls or gets a push when it is done, and workers autoscale on the age of the oldest job.

In the interview
  • "POST returns 202 with a job
  • id. The API writes the job
  • and enqueues it. Workers
  • lease jobs, heartbeat, and
  • checkpoint; failed jobs back
  • off and land in a DLQ. The
  • client polls or gets a push.
  • Workers autoscale on the
  • oldest job age."

Interview questions

How would you design an API for a task that takes ten minutes?

Make submission asynchronous: POST creates a job and returns 202 Accepted with a job ID, a GET endpoint returns status and progress, and completion is signalled by polling, a push over SSE or WebSocket, or a webhook, with the result stored in object storage.

What happens if a worker crashes halfway through a job?

The job was leased rather than deleted, so when the visibility timeout or lease expires without an acknowledgement it becomes available again and another worker picks it up; checkpoints let it resume rather than restart.

Why must jobs be idempotent?

Queues deliver at least once, so a job may run again after a crash, a lease expiry or a retry. Idempotent jobs, for example writing output to a deterministic key with upserts, produce the same result no matter how many times they run.

How do you handle a job that keeps failing?

Retry transient failures with exponential backoff and jitter up to a limit, fail permanent errors immediately, and move jobs that exhaust their retries to a dead-letter queue with an alert so they do not block workers.

How do you decide how many workers to run?

Use Little’s law: concurrent jobs equal arrival rate times average duration, then autoscale on queue depth or the age of the oldest waiting job, with separate pools for job types with very different durations.

How do you keep one customer from monopolising the workers?

Use per-tenant queues or quotas with fair scheduling across tenants, cap each tenant’s concurrent jobs, rate limit submissions, and keep a higher-priority lane for interactive work.

How do you support cancelling a running job?

Mark the job as cancelling in the jobs table; workers check that flag at each checkpoint or heartbeat, stop at a safe point, clean up partial outputs, and record a final cancelled status.