Design a Distributed Job Scheduler: Exactly-Once Is a Lie, and Other Things to Say Out Loud

“Design a job scheduler” sounds like a question about cron. It is really a question about distributed coordination: how do you make sure a job runs when it should, runs once when you only want it once, still runs when the machine that was supposed to run it dies, and does not run twice when two machines both think it is their turn. Candidates who treat it as “a table of jobs and a loop” get taken apart by the third follow-up.

By the end of this post you will have a structure that puts delivery semantics, leasing, and failure recovery at the center, where the interviewer is looking, and the numbers and vocabulary to make each choice sound deliberate.

The Question

The direct forms:

• “Design a distributed job scheduler.”
• “Design a cron service for the company.” (Google’s version of this question is a whole SRE book chapter.)
• “Design a system that runs a task at a given time, or on a recurring schedule.”
• “Design a delayed job queue.” (the one-off variant)

The cousins: “design a workflow orchestrator like Airflow” adds dependencies between jobs and is a bigger question. “Design a task queue like Sidekiq or Celery” is the same question without the time dimension. If the interviewer does not say which, ask. The scope changes the design.

What They Are Really Asking

1. Do you understand delivery guarantees? The interviewer is waiting for you to say “at-least-once with idempotent jobs” unprompted. A candidate who promises exactly-once is about to have a bad ten minutes.
2. Can you coordinate without a single point of failure? One scheduler process is easy to reason about and dies at the worst time. Several schedulers double-fire unless something stops them. The interviewer wants to hear leases, claims, or partition ownership.
3. What happens when things die mid-flight? A worker takes a job and disappears. A scheduler goes down for twenty minutes and comes back. These are the follow-ups, and a strong answer preempts them.
4. Can you find what is due efficiently? “Scan the whole table every second” works for a thousand jobs and dies at a million. The interviewer wants to see you know the options.
5. Do you separate the scheduler from the executor? Deciding when something runs and actually running it are different concerns with different scaling. Blending them into one box is a common tell.

A weak answer is a jobs table, a loop that polls it, and workers that run whatever they are handed. A strong answer is the same sketch with an explicit claim step, a lease with heartbeats, an idempotency key on every execution, a retry policy with a dead-letter queue, and a stated answer for scheduler downtime.

The Gotchas

Gotcha 1: Promising exactly-once. A worker runs the job, makes its side effect, and crashes before acknowledging. The scheduler cannot tell the difference between “ran and crashed” and “never ran.” It will retry. That is the whole story, and no amount of engineering removes it. The answer is at-least-once delivery plus idempotent jobs, with an idempotency key made from the job ID and the scheduled time. Say this early and you have answered the hardest question in the interview.

Gotcha 2: One scheduler, or several with no coordination. A single scheduler is a single point of failure. Two schedulers scanning the same table both find the same due job and both enqueue it. You need one of: leader election so only one scans, partitioning so each scheduler owns a slice of jobs, or an atomic claim in the store so only one scan wins each job. Name the one you are using.

Gotcha 3: Polling without a plan. SELECT * FROM jobs WHERE next_run_at <= now() across a million rows every second is a full scan. You need an index on the due time and a way to claim a batch atomically. In Postgres that is FOR UPDATE SKIP LOCKED. Alternatively, keep due times in a sorted structure built for this. Know at least two options.

Gotcha 4: No misfire policy. The scheduler was down from 02:00 to 02:20. A job scheduled every minute missed twenty runs. Do you run all twenty, run one, or skip them? There is no universally right answer, which means it must be per-job configuration, and you should say so.

Gotcha 5: Leases that expire while the job is still running. A worker takes a five-minute lease and the job takes eight minutes. At minute five another worker claims it and now the job is running twice. The fix is heartbeats that extend the lease while the worker is alive, plus a fencing token so a stale worker’s writes are rejected if it comes back from a pause.

Gotcha 6: Everything at midnight. Real schedules cluster on the hour and at 00:00. A design that handles the average rate and not the top-of-the-hour spike will fall over in production. Estimate the peak, and mention jitter for jobs that do not need the exact minute.

Gotcha 7: Time zones and daylight saving. A “daily at 02:30” job in a zone that skips 02:30 once a year either does not run or runs twice, depending on your library. Store schedules with an explicit time zone, compute the next run in that zone, and store the result in UTC. Say this in one sentence and move on; it shows you have been burned.

Gotcha 8: Conflating scheduling, execution, and orchestration. The scheduler decides when. The executor runs the code. The orchestrator handles dependencies between jobs. Scope the question to the first two unless the interviewer asks for the third, and say that is what you are doing.

Gotcha 9: No history. “Why did my job not run at 09:00?” is the first support ticket every scheduler receives. Execution history with status, attempts, worker, and timing is a requirement, not a nice-to-have.

How to Answer

Step 1: Clarify the scope

Question	Why it matters
One-off delayed jobs, recurring cron jobs, or both?	Recurring adds next-run computation and misfire policy
Do jobs depend on each other?	If yes, it is an orchestrator. Scope it out or in, explicitly
What scale? Job definitions, executions per day, peak rate?	Decides polling strategy and partitioning
What precision? To the second, or within a minute is fine?	Second-level precision rules out lazy polling intervals
How long do jobs run? Seconds or hours?	Long jobs need heartbeats; short ones can use a fixed lease
Who runs the code? Our workers, or do we call a webhook?	Changes the execution side entirely
Is at-least-once acceptable?	It has to be. Confirm it so you can say why
Multi-tenant?	Adds fairness and per-tenant limits

Defaults if the interviewer says “you choose”: both one-off and recurring, no dependencies, 1 million job definitions, 10 million executions per day, precision within a few seconds, jobs run from seconds to an hour, our own worker fleet, at-least-once, multi-tenant.

Step 2: Do the math

Quantity	Estimate
Average execution rate	10M / 86,400 ≈ 116 per second
Peak rate	Top of the hour clusters. Plan for 10x → 1,200 per second, and say midnight may be worse
Job definitions	1M × ~1 KB ≈ 1 GB. One database
Execution history	10M / day × ~500 B ≈ 5 GB per day. 30 days ≈ 150 GB. Lives in a separate store with a TTL, not the hot jobs table
Workers	If the average job takes 10 seconds, 116/s × 10 s ≈ 1,160 concurrent jobs. Size the fleet for the peak, or let the queue absorb it

The conclusions to say out loud: the job definitions fit anywhere; the execution history is the storage problem; the peak rate is the scheduling problem.

Step 3: Define the data model

jobs (
  id              uuid primary key,
  tenant_id       uuid,
  schedule        text,          -- cron expression, or null for one-off
  timezone        text,          -- IANA zone, e.g. 'America/Chicago'
  run_at          timestamptz,   -- for one-off jobs
  next_run_at     timestamptz,   -- computed, indexed
  payload         jsonb,
  max_attempts    int,
  backoff         text,          -- e.g. 'exponential:30s:1h'
  misfire_policy  text,          -- 'skip' | 'fire_once' | 'fire_all'
  overlap_policy  text,          -- 'skip' | 'queue' | 'allow'
  status          text,          -- 'active' | 'paused'
  version         int            -- bumps on edit; stale executions check it
)
create index on jobs (next_run_at) where status = 'active';

executions (
  id              uuid primary key,
  job_id          uuid,
  job_version     int,
  scheduled_for   timestamptz,   -- (job_id, scheduled_for) is the idempotency key
  attempt         int,
  status          text,          -- 'queued' | 'running' | 'succeeded' | 'failed' | 'dead'
  worker_id       text,
  lease_until     timestamptz,
  started_at      timestamptz,
  finished_at     timestamptz,
  error           text,
  unique (job_id, scheduled_for, attempt)
)

Two things to point at. The partial index on next_run_at is what makes “find what is due” cheap. The unique constraint on (job_id, scheduled_for, attempt) is what makes a double-enqueue harmless: the second insert fails.

Step 4: Draw the architecture

flowchart LR
    U[API: create / edit / pause jobs] --> DB[(Jobs DB)]
    S1[Scheduler shard 1] & S2[Scheduler shard 2] -->|claim due jobs<br/>SKIP LOCKED| DB
    S1 & S2 -->|enqueue execution| Q[[Queue]]
    Q --> W1[Worker] & W2[Worker] & W3[Worker]
    W1 & W2 & W3 -->|lease + heartbeat<br/>ack / fail| DB
    W1 & W2 & W3 -.->|after max attempts| DLQ[[Dead-letter queue]]
    DB -->|history, TTL| H[(Execution history)]
    CO[(Coordination store:<br/>shard ownership leases)] -.-> S1 & S2

Talk through the separation: the API writes job definitions. Schedulers find due jobs and turn them into executions. The queue decouples scheduling rate from execution rate. Workers run jobs under a lease. History goes to its own store.

Step 5: Find what is due, and claim it atomically

Compare the options before picking one.

Approach	How it works	Pros	Cons
Database polling with SKIP LOCKED	Each scheduler runs SELECT ... WHERE next_run_at <= now() FOR UPDATE SKIP LOCKED LIMIT 100, marks them claimed, enqueues	Atomic claims for free, no extra infrastructure, easy to reason about	Polling interval bounds precision; hot index contention at very high rates
Sorted set in a cache (e.g. Redis ZRANGEBYSCORE)	Due time is the score; pop a range atomically with a script	Very fast; natural for one-off delayed jobs	Durability depends on the cache config; recurring schedules still live elsewhere
In-memory timing wheel per scheduler shard	Load the next N minutes of jobs into a hierarchical timer	Sub-second precision, minimal store load	Must reload on restart; needs partition ownership so shards do not overlap
Delayed messages in a broker	Publish with a delay header	Simple for one-off jobs	Most brokers do it badly or with coarse granularity; no good for cron

A defensible recommendation at the stated scale: database polling with SKIP LOCKED every second, with scheduler shards that each own a hash range of job IDs. It handles 1,200 per second comfortably, it is one system instead of two, and the claim is atomic by construction. Say what would change your mind: at ten times the rate, or with a sub-second precision requirement, move the due index to a sorted set or a timing wheel, keeping the database as the source of truth.

The claim and the next-run computation must be one transaction. Claim the row, insert the execution, compute and write the new next_run_at, commit. If the scheduler dies between the claim and the enqueue, the row is still claimed but no execution exists. Handle that with a reaper that finds claimed-but-not-enqueued rows older than a few seconds and re-releases them. Mention the reaper; it is the detail that shows you have run one of these.

Step 6: Leases, heartbeats, and fencing on the worker side

sequenceDiagram
    participant Q as Queue
    participant W as Worker
    participant D as Jobs DB
    participant T as Target system
    Q->>W: execution {job_id, scheduled_for, attempt}
    W->>D: set status=running, worker_id, lease_until=now+60s, fence=N
    loop every 20s while running
        W->>D: extend lease_until (only if fence == N)
    end
    W->>T: do the work (idempotency key = job_id + scheduled_for)
    T-->>W: ok
    W->>D: status=succeeded (only if fence == N)
    Note over D: Reaper re-queues any execution<br/>whose lease_until < now and status=running

The points to say:

• The lease is short and renewed, not long and fixed. A 60-second lease renewed every 20 seconds means a dead worker’s job is picked up within a minute, and a live worker running for an hour keeps it.
• The reaper re-queues expired leases. That is how a dead worker’s job gets retried without anyone noticing the worker died.
• The fencing token is the execution’s lease generation. If a worker was paused (garbage collection, network partition), its lease expired, and another worker took over, the first worker’s late writes carry a stale token and are rejected. Without this, the retry and the original can both “succeed.”
• The idempotency key goes to the target. If the job sends an email or charges a card, the downstream must dedupe on (job_id, scheduled_for). The scheduler can only guarantee at-least-once; the job makes it effectively once.

Step 7: Retries, backoff, misfires, and overlap

Policy	Options	Default and why
Retry	Max attempts, exponential backoff with jitter, cap	5 attempts, 30 s doubling to 1 h, with jitter so a failing dependency does not get hit by every retry at once
Dead letter	After max attempts, move to a dead-letter queue and alert	Never silently drop. Someone needs to look
Misfire (scheduler was down)	Skip missed runs; fire once now; fire all missed runs	Per job. A report job wants fire once. A billing job may want fire all. A heartbeat wants skip
Overlap (previous run still going when the next is due)	Skip; queue; allow concurrent	Per job. Default skip, because most recurring jobs are not safe to overlap

The interviewer is checking that you know these are policies, not defaults you hard-code.

Step 8: Scale and multi-tenancy

• Scheduler shards. Partition job IDs by hash into N ranges. Each scheduler instance leases one or more ranges in a coordination store (etcd, ZooKeeper, or a table with leases). If an instance dies, its ranges are re-leased within seconds. This avoids a single leader and avoids double scanning.
• Workers autoscale on queue depth. The queue is the buffer that absorbs the top-of-hour spike.
• Priority. Separate queues per priority, workers drain high first. Guard against starvation with a minimum share for low priority.
• Tenant fairness. Per-tenant concurrency caps so one tenant with a million jobs at midnight does not delay everyone else’s. A token bucket per tenant on the enqueue side works. If you wrote a rate limiter answer, this is where you reuse it.
• History. Write executions to a time-series or wide-column store with a TTL. Keep the hot table small.

Step 9: Say what you would buy

An architect interview rewards knowing when not to build. Name the options: managed schedulers from the cloud providers for cron-with-webhook, Temporal or similar for durable workflows with dependencies and long-running state, and the language-ecosystem task queues (Sidekiq, Celery, and their peers) for single-application background jobs. Then say why you would still build: multi-tenant isolation, custom misfire semantics, or scale beyond what the managed option handles. Interviewers trust a candidate who can explain what the managed offering does not give them.

Follow-Up Questions to Expect

• “We really need exactly-once.” Explain why the scheduler cannot provide it, then show how idempotency at the target achieves the same observable result. If they push, ask what side effect they are worried about and design the dedupe for it.
• “A job takes longer than its interval.” Overlap policy. Default skip. Mention that the skipped run should be recorded in history so it is visible.
• “What if 100,000 jobs are due at midnight?” The queue absorbs it, workers autoscale, and jobs that do not need exactly midnight get jitter at definition time. Also mention per-tenant caps.
• “A scheduler shard dies mid-scan.” Its claimed rows are re-released by the reaper, its shard range is re-leased by another instance, nothing is lost and nothing double-fires because claims were atomic.
• “Someone edits a job’s schedule while a run is queued.” The execution carries the job version. Workers check the version on start and skip if it is stale, or run it if the policy says in-flight executions complete. Say which and why.
• “How do you add dependencies between jobs?” Now it is an orchestrator. Sketch a DAG (Directed Acyclic Graph) where a job’s completion event enqueues its dependents, and say that is a different post.
• “How do you test it?” Property tests on next-run computation across DST transitions, a chaos test that kills workers mid-job and confirms exactly one success at the target, and a load test at the midnight spike.

Key Takeaways

• Say “at-least-once with idempotent jobs” in the first five minutes. It is the answer to the hardest question.
• Separate the scheduler (when), the executor (what), and the orchestrator (dependencies). Scope explicitly.
• Find due jobs with an index and claim them atomically. SKIP LOCKED is the simplest correct answer at moderate scale.
• Leases are short and renewed by heartbeat. A reaper re-queues expired ones. A fencing token rejects stale writers.
• Misfire and overlap are per-job policies, not global defaults.
• Estimate the top-of-hour peak, not the average, and let a queue absorb it.
• Execution history is a requirement. It answers the first support ticket.
• Know what you would buy and why you would still build.

Further Reading

• Google SRE Book, Distributed Periodic Scheduling with Cron
• Martin Kleppmann, How to do distributed locking (the fencing token argument)
• PostgreSQL docs on SELECT ... FOR UPDATE SKIP LOCKED
• Quartz Scheduler documentation on misfire instructions
• Temporal, What is a durable execution
• The foundation posts this design leans on: first five minutes, numbers, idempotency for the execution-level dedupe, isolation for SKIP LOCKED, and sagas and the outbox for what a job that spans services needs