Post

System Shapes: Coordination (Schedulers, Workflow Engines, Locks)

Coordination systems move little data, but every operation must happen once in effect while processes die and clocks lie. Here is the shape, the arithmetic of leases and heartbeats, and a deep dive on schedulers, workflow engines, and distributed locks.

System Shapes: Coordination (Schedulers, Workflow Engines, Locks)

Some systems are hard because of volume. Coordination systems are hard because of disagreement. A scheduler fires a few hundred jobs a second; a lock service grants a few thousand locks a second. Neither number would trouble a single database. What makes them hard is that every one of those operations has to happen once in effect, while the machines doing them pause, crash, lose the network, and disagree about what time it is.

This post is the deep dive on the coordination shape from the first five minutes post. It covers what makes a system coordination-shaped, the generic architecture and the arithmetic that matters (lease lengths and heartbeat load, not requests per second), and then the three standard examples one at a time: the scheduler, where time is the trigger; the workflow engine, where state lives for days; and the lock, where the guarantee itself is the product. The scheduler already has its own full design post, so it gets a shape-level summary here, and the depth goes to workflow engines and locks.

Where This Shows Up

The direct prompts:

  • “Design a distributed job scheduler” or “design cron for the company.”
  • “Design a workflow engine” or “design something like Airflow” or “design an order fulfilment pipeline that runs for days.”
  • “Design a distributed lock service.”
  • “How would you do leader election?”

And the follow-ups that turn another design into a coordination question:

  • “Two instances of this worker both pick up the same message. What happens?”
  • “This step waits three days for a human to approve. Where does that state live?”
  • “You deploy new code while ten thousand workflows are halfway through. What breaks?”

What They Are Really Checking

  1. Do you know that exactly-once is not available, and what to say instead? At-least-once delivery plus idempotent effects, or a claim protected by a fencing token. The interviewer is waiting for one of those sentences.
  2. Do you understand that a lease is a guess about time? A process cannot know its lease has expired if it was frozen when it expired. Candidates who treat “I hold the lock” as a fact rather than a belief are about to be walked into a split brain.
  3. Can you say where the guarantee is enforced? Not “the lock service guarantees it,” but “the storage layer rejects writes with a stale token.” The enforcement point is the design.
  4. Do you have a failure story for every actor? The worker dies, the coordinator dies, the network splits, the clock jumps. Each needs a sentence about who notices, how long it takes, and what prevents double work.
  5. Do you prefer avoiding coordination to doing it well? The strongest answer often removes the lock: partition ownership, a conditional write, a unique constraint. Interviewers at architect level reward that instinct.

The Gotchas

Gotcha 1: Promising exactly-once. A worker performs a side effect and crashes before recording that it did. Nobody can distinguish “did it and died” from “never did it,” so the work will be retried. Exactly-once delivery does not exist; exactly-once effect does, and it comes from idempotency at the target (keyed on something like job ID plus scheduled time, or workflow ID plus step ID). Say this first and the rest of the interview gets easier.

Gotcha 2: Treating a lease as a lock. A lease is a lock with an expiry, granted so that a dead holder does not block everyone forever. The price is that a live but paused holder (a garbage collection pause, a virtual machine migration, a swapped-out process) can wake up after its lease has been given to someone else and carry on as if it still owns the resource. Every lease design must answer “what happens when the old holder comes back.”

Gotcha 3: Trusting wall clocks. Wall clocks jump (NTP, Network Time Protocol, corrections and misconfigured hosts), and clocks on different machines disagree by milliseconds on a good day and seconds on a bad one. Let one clock decide expiry, measure durations locally with a monotonic clock, and leave a margin for drift.

Gotcha 4: No fencing token at the resource. The fix for Gotcha 2 is a token that increases every time the lease changes hands, passed with every write, and checked by the resource being protected. Without the check at the resource, the token is decoration. If the storage cannot check a token, the lock is an efficiency hint, not a correctness guarantee, and you should say which one you are building.

Gotcha 5: A single Redis key as a correctness lock. SET lock:x token NX PX 30000 is fine for “avoid doing the same expensive work twice.” It is not fine for “two holders corrupts data,” because Redis replication is asynchronous: the primary grants the lock, dies before replicating, and the promoted replica grants it again. Redlock (the multi-node Redis algorithm) is contested precisely because it depends on bounded clock drift and pauses. For correctness, use a consensus store and fencing.

Gotcha 6: Putting the coordination store on the data path. ZooKeeper and etcd hold a few gigabytes of critical metadata, with every write going through consensus. Ten thousand workers heartbeating per-job leases into etcd turns the most fragile component into the busiest one. Keep per-item claims in the database and use the consensus store for membership and shard ownership.

Gotcha 7: Non-deterministic workflow code. Durable execution engines rebuild state by replaying workflow code against its recorded history. If that code reads the clock, generates a random number, iterates an unordered map, or calls an API directly, the replay takes a different path from the original and the engine either fails the workflow or, worse, silently diverges. Side effects belong in activities; workflow code must be deterministic.

Gotcha 8: Deploying over running workflows. A workflow started last Tuesday is replaying against the code you shipped this morning. Reorder two steps and every in-flight workflow fails replay. Versioning running workflows (patch markers, or pinning workflows to the worker version they started on) is a design requirement from day one, not an operational afterthought.

Gotcha 9: Recovery storms. The scheduler comes back after twenty minutes and fires every missed run at once; the lock service recovers and ten thousand clients retry in the same second. Coordination systems fail quietly and recover loudly. Misfire policies, jittered backoff, and rate-limited reaping keep recovery from becoming the second outage.

Gotcha 10: Reaching for a lock first. “Take a distributed lock, then update the row” is usually a worse version of “update the row where the version is what I read.” Locks add a dependency, a failure mode, and a pause-shaped hole in the guarantee. Conditional writes, unique constraints, and single-owner partitions solve most of the same problems with fewer moving parts.

How to Answer

What defines the shape

Property Coordination systems Why it matters
Volume Low: tens to thousands of operations per second Throughput is rarely the bottleneck
Per-operation stakes High: a duplicate charges a card twice, a missed run skips payroll Effort goes into correctness under failure
Access pattern Claim, renew, complete, release Every operation is a conditional write; contention is on ownership
Data size Small: job definitions, workflow histories, lock records Fits in one database
Skew In time, not in keys: everything due at 00:00, everything retrying after an outage Peaks come from the calendar and from recovery
Consistency tolerance None for ownership. Two owners is the bug Ownership decisions need linearizability, per the consistency post
Failure model Processes pause, clocks drift, networks split A participant can be wrong about its own state

The one-line definition: low volume, every operation must happen once in effect, and the core problem is agreeing who owns what when processes die and clocks lie.

The generic architecture

flowchart LR
    API[API: define jobs,<br/>start workflows] --> DB[(State store<br/>claims, leases, history)]
    C1[Coordinator 1] & C2[Coordinator 2] -->|own shards via lease| CS[(Consensus store<br/>etcd / ZooKeeper)]
    C1 & C2 -->|claim due work<br/>conditional update| DB
    C1 & C2 --> Q[[Task queue]]
    Q --> W1[Worker] & W2[Worker]
    W1 & W2 -->|heartbeat, complete<br/>with fencing token| DB
    W1 & W2 -->|side effect +<br/>idempotency key| T[Target systems]
    R[Reaper] -->|requeue expired leases| DB

Four ideas carry the whole shape:

  1. Claim with a conditional update. Ownership changes hands only through an atomic compare-and-set (below), or SELECT ... FOR UPDATE SKIP LOCKED to grab a batch. The database’s atomicity, not application logic, ensures one winner.
  2. Lease, do not lock. Every claim expires unless renewed. A dead owner’s work is recovered by a reaper, not by a human.
  3. Fence every write. The claim returns a monotonically increasing token. Every subsequent write carries it, and the store rejects anything older than the latest.
  4. Make the effect idempotent. Because retries are guaranteed, the side effect carries a deterministic key and the target dedupes, per the idempotency post.

A fenced claim in SQL:

1
2
3
4
5
6
7
8
9
-- claim: returns the new fence, or zero rows if someone else holds a live lease
UPDATE tasks
SET owner = 'worker-17', fence = fence + 1, lease_until = now() + interval '30 seconds'
WHERE id = 42 AND (owner IS NULL OR lease_until < now())
RETURNING fence;

-- every later write is conditional on the fence
UPDATE tasks SET status = 'done'
WHERE id = 42 AND fence = 7;   -- zero rows means: you lost the lease, stop

The now() here is the database’s clock, which is the point: one clock decides expiry, not every worker’s.

The arithmetic that matters

Requests per second is the wrong number for this shape. These are the right ones.

Quantity Example What it tells you
Failover time Lease 30 s, renew every 10 s, reaper scans every 5 s: a dead worker’s task restarts within about 35 s The worst-case delay a crash adds. Shorter leases mean faster recovery
False-expiry risk A 30 s lease survives any pause under 20 s (one missed renewal plus margin). A 5 s lease is lost to one long garbage collection pause Shorter leases mean more double-execution attempts, which fencing must catch
Heartbeat load 10,000 running tasks renewing every 10 s = 1,000 writes per second Fine for Postgres. Too much for a consensus store you also use for elections
Consensus write latency A quorum write is a round trip plus a disk sync: a few milliseconds in one region, 100 ms or more across regions Why the consensus store holds membership and shard leases, not per-task claims
Calendar peak 10M runs per day averages 116 per second; top of the hour can be 10x The peak is a calendar effect; a queue absorbs it

The lease trade-off in one sentence: lease length trades recovery time against the chance of a false expiry, and fencing makes a false expiry safe, so with fencing you can afford short leases. Renewing at one third of the lease length is the common rule: two renewals can fail before the lease is lost.

Standard problems

Problem What goes wrong Usual fix
Duplicate execution Retry after a crash runs the side effect twice At-least-once delivery plus an idempotency key at the target
Two owners A paused holder resumes after its lease was reassigned Fencing token checked by the resource
Lost work Owner dies holding a claim; nobody resumes it Lease expiry plus a reaper that requeues
Clock disagreement Worker thinks its lease is valid; the store thinks it expired One authority for expiry (the store’s clock), monotonic clocks locally
Single coordinator One scheduler or leader dies and everything stops Several coordinators with shard ownership or leader election, all fenced
Split brain A partition leaves two leaders both acting Majority quorum, epoch numbers, the old leader steps down when it cannot renew
Stuck work A task holds a lease forever because it keeps heartbeating but never progresses Heartbeats carry progress; a maximum runtime per attempt

Example 1: the scheduler (time is the trigger)

Special property: nothing external starts the work. The clock does. A queue worker reacts to a message; a scheduler has to notice that a moment has arrived, and that makes time itself the source of most of its problems.

The coordination part is standard: several scheduler instances own shards of the job table through leases, claim due rows with FOR UPDATE SKIP LOCKED, insert an execution with a unique (job_id, scheduled_for) key, and hand it to workers that hold a renewed, fenced lease. The scheduler design post walks through every piece. What the time trigger adds:

Decision forced by “time is the trigger” The question Typical answer
Finding due work How do I know what is due without scanning everything? Partial index on next_run_at; sorted set or timing wheel at higher rates
Misfire policy The scheduler was down 20 minutes. Do missed runs fire? Per job: skip, fire once, or fire all. A report fires once; billing may fire all; a health check skips
Thundering herd at :00 Humans write 0 * * * * and 0 0 * * * Queue absorbs the spike; add jitter at definition time for jobs that do not need the exact minute
Time zones and DST (Daylight Saving Time) “Daily at 02:30” on a day 02:30 does not exist, or happens twice Store the schedule with an IANA zone, compute the next run in that zone, store the result in UTC
Overlap The 10:00 run is still going at 11:00 Per job: skip, queue, or allow; default skip
The idempotency key What makes a duplicate harmless? (job_id, scheduled_for): the scheduled time, not the actual start time, so a retry has the same key

The subtle point to make out loud: the idempotency key includes the scheduled time, which is a fact, not the execution time, which is an accident. A retry at 10:03 of the 10:00 run is the same run.

Example 2: the workflow engine (state that lives for days)

Special property: a single unit of work spans many steps, many processes, and possibly weeks. An order flow charges a card, waits three days for delivery, and sends a review request a week later. No process lives that long, so “where am I in this flow” must be durable, and resuming must be safe after a crash at any point.

There are four broad answers to “where does the state live”:

Approach How state is kept Example Fits
Explicit state machine A status column per instance; each transition is a conditional update plus an outbox event Hand-built saga orchestrator, per the sagas post A few well-known flows that rarely change
Declarative state machine A JSON or YAML definition of states and transitions, interpreted by a managed service AWS Step Functions Flows that are mostly calls to other services with retries and waits
Durable execution Ordinary code; the engine records every result in an event history and replays the code to rebuild state Temporal, Cadence Complex flows with branches, loops, long waits, and frequent change
DAG (Directed Acyclic Graph) scheduler A graph of tasks run per schedule interval; task state in a metadata database Apache Airflow Batch data pipelines, not per-request business processes

The distinction interviewers like: Airflow schedules batch graphs; Step Functions interprets a state machine; Temporal makes ordinary code durable. Use Airflow for “run this pipeline every night for yesterday’s data,” not for “process this order.”

How durable execution works

The engine stores an append-only event history per workflow: activity scheduled, activity completed with result, timer fired, signal received. When a worker picks up a workflow (after a crash, or because a timer woke it), it runs the workflow code from the beginning, and every activity result already in the history is returned from the record instead of re-run. The code fast-forwards to where it left off and continues.

sequenceDiagram
    participant W as Worker (new process)
    participant E as Engine
    participant H as Event history
    E->>W: workflow task for order-123
    W->>H: load history
    H-->>W: Started, ChargeCard completed (ok), Timer(3 days) started, Timer fired
    Note over W: Replay: chargeCard() returns recorded "ok"<br/>sleep(3 days) returns immediately
    W->>E: next command: schedule activity SendReviewRequest
    E->>H: append ActivityScheduled

That mechanism forces the decisions that make up the rest of the design.

Determinism constraints. Replay only works if the code takes the same path every time.

Do not do in workflow code Why it breaks Do instead
Read the system clock Replay sees a different time Use the engine’s workflow time, recorded in history
Generate random numbers or UUIDs (Universally Unique Identifiers) Different value on replay, different branch Use the engine’s deterministic random or generate it in an activity
Call a database or HTTP API Result differs on replay, and the call repeats Put it in an activity; the result is recorded
Iterate an unordered map, or use threads directly Order differs between runs Ordered collections; the engine’s own concurrency primitives
Sleep with the language’s sleep Not recorded; lost on crash The engine’s durable timer

Activities are at-least-once. An activity can complete its side effect and the worker can die before reporting it, so the engine retries it. Every activity therefore needs an idempotency key, and the natural one is the workflow ID plus the activity’s position in the workflow. The engine gives you exactly-once workflow logic; it gives you at-least-once activities. Say both halves.

Durable timers. sleep(3 days) writes a timer event and frees the worker. The engine’s timer service fires it later and the workflow resumes on whichever worker is available. A million sleeping workflows cost a million timer rows, not a million threads. The engine’s timer service is itself a scheduler: sharded, leased, fenced.

Human steps and external signals. A workflow waiting for approval blocks on a signal (Temporal) or a task token callback (Step Functions). Always pair the wait with a timeout and an escalation branch: “if no approval in 48 hours, notify the manager; after 7 days, cancel and compensate.” A wait without a timeout is a workflow that lives forever.

Versioning running workflows. Two strategies. Patch markers: the code branches on “was this workflow started before patch X?”, the marker is recorded in history so replay takes the old branch, and the old branch is deleted once every old workflow finishes. Worker versioning: pin each workflow to the code version it started on and run old and new workers side by side until the old ones drain. Either way, a change that alters the sequence of commands breaks in-flight workflows; changing an activity’s implementation does not.

Unbounded histories. A workflow that loops forever (a subscription billing monthly for years) grows its history without limit, and replay gets slower with every event. Engines cap history size (Temporal warns at about ten thousand events and fails at a hard limit beyond that) and offer “continue as new”: end this run and start a fresh one carrying forward only the current state.

Compensation. A workflow engine is the natural home for a saga orchestrator; the sagas post covers compensation design, and the engine makes the orchestrator crash-proof.

The engine’s own coordination. Temporal shards workflows by ID across history shards, each owned by one host at a time under a lease, and every write to a shard is conditional on a range ID that increases with each ownership change: a fencing token. The generic architecture, applied to the engine itself.

The arithmetic for this example is about state, not throughput: 1 million active workflows averaging 100 history events of 1 KB is about 100 GB of history, almost all of it idle. Most workflows are sleeping. The engine is a storage system with a timer service attached.

Example 3: the lock (the guarantee is the product)

Special property: the lock does no work of its own. Its only output is a promise, and everything downstream is only as correct as that promise. A scheduler that double-fires produces a duplicate email. A lock service that grants twice can corrupt every system relying on it.

The first question to ask is Kleppmann’s: is this lock for efficiency or for correctness?

  Efficiency lock Correctness lock
Purpose Avoid doing the same work twice (rebuilding a cache entry, sending a digest) Prevent two writers from corrupting shared state
Cost of two holders Some wasted work, maybe a duplicate notification Data corruption, double spending, a split brain
Good enough A single Redis key with SET NX PX Consensus-backed lease plus fencing tokens checked at the resource

The options, compared:

Option How it works Safety Failure behaviour Use for
Redis SET key token NX PX 30000 Set if absent with a TTL (time to live); release deletes only if the token matches Async replication can lose the grant on failover Fast; occasionally grants twice Efficiency locks
Redlock Acquire on a majority of N independent Redis nodes within a time bound Disputed: assumes bounded pauses and clock drift; no fencing token Better than one node Efficiency locks needing more availability
Database row or advisory lock SELECT ... FOR UPDATE, or Postgres pg_advisory_lock As strong as the primary Failover drops the locks Resource in the same database (then it is just a transaction)
ZooKeeper Ephemeral sequential node; lowest sequence holds the lock; others watch their predecessor Consensus; the transaction ID serves as a fencing token Session expiry deletes the node; majority loss stops grants Correctness locks, leader election
etcd Key attached to a lease kept alive by the client; the key’s revision is a fencing token Consensus (Raft) Lease expiry deletes the key; majority loss stops grants Correctness locks, leader election

The fencing token, step by step

sequenceDiagram
    participant A as Client A
    participant L as Lock service
    participant B as Client B
    participant S as Storage
    A->>L: acquire
    L-->>A: granted, token 33
    Note over A: GC pause 40 s
    Note over L: lease expires
    B->>L: acquire
    L-->>B: granted, token 34
    B->>S: write (token 34)
    S-->>B: ok, highest seen = 34
    Note over A: resumes, still believes it holds the lock
    A->>S: write (token 33)
    S-->>A: rejected: 33 < 34

Three things to say. The pause is unavoidable: garbage collection, a virtual machine migration, or a slow network can exceed any lease. Client A cannot detect it, because from its point of view no time passed between checking the lease and writing. And only the storage can stop A, which is why the token check lives there; if the storage cannot do a conditional write on a token, no lock service can make the system safe.

Leader election is a lock

Electing a leader is acquiring a lock called “leader” and renewing it. Everything above applies, with one renaming: the fencing token becomes the epoch or term. Raft’s term number, ZooKeeper’s epoch, and Kubernetes’ Lease object all serve the purpose. A new leader increments the epoch; followers and storage reject commands from older epochs. And the old leader has a duty of its own: step down when you cannot renew, rather than waiting to be told. A leader that cannot reach a majority must assume it has been replaced.

The lock service as a single point of failure

A consensus store with three nodes survives one failure; with five, two. Lose the majority and no new locks are granted anywhere. Two consequences to state:

  • Decide what clients do when the lock service is unreachable. For a correctness lock, fail closed: stop work when the lease cannot be renewed. For an efficiency lock, fail open: proceed and accept some duplicate work.
  • Use coarse locks. The lesson of Google’s Chubby paper: lock services work best for locks held for seconds to hours (leader of a shard), not milliseconds (per request). Fine-grained locking through a consensus store puts a consensus round trip on every request.

The best lock is no lock

Most interview problems that seem to need a distributed lock have a better answer:

Instead of a lock Use Example
Lock, read, modify, write, unlock Conditional write: UPDATE ... WHERE version = ? Updating an account balance; retry on conflict
Lock to prevent duplicate creation Unique constraint; the second insert fails One execution per (job_id, scheduled_for)
Lock per item across many workers Partitioned ownership: each partition has one owner, so items in it need no lock Kafka consumer groups; scheduler shards
Lock to prevent a duplicate side effect Idempotency key at the target Payment provider dedupes on the key
Lock around a queue of work SKIP LOCKED claim Workers pulling tasks from a table

Partitioned ownership deserves the emphasis. It replaces many fine-grained locks with a few coarse leases (one per partition), which is exactly the usage pattern a consensus store is good at. Keep the lock service for the questions only it can answer: who leads this shard, who is a member of this cluster.

Side by side

  Scheduler Workflow engine Lock
Special property Time is the trigger State lives across steps and days The guarantee is the product
What is claimed One run of a job One step of a workflow, or a history shard A named resource
Exactly-once in effect via (job_id, scheduled_for) idempotency key Replay for workflow logic; idempotency keys for activities Fencing token checked at the resource
Hardest failure Scheduler down at 00:00; misfires on recovery Code deployed under running workflows Paused holder resumes after expiry
Signature decision Misfire and overlap policy Determinism and versioning Efficiency versus correctness; whether you need a lock at all
Usually bought, not built Managed cron, or the scheduler inside a workflow engine Temporal, Step Functions, Airflow (by use case) etcd or ZooKeeper; never a home-grown consensus

Where each example stops being coordination

Example Turns into When
Scheduler Write-heavy ingest One-off delayed jobs at hundreds of thousands per second (delayed messages, reminders for every user); the problem becomes ingest and storage of timers
Scheduler Fan-out One job’s output is “notify ten million users”; the trigger is coordination, the delivery is fan-out
Workflow engine Contended writes Many workflows compete for the same scarce thing (the last seat, the last unit); the hard part moves to the hot row
Lock Contended writes The lock protects a hot business object; the answer is usually a conditional write on that row, which is the contended-writes post’s territory
Lock Real-time Ownership of a live session or connection (which server holds this document, this driver); the lock becomes routing in a stateful real-time system

The pattern: coordination is the control plane. When the work it controls becomes the hard part, the shape of that work takes over.

The sentence to say in the interview

This is a coordination problem: the volume is low, but every operation has to happen once in effect while workers pause and crash. I will claim work with a conditional update, give every claim a short lease renewed by heartbeat, fence every write with a token the store checks, and make every side effect idempotent so the inevitable retry is harmless. Per-item claims live in the database; a consensus store only holds shard ownership. And wherever I can, I will replace a lock with a conditional write or a single owner per partition.

The question that impresses: “What happens when the process that thinks it owns this is wrong?” Ask it of your own design before the interviewer does.

Follow-Up Questions to Expect

  • “Can you give me exactly-once?” Not delivery. Effect, yes: at-least-once execution plus an idempotency key at the target, or a fencing token at the store. Ask which side effect worries them and design the dedupe for that one.
  • “The lock holder pauses for a minute. What happens?” Its lease expires, someone else acquires it with a higher token, and when the first holder resumes its writes are rejected by the storage. Without the token check, both write, and that is why the check exists.
  • “Why not just use Redis for the lock?” For efficiency, fine. For correctness, a single Redis loses grants on failover, and Redlock relies on timing assumptions that pauses violate and provides no fencing token. Use etcd or ZooKeeper and check tokens at the resource.
  • “How long should the lease be?” Long enough to survive a normal pause with a renewal to spare, short enough that recovery time is acceptable. With fencing, err short.
  • “How does a workflow survive a crash three days in?” Any worker replays the code against the durable history; recorded results are returned without re-running activities.
  • “Temporal, Step Functions, or Airflow?” Airflow for scheduled batch pipelines. Step Functions for managed state machines gluing cloud services together. Temporal for complex, long-running business processes written as code.
  • “What if the consensus store loses its majority?” No new grants or elections. Existing holders fail closed when they cannot renew. The system stops rather than splits, which is the point of using consensus.
  • “Do we need a lock here at all?” Often not. Offer the conditional write, the unique constraint, or the partition owner first.

Key Takeaways

  • Coordination is low volume and high stakes: the problem is agreeing who owns what when processes die and clocks lie.
  • Exactly-once delivery does not exist. Exactly-once effect is at-least-once plus idempotency, or a claim plus a fencing token.
  • A lease is a lock that expires. It recovers from dead holders and creates paused ones. Fencing tokens checked at the resource make the paused holder harmless.
  • The arithmetic is lease length, failover time, and heartbeat load. Per-item claims go in the database; only shard ownership goes in the consensus store.
  • Schedulers: time is the trigger. Misfire policy, the :00 herd, time zones, and the scheduled time in the idempotency key.
  • Workflow engines: state lives for days. Replay, deterministic workflow code, at-least-once activities, durable timers, and versioning from day one.
  • Locks: decide efficiency or correctness first. Redis for efficiency; consensus plus fencing for correctness. Leader election is a lock with an epoch.
  • The best lock is often no lock: conditional writes, unique constraints, partitioned ownership.

Further Reading

The Six Shapes

Each shape has its own deep dive. Start from the first five minutes post to see how naming the shape fits into the opening of an interview.

Shape Examples Where the difficulty lives
Read-heavy lookup URL shortener, product catalog, profile service Cache and read path; writes are trivial
Write-heavy ingest Metrics, logging, clickstream Ingest path, storage tiering, downsampling
Fan-out News feed, notifications, chat Delivery to many recipients; hot users
Contended writes Ticketing, inventory, auctions Consistency on a hot row; the peak moment
Coordination (you are here) Scheduler, workflow engine, locks Leases, exactly-once claims, failure recovery
Real-time Ride matching, live location, collaborative editing Latency, ordering, stateful connections
This post is licensed under CC BY 4.0 by the author.