Sagas and the Outbox Pattern: How to Do a Transaction Across Services Without a Distributed Transaction

Three posts on this blog have ended a paragraph with “no isolation spans services; that is sagas and the outbox, and it is a different post.” This is that post.

The thesis is in the title’s second half. Most candidates prepare the saga: the multi-step process with compensations, the choreography-versus-orchestration debate. Far fewer can explain the outbox, and the outbox is the part that goes wrong in production. Almost every event-driven system has a moment where a service writes to its database and then publishes a message, and the gap between those two operations is where orders vanish and payments get charged twice. Fix that gap first. The saga is what you build on top of it.

Where This Shows Up

• “The order service saves the order and publishes an event. The publish fails. What happens?”
• “How do you keep the inventory service and the order service consistent?”
• “Why not use two-phase commit?”
• “Design checkout across order, payment, and inventory services.”
• “The payment succeeded but the inventory reservation failed. Now what?”
• “What is the difference between choreography and orchestration?”

What They Are Really Checking

1. Do you see the dual-write problem? Write to a database, then publish to a broker. Two systems, no shared transaction. The interviewer wants you to name it before they do.
2. Do you know why two-phase commit is not the answer? It exists. It has real costs. Managed services and message brokers mostly do not participate in it. Know the argument, not just the conclusion.
3. Can you explain the outbox mechanically? Same transaction, relay, at-least-once, idempotent consumer. Four parts, in order.
4. Do you know that compensation is not rollback? A refund is a new transaction that can fail. The customer saw the charge. This is where the saga stops being a diagram and starts being a design.
5. Can you say what you lose? Sagas give up isolation. Intermediate states are visible. The interviewer wants the countermeasures.
6. Do you know when not to use one? If every operation needs a saga across A and B, A and B should be one service.

The Gotchas

Gotcha 1: The dual write. Save the order, then publish the event. Crash in between: the order exists and nobody knows. Publish first, then save: the event is out and the order never lands. Wrapping both in a try-catch does not help; the crash can be a process kill. This is the bug, and the outbox is the fix.

Gotcha 2: Saying “two-phase commit” as if it were available. XA-style distributed transactions require every participant to speak the protocol. Your relational database might. Your message broker, your search index, your third-party payment API, and most managed cloud services do not. Where it does work, the coordinator is a single point of failure and participants hold locks while waiting for it. Say this in two sentences and move on.

Gotcha 3: Compensation as rollback. “If payment fails, we roll back the inventory reservation.” There is no rollback. There is a new transaction called “release reservation,” and it can fail, be delayed, or arrive after someone else took the stock. Compensations are forward actions that semantically undo, and every one of them needs its own retry and failure story.

Gotcha 4: Non-idempotent steps. The outbox relay delivers at least once. Every consumer will eventually see a duplicate. If “charge the card” runs twice, the saga has created the bug it was meant to prevent. Every step is idempotent, enforced by an inbox table, per the idempotency post.

Gotcha 5: Choreography spaghetti. Five services each reacting to each other’s events. Nobody can say what the checkout flow is without reading five codebases. Cyclic dependencies appear. Debugging is grepping logs for a correlation ID and hoping. Choreography is fine for two or three steps; past that it needs an owner.

Gotcha 6: The orchestrator as a god service. The other failure mode. One service that knows every business flow, calls everyone, and becomes the thing every team waits on. Keep the orchestrator to sequencing and state; the business logic stays in the participants.

Gotcha 7: Forgetting that intermediate states are visible. The order is created and inventory reserved, and payment is still pending. A shipping process reads “order exists” and ships it. Sagas have no isolation, so the design has to mark in-flight state explicitly and other transactions have to respect it.

Gotcha 8: Designing only the happy path. Every step needs a compensation or an explicit “cannot be compensated, so it runs last.” And every compensation needs an answer to “what if this fails,” which is usually retry until it succeeds, then alert a human.

Gotcha 9: Losing ordering in the relay. Two events for the same order published out of order: “order cancelled” before “order created.” The relay must preserve order per aggregate, which means partitioning by aggregate ID and publishing one aggregate’s events serially.

Gotcha 10: Using a saga where the boundary is wrong. If order and inventory are so entangled that every operation spans both, the saga is a symptom. The monolith post makes the argument: a service that cannot commit its own invariants alone is not a bounded context.

How to Answer

Step 1: Split the problem in two

Say this first, because it organizes everything after it:

There are two problems here. The first is atomicity between a local database write and a message: I need “the order is saved” and “the world is told” to happen together or not at all. The second is a business process that spans several services and must end in a consistent state even when a step fails. The outbox solves the first. The saga solves the second, and it depends on the first.

Step 2: Compare the options for the first problem

Option	Atomic?	Availability	Latency	Operational cost	Verdict
Dual write (save, then publish)	No	High	Low	None	The bug. Never
Two-phase commit (XA)	Yes, among participants that support it	Coordinator is a single point of failure; participants block while waiting	High: two round trips plus lock hold time	High, and most brokers, APIs, and managed services cannot participate	Rarely available; rarely worth it
Transactional outbox with a relay	Yes: the event is committed with the data	High	Low on the write path; relay adds delivery lag of milliseconds to seconds	An outbox table and a relay process	The default answer
Outbox via change data capture (CDC)	Yes	High	Low	Runs a CDC connector on the database log	The outbox at scale; same idea, log-based relay
Event sourcing (the events are the state)	Yes, trivially	High	Low	A different data model for the whole service	Right when you wanted event sourcing anyway; not a fix to bolt on
Redraw the boundary so it is one database	Yes	High	Lowest	None	Right more often than people admit

Step 3: The transactional outbox

sequenceDiagram
    participant A as Order service
    participant DB as Order DB
    participant R as Relay
    participant Q as Broker
    participant C as Inventory service
    participant CDB as Inventory DB
    A->>DB: BEGIN
    A->>DB: INSERT orders (...)
    A->>DB: INSERT outbox (event_id, aggregate_id, type, payload)
    A->>DB: COMMIT
    Note over DB: Order and event are atomic
    R->>DB: SELECT unpublished outbox rows, ordered per aggregate
    R->>Q: publish(event_id, payload), key = aggregate_id
    Q-->>R: ack
    R->>DB: mark published
    Note over R,Q: At-least-once: a crash after publish<br/>and before mark means a duplicate
    Q->>C: deliver event
    C->>CDB: BEGIN
    C->>CDB: INSERT inbox (event_id) — unique
    alt already seen
        C->>CDB: ROLLBACK, ack the message
    else first time
        C->>CDB: reserve inventory
        C->>CDB: INSERT outbox (InventoryReserved ...)
        C->>CDB: COMMIT
    end

The outbox row:

outbox (
  event_id      uuid primary key,   -- consumers dedupe on this
  aggregate_type text,              -- 'order'
  aggregate_id   uuid,              -- partition key; preserves per-order ordering
  event_type     text,              -- 'OrderCreated'
  payload        jsonb,
  created_at     timestamptz,
  published_at   timestamptz        -- null until the relay confirms
)

The four things to say:

1. Same transaction. The business row and the outbox row commit together. There is no window.
2. The relay. A poller that selects unpublished rows, or a CDC connector that tails the database log. CDC has lower lag and no polling load; polling is simpler and fine at moderate volume. Either way the relay publishes with the aggregate ID as the partition key so one order’s events stay in order.
3. At-least-once. The relay can crash between publishing and marking published. Duplicates are a certainty over time, not a possibility.
4. The inbox. The consumer inserts the event ID into an inbox table in the same transaction as its processing. The unique constraint makes the duplicate a no-op. If the consumer’s processing emits its own events, they go into its own outbox in that same transaction, and the chain continues.

That is the whole mechanism. A candidate who can draw it and say those four sentences has demonstrated more than most.

Step 4: The saga

A saga is a sequence of local transactions, each in one service, where each step has a compensating transaction that semantically undoes it. If step N fails, run the compensations for steps N-1 down to 1.

Checkout as the example:

flowchart LR
    T1[1. Create order<br/>status = PENDING] --> T2[2. Reserve inventory]
    T2 --> T3[3. Charge payment<br/>pivot]
    T3 --> T4[4. Confirm order<br/>status = CONFIRMED]
    T4 --> T5[5. Send confirmation email]
    T2 -.->|fails| C1[Cancel order]
    T3 -.->|fails| C2[Release inventory] -.-> C1
    style T3 fill:#7a4a00,color:#fff

Three kinds of step, and naming them is what separates a memorized diagram from an understood one:

Step kind	Meaning	In the example
Compensable	Can be undone by a later compensation	Create order, reserve inventory
Pivot	The point of no return. Once it commits, the saga must complete forward; nothing after it is compensated, nothing before it is compensated either	Charge payment
Retryable	Runs after the pivot; cannot fail permanently, only be retried until it succeeds	Confirm order, send email

The ordering rule that follows: put the steps most likely to fail first, the pivot as late as possible, and steps that cannot be compensated after the pivot. You do not want to refund a card because an email service was down. You want the email last, retried forever.

Step 5: Choreography or orchestration

	Choreography	Orchestration
How it works	Each service reacts to events and emits its own; no central coordinator	A saga orchestrator sends commands to each service and tracks state
Where the flow lives	Nowhere explicit; it emerges from subscriptions	In one place, as a state machine
Coupling	Services know event names, not each other	Services know the orchestrator; the orchestrator knows everyone
Failure handling	Each service must know its own compensation trigger	Orchestrator runs compensations in order
Visibility	Correlation ID and log aggregation	The orchestrator’s state table is the status of every saga
Testing	Integration-heavy	The state machine is unit-testable
Risk	Spaghetti past three or four steps; cycles	The orchestrator becomes a god service or a bottleneck
Use for	Two or three steps, simple reactions, teams that own adjacent services	Anything with branches, more than three steps, or a flow someone has to explain to a product manager

A defensible recommendation: orchestration for checkout-shaped flows, with the orchestrator’s state persisted in its own database and every transition idempotent. The orchestrator itself is driven by events through the same outbox and inbox machinery, so it is not a synchronous single point of failure. If the organization already runs a durable workflow engine, the orchestrator is a workflow definition and the engine handles retries and timers. That connects to the scheduler post and its “what you would buy” section.

Step 6: What you lose, and the countermeasures

Sagas have atomicity (eventually, every step or every compensation), consistency (eventually), and durability. They do not have isolation. Other transactions can see the order between steps. The countermeasures, from Richardson’s catalogue:

Anomaly	Countermeasure	How
Another process acts on an in-flight order	Semantic lock	The PENDING status is the lock. Shipping only reads CONFIRMED. Every reader respects the flag
Two sagas update the same row and one overwrites the other	Commutative updates	Make the operations order-independent: reserve and release by quantity, not by setting a total
A compensation undoes something a later, unrelated update depended on	Pessimistic view	Reorder the saga so the step that risks dirty reads runs after the pivot
A step reads a value that a concurrent saga is changing	Reread the value	Compare-and-set on the step’s write: WHERE version = ?
A user sees a half-finished state	Version file, or just tell them	Show “processing” states in the interface. Sometimes the countermeasure is honesty

Naming two of these unprompted is enough. The interviewer is checking that you know isolation was lost, not that you have memorized the list.

Step 7: Failure, end to end

• A step fails transiently. Retry with backoff under the same idempotency key. The inbox makes the retry safe.
• A step fails permanently before the pivot. Run compensations in reverse order. Each compensation is itself a retried, idempotent step.
• A compensation fails. Retry until it succeeds. If it cannot, park the saga in a NEEDS_ATTENTION state, alert, and give a human a tool. Compensations must not fail silently, and the design should say so.
• A step after the pivot fails. Retry forever. It is retryable by construction, or it should not be after the pivot.
• The saga stalls. Every saga has a timeout. A saga that has been PENDING past it is either completed by a sweeper that resumes it or compensated. The scheduler post’s reaper is the same shape.
• Observability. The saga ID is the correlation ID on every message and log line. The orchestrator’s state table, or a saga log in choreography, answers “where is order 123 right now.”

Step 8: The sentence to say in an interview

Checkout is a saga across order, inventory, and payment. Each service commits its own step locally and publishes its event through a transactional outbox, so the write and the event are atomic, and a relay delivers at-least-once to consumers that dedupe with an inbox. An orchestrator with persisted state drives the steps: create the order as pending, reserve inventory, charge payment as the pivot, confirm, then email. Inventory and order creation are compensable; payment is the point of no return; confirmation and email are retried until they succeed. The pending status is the semantic lock, so nothing ships until the saga completes. If a compensation fails it retries and then parks for a human. I would only reach for this if order and inventory genuinely belong to different services; if every operation crosses that line, I would question the boundary first.

Follow-Up Questions to Expect

• “Why not two-phase commit?” Participants must support the protocol, most brokers and APIs do not, the coordinator is a single point of failure, and participants hold locks while blocked on it. Where all participants are one vendor’s database, it is fine; that is rarely the situation.
• “The refund fails. Now what?” Retry with backoff under the same key. If it keeps failing, park the saga, alert, and give operations a tool. The customer has been charged; this is the case that must never be silent.
• “The customer sees the order as placed while payment is still pending.” That is isolation loss. Show the real state: “processing.” The semantic lock stops downstream processes from acting on it.
• “What order do you run the steps?” Most likely to fail first, pivot as late as possible, non-compensable steps after the pivot, retried until success.
• “What if the relay publishes the same event twice?” It will. That is why the consumer has an inbox keyed on event ID.
• “Outbox versus event sourcing?” The outbox bolts reliable publishing onto a conventional data model. Event sourcing makes the events the data model. Choose event sourcing for its own reasons, not to solve this problem.
• “When is a saga the wrong answer?” When the services involved cannot enforce their own invariants without each other. Merge them.

Key Takeaways

• Two problems: atomic write-plus-publish, and a multi-service process that ends consistent. The outbox solves the first; the saga is built on it.
• The dual write is the bug. The transactional outbox is the fix: same transaction, relay, at-least-once, inbox.
• Two-phase commit exists and is usually unavailable or too costly. Say why in two sentences.
• Compensation is a new forward transaction, not a rollback. It can fail. Design for that.
• Name the pivot. Order steps so the pivot is late and non-compensable steps come after it.
• Orchestration for real flows; choreography for two or three simple steps.
• Sagas lose isolation. Use a pending status as a semantic lock and make updates commutative where you can.
• If everything needs a saga, the boundary is wrong.

Further Reading

• Garcia-Molina and Salem, Sagas (SIGMOD 1987; the original paper)
• Chris Richardson, Transactional Outbox and Saga at microservices.io, and Microservices Patterns chapters 3 and 4 for the countermeasures
• Debezium, Outbox Event Router (CDC-based outbox in practice)
• Pat Helland, Life Beyond Distributed Transactions (why you end up here)
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 9 on two-phase commit and chapter 11 on stream processing