Idempotency Keys, Properly: Why "Just Make It Idempotent" Is Harder Than It Sounds

Four posts on this blog have already said “at-least-once plus idempotent operations” as if that settled something. It settles the delivery-guarantee argument. It does not settle how you actually build the idempotent operation, and that is the follow-up interviewers use to find out whether a candidate has shipped one or only read about it.

“Just make it idempotent” hides at least eight decisions: who generates the key, what the key is scoped to, where the record of it lives, how the check and the side effect are made atomic, what happens when two requests with the same key arrive at once, what you return on a replay, what you do when the same key arrives with a different payload, and how long you keep it all. By the end of this post you will be able to walk through every one of them in the order an interviewer will ask.

Where This Shows Up

It is rarely the headline question. It is the second or third follow-up to almost any design:

• “The client times out and retries the payment. What happens?”
• “Your scheduler delivers at-least-once. How does the job avoid running twice?”
• “The message broker redelivers. How does the consumer handle it?”
• “The user double-clicks Submit.”
• “Design a payments API.” (this is most of the question)
• “How do you get exactly-once?” (the honest answer routes through here)

What They Are Really Asking

1. Do you know that idempotency is a mechanism, not an adjective? Saying an operation “is idempotent” is a claim. The interviewer wants the design that makes it true.
2. Do you know who generates the key and when? This is the single most common gap. A key generated by the server cannot protect against the lost-response case.
3. Can you make the check and the side effect atomic? Check-then-act is a race. The interviewer wants to hear how you reserve the key before doing the work.
4. Do you know what to return on a replay? Not “re-run it” and not “generic 200.” The same response as the first time.
5. Do you carry the idea past HTTP? Queue consumers, scheduled jobs, and webhooks are the same problem wearing different clothes.

A weak answer is “we use an idempotency key header and check it in Redis.” A strong answer walks the request through reservation, execution, response storage, and replay, names the race and the crash window, and says what the key is scoped to and how long it lives.

The Gotchas

Gotcha 1: Claiming natural idempotency that is not there. “Set status to paid” is idempotent. “Send the confirmation email” is not. “Insert the order” is not. “Append to the ledger” is not. Most real operations bundle an idempotent state change with a non-idempotent side effect, and the side effect is the part that gets retried into duplicates.

Gotcha 2: Letting the server generate the key. The dangerous failure is: request reaches the server, server does the work, response is lost on the way back, client retries. If the server minted the key on the first attempt, the client never had it, and the retry is a brand-new request. The client must generate the key before the first attempt and reuse it on every retry.

Gotcha 3: Getting the scope wrong. A key that is globally unique across all tenants means one tenant’s key can collide with, or be guessed to replay, another’s. Scope the key to the caller and the operation: (tenant or user, endpoint or operation name, key).

Gotcha 4: Check-then-act. Two requests with the same key arrive on two servers. Both look up the key, both see nothing, both execute. The fix is to reserve the key first with an atomic insert that fails on conflict, then do the work. The reservation is the lock.

Gotcha 5: Storing the key somewhere other than the side effect. Key in a cache, order in the database. The server writes the order, crashes before writing the key. The retry finds no key and creates a second order. Or the reverse: key written, crash, no order, and every retry returns “already done.” If you can, write the key record and the business record in the same transaction. If the side effect is external, propagate the key to it.

Gotcha 6: Not storing the response. A replay must return exactly what the first attempt returned: the same status code, the same body, the same created resource ID. If the replay re-executes, you have lost. If it returns a generic acknowledgment, the client cannot get the order ID it needed.

Gotcha 7: Same key, different payload. A client bug, or a malicious caller, sends the same key with a different amount. If you blindly return the stored response, you have silently ignored a request. Store a fingerprint of the original request and reject mismatches with an explicit error.

Gotcha 8: Caching failures forever. The first attempt failed because a downstream timed out. A retry with the same key should be allowed to try again. But a first attempt that failed validation should return the same validation error. You need to distinguish retryable failures from permanent ones in the key record.

Gotcha 9: Forgetting the second line of defense. Keys expire. Clients lose them. A unique constraint on a natural business key, such as one order per cart or one payout per invoice, catches what the idempotency layer misses. Both layers, not one.

Gotcha 10: Idempotent is not the same as ordered. Two updates with different keys arriving out of order are each idempotent and together produce the wrong final state. That is a versioning problem, solved with conditional writes or version numbers. Do not let the interviewer conflate them, and do not conflate them yourself.

How to Answer

Step 1: Classify the operation

Operation shape	Naturally idempotent?	What to do
Set an absolute value (PUT full replace, status = 'paid')	Yes	Nothing extra, but check the side effects it triggers
Delete by ID	Yes, if a second delete returns success rather than 404	Decide the second-delete response deliberately
Create (POST that mints an ID)	No	Idempotency key
Increment, append, transfer	No	Idempotency key
Send (email, SMS, webhook, payment)	No	Idempotency key, propagated to the provider
Conditional update (WHERE version = 7)	Yes	Version numbers handle both retries and ordering

Say the classification out loud. It shows you know idempotency is a property of specific operations, not of “the API.”

Step 2: Key generation and scope

• The client generates it, before the first attempt, as a random 128-bit value (a UUID is fine) or a deterministic hash of a business fact that uniquely identifies the intent, such as cart_id + checkout_attempt.
• The client reuses it on every retry of that intent. A fresh key on retry defeats the entire mechanism. This is an SDK design rule: the retry loop lives inside a function that was handed the key once.
• If the client can crash between attempts, the client persists the key with the pending intent. A mobile app writes it to local storage before the first call.
• Scope is (caller identity, operation, key). Two callers can use the same key value without colliding. The same caller cannot reuse a key across different operations.
• Transport is a header, conventionally Idempotency-Key, which the IETF (Internet Engineering Task Force) draft standardizes.

Step 3: The record

idempotency_keys (
  scope_id         text,         -- tenant or user id
  operation        text,         -- e.g. 'POST /v1/payments'
  key              text,
  request_hash     bytea,        -- sha256 of method + path + canonical body
  status           text,         -- 'in_progress' | 'succeeded' | 'failed_retryable' | 'failed_permanent'
  response_code    int,
  response_body    jsonb,
  locked_at        timestamptz,  -- for detecting stuck in_progress rows
  created_at       timestamptz,
  expires_at       timestamptz,
  primary key (scope_id, operation, key)
)

The primary key is the reservation. The request_hash is the payload-mismatch check. The stored response is the replay. The status is the failure-semantics decision. Point at each column and say what gotcha it prevents.

Step 4: The protocol

sequenceDiagram
    participant C as Client
    participant S as Server
    participant K as Key table
    participant B as Business tables
    participant P as External provider
    C->>S: POST /payments  Idempotency-Key: k1
    S->>K: INSERT (scope, op, k1, hash, in_progress)
    alt insert succeeds (first time)
        S->>P: charge, idempotency key = scope+k1
        P-->>S: ok, charge_id
        S->>B: INSERT payment row
        S->>K: UPDATE status=succeeded, response=201 {payment_id}
        Note over B,K: same transaction
        S-->>C: 201 {payment_id}
    else insert conflicts (retry or duplicate)
        S->>K: SELECT existing row
        alt hash mismatch
            S-->>C: 422 key reused with different payload
        else status in_progress
            S-->>C: 409 in progress, Retry-After
        else status succeeded
            S-->>C: stored 201 {payment_id}
        else status failed_retryable
            S->>K: UPDATE status=in_progress, locked_at=now
            Note over S: take over and execute
        else status failed_permanent
            S-->>C: stored error
        end
    end

Walk it in order:

1. Reserve. Insert the key row with in_progress. The primary key makes this atomic. If it succeeds, you own this key. If it conflicts, someone got there first, and you branch on what they left behind.
2. Execute. Do the work. If it involves an external provider, send the provider your key (or a derivation of it) so its own idempotency layer dedupes if you crash and retry.
3. Commit the result and the response together. The business row and the key row’s transition to succeeded with the stored response go in one transaction. This closes the crash window between “did the thing” and “recorded that I did the thing.”
4. Replay. A conflict with succeeded returns the stored response. Same code, same body. The client cannot tell it was a replay, which is the point.

Step 5: Where the record lives

Choice	Crash safety	Notes
Same database as the business data, same transaction	Best. No window between side effect and key record	The default. Use it whenever the side effect is in your own database
Separate key store (cache) plus business database	A window exists between the two writes	Acceptable only if the business table also has a natural unique constraint as a backstop
Business change plus outbox row in one transaction, key propagated in the event	Good. The downstream dedupes on the event ID	This is how idempotency survives crossing a queue
External provider with its own idempotency support	Good, if you send the key before the crash window	Payment providers accept idempotency keys for exactly this reason. Use them

The line to say: the idempotency record should be committed in the same transaction as the side effect it protects. Everything else is a compromise you should name.

Step 6: Concurrency, and the stuck reservation

Two requests with the same key at the same instant: one wins the insert, the other conflicts and sees in_progress. Two reasonable responses:

• Reject with 409 and Retry-After. Simple, stateless, what most payment APIs do. The client retries in a second and gets the stored response.
• Wait for the first to finish and return its response. Friendlier, but holds a connection and needs a timeout. Fine for internal services, risky at the edge.

Then the case candidates forget: the server that reserved the key crashed mid-execution. The row is in_progress forever. Use locked_at: a reservation older than the operation’s maximum duration is considered abandoned and the next request with that key takes it over. This is the same lease-and-reaper idea from the job scheduler post.

Step 7: Failure semantics

First attempt outcome	Key status	Replay behavior
Succeeded	succeeded	Return stored response
Validation error, business rule rejection	failed_permanent	Return stored error
Downstream timeout, transient error	failed_retryable	Allow re-execution under the same key
Server crashed mid-flight	in_progress past locked_at threshold	Allow takeover

Say that the difference between the last two rows and the second row is the whole reason the status column has four values instead of two.

Step 8: Retention and the second line of defense

• Keep keys for the client’s maximum retry horizon. Twenty-four hours is a common choice. Longer if clients can queue offline.
• Size it. 10 million requests a day with a 1 KB stored response is 10 GB a day, so 24-hour retention is 10 GB and 72-hour retention is 30 GB. It fits, and it is worth stating.
• Back it with a natural unique constraint. One order per cart, one payout per invoice, one enrollment per user per course. When the key has expired and a very late retry arrives, the constraint catches it and you can map the conflict to the existing record.

Step 9: Beyond HTTP

The same design, with the key renamed:

Context	The key	Where the record lives
Queue consumer	Message ID, or a producer-supplied dedupe ID	An inbox table in the consumer’s database, written in the same transaction as the processing. This is the inbox pattern, the mirror of the outbox
Scheduled job	(job_id, scheduled_for)	The execution table, as in the job scheduler post
Webhook receiver	The provider’s event ID	An events-seen table with a TTL
Retry within a saga step	Step ID plus saga instance ID	The saga’s state store

The interviewer’s question “how does the consumer handle redelivery” has the same answer as “how does the API handle a client retry.” Saying so shows you see the pattern rather than the instance.

Follow-Up Questions to Expect

• “What if the client does not send a key?” For unsafe operations, require it and return 400. Do not try to infer one from the payload; a fingerprint-based fallback with a short window is a last resort with false positives.
• “The client reuses a key with a different amount.” 422 with a clear message. Never silently return the old response.
• “The payment provider does not support idempotency keys.” Query before acting, using a reference you control, and run a reconciliation job. Say that this is strictly weaker and why.
• “Isn’t Kafka exactly-once?” Within Kafka, with transactional producers and consumers, yes. The guarantee ends at the first side effect outside Kafka. That side effect needs this design.
• “The key store is down.” For money-moving operations, fail closed. For others, decide per operation and say so. Same argument as the rate limiter post.
• “How do you test it?” Kill the process between each step of the protocol and assert exactly one side effect. Send the same key from two threads and assert one execution. Replay after expiry and assert the natural constraint catches it.

Key Takeaways

• Idempotency is a mechanism with eight decisions, not a property you assert.
• The client generates the key before the first attempt and reuses it on every retry. A server-minted key cannot survive a lost response.
• Scope the key to caller and operation. Store a request fingerprint and reject mismatches.
• Reserve the key with an atomic insert before doing the work. That insert is the lock.
• Commit the key record and the side effect in the same transaction. If the side effect is external, send the key downstream.
• Store the response and return it verbatim on replay.
• Four statuses: succeeded, permanent failure, retryable failure, in progress. The last two allow re-execution.
• Retain for the retry horizon, and back it with a natural unique constraint for everything after that.
• Queue consumers, scheduled jobs, and webhooks are the same design with a different key.

Further Reading

• Stripe, Designing robust and predictable APIs with idempotency
• Brandur Leach, Implementing Stripe-like idempotency keys in Postgres
• IETF HTTP APIs working group, The Idempotency-Key HTTP header field
• Amazon Builders’ Library, Making retries safe with idempotent APIs
• Martin Kleppmann, Designing Data-Intensive Applications, chapter 11 on exactly-once and idempotence