System Shapes: Contended Writes (Ticketing, Inventory, Auctions)
Contended writes is the shape where many users race to change the same small piece of state at the same moment. Here is the hot-row arithmetic, the four ways to serialize a write, and what ticketing, inventory, and auctions each force you to decide differently.
Some systems are hard because there is a lot of data, and some are hard because there are a lot of readers. Contended-writes systems are hard because a lot of people want to change the same thing at the same instant, and only some of them are allowed to win. The total traffic is often small. The traffic on one row, for one minute, is the whole problem.
The first five minutes post lists this shape as “ticketing, inventory, auctions” and says the difficulty is “consistency on a hot row; the peak moment.” This post goes through the shape in depth: what defines it, the arithmetic that sizes it, the standard problems and their fixes, and then each of the three examples, because each one has a special property that changes the design. Ticketing already has a full design post, so it gets a shape-level summary here; inventory and auctions get the depth.
Where This Shows Up
The direct prompts:
- “Design a ticket booking system.” / “Design Ticketmaster.”
- “Design the inventory service for an e-commerce site.”
- “Design a flash sale.” / “Design a limited sneaker drop.”
- “Design an online auction.” / “Design eBay bidding.”
- “Design hotel, airline, or restaurant reservations.”
And the follow-ups that turn a different question into this one:
- “What happens when a celebrity’s product goes on sale?”
- “Two users buy the last unit at the same instant. Walk me through it.”
- “How do you make sure you never oversell?”
- “Two bids arrive in the same millisecond. Who wins?”
- “Your database has one row that everyone is updating. What is its throughput?”
What They Are Really Checking
- Can you find the unit of contention? One seat, one SKU (Stock Keeping Unit) counter, one auction’s high bid. Everything else in the design is easy. The interviewer wants to hear you name the hot row before you draw boxes.
- Do you know what a single row can sustain? “Use a database transaction” is not an answer until you say how many per second that row can take and whether the peak exceeds it.
- Can you pick a serialization mechanism and defend it? Pessimistic lock, conditional write, atomic decrement, or a single writer per key. Each has a contention profile. Choosing by name is junior; choosing by the arithmetic is senior.
- Do you separate the display from the decision? The stock badge, the seat map, and the current-bid banner can be stale. The write that commits a unit to a buyer cannot.
- Can you shape the peak instead of absorbing it? Admission control, waiting rooms, and holds turn a burst you cannot serve into a rate you chose.
- Do you treat oversell as a business decision? Some domains tolerate it deliberately. Asking which one you are in is the question that shows product judgment.
The Gotchas
Gotcha 1: Sizing for the average. A ticketing site averages three purchases a second and takes thousands per second on one event at the on-sale. A store averages a few orders a second per SKU and takes ten thousand a second during a flash sale. The design is sized for the peak on one key, and the average is noise.
Gotcha 2: Read-modify-write in the application. Read the count, subtract in code, write it back. Two requests, one unit, both succeed. At the default isolation level this is a lost update, and it is the most common oversell bug in production. Make the check and the change one atomic statement.
Gotcha 3: Holding a lock across a slow call. SELECT ... FOR UPDATE, then call the payment provider, then commit. A row’s throughput is one divided by how long the lock is held, so a three-second payment call turns a row that could take a thousand writes a second into one that takes a third of a write per second.
Gotcha 4: Optimistic concurrency on a hot key. Version checks with retries are excellent when conflicts are rare. With 100 clients racing on one row, each round has one winner and 99 retries, and serving all 100 costs about 5,000 attempts. Optimistic concurrency degrades quadratically under exactly the contention this shape is about.
Gotcha 5: Treating fungible units as distinct, or distinct units as fungible. A count of 1,000 identical phones can be split across buckets and decremented anywhere. A seat cannot: seat A-12 is one row no matter what. Getting this wrong either leaves easy throughput on the table or sells the same seat twice.
Gotcha 6: Holds that never expire, or expire by cron. Carts and checkouts that reserve stock without a deadline turn abandoned sessions into dead inventory during the exact minutes everyone wants it. Expiry belongs inside the conditional write, with a sweeper only for cleanup.
Gotcha 7: Assuming oversell is always forbidden. Airlines and hotels overbook on purpose. Retailers accept orders and backorder. A flash sale for a limited-edition item cannot. Ask before designing for zero oversell, because zero oversell is the most expensive point on the curve.
Gotcha 8: Using the client’s timestamp to order bids. Clocks on phones are wrong and clients lie. The order that counts is the order the serialization point commits the writes. A bid “placed” at 11:59:59 on the client and committed at 12:00:00.2 is late.
Gotcha 9: Forgetting that the close is a write. An auction does not end because a timer fires in someone’s browser. It ends when one write flips its state to closed, and every bid must be conditional on that state. Without it, a bid can be accepted after the winner was announced.
Gotcha 10: Letting the peak reach the database. A hundred thousand arrivals a minute on one event or one SKU will exhaust connection pools and the row lock queue long before the database’s raw capacity matters. Admission control at the edge is part of the write path design, not an afterthought.
How to Answer
What defines the shape
| Property | Contended writes | Compare with |
|---|---|---|
| Read to write ratio | Reads still dominate overall (10:1 to 100:1), but the writes are what matter | Read-heavy lookup, where writes are trivial |
| Access pattern | Many writers target the same few keys at the same moment | Write-heavy ingest, where writes are spread across keys and append-only |
| Data size | Small. 50,000 seats is 10 MB. A catalog’s stock table is a few GB | Size is never the problem here |
| Skew | Extreme and time-bound. One event, one SKU, one auction takes most of the traffic for minutes | Zipf skew in read-heavy systems is steady; here it spikes |
| Consistency tolerance | Zero on the decision (who got the unit), generous on the display (how many are left) | Most shapes can be eventual nearly everywhere |
| Time profile | A scheduled or predictable peak: on-sale, drop, auction close | Ingest is a steady firehose; fan-out spikes on hot users |
So the design effort goes into one write path: how a single key serializes its writes, how fast it can do so, and how the peak is throttled down to that rate before it arrives.
The generic architecture
flowchart LR
U[Clients] --> E[Edge / CDN]
E -->|display reads| RC[(Read model<br/>cache, ~1s stale)]
E --> AC[Admission control<br/>waiting room, rate limit]
AC --> API[Stateless API]
API -->|serialized write per key| HS[(Contended store<br/>partitioned by unit of contention)]
HS -.->|change stream| RC
API --> OUT[(Outbox)]
OUT --> Q[[Broker]]
Q --> D[Payment, email, analytics]
Five parts, and each one exists because of the shape:
- A read model for the display. Seat maps, stock badges, and current bids are served from a cache fed by a change stream. The decision path never reads from it.
- Admission control in front of the write path. It converts an unbounded burst into a fixed rate, and it is where fairness and bot control live.
- A store partitioned by the unit of contention. Event, SKU, or auction. You cannot shard below that unit without changing the semantics, which is why the hot key stays hot.
- A serialized write per key: one of the four mechanisms in the next table.
- Everything else asynchronous. Payment, confirmation email, and analytics hang off an outbox so nothing slow sits inside the critical section.
The arithmetic that matters
The number that sizes this shape is how many writes per second one key can take, and it is set by how long each write holds the key:
Throughput of one key ≈ 1 / (time the key is held per write)
| What holds the key | Hold time | Max writes per second on that key |
|---|---|---|
| In-memory atomic operation (Redis, single thread) | ~10 microseconds | ~100,000 |
| Single-statement conditional update, local commit with group commit | ~0.5 to 1 ms | ~1,000 to 2,000 |
| Same, with synchronous replication to another zone | ~2 to 3 ms | ~300 to 500 |
| Multi-statement transaction with application round trips | ~10 ms | ~100 |
| Transaction held across a payment call | ~3 s | ~0.3 |
Say the arithmetic out loud: “A popular on-sale admits 100,000 buyers. If they all reach the row in one minute, that is about 1,700 writes per second on one key. A synchronously replicated conditional update tops out around 400. So either admission control holds the rate under 400, or the key has to be split, or the writes have to be batched.” Every decision in the rest of the design follows from that sentence.
Batching is the lever people forget. A single writer that drains a queue and applies 100 decrements in one transaction pays the commit cost once, so the same 2 ms hold time serves 50,000 requests per second. That is the idea behind the LMAX (London Multi-Asset Exchange) single-writer architecture, and behind every exchange matching engine.
The four ways to serialize a write
| Mechanism | How | Under contention | Use when |
|---|---|---|---|
| Pessimistic lock | SELECT ... FOR UPDATE, decide, update, commit |
Waiters queue; throughput is 1 / transaction time; deadlocks with multiple rows unless locked in a fixed order | The decision needs several reads and cannot be one statement, and the transaction is short |
| Optimistic version (compare-and-set) | Read with a version, UPDATE ... WHERE version = :v, retry on zero rows |
Fine at low contention; wasted work grows quadratically on a hot key | Conflicts are rare: profile edits, admin changes, cold SKUs |
| Atomic conditional statement | UPDATE ... SET n = n - 1 WHERE id = ? AND n >= 1 in one statement |
The database’s row lock is held for microseconds; losers get a clean zero-row result, no retry loop | The default. Seats, counts, bids that fit in one predicate |
| Single writer per key | Partitioned queue or actor per key; one consumer applies writes in order | No contention at all; latency is queue depth; batching multiplies throughput | Extreme peaks, strict ordering (auctions), or logic too complex for one statement |
The isolation post covers the database side of this toolbox (why read committed permits lost updates, and why a conditional statement or a unique constraint beats raising the isolation level). This table is the same toolbox seen from the throughput side.
The standard problems
| Problem | What goes wrong | Usual fix |
|---|---|---|
| Lost update / oversell | Two writers read, both decide, both write | Atomic conditional statement or unique constraint |
| Hot row saturation | Lock queue grows, latency climbs, connection pool exhausts | Admission control below the row’s rate; split fungible counters; batch in a single writer |
| Retry storm | Optimistic retries and client retries multiply load at the worst moment | Conditional statement instead of compare-and-set; capped retries with jitter; return “sold out” fast |
| Abandoned holds | Carts and checkouts reserve units that never convert | Hold with expiry checked inside the write; short time to live (TTL); sweeper for cleanup |
| Display disagrees with decision | Badge says “3 left,” purchase fails | Expected and fine; the write is authoritative and the interface handles the conflict |
| Unfairness and bots | Fastest scripts win; real users see “sold out” instantly | Waiting room with randomized initial order, per-account limits, challenges at entry |
| Double charge on retry | Client retries a purchase after a timeout | Idempotency key on the purchase and on the charge |
| Multi-unit deadlock | Two buyers lock seats A then B, and B then A | One statement over all units, or lock in sorted key order |
Example 1: Ticketing, where every unit is distinct
Special property: each unit is a distinct item. Seat A-12 is not interchangeable with seat A-13, so the unit of contention is a single row and it cannot be split. Contention is per seat, the peak is the most extreme of the three examples (the entire inventory is wanted in the first minute), and the buyer often wants several adjacent units at once.
What that property forces:
- The row cannot be sharded, so the rate must be throttled. You can partition an event by section across shards, but not a seat. The waiting room is therefore not optional: it is the mechanism that keeps hold attempts below what the seat rows can take.
- The hold is a state in the row with an expiry, taken by one conditional update that succeeds if the seat is available or its hold has lapsed. No lock survives past the statement, and expiry needs no cron job to be correct.
- Multi-seat holds are all-or-nothing in one statement over all requested seats, checked by row count. That also avoids the lock-ordering deadlock.
- The sale after payment uses the hold ID as a fencing token, so a hold that expired during a slow charge is detected and refunded instead of double-sold.
- “Best available” is a search on the display followed by a conditional write, retried on the next candidates. Never a serializable read-then-write.
- General admission collapses into the inventory case: one counter per tier, decremented atomically.
The ticket booking post walks through all of this with the schema, the SQL, the purchase saga, and the on-sale numbers. For the shape, the one-line version is: distinct units mean the hot row cannot be split, so admission control sets the write rate, and the hold-with-expiry conditional update is the decision.
Example 2: Inventory, where units are fungible
Special property: units are interchangeable, so the state is a count, not a set of rows. One iPhone in a given color and capacity is as good as any other. That single fact unlocks options ticketing does not have: you can split the counter, reserve without naming the unit, choose how much oversell to tolerate, and decide which warehouse fulfills the order after the sale.
The baseline: one row per SKU per location
1
2
3
4
5
6
7
8
9
10
-- stock per SKU per warehouse
inventory (sku, warehouse_id, on_hand, reserved, version,
primary key (sku, warehouse_id))
-- reserve n units at checkout: one atomic statement
UPDATE inventory
SET reserved = reserved + :n
WHERE sku = :sku AND warehouse_id = :wh
AND on_hand - reserved >= :n;
-- 1 row: reserved. 0 rows: not enough available at this location.
Available-to-sell is on_hand - reserved. For 99 percent of SKUs this row sees a write every few minutes, and the design is done. The interview is about the other 1 percent: the SKU on the homepage during a flash sale.
Reserve, then commit
Inventory has a two-phase lifecycle that is easy to skip and expensive to get wrong:
| Step | Effect on the row | Triggered by |
|---|---|---|
| Reserve | reserved += n |
Checkout begins (or add-to-cart, for flash sales) |
| Commit | on_hand -= n, reserved -= n |
Payment succeeds |
| Release | reserved -= n |
Payment fails, cart abandoned, or reservation expires |
| Receive / adjust | on_hand += n (or -= for damage and shrinkage) |
Warehouse events |
Each reservation is its own record with an expiry (reservations(id, sku, warehouse_id, n, expires_at, status)) so that releases are idempotent and a sweeper can find lapsed ones. When to reserve is a product decision with a direct throughput cost:
| Reserve at | Buyer experience | Inventory effect | Contention |
|---|---|---|---|
| Add to cart | Item is “yours” while browsing | Abandoned carts lock stock; needs a short TTL (10 to 15 minutes) | Highest: every cart add is a write on the hot row |
| Checkout start | Rare surprise at payment | Standard for most retail | Moderate |
| Payment success only | Occasional “sorry, it sold out” after entering a card | No dead stock | Lowest, but oversell risk shifts to the payment window |
Most retailers reserve at checkout. Flash sales reserve at cart with a short timer, because the product promise is “if it is in your cart, you got one.”
Splitting the hot counter
Because units are fungible, one hot row can become many cool ones. Three standard ways:
| Technique | How | Cost |
|---|---|---|
| Sharded counter (buckets) | 1,000 units become 10 rows of 100. A request picks a random bucket and decrements it; on zero, it tries another | “Sold out” requires checking every bucket; the last few units are found by scanning; total is a sum |
| Escrow / quota allocation | Each app server, region, or channel is granted a quota (say 50 units) it can sell locally without coordination; it asks the central row for more when low | Units stranded in a quota that has no buyers; rebalance near the end |
| Pre-minted tokens | Put 1,000 tokens in a list (a Redis list or a queue). Claiming is a pop, which is atomic and fast | The token store becomes the source of truth for the sale window; must be reconciled to the database after |
Ten buckets at 400 writes per second each give 4,000 per second. The escrow idea is old and formal: Patrick O’Neil’s escrow transactional method (1986) lets concurrent transactions decrement a quantity as long as the worst case still satisfies the constraint. The interview sentence is “the units are fungible, so I can split the counter, and the only cost is that ‘sold out’ becomes a scatter-gather at the very end.”
For a true flash sale (10,000 units, 1,000,000 people, starting at noon), the usual production answer combines these: a waiting room in front, an in-memory atomic decrement (a Redis DECR or a short Lua script that checks and decrements) as the gate, an order record written asynchronously behind it, and a reconciliation job that compares the gate’s count to the database. The in-memory gate takes 100,000 decisions a second. The database only sees the 10,000 winners.
1
2
3
4
5
6
-- flash-sale gate: atomic check-and-decrement in Redis
local left = tonumber(redis.call('GET', KEYS[1]) or '0')
if left >= tonumber(ARGV[1]) then
return redis.call('DECRBY', KEYS[1], ARGV[1])
end
return -1
Say the risk out loud: if the Redis node fails over and loses the last second of writes, the gate may sell a few units twice. Either accept that and cap it with the reconciliation step (cancel and apologize), or make the gate durable (a replicated log, or a single writer with a write-ahead log), which costs latency. That is a business decision, which leads to the next point.
Oversell tolerance is a business decision
| Domain | Oversell policy | Why |
|---|---|---|
| Airline seats, hotel rooms | Deliberate overbooking, a few percent | No-shows are statistically predictable; an empty seat is lost revenue forever |
| General retail | Small oversell tolerated; backorder or cancel with apology | Counts are already wrong (shrinkage, damage, miscounts); restock is coming |
| Limited edition, flash sale | Zero, or as close as you can afford | There is no restock; overselling is a public failure |
| Regulated goods, pharmacy | Zero | Legal and safety constraints |
Asking “what happens if we sell one more than we have?” before designing is the most senior question in an inventory interview. The stored count is already an estimate (shrinkage, miscounts), so perfect concurrency control on it is often effort in the wrong place.
Multiple warehouses
Availability shown to the buyer is the sum across locations. Allocation (which warehouse ships) can be decided at reserve time or later:
- Reserve against a specific warehouse (nearest with stock): simple, accurate, but a buyer may be told “out of stock” while another warehouse has units.
- Reserve against a network-level pool, allocate later: a single counter per SKU for available-to-promise, with an order-routing step after payment that picks the warehouse. This is more flexible and is what large retailers do, but now there are two counters (network and location) that must reconcile.
Either way, the location rows see far less contention than the network row, so the hot key is the network-level available-to-promise counter, and that is the one you split.
Example 3: Auctions, where validity depends on the current value
Special property: whether a write is allowed depends on the current state in a way that changes with every write. A bid is valid only if it exceeds the current highest bid by the minimum increment, and only if the auction is still open. Unlike inventory, you cannot split the state: there is exactly one highest bid. Unlike ticketing, the contention is not “first one wins and the rest go away”; every accepted bid raises the bar for the next, and the losers may immediately bid again.
The bid as a conditional write
1
2
3
4
5
6
7
8
UPDATE auctions
SET high_bid = :amount, high_bidder = :user_id, bid_count = bid_count + 1
WHERE id = :auction_id
AND status = 'OPEN'
AND ends_at > now()
AND :amount >= high_bid + min_increment;
-- 1 row: accepted, and append to bids(auction_id, seq, user_id, amount, accepted_at)
-- 0 rows: outbid or closed; re-read and tell the bidder which
The status and the deadline are part of the predicate, which is what makes the close safe. The bid history table is append-only and becomes the audit log: in a dispute, it is the evidence.
Ordering and tie-breaking
Two bids of the same amount arrive within a millisecond. The rule must be explicit and must be the commit order at the serialization point: the first one the database (or the single writer) applies wins, the second fails the predicate because high_bid + min_increment has moved. Client timestamps are never used. This is also why auctions are a natural fit for a single writer per auction: a partitioned log keyed by auction ID gives every bid a sequence number, the consumer applies them in order, and the log is the history. At a few hundred bids per second on the hottest auction, either mechanism works; the log becomes worthwhile when you also want replay and a provable order.
Proxy bidding changes what a “bid” writes
On eBay-style auctions, the bidder submits a maximum, and the system bids on their behalf. The displayed price is the second-highest maximum plus one increment. That means:
- The row stores the leader’s hidden maximum in addition to the visible price.
- A new bid is compared to the hidden maximum, not the visible price. If it is lower, the visible price rises to just above it and the leader stays the leader; the new bidder is told they were outbid instantly.
- One incoming bid can produce several logical bid events (yours, then the proxy’s response), all inside one atomic write.
- Ties on maximum go to the earlier maximum, which again means commit order.
Proxy bidding also reduces contention: a bidder who sets a high maximum does not need to hammer the row in the last seconds.
Sniping and the close moment
The peak in an auction is not the start; it is the last few seconds, when bidders who were waiting all fire at once to deny others time to respond. That is sniping, and the platform chooses how to handle it:
| Close rule | Behavior | Effect on the system |
|---|---|---|
| Hard close | Ends at a fixed instant; the last valid bid before it wins | Extreme spike in the final seconds; latency to the serialization point decides winners, so fairness depends on geography |
| Soft close (extension) | Any bid in the last N minutes extends the end by N minutes | Spike flattens; the deadline becomes part of the state the bid updates (ends_at = greatest(ends_at, now() + interval '2 minutes')) |
| Sealed bid | Bids are hidden until the close | No sniping and no visibility fan-out; contention collapses to “append bid, compute winner at close” |
The research on this is real: Roth and Ockenfels compared eBay (hard close) with Amazon’s former auctions (soft close) and found far more last-minute bidding under the hard close. In an interview, the soft close is the answer that both improves fairness and removes the system’s worst spike.
The closing write is the true serialization point
The close is a write, not a timer: one statement sets status = 'CLOSED' where it is still OPEN and ends_at <= now(). Because every bid is conditional on status = 'OPEN' AND ends_at > now(), there is no window where a bid is accepted after the close, and the clock that matters is the one at the store, not the client’s or the app server’s. A scheduler triggers the close (that part is the coordination shape), but correctness comes from the predicate. After the close, the winner is notified, payment is requested, and a non-paying winner triggers a second-chance offer to the runner-up, which is a small saga.
Bid visibility is a different shape
Thousands of watchers want to see the current price update within a second. That is not contended writes; it is fan-out. Publish accepted bids from the change stream to a pub/sub channel per auction and push to watchers over WebSockets. Say the fairness caveat: a bidder with slower updates sees an older price, so the write must always validate against the real current value and return the actual state on rejection.
Side by side
| Ticketing | Inventory | Auctions | |
|---|---|---|---|
| Unit of contention | One seat (distinct) | One SKU count (fungible) | One auction’s high bid (single value) |
| Can the hot key be split? | No; partition by section at most | Yes: buckets, escrow, tokens | No |
| What a write checks | Seat available or hold expired | Enough units available | Amount beats current high plus increment; still open |
| Losers | Try another seat | Sold out, or backorder | Bid again, higher |
| When the peak happens | On-sale start | Drop or flash-sale start | Final seconds |
| Oversell / wrong-winner tolerance | Zero | A business decision, often small but nonzero | Zero |
| Primary peak control | Waiting room | Waiting room plus in-memory gate | Soft close; proxy bidding |
| Default mechanism | Conditional update with hold expiry | Atomic decrement with reserve and commit | Conditional update, or single writer per auction |
| The twist | Holds, fencing token, payment saga | Reserve timing, split counters, multi-warehouse | Proxy maximums, tie-breaking, the closing write |
Where each example stops being this shape
- Ticketing becomes fan-out when you push live seat-map updates to 100,000 admitted buyers, and coordination in the payment saga and the hold sweeper. The seat map for browsers is a read-heavy lookup.
- Inventory becomes write-heavy ingest on the warehouse side: every scan, pick, receipt, and adjustment is an event, and the stock ledger is an append-only stream aggregated into the count. The “in stock” badge on the product page is read-heavy, which the read-heavy post discusses as the part of the catalog that must be split off. Most SKUs, most of the time, are not contended at all.
- Auctions become fan-out for bid visibility to watchers, and real-time for live auctions with an auctioneer and streamed bidding, or for ad exchanges, where a sealed-bid auction is decided in under 100 milliseconds by one process and there is no shared hot row at all.
The sentence to say in the interview
This is a contended-writes problem. The unit of contention is the [seat / SKU count / auction], and one key can take roughly a thousand conditional updates a second, a few hundred with synchronous replication. The peak is [number], so I will use admission control to hold the write rate under that, make every decision a single conditional statement so no lock outlives it, and serve the display from a cache that is allowed to be a second stale. [For fungible stock: I can split the counter if the peak exceeds one row.] Before I go further: what does the business want if we sell one more than we have?
The question that impresses is the last one. It shows you know that the cost of zero oversell is real and that it is the product’s call.
Follow-Up Questions to Expect
- “Why not just use serializable isolation?” It does not raise the hot row’s throughput; it lowers it and adds retries. The invariant fits in one conditional statement, which is correct at read committed. See the isolation post.
- “Why not a distributed lock in Redis?” It adds a second source of truth and still serializes on one key. If the store can do a conditional write, the lock adds failure modes without adding throughput.
- “The flash-sale gate is in Redis. What if it loses writes on failover?” A few units can be sold twice. Either accept it and reconcile (cancel and apologize, within the business’s oversell tolerance), or put a durable single writer behind the gate. State which you chose and why.
- “How do you show ‘only 3 left’?” From the read model, rounded and stale. Some sites deliberately show a coarse value (“low stock”) because a precise number invites refresh storms.
- “An auction’s last ten seconds get 5,000 bids a second. What breaks?” A single synchronous row tops out near that. Options: soft close to flatten it, a single writer per auction that rejects most bids cheaply in memory (anything below the current high plus increment never needs a durable write), and proxy bidding to reduce how many humans are hammering.
- “How do you test it?” A load test that fires the peak at one key and asserts the invariant: units sold plus units available equals the starting count, exactly one winner per auction, no seat sold twice. Then a chaos test that kills the writer mid-peak and checks the same invariant.
Key Takeaways
- Contended writes means many writers on one key at one moment. Find the unit of contention first.
- One key’s throughput is one divided by its hold time: about 1,000 per second for a single conditional statement, a few hundred with synchronous replication, a third of one if you hold it across a payment.
- The default mechanism is a single atomic conditional statement. Optimistic retries collapse on a hot key; pessimistic locks must never span a slow call.
- Shape the peak: admission control sets the write rate, holds with expiry stop abandoned sessions from eating stock, and batching in a single writer multiplies throughput.
- Ticketing: distinct units, cannot split the row, so the waiting room sets the rate.
- Inventory: fungible units, so split the counter, reserve then commit, and treat oversell tolerance as a business decision.
- Auctions: validity depends on the current value, the close is a write in every bid’s predicate, and commit order breaks ties.
- The display is eventual; the decision is linearizable.
Further Reading
- Martin Kleppmann, Designing Data-Intensive Applications, chapter 7 (preventing lost updates, write skew, and the atomic-operation toolbox)
- Patrick O’Neil, “The Escrow Transactional Method,” ACM Transactions on Database Systems, 1986 (the formal basis for splitting a fungible quantity)
- Martin Fowler, The LMAX Architecture (a single writer, in memory, batching its way past the hot-key ceiling)
- Alvin Roth and Axel Ockenfels, “Last-Minute Bidding and the Rules for Ending Second-Price Auctions: Evidence from eBay and Amazon Auctions on the Internet,” American Economic Review, 2002 (hard close versus soft close, measured)
- Pat Helland, “Life beyond Distributed Transactions: an Apostate’s Opinion,” CIDR 2007 (why the unit of contention becomes the unit of partitioning)
- Cloudflare, Waiting Room documentation (what managed admission control does)
The Six Shapes
Each shape from the first five minutes post has its own deep dive:
| Shape | Examples | Where the difficulty is |
|---|---|---|
| Read-heavy lookup | URL shortener, product catalog, profile service | Cache and read path |
| Write-heavy ingest | Metrics, logging, clickstream | Ingest path, storage tiering, downsampling |
| Fan-out | News feed, notifications, chat | Delivery to many recipients; hot users |
| Contended writes (you are here) | Ticketing, inventory, auctions | Consistency on a hot row; the peak moment |
| Coordination | Scheduler, workflow engine, locks | Leases, exactly-once claims, failure recovery |
| Real-time | Ride matching, live location, collaborative editing | Latency, ordering, stateful connections |