Post

System Shapes: Read-Heavy Lookup (URL Shortener, Product Catalog, Profile Service)

A read-heavy lookup system spends all of its design effort on the read path. Here is what defines the shape, the cache arithmetic that matters, the six standard problems, and how a URL shortener, a product catalog, and a profile service each bend the same architecture in a different direction.

System Shapes: Read-Heavy Lookup (URL Shortener, Product Catalog, Profile Service)

The first five minutes post says to name the shape of the system in the first minute, because the shape tells you where the difficulty lives. This is the first of six posts that take one shape each and go deep: what defines it, the generic architecture with real numbers, the standard problems, and then each of the classic examples, with the one property that makes it different from its siblings.

The read-heavy lookup is the shape most candidates think they already know. “Put a cache in front of it” is the whole answer for many of them, and it is not enough. The writes are trivial, so the entire interview is about the read path: how many cache layers, how they stay correct, what happens when a cache misses, expires, or dies, and who notices when it is stale. Three systems share this shape (a URL shortener, a product catalog, and a profile service), and each one breaks the generic design in a different place. Knowing where is what turns “add a cache” into a design.

Where This Shows Up

The prompts that are this shape, more or less directly:

  • “Design a URL shortener” (or Pastebin, or a QR code redirect service).
  • “Design the product detail page for an e-commerce site.”
  • “Design a user profile service that every other team calls.”
  • “Design a configuration or feature flag service.”
  • “Design a DNS-like lookup service” or “a geolocation-by-IP lookup.”

And the follow-ups that pull any design into this shape:

  • “Reads are a thousand times writes. What changes?”
  • “What is your cache hit rate, and what happens to the database if it drops?”
  • “One key gets ten percent of all traffic. Now what?”
  • “A user edits their profile and refreshes. What do they see?”
  • “The cache cluster restarts. Walk me through the next sixty seconds.”
  • “How do you invalidate it?”

What They Are Really Checking

  1. Can you recognize the shape and spend your time accordingly? The write path of a URL shortener is one insert. A candidate who spends twenty minutes on it has optimized the wrong half of the system.
  2. Can you do the cache arithmetic? Hit rate times read rate equals database load. Saying “99 percent hit rate leaves 400 reads per second for the database” is a design decision. Saying “we add Redis” is not.
  3. Do you know how caches fail? Stampedes, hot keys, negative lookups, cold starts, and the cache-aside race. Each has a standard fix, and the interviewer wants to hear you name the failure before they do.
  4. Can you say who notices staleness? Staleness is usually fine. The skill is spotting the one place where it is not: the owner reading their own edit, the price at checkout, the link that was taken down for abuse.
  5. Do you know where the shape ends? Every one of these systems has a part that is not read-heavy (click analytics, inventory, the social graph). Spotting it and moving it out of the read path is the senior move.

The Gotchas

Gotcha 1: Adding a cache without the hit-rate arithmetic. “We put Redis in front of the database” is a box, not a design. The number that matters is the miss rate times the read rate, because that is the database load. At 40,000 reads per second, the difference between a 99 percent and a 90 percent hit rate is the difference between one database and five. Say the number.

Gotcha 2: Assuming the cache never fails. A cache that absorbs 99 percent of reads is now a dependency the database cannot survive without. When the cache cluster restarts or a deploy flushes it, 100 percent of reads hit a database sized for 1 percent. Either the database tier can absorb the full read rate for a few minutes, or you need load shedding and a gradual warm-up. Pick one out loud.

Gotcha 3: Treating cache-aside as correct. Reader A misses, reads the old value from the database, and pauses. Writer B updates the database and deletes the cache key. Reader A wakes up and writes the old value into the cache, where it stays until the TTL (time to live) expires. Delete-on-write does not close this race. A short TTL bounds it; a version check or a lease closes it.

Gotcha 4: The cache stampede. A hot key expires. Ten thousand requests miss at the same instant and all of them go to the database to rebuild the same value. The fix is request coalescing (one rebuild per key, everyone else waits for it), early probabilistic refresh, and jittered TTLs so keys do not all expire together.

Gotcha 5: Forgetting negative lookups. A request for a key that does not exist is always a cache miss and always a database read. Bots scanning random short codes, or an attacker requesting random user IDs, will walk straight past a 99 percent hit rate. Cache “not found” with a short TTL.

Gotcha 6: Ignoring the person who just wrote. Staleness of a few seconds is invisible to everyone except the user who made the change. They edit their name, refresh, see the old one, and file a bug. Read-your-writes for the author is a requirement even when eventual consistency is fine for everyone else.

Gotcha 7: One TTL for the whole record. A product record holds a description that changes yearly, a price that changes daily, and a stock count that changes every second. Any single TTL is wrong for two of the three. Split the record by rate of change and cache each part on its own terms.

Gotcha 8: Making the decision from the cache. The product page can show a cached price. The checkout cannot charge it. The display is eventual; the decision is read from the source of truth. A design that charges whatever the cache said has a pricing bug waiting for the first flash sale.

Gotcha 9: Putting analytics on the read path. A URL shortener that increments a click counter in the database on every redirect has turned a read-heavy system into a write-heavy one, on the hot path. Emit an event to a queue and aggregate somewhere else.

Gotcha 10: Caching per-viewer results. A profile looks different to a friend, a stranger, and a blocked user. Caching the rendered view per viewer multiplies the cache size by the number of viewers and leaks private fields when the key is wrong. Cache the raw record once and apply visibility rules after the read.

How to Answer

Step 1: Recognize the shape

Five properties define it. If the prompt has most of them, say “this is a read-heavy lookup” in the first minute.

Property Typical value What it implies
Read to write ratio 100:1 to 10,000:1 Writes are trivial. The design effort goes on the read path
Access pattern By key: a short code, a SKU (Stock Keeping Unit), a user ID Point lookups. No joins or range scans on the hot path, so any key-value store or a single indexed table works
Record size Hundreds of bytes to a few kilobytes The working set is modest and often fits in memory
Popularity Skewed (Zipf): a few percent of keys take most of the traffic Caching works extremely well, and hot keys are a risk
Staleness tolerance Seconds to hours for most readers Caches with TTLs are acceptable; you only need to find the exceptions

The last row is the one to probe. Ask: “Is anything here mutable, and who notices when the cache is stale?” The answer decides how hard invalidation is, and invalidation is where these designs differ.

Step 2: The generic architecture

flowchart LR
    C[Client] --> E[CDN / edge<br/>optional]
    E --> A[App servers<br/>stateless, small in-process cache]
    A --> R[(Distributed cache<br/>Redis, cache-aside)]
    A --> P[(Primary DB<br/>source of truth)]
    A --> RR[(Read replicas)]
    P -. replication .-> RR

Each layer absorbs part of the traffic and costs something in correctness:

Layer Typical latency What it absorbs What it costs
CDN (Content Delivery Network) / edge 10 to 50 ms from the user, zero load on you Public, identical-for-everyone responses Purging is slow (seconds to minutes) and you lose per-request visibility
In-process cache ~100 ns The hottest few thousand keys Every server has its own copy; invalidation must reach all of them, so keep TTLs to seconds
Distributed cache (Redis-class) ~0.5 ms in the same data center The working set One more system to keep consistent; the cache-aside race
Read replicas 1 to 5 ms Misses, at scale Replication lag: reads may be behind the primary
Primary database 1 to 5 ms Writes, and reads that must be current The only place that is always right, and the scarcest capacity

The pattern on the hot path is cache-aside: look in the cache; on a miss, read the database and populate the cache with a TTL. On a write, update the database and delete the cache key. Simple, and it has the race in Gotcha 3, which you should mention unprompted.

Step 3: Do the arithmetic that matters

Take 40,000 reads per second at peak and 100 writes per second. The writes need no discussion: one primary handles them with room to spare. The reads are the design.

Hit rate Reads reaching the database What the database tier needs
99.9% 40 per second Nothing special
99% 400 per second One primary handles it easily
95% 2,000 per second Still one node, with less headroom
90% 4,000 per second Read replicas, or a bigger node
0% (cache down) 40,000 per second Four or five replicas at roughly 10,000 point reads each, per the numbers post

Two conclusions to say out loud. First, every point of hit rate is worth real hardware, so the things that raise it (longer TTLs, negative caching, a local cache for hot keys) are worth discussing. Second, the bottom row is the failure mode. Either provision for it, or have a plan: shed load, serve stale, and warm the cache gradually.

Size the cache the same way. Say 100 million records at 1 KB each is 100 GB. With Zipf-shaped traffic, the hottest 20 percent of records (20 GB) serves well over 80 percent of reads, which fits in one Redis-class node with room for replicas. Then say “so”: the cache is sized for the working set, not the dataset, and a cluster of three nodes gives headroom and failover rather than capacity.

Step 4: The standard problems

Every read-heavy design meets the same six problems. Name them, give the fix, and move on.

Problem What goes wrong Usual fix
Invalidation The cache serves old data after a write A TTL as the backstop, plus delete-on-write, or event-driven invalidation from a change stream
Cache stampede A hot key expires and thousands of requests hit the database at once Request coalescing (single-flight), early probabilistic refresh, jittered TTLs, serve-stale-while-revalidating
Hot key One key overloads one cache node, because a key lives on exactly one shard A small in-process cache in front, or replicate the key across nodes with a suffix (key#1 to key#8)
Negative lookups Misses for nonexistent keys always reach the database, and are an attack vector Cache “not found” with a short TTL; a Bloom filter for very large keyspaces
Replica lag A read from a replica right after a write returns the old value Read-your-writes routing: send the writer’s reads to the primary for a short window
Cold start A deploy or a cache flush makes everything miss Warm the cache from logs of hot keys, roll out gradually, and cap concurrency to the database

The invalidation options deserve one more table, because the right choice differs per example:

Strategy How it works Staleness bound Good for
TTL only Entries expire after N seconds N seconds Data where N seconds of staleness is fine for everyone
Delete on write The writer deletes the key after the database commit The cache-aside race window, bounded by the TTL Single-writer data with one cache
Write-through The writer updates cache and database together Near zero, if the write succeeds in both Small values written by the owner
Event-driven A change stream (CDC, Change Data Capture, or an outbox event) drives purges in the cache, CDN, and search index Pipeline lag, usually seconds Data with many derived copies

Example 1: URL shortener, the purest case

Special property: the mapping is immutable. A short code never changes its target. That means the cache can never be wrong about a code that exists, so invalidation only matters for the rare exceptions: deletion, expiry, and abuse takedowns. Everything else follows from this.

Caching is easy and the hit rate is very high. Use long TTLs (a day or more) because there is nothing to go stale. The traffic is extremely skewed: a link shared in a viral post gets millions of clicks in an hour and almost nothing a week later, and the viral links are exactly the ones sitting in cache. Hit rates above 99 percent are realistic. The URL shortener post works the full numbers: about 4,000 reads per second at peak, and a few gigabytes of hot set that a single cache node holds.

301 versus 302 is a read-path decision, not a detail.

Redirect What the browser does Effect on load Effect on the business
301 (Moved Permanently) Caches the redirect; repeat clicks never reach you Much lower No analytics on repeat clicks, and you cannot change or take down the link for anyone who cached it
302 (Found) or 307 Asks you every time Every click hits your servers Full analytics and full control, including takedowns

Most shorteners exist for the analytics, so 302 is the usual answer. Say what you are choosing and why. The browser cache is the outermost cache layer in this design, and you are deciding whether to use it.

Takedowns are the one invalidation that must be synchronous. When a link is reported as phishing, delete the cache entry and purge the CDN in the same request that marks the row. A takedown that waits for a day-long TTL is a takedown that did not happen. This is also a second reason for 302: a 301 sitting in a browser cache cannot be taken down at all.

Negative caching matters more here than anywhere. The keyspace of seven base62 characters is about 3.5 trillion codes, almost all unused. Bots and scanners request random codes, and every one is a guaranteed miss. Cache the 404 for a minute, rate limit per client (see the rate limiter post), and for a very large keyspace consider a Bloom filter that answers “definitely not present” without touching the database.

The remaining hard part is on the write side. Generating unique short codes without a coordination bottleneck (counter plus base62, pre-allocated ranges, random with retry, or hash with a collision check) is covered in depth in the URL shortener post. It is the one place the write path deserves time, and it is still a few hundred writes per second.

The trap: click analytics. Analytics is a write-heavy ingest problem bolted onto a read-heavy system. If the redirect handler increments a counter in the database, every read is now also a write, and the 99 percent hit rate no longer protects the database. Keep it off the redirect path: emit a click event to a queue, return the redirect, and aggregate elsewhere.

flowchart LR
    U[Browser] -->|GET /aB3xK9q| A[Redirect service]
    A -->|lookup| C[(Cache)]
    C -. miss .-> D[(DB)]
    A -->|302 Location| U
    A -. click event, async .-> Q[[Queue]]
    Q --> AG[Aggregator<br/>write-heavy ingest]

Example 2: Product catalog, mutable with mixed freshness

Special property: one product record mixes data that changes at very different rates. The title changes once a year, the price changes daily or during a sale, and the stock count changes with every purchase. Any single caching policy is wrong for most of the record. The design move is to split the record by rate of change and serve each part from the layer that matches its staleness tolerance.

Data Changes Staleness tolerance Served from
Title, description, images, specs Rarely Hours CDN for images; cache with a long TTL for the rest
Price, promotions Daily, or every few minutes in a sale Minutes on the page; never at checkout Cache with a short TTL, or event-driven invalidation
Inventory Constantly “In stock” on the page can lag; the purchase cannot A separate service. This part is not read-heavy at all
Reviews and ratings Continuously, in aggregate Hours for the average; minutes for “your review” Precomputed aggregate, cached

The display is eventual; the decision is linearizable. The product page shows a cached price and a cached “in stock” badge. The checkout re-reads the price and attempts a conditional stock decrement against the source of truth. If the price changed since the page loaded, checkout says so. This is the same split as the seat map in the ticket booking post, and the consistency post explains why it is the most common pattern in system design: a few linearizable decisions surrounded by eventually consistent views.

Invalidation is event-driven, because there are many copies. A product exists in the database, the distributed cache, the CDN, the search index, and maybe a recommendations feature store. A merchant edit must reach all of them. Delete-on-write from the API cannot reliably do that, because the write and the purges are a dual write. Instead, publish a ProductUpdated event through an outbox or a CDC stream (see the sagas and outbox post), and let each copy’s owner consume it: one consumer purges the cache and CDN, another reindexes search. The staleness bound is the pipeline lag, typically a few seconds, plus the TTL as a backstop if an event is lost.

flowchart LR
    M[Merchant edit] --> DB[(Catalog DB)]
    DB -->|outbox / CDC| K[[ProductUpdated]]
    K --> P1[Cache purger] --> R[(Cache)]
    P1 --> CDN[CDN purge]
    K --> P2[Indexer] --> S[(Search index)]

Browse and search are not key lookups. “Red running shoes under $50, size 10” is a query with filters, facets, and ranking. It goes to a search index (Elasticsearch or OpenSearch) fed by the same change stream: another eventually consistent read model. The product page itself is still a lookup by SKU. Do not try to serve faceted search from the key-value cache.

Flash sales are the stampede case. At noon thousands of hot prices change at once, exactly when traffic peaks. Pre-warm the new prices (write them with a future effective time) rather than purging and letting the first readers rebuild.

Example 3: Profile service, the user sees their own writes

Special property: the person who writes the data reads it immediately. For the rest of the world, a profile that is ten seconds stale is invisible. For the owner, who just changed their name and pressed refresh, it is a broken product. This is the read-your-writes guarantee from the consistency post, and it is the decision this example forces.

Three ways to provide it, in order of preference:

Mechanism How it works Cost
Update the cache in the write path On edit, write the database, then overwrite (not just delete) the cache entry with the new value and a version Two writes; needs a version so a slow reader cannot overwrite it with an older value
Route the owner to the primary For 30 to 60 seconds after an edit, that user’s reads of their own profile skip the cache and replicas Sticky routing; a little extra primary load, limited to recent editors
Version token The write returns a version; the client sends it on later reads; a replica or cache that is behind that version falls back to the primary A token in the session; most precise, most plumbing
sequenceDiagram
    participant U as Owner
    participant API
    participant P as Primary DB
    participant R as Replica
    U->>API: PATCH /me {name}
    API->>P: UPDATE ... version = 42
    API-->>U: 200 {version: 42}
    U->>API: GET /me (min_version 42)
    API->>R: read profile
    R-->>API: version 41 (lagging)
    API->>P: fall back to primary
    P-->>API: version 42
    API-->>U: new name

Everyone else reads from the cache with a TTL of a minute or so, and nobody can tell.

Fan-in reads: the N+1 problem. A feed or comment page shows 50 authors. A naive implementation calls the profile service 50 times, sequentially, per page view. At 0.5 ms per cache hit plus a network hop, that is 50 to 100 ms of latency for something that should take 2 ms. The profile service must offer a batch get (GET /profiles?ids=...), and the cache must support multi-get. This is the most common performance bug in services built around this shape.

Hot keys: celebrity profiles. A celebrity profile might get 50,000 reads per second, all on one key, all on one Redis shard. Put a small in-process cache in front with a TTL of a few seconds. With 200 app servers, each refreshing a hot key every five seconds, that key costs Redis 40 reads per second instead of 50,000. Seconds of staleness on a celebrity’s bio is harmless.

Privacy: visibility depends on the viewer. The same profile shows a phone number to friends, a public subset to strangers, and nothing to a blocked user. Cache the raw profile once per user, then apply visibility rules in the service after the read, using the viewer’s relationship (which may itself be a cached lookup). Do not cache per-viewer renderings: the cache grows by the number of viewer combinations, and a key mistake serves one viewer’s view to another, which is a privacy incident.

Deletion is invalidation with legal consequences. Under GDPR (General Data Protection Regulation), “delete my account” must reach the database, every cache layer, the CDN copy of the avatar, the search index, and any replicas or analytics copies. A TTL is not good enough if it is a week long. Drive deletion through the same event stream as updates, and give it an explicit completion check rather than trusting that every consumer ran.

Side by side

  URL shortener Product catalog Profile service
Special property Immutable mapping Mixed rates of change in one record The writer reads immediately
Mutability Never, except delete and takedown Occasionally, at different rates per field Edited by the owner
Invalidation difficulty Almost none; takedowns synchronous Event-driven, per field, many copies Must provide read-your-writes
Hot keys Viral links Sale items, launches Celebrities
The twist 301 vs. 302; ID generation Split off inventory; search index; display vs. decision Read-your-writes; N+1 fan-in; privacy; deletion
Where it stops being read-heavy Click analytics Inventory and checkout The social graph and activity

Where each example stops being this shape

Every one of these systems has a part that is a different shape. Naming the boundary, and keeping that part off the read path, is what the interviewer is waiting for.

  • URL shortener → write-heavy ingest. Click analytics is an event per redirect, at the full read rate, with aggregation by link, time, country, and referrer. That is a metrics pipeline: a queue, a stream processor, and a time-series or columnar store. The redirect emits the event and forgets it.
  • Product catalog → contended writes. Inventory during a sale is many buyers decrementing the same counter. That is a conditional update or a reservation against the source of truth, the same problem as the last seat in the ticket booking post. The catalog shows “in stock” from cache; the inventory service decides.
  • Profile service → fan-out. The profile itself stays read-heavy. But “who follows whom” and “what my friends did” are the social graph and the activity feed, and those are a fan-out problem with hot users. If the interviewer adds “show recent activity on the profile,” say that you have moved shapes.

The sentence to say in the interview

This is a read-heavy lookup: reads outnumber writes by a thousand to one, every read is a point lookup by key, and popularity is skewed, so I will spend the deep dive on the read path. Cache-aside in Redis with a small in-process cache for hot keys; at 40,000 reads per second a 99 percent hit rate leaves 400 for the database, and I will size the database so a cold cache does not take it down. The question that decides the rest is: is anything here mutable, and who notices when the cache is stale?

Then answer that question for the system in front of you. For a shortener: nothing is mutable except takedowns, which purge synchronously. For a catalog: the price and stock are, so I split them out and the checkout reads the source of truth. For a profile: the owner notices, so the owner gets read-your-writes.

Follow-Up Questions to Expect

  • “Why not just make the TTL very short?” A short TTL lowers the hit rate, which raises database load, and it does not fix read-your-writes for the owner (one second is still long enough to refresh). TTLs are a backstop for missed invalidations, not the invalidation strategy.
  • “How do you prevent a stampede when a hot key expires?” Single-flight: the first miss takes a short lock or lease on the key and rebuilds it; other requests wait briefly or get the stale value. Add jitter to TTLs, and for the hottest keys refresh in the background before expiry.
  • “The cache cluster just restarted. What happens?” Every read misses. If the database tier cannot absorb the full read rate, shed load (serve errors or stale data for the long tail), limit concurrent rebuilds per key, and warm the hot keys first from a list of recently popular ones.
  • “Would you use a CDN for the profile?” For the public avatar image, yes. For the profile JSON, usually not: it is per-viewer after visibility rules, and CDN purges are too slow for read-your-writes.
  • “How do you go multi-region?” Reads are easy: a cache and replicas in every region, with regional TTLs. Writes go to a home region per key. Read-your-writes across regions means routing the owner to the home region for a short window after an edit, or carrying a version token.

Key Takeaways

  • A read-heavy lookup has a high read to write ratio, key-based access, small records, skewed popularity, and tolerance for staleness. Name it in the first minute and spend the deep dive on the read path.
  • The number that matters is hit rate times read rate. At 40,000 reads per second, 99 percent is one database and 90 percent is several. Plan for the cache being down, too.
  • Six standard problems: invalidation, stampede, hot keys, negative lookups, replica lag, cold start. Know the fix for each.
  • Cache-aside has a race. A TTL bounds it; a version check closes it.
  • URL shortener: immutable, so caching is easy. The decisions are 301 versus 302, synchronous takedowns, and keeping analytics off the redirect path.
  • Product catalog: split the record by rate of change. The display is eventual; the checkout reads the source of truth. Invalidate through events because there are many copies.
  • Profile service: the owner needs read-your-writes. Batch gets for fan-in, a local cache for celebrities, visibility rules after the read, and deletion that reaches every copy.
  • The question that impresses: “Is anything here mutable, and who notices when the cache is stale?”

Further Reading

  • Martin Kleppmann, Designing Data-Intensive Applications, chapter 5 (replication lag and read-your-writes) and chapter 11 (derived data and change streams)
  • Rajesh Nishtala et al., Scaling Memcache at Facebook (NSDI 2013): leases for the cache-aside race and for stampedes, regional invalidation
  • Nathan Bronson et al., TAO: Facebook’s Distributed Data Store for the Social Graph (USENIX ATC 2013): a read-optimized cache tier in front of a database, at the scale of billions of reads per second
  • Matt Brown, Caching challenges and strategies (Amazon Builders’ Library): cold starts, negative caching, and the cache as a dependency
  • Andrea Vattani, Flavio Chierichetti, and Keegan Lowenstein, Optimal Probabilistic Cache Stampede Prevention (VLDB 2015): the early-refresh technique
  • Related posts on this blog: URL shortener, consistency, numbers, sagas and the outbox

The Six Shapes

Each shape gets its own deep dive. Start from the first five minutes post for how to classify a prompt.

Shape Examples Post
Read-heavy lookup URL shortener, product catalog, profile service You are here
Write-heavy ingest Metrics, logging, clickstream System Shapes: Write-Heavy Ingest
Fan-out News feed, notifications, chat System Shapes: Fan-Out
Contended writes Ticketing, inventory, auctions System Shapes: Contended Writes
Coordination Scheduler, workflow engine, locks System Shapes: Coordination
Real-time Ride matching, live location, collaborative editing System Shapes: Real-Time
This post is licensed under CC BY 4.0 by the author.