System Shapes: Write-Heavy Ingest (Metrics, Logging, Clickstream)
Metrics, logging, and clickstream look alike: millions of small events a second, written once, read rarely. Here is the generic ingest architecture with the arithmetic, the standard problems, and the property that makes each example different: cardinality, burstiness, and late events.
The first five minutes post asks you to name the shape of the system before you draw anything. This post is the deep dive on the second shape: write-heavy ingest. Metrics, logging, and clickstream are its standard examples, and they show up constantly, either as the whole question (“design a metrics system”) or as a side path bolted onto another design (“and we want click analytics”).
The shape is the mirror image of the read-heavy lookup. There, writes were trivial and the design lived in the cache. Here, almost every request is a write, nobody reads any single event, and the reads that do happen are large aggregations over billions of rows. The design lives in three places: the ingest path, the storage tiers, and the reduction of data over time. Each of the three examples stresses a different one, and knowing which is what this post is for.
Where This Shows Up
- “Design a metrics and monitoring system like Datadog or Prometheus.”
- “Design a centralized logging system for 10,000 services.”
- “Design clickstream analytics for a large website or mobile app.”
- “Design an ad click counter.” (Clickstream with money attached.)
- “Design the analytics for this URL shortener.” (The ingest path hiding inside a read-heavy question.)
- “We ingest a million events a second. What happens when the database falls behind?”
- “During an incident, log volume goes up ten times. What breaks?”
- “How long do you keep the data, and what does it cost?”
What They Are Really Checking
- Do you decouple the producer from the store? Producers cannot wait on a database. The interviewer wants a durable buffer (a log such as Kafka) between them, and a reason for it.
- Can you do the volume arithmetic? Events per second times bytes per event, per day, per retention period, after compression and replication. The storage choice and the bill fall out of that number.
- Do you have a plan for overload? Backpressure, dropping, or sampling. Each is a legitimate answer for some data, and “the queue absorbs it” is only true until the queue fills.
- Do you reduce data over time? Nobody needs per-second data from two years ago. Tiering and downsampling are the design, not an afterthought.
- Do you know what makes each example different? Cardinality for metrics, bursts and full-text search for logs, late and duplicate events for clickstream. A generic “Kafka plus a database” answer that ignores these is the mid-level answer.
- Do you separate the write model from the read model? Ingest is append-only and partitioned by arrival. Queries are aggregations by time and dimension. One storage layout rarely serves both.
The Gotchas
Gotcha 1: Writing each event synchronously to the database. A million single-row inserts per second is a million round trips and a million index updates. The fix is batching at every hop, and a log in front of the store so producers never wait on it.
Gotcha 2: Treating the queue as infinite. Kafka absorbs bursts, but only for as long as its retention and disk allow. If consumers fall behind for longer than that, data is lost silently. Say what happens when the buffer fills: block, drop, or sample, and for which data.
Gotcha 3: Choosing the partition key without thinking. Partition by service and one noisy service melts one partition. Partition randomly and you lose ordering and locality that sessionization or compression needs. The key is a decision with a cost on each side.
Gotcha 4: Ignoring cardinality in metrics. Adding a user_id label to a metric turns one time series into a million. Storage of the samples is cheap; the index of series is not. This is the most common way real metrics systems fall over.
Gotcha 5: Averaging averages when downsampling. A one-hour rollup that stores the mean of one-minute means is wrong whenever the minutes have different counts, and percentiles cannot be averaged at all. Store sum, count, min, max, and a mergeable histogram or sketch.
Gotcha 6: Indexing every word of every log line. A full-text inverted index can cost as much storage as the raw logs and most of the ingest CPU. Index the structured fields you filter by (service, level, trace ID) and scan the rest.
Gotcha 7: Designing logging for the average day. Log volume spikes during incidents, which is exactly when you need the logs. If the pipeline drops data or falls an hour behind under a 10x burst, it fails at its one job.
Gotcha 8: Using processing time for clickstream. Mobile clients batch and go offline, so events arrive minutes or days late. Counting by arrival time puts Tuesday’s purchases in Wednesday’s report. Window by event time, with a watermark and a lateness policy.
Gotcha 9: Assuming exactly-once delivery. Every hop retries, so every hop can duplicate. Metrics tolerate it naturally, logs mostly tolerate it, clickstream feeding billing does not. Dedup by a client-generated event ID where counts matter.
Gotcha 10: Forgetting deletion in an append-only system. Retention is easy if storage is partitioned by time: drop old partitions. A user’s privacy deletion request is hard, because their events are scattered across thousands of immutable files. Plan for it before the first byte lands.
How to Answer
What defines the shape
| Property | Typical value | Consequence |
|---|---|---|
| Write to read ratio | 1,000:1 or more by request count | The write path is the hot path; reads are rare batch queries |
| Access pattern | Append-only, by time; no updates, no point reads | Log-structured storage and time-partitioned files |
| Event size | 100 bytes to 2 KB | Per-request overhead dominates unless you batch |
| Volume | 100,000 to 10 million events per second | Tens of terabytes per day raw; storage cost drives the design |
| Skew | Some sources are 100x louder than others; bursts of 5x to 10x | Hot partitions; buffer sizing |
| Consistency tolerance | Very high. Seconds of delay, small loss or duplication often acceptable | You can trade correctness for availability, deliberately and per data type |
| Value over time | Falls steeply with age | Tiering and downsampling |
Say the last row out loud: an event goes from “I am debugging with it” to “one row in a quarterly trend” within days, and storage should follow that curve.
The generic architecture
flowchart LR
P[Producers<br/>agents, SDKs, browsers] -->|batched| C[Collectors<br/>stateless, validate, enrich]
C -->|batched, keyed| K[(Durable log<br/>Kafka, partitioned)]
K --> S1[Stream processor<br/>aggregate, dedup, window]
K --> S2[Sink to hot store<br/>TSDB / search index / OLAP]
K --> S3[Sink to object storage<br/>Parquet, columnar, cold]
S1 --> H[(Hot store<br/>days)]
S2 --> H
H -->|age out, downsample| W[(Warm<br/>weeks to months)]
S3 --> O[(Cold: object storage<br/>years)]
H --> Q[Query layer]
W --> Q
O --> Q
Five decisions make this a design rather than a diagram:
- Batch at every hop. The client sends 100 events per request, the collector writes 1,000 per Kafka produce call, the sink writes a file of millions. A batch trades a few hundred milliseconds of latency for orders of magnitude fewer operations.
- The log is the shock absorber. Kafka (or Kinesis, or Pulsar) decouples the rate producers write from the rate stores can absorb. It also lets you replay: a bug in the aggregator is fixed by rewinding the consumer offset and reprocessing.
- Multiple consumers, multiple read models. The same stream feeds a hot store (a TSDB, a search index, or an OLAP (Online Analytical Processing) store) for recent queries, an aggregator for dashboards, and object storage for the long tail. Each consumer is independent and can fall behind without hurting the others.
- Time-partitioned, append-only storage. Log-structured merge (LSM) trees, time-series databases (TSDBs), and columnar files all turn random writes into sequential ones. Partitioning by time makes retention a matter of deleting a directory.
- Tiers by age. Hot on SSD for days, warm on cheaper disks or object storage with a local cache for weeks, cold in object storage for years. Downsample as data moves down.
The arithmetic that matters
Use the back-of-envelope numbers: a day is 100,000 seconds.
1
2
3
4
5
6
7
8
1,000,000 events/s x 500 bytes = 500 MB/s raw
500 MB/s x 100,000 s/day = 50 TB/day raw
x 3 Kafka replicas = 1.5 GB/s written across the cluster
Kafka at ~100-200 MB/s per broker = ~10-15 brokers for writes, plus headroom
Compressed 10:1 (columnar, repetitive) = 5 TB/day stored
Hot tier: 7 days x 5 TB x 3 replicas = ~100 TB of SSD
Cold tier: 1 year x 5 TB = ~1.8 PB in object storage
1.8 PB x $0.02/GB-month = ~$36,000/month for cold storage
Two things to say. Without batching, a million events per second is a million requests per second, hundreds of app servers; with batches of 1,000 it is 1,000 requests per second, a handful. And the storage bill, not compute, eventually dominates, which makes retention an architecture decision.
The standard problems
| Problem | What goes wrong | Usual fix |
|---|---|---|
| Per-event overhead | One request and one index update per event; the store saturates at a fraction of the volume | Batch at the client, the collector, and the sink |
| Consumer falls behind | Lag grows; once it exceeds log retention, data is lost | Monitor consumer lag as the primary health metric; scale consumers up to the partition count; size retention for the longest outage you will tolerate |
| Burst exceeds capacity | Producers time out, agents fill their local buffers and drop | Backpressure for data that must arrive, dropping or sampling for data that need not; per-tenant quotas (a rate limiter at the collector) |
| Hot partition | One loud source overloads one partition and one consumer | Key by source plus a salt; split hot keys; or key randomly where ordering does not matter |
| Duplicates | Retries at each hop deliver the same event twice | Idempotent writes (metrics), tolerate (logs), dedup by event ID in a window (clickstream) |
| Retention by delete | Row-level TTL deletes create tombstones and compaction storms | Partition by time and drop whole partitions |
| Read model mismatch | A query for “errors by service last week” scans everything | Pre-aggregate in the stream; columnar storage partitioned by date; materialized rollups |
| Schema drift | A producer renames a field; downstream jobs break silently | Schema registry with compatibility checks at the collector |
Backpressure, dropping, or sampling
This choice comes up in every ingest interview, and the right answer is “it depends on the data,” followed by saying what it depends on.
| Strategy | What it does | Use it for | Cost |
|---|---|---|---|
| Backpressure | Slow the producer: block, return 429, let the agent buffer locally | Data that must arrive: billing events, audit logs, purchases | Producer latency grows; if the producer is the user’s request path, users feel it |
| Drop | Discard newest or oldest when the buffer is full | Debug logs, high-frequency gauges that will be resent in 10 seconds | Gaps, possibly during the incident you are investigating |
| Sample | Keep 1 in N, chosen deterministically (by trace ID or user ID) | High-volume, statistically useful data: info logs, page views at extreme scale | Rare events can vanish; counts must be scaled up by N |
The sentence that lands: “Errors and purchases get backpressure and never drop; debug logs drop first; page views sample deterministically by user so sessions stay whole.”
Example 1: Metrics, where the difficulty is cardinality
Special property: the data is tiny and highly compressible, but the number of distinct series is unbounded. A metric sample is a series identifier, a timestamp, and a number. Every sample for a given series looks like the last one. The problem is not the samples, it is the series.
A time series is a metric name plus a set of label values: http_requests_total{service="checkout", region="us-east", status="500"}. Each unique combination is a separate series with its own index entry and its own in-memory buffer.
1
2
3
10,000 hosts x 1,000 series per host = 10 million active series
Scraped every 10 seconds = 1 million samples/s
Compressed to ~1.4 bytes per sample = 1.4 MB/s, ~140 GB/day
The 1.4 bytes is from Facebook’s Gorilla paper: delta-of-delta timestamps (regular scrapes make most deltas zero) and XOR-encoded values. The samples are cheap. Now a developer adds a user_id label to http_requests_total, and the service has a million active users:
1
1 metric x 50 combinations of other labels x 1,000,000 users = 50 million new series
The series count has gone from 10 million to 60 million from one line of code. Each active series costs a few kilobytes of memory in the ingester for its buffer and index, so the ingester tier just needed several hundred gigabytes more memory, and every query that touches that metric now scans 50 million series. This is cardinality explosion, and it is how real metrics systems go down.
The decisions it forces:
| Decision | Choice | Why |
|---|---|---|
| Label policy | No unbounded values (user ID, request ID, URL with IDs, email) as labels | Those belong in logs or traces, which are indexed per event, not per series |
| Enforcement | Per-tenant series limits at ingest; reject new series past the limit and alert the owner | One team’s mistake should not take down everyone’s dashboards |
| Partition key | Hash of the series identifier | All samples of a series land on the same ingester, so it can compress them together and hold one buffer |
| Write path | Ingester keeps recent samples in memory with a write-ahead log (WAL), flushes compressed blocks every hour or two | Compression needs many consecutive samples; the WAL covers crashes |
| Duplicates | Last write wins on (series, timestamp) | The write is naturally idempotent; retries are free |
| Replication | Write to 3 ingesters, read with dedup | Losing one ingester loses no recent data |
Downsampling is the second half of the metrics design. One series at 10-second resolution is 3.15 million points a year. Nobody reads a two-year chart at 10-second resolution; the screen has about 1,000 pixels across.
| Tier | Resolution | Retention | Points per series per year |
|---|---|---|---|
| Hot | 10 seconds (raw) | 15 days | 3,150,000 (if kept a year) |
| Warm | 1 minute | 90 days | 525,000 |
| Cold | 1 hour | 2 years | 8,760 |
That is a 360x reduction from raw to hourly. The rollup must store sum, count, min, and max per interval, so that the average can be recomputed correctly and spikes are not smoothed away. For latency percentiles, store a histogram (fixed buckets, or a mergeable sketch such as DDSketch or t-digest), because the 99th percentile of an hour is not the average of sixty 99th percentiles. That one fact separates candidates who have operated a metrics system from those who have read about one.
The read side. During an incident, 200 engineers open the same dashboard, each panel aggregates thousands of series, and it refreshes every 30 seconds. Cache results by (query, step-aligned time range) and serve old ranges from rollups: a small read-heavy problem inside the ingest system.
Example 2: Logging, where the difficulty is bursts and search
Special property: the volume is spiky and correlated with failure, and the reads are needle-in-a-haystack searches over unstructured text. Metrics are small and regular. Logs are large, irregular, and most valuable at exactly the moment the system is least healthy.
1
2
3
10,000 services, ~1,000,000 log lines/s average x 500 bytes = 500 MB/s, 50 TB/day
Incident: one service in a retry loop logs 10x for an hour
= 5 GB/s burst
Burstiness drives the ingest design. A service failing a dependency logs a stack trace on every request, and every retry produces another one. The burst is concentrated on a few services, so it is also a hot-partition problem.
| Decision | Choice | Why |
|---|---|---|
| Agent behavior | Local disk buffer on each host (hundreds of MB), batch and compress before sending | Survives a collector outage of minutes without loss |
| Buffer sizing | Kafka retention of 24 to 72 hours, sized for the burst, not the average | The indexer can fall behind during the incident and catch up after |
| Partition key | Service plus a random salt, or round robin | A single noisy service spreads across partitions instead of melting one; per-service ordering is rarely needed because each line has a timestamp |
| Overload policy | Per-service quotas; drop DEBUG first, then INFO; never drop ERROR or audit | Keep the signal; shed the noise |
| Deduplication | Repeated identical lines collapsed to one line plus a count at the agent | A retry loop producing 50,000 copies of one stack trace is one fact |
| Priority lanes | Separate topic for ERROR and audit logs | The important logs never queue behind the flood |
The indexing decision is the cost decision. There are two families, and you should name both:
| Approach | How it works | Ingest cost | Storage cost | Query |
|---|---|---|---|---|
| Full-text index (Elasticsearch, OpenSearch) | Tokenize every line into an inverted index | High: tokenizing is most of the CPU | Index often 50 to 100 percent of raw size, times replicas | Fast arbitrary search over any word |
| Label index only (Loki style), or columnar (ClickHouse style) | Index a few fields (service, level, host); store compressed chunks; scan at query time | Low | Close to the compressed raw size, on object storage | Fast when you filter by labels first, slower for “find this string anywhere” |
The answer most interviewers want is the middle path: structured logs. If services emit JSON with service, level, trace_id, user_id, and message, you index the fields that people filter by and leave the message body for scanning. The trace ID is the most valuable field of all: it joins a log line to the metrics spike and the distributed trace. Schemaless ingest is convenient, but an unbounded set of field names is the logging version of cardinality explosion (the index mapping grows without limit), so cap the field count per service.
Tiering for logs follows how people debug:
| Tier | Age | Store | Query latency |
|---|---|---|---|
| Hot | 0 to 7 days | Indexed, SSD | Seconds |
| Warm | 7 to 30 days | Compressed chunks, object storage with a local cache | Tens of seconds |
| Cold / archive | 30 days to years (compliance) | Object storage archive class | Minutes to hours; rehydrate on request |
A full-text index roughly cancels out compression, so indexed logs take about the raw size: 50 TB a day, 100 TB with one replica. Thirty days hot is 3 PB of SSD, about $240,000 a month at $0.08 per GB-month before the nodes to run it. Seven days hot is 700 TB (about $56,000), and the other 23 days as compressed chunks on object storage are about 115 TB (about $2,300). Say the number; it is the argument.
Also redact secrets and personal data at the collector, before anything is written; removing it from an append-only archive later is very expensive. And derive counts from logs in the stream (ERROR lines per service), so dashboards stay right even when the logs themselves are sampled.
Example 3: Clickstream, where the difficulty is time and identity
Special property: the producers are untrusted, unreliable clients, and the events describe user behavior that must be reassembled in order. Metrics and logs come from your own servers on your own network. Clickstream comes from browsers and phones that go offline, retry, run ad blockers, get impersonated by bots, and belong to people with privacy rights.
1
2
3
50 million daily active users x 200 events/day = 10 billion events/day
10 billion / 100,000 s = 100,000 events/s average, ~300,000 peak
x 1 KB (properties, device, page, referrer) = 10 TB/day raw, ~1 TB/day as Parquet
Client-side batching and late data. The browser SDK (Software Development Kit) queues events and flushes every few seconds, or on page hide using the browser’s beacon API so the last events survive the tab closing. The mobile SDK persists events to local storage and flushes when it has a network. A phone on a flight sends six hours of events at once when it lands. So:
| Decision | Choice | Why |
|---|---|---|
| Event identity | Client generates a UUID (Universally Unique Identifier) per event; the client timestamp travels with it | Retries resend the same ID, so the pipeline can dedup; event time is when it happened, not when it arrived |
| Clock trust | Record client time, server receive time, and the client’s send time; correct skew by their difference | Phone clocks are wrong; the offset between send time and receive time estimates the error |
| Windowing | Aggregate by event time, with a watermark (for example, “events up to 10 minutes late go into the live window”) | Processing time would put events in the wrong hour or day |
| Late events | Past the watermark, write to the raw table anyway and recompute the affected daily partitions in a batch job | The real-time dashboard is approximate; the daily report is correct |
| Dedup | Keyed state on event ID for a window (hours), then rely on the batch layer’s dedup for anything older | Exact dedup forever is unbounded state |
This is the point to say the words event time versus processing time and watermark. It is the clickstream-specific competency.
Sessionization forces the partition key. A session is a run of a user’s events with no gap longer than 30 minutes. To compute it in a stream, all events for one user (or anonymous device ID, before login) must reach the same processor, so the partition key is the user or device ID. That is the opposite of the logging choice (spread a loud key) and it carries the opposite risk: a bot or a load-testing script with one device ID is a hot partition. Session windows also interact with late data: an event arriving an hour late can merge two sessions into one, which is another reason the authoritative session table is rebuilt in batch.
Bot filtering and trust. Much web traffic is automated. Filter in layers: known bot user agents and data center IP ranges at the collector, behavior heuristics (200 page views a minute, no mouse events) in the stream, and flag rather than delete so analysts can see what was removed. Never let a client-reported field such as a price drive a financial decision: the client is not trusted.
Privacy and consent. Consent is checked at the collector, before persistence: events from users who declined tracking are dropped or stripped of identifiers. Deletion requests (the right to erasure under the GDPR (General Data Protection Regulation), for example) are hard in an append-only lake partitioned by date, because one user’s events are in every daily partition. The common answers are to keep identifiers in a separate mapping table and delete the mapping (pseudonymization), to encrypt per user and delete the key (crypto-shredding), or to use a table format that supports row-level deletes with periodic rewrites (Delta Lake, Apache Iceberg, Apache Hudi). Say one of these and that it was decided up front.
Feeding the warehouse. The long-term home is columnar files in object storage, partitioned by event date, queried by a warehouse engine (BigQuery, Snowflake, Trino). The stream serves the live dashboard; a batch job recomputes yesterday correctly from raw events. That fast-approximate plus slow-correct split is the Lambda architecture; Kappa replays the log through the same stream job instead. Either is fine if you can say why.
Side by side
| Metrics | Logging | Clickstream | |
|---|---|---|---|
| Producer | Your servers, agents scraping or pushing | Your servers, agents tailing files | Untrusted browsers and phones |
| Event size | ~16 bytes raw, ~1.4 compressed | 200 bytes to several KB | ~1 KB with properties |
| Special property | Unbounded series cardinality | Bursts during failures; free-text search | Late, duplicated, out-of-order events from users |
| Partition key | Series hash (locality for compression) | Service plus salt (spread the loud ones) | User or device ID (sessionization needs it) |
| Duplicates | Idempotent by (series, timestamp) | Mostly tolerated; collapse repeats | Dedup by client event ID |
| Loss tolerance | Small gaps fine; next scrape in 10 seconds | Debug fine, errors and audit never | Fine for page views, never for purchases or billed clicks |
| Reduction over time | Downsample to rollups (sum, count, min, max, histogram) | Tier to object storage; drop debug early | Aggregate to daily tables; keep raw in columnar files |
| Read pattern | Dashboards and alert rules over time ranges | Search by trace ID, service, string | Analyst SQL, funnels, cohorts |
| Hard part in one phrase | The index of series | The incident burst and the index cost | Event time and identity |
Where each example stops being this shape
| Example | The pressure | It becomes |
|---|---|---|
| Metrics | Alert rules evaluated on a schedule, each exactly once, with a single owner per rule | Coordination: leases and failover for the rule evaluators |
| Metrics | An alert fires and must page the right people across email, SMS, and push | Fan-out: delivery to many recipients |
| Logging | Audit logs that must never be lost or altered | Still ingest, but with backpressure instead of dropping, synchronous replication, and tamper evidence (hash chaining, write-once storage) |
| Logging | Live tail of a service’s logs during an incident | Real-time: streaming to connected clients |
| Clickstream | Billing per ad click, or a budget that stops a campaign at $10,000 | Contended writes: exactly-once counting and a hot counter per campaign |
| Clickstream | Personalizing the next page from the last click | Real-time: the event must influence a response within milliseconds |
The ad click case deserves a sentence in any clickstream answer: once a count is money, loss and duplication stop being acceptable, and the counting path needs dedup by event ID, idempotent sinks (per the idempotency post), and reconciliation against the batch count.
The sentence to say in the interview
This is a write-heavy ingest problem: about a million small events a second, written once and read rarely, in aggregate. So I will spend the design on three things: an ingest path that batches at every hop with Kafka as the buffer, a storage layout partitioned by time with hot, warm, and cold tiers, and a plan for reducing the data as it ages. The one property I want to pin down first is what makes this data different: for metrics that is label cardinality, for logs it is the incident burst, for clickstream it is late and duplicate events from clients.
The question that impresses: “What can we afford to lose, and what must never be dropped?” It sets the overload policy, the delivery guarantees, and half of the cost, in one sentence.
Follow-Up Questions to Expect
- “Why not write straight to the database?” Because producers then wait on the slowest component, a database hiccup becomes data loss at every producer, and you cannot replay. The log decouples rates, absorbs bursts, and lets several consumers read the same stream independently.
- “How many Kafka partitions?” Enough that one consumer per partition keeps up at peak: if one consumer handles 20 MB per second and peak is 1 GB per second, that is at least 50, and I would provision 2 to 3 times that for growth, because adding partitions later reshuffles keys.
- “A consumer has been down for an hour. What happens?” Producers are unaffected. Lag grows; when the consumer returns it catches up faster than real time because it reads in large batches. If the outage exceeds log retention, the oldest data is gone, so retention is sized for the longest outage we will tolerate, and lag is alerted on well before that.
- “Exactly-once?” End to end, only with an idempotent or transactional sink. Kafka transactions give exactly-once within Kafka; writing to an external store needs either an idempotent write keyed by event ID or offsets committed in the same transaction as the output. For metrics and most logs I would not pay for it; for billed clicks I would.
- “How do you compute a 99th percentile across a fleet?” Not by averaging per-host percentiles. Each host reports a histogram or sketch; histograms with the same buckets are summed, and the percentile is computed from the merged result.
- “How do you control cost?” Retention by tier, downsampling, sampling of high-volume low-value data, compression in columnar formats, and per-team quotas with a usage report. The biggest lever is usually how long data stays in the hot, indexed tier.
Key Takeaways
- Write-heavy ingest: small events at a high rate, append-only, read rarely and in aggregate, value falling fast with age.
- Batch at every hop, put a durable log in front of every store, and do the storage arithmetic: cost drives the design.
- Decide per data type what happens under overload: backpressure for what must arrive, drop or sample the rest.
- The partition key differs by example: series hash for metrics, salted service for logs, user ID for clickstream.
- Metrics: the samples are cheap and the series are expensive. Police cardinality and downsample with sum, count, min, max, and mergeable histograms.
- Logs: design for the incident burst, index structured fields rather than every word, and tier to object storage after days.
- Clickstream: event time, watermarks, client event IDs for dedup, sessionization by user, bot filtering, and consent before persistence.
- The moment a count becomes money, the example leaves the shape and needs exactly-once counting.
Further Reading
- Martin Kleppmann, Designing Data-Intensive Applications, chapter 3 (LSM trees and column storage) and chapter 11 (stream processing, event time, exactly-once)
- Jay Kreps, The Log: What every software engineer should know about real-time data’s unifying abstraction (LinkedIn Engineering)
- Tuomas Pelkonen et al., Gorilla: A Fast, Scalable, In-Memory Time Series Database (VLDB 2015; the compression behind the 1.4 bytes per sample)
- Tyler Akidau, Streaming 101: The world beyond batch (event time, processing time, watermarks)
- Prometheus documentation, Metric and label naming (the cardinality warning in the official guidance)
The Six Shapes
Each shape has its own deep dive; the first five minutes post explains how to classify a question in the first minute.
| Shape | Examples | Post |
|---|---|---|
| Read-heavy lookup | URL shortener, product catalog, profile service | Read-heavy lookup |
| Write-heavy ingest | Metrics, logging, clickstream | You are here |
| Fan-out | News feed, notifications, chat | Fan-out |
| Contended writes | Ticketing, inventory, auctions | Contended writes |
| Coordination | Scheduler, workflow engine, locks | Coordination |
| Real-time | Ride matching, live location, collaborative editing | Real-time |