Design Monitoring and Alerting: Millions of Time Series, Fast Dashboards and Pages That Matter

Listen

Listen. We collect numeric metrics, such as request counts, latency and CPU, each tagged with labels like service and host. We keep raw data for fifteen days and downsampled data for a year. Users query and graph rates, sums and percentiles grouped by label. Alert rules fire when an expression stays true long enough, and alerts are routed, grouped and silenced. Logs and traces are out of scope.

The qualities. Ingest is constant and huge, and must never slow down the services it watches. Dashboards should load in about a second for an hour of data. Alerts should fire within a minute of a problem. The monitoring system must be more available than everything it monitors. Losing a few samples is acceptable. Losing a page is not.

Ask scale

Scale. A hundred thousand hosts with a thousand series each is a hundred million active time series. Sampling every fifteen seconds gives almost seven million samples a second. That sounds scary, but good compression stores a sample in about one and a half bytes, so roughly ten megabytes a second, under a terabyte a day. Dashboards add about seven thousand queries a second.

100,000 hosts × 1,000 series
  = 100M active series
one sample every 15 s
  → 100M / 15 ≈ 6.7M samples/s

compressed ≈ 1.5 bytes/sample
  6.7M × 1.5 B ≈ 10 MB/s
  ≈ 860 GB/day, 15 days ≈ 13 TB

queries: 10K open dashboards
  × 20 panels / 30 s ≈ 7,000/s

Define API

The API. Writes are batched and compressed, thousands of samples per call. Queries come in two forms: an instant query for one moment, and a range query with a start, an end and a step, which is what a graph needs. Rules have an expression, a for duration, a severity and a link to a runbook. Silences mute matching alerts during planned work.

POST /v1/write   (batched,
  snappy-compressed samples)
GET  /v1/query?expr=…&time=…
GET  /v1/query_range?expr=…
     &start=…&end=…&step=60s
POST /v1/rules
  { expr, for: "5m", severity,
    labels, runbook_url }
POST /v1/silences
  { matchers, until, reason }

Data model

Data model. A series is a metric name plus a sorted set of labels, hashed into a series id. Each sample is just a timestamp and a floating point value, so samples are packed into compressed chunks of about two hours per series. An inverted index maps each label value to the list of series that have it, which is how queries find their series.

series = name + sorted labels
  http_requests_total
  {service="cart", code="500"}
  → series_id = hash(...)

samples: (series_id, ts, value)
  packed in 2-hour chunks

inverted index:
  service="cart" → [ids …]
  code="500"     → [ids …]

High-level design

The write and read paths. Collectors scrape each host every fifteen seconds, or receive pushed samples, and write batches to Kafka. Ingesters consume Kafka and write each series to its shard of the time-series database. Dashboards send queries to a query service that fans out to the shards, merges results and caches them.

Alerting reuses the store. A rule evaluator runs each rule's query every thirty seconds and sends firing alerts to an alert manager, which groups, deduplicates and routes them to the right on-call person. Older data is compacted and copied to object storage, the long-term store, at lower resolution.

Deep dive: collection

Deep dive one: collection. With pull, collectors scrape each target's metrics endpoint, found through service discovery. A failed scrape tells you the target is down, which is useful. With push, agents send samples, which suits short-lived jobs and devices behind firewalls, but silence is ambiguous. Many systems pull by default and offer a push gateway for batch jobs.

Deep dive: storage

Deep dive two: storage. Recent samples are appended to an in-memory head block, protected by a write-ahead log. Every two hours the head becomes an immutable block on disk, and compaction merges small blocks into bigger ones. Compression is the trick: timestamps arrive every fifteen seconds, so the delta of the delta is usually zero, about one bit. Values are XORed with the previous one, and most bits cancel.

The cardinality trap. The number of series is the product of label values. Service, status code and path might give ten thousand series. Add a user id label, with a million users, and you have ten billion series, and the index runs out of memory. Enforce series limits per team, reject labels like user id or request id, and put per-user detail in logs or traces instead.

series = product of label values

requests{service, code, path}
  20 × 10 × 50 = 10,000 series

add user_id (1M users):
  → 10 billion series
  → index and memory explode

fix: per-tenant series limits,
  reject high-cardinality labels

Deep dive: queries

Deep dive three: queries. First intersect the posting lists for each label, which narrows a hundred million series to, say, forty. Then read only those series' chunks for the requested range and compute the rate, sum or percentile. For thirty days, read five-minute or hourly rollups instead of raw samples. Cache results keyed by expression, step and an aligned time window.

Downsampling. Keep raw fifteen-second data for fifteen days, five-minute rollups for ninety days, and hourly rollups for a year or more. Each rollup stores min, max, sum and count, so averages stay correct and short spikes survive in the max. One important rule: you cannot average percentiles. Store latency as histograms, which can be added up, and compute p ninety-nine from the merged histogram.

raw 15 s     → keep 15 days
5-minute     → keep 90 days
1-hour       → keep 1 year+

each rollup stores
  min, max, sum, count
  → avg = sum / count
  → spikes survive (max)

percentiles: store histograms,
  never average p99s

Deep dive: alerting

Deep dive four: alerting. A good rule uses a rate over a window, like errors divided by requests over five minutes above two percent. The for clause makes it pending first and fire only after five minutes of being true, which stops flapping. Page on symptoms users feel, such as errors, latency or a fast error budget burn. Causes like disk at eighty percent should open a ticket, not wake someone.

The alert manager turns noise into one message. It deduplicates the same alert from two redundant evaluators. It groups by cluster and alert name, so a hundred and twenty hosts going down becomes one page. Inhibition mutes everything inside a zone that is already known to be down. Silences cover planned maintenance, and routing sends each alert to the owning team's rota.

alert: CartErrorsHigh
expr:  rate(errors[5m])
     / rate(requests[5m]) > 0.02
for:   5m   (pending → firing)
labels: severity=page, team=cart
runbook: wiki/cart-errors

page on symptoms users feel
  (errors, latency, SLO burn)
ticket on causes (disk 80%)

Deep dive: meta

Last hard part: who monitors the monitor? Run it in a separate failure domain, so the outage it should report cannot also take it down. Run two rule evaluators and a small cluster of alert managers that deduplicate between them. Add a dead man's switch: an alert that always fires, sent to an outside service that pages you if it ever stops arriving.

Trade-offs

Failure modes. If ingesters fall behind, Kafka buffers and dashboards lag, but nothing is lost. A dead database shard is covered by a replica. A single enormous query can hurt everyone, so limit series and samples per query and time out. Under overload, drop low-priority metrics first and protect the series that alert rules depend on.

Monitor ingest lag and dropped samples, active series per team, query p ninety-nine, rule evaluation delay, and the human metric: pages per week and how many were actionable. At ten times the scale, add shards by series hash, give each team its own tenant with quotas, and compute rollups in the stream instead of later.

Say it like this

In the interview: pull metrics through service discovery, buffer in Kafka, and store series in a time-series database with an in-memory head, a write-ahead log and compressed two-hour blocks. Find series through an inverted label index, and cap cardinality. Downsample with min, max, sum and count, and use histograms for percentiles. Alert on symptoms with a for duration, and let an alert manager group and route.