Notes
Requirements
Requirements. Collect metrics, logs and traces from thousands of hosts. Dashboards over the last hour should load in under a second, and alerts should fire within about a minute. Keep metrics for a year or more, downsampled, and logs for a few weeks. It has to be cheap per gigabyte, survive traffic spikes, stay up exactly when production is falling over, and never slow down the services it watches.
- collect metrics, logs, traces
- from thousands of hosts
- dashboards: last hour in < 1 s
- alert within ~1 minute
- keep: metrics 13 months
- (downsampled), logs 14-30 days
- cheap per GB, survive spikes
- stay up when prod is down
- never slow the services
Collection
Collection. Metrics are often pulled: a collector scrapes each service's metrics endpoint every fifteen seconds, so if a scrape fails it knows the target is down, and the service does almost nothing extra. Push suits short-lived jobs and devices behind NAT. Logs and traces are pushed: the app writes to stdout or an OpenTelemetry SDK, and a local agent batches, compresses and forwards them, so a slow backend never blocks the app.
- pull (Prometheus): collector
- scrapes /metrics every 15 s
- + knows when a target is down
- + service does nothing extra
- - needs service discovery
- push (OTLP, StatsD): app or
- agent sends to collector
- + short jobs, behind NAT
- logs: app → stdout → agent
Pipeline
Between agents and storage, put a buffer like Kafka. Incidents cause log storms, ten times normal volume, at exactly the moment you need the data. The buffer absorbs the burst, lets the stores catch up, and replays after a storage outage. It can also feed other consumers, like security analytics. Decide the overload policy in advance: drop debug logs first, sample info, and always keep errors.
- incidents cause log storms:
- 10x volume exactly when you
- need the data
- Kafka between agents and
- stores: absorbs bursts,
- replays after store outages,
- feeds several consumers
- overload: drop DEBUG first,
- sample INFO, keep ERROR
Time-series storage
Inside a time-series database, a series is a metric name plus labels, like requests total for the cart service, route pay, status five hundred. Each sample is a timestamp and a value. Because samples arrive at regular intervals and change slowly, timestamps compress with delta-of-delta encoding and values by XOR with the previous one, down to about one or two bytes per sample. Recent data sits in memory with a write-ahead log, older data in compacted blocks on disk.
series = name + labels:
http_requests_total{svc="cart",
route="/pay", code="500"}
sample = (timestamp, value)
timestamps: delta-of-delta
values: XOR with previous
→ ~1.3 bytes per sample
recent block in RAM + WAL,
2 h blocks on disk, compacted Cardinality
The thing that kills metric systems is cardinality: the number of unique series. Fifty routes times ten status codes times two thousand pods is already a million series. Add a user id label with ten million values and you have billions, and the database runs out of memory. Never put ids, raw URLs or emails in labels; that's what logs and traces are for. Enforce per-team series quotas at ingest.
- cost ∝ number of series
- route (50) × code (10) ×
- pod (2,000) = 1M series
- add user_id label (10M)
- → billions of series → OOM
- never label with ids, URLs
- with ids, emails, timestamps
- per-team series quotas
Logs
For logs, there are two main approaches. Full-text indexing, as in Elasticsearch, makes any search fast, but the index is roughly as big as the data, which is expensive at terabytes a day. Or index only a few labels, like service, level and pod, keep compressed chunks in object storage, and scan the matching chunks at query time. That's often around ten times cheaper and slower only for broad searches. Columnar stores like ClickHouse are a third option when you want SQL over logs.
- full-text index (Elastic):
- fast arbitrary search
- index ≈ size of the data,
- expensive at TB/day
- label index only (Loki-style):
- index {svc, level, pod},
- store compressed chunks in
- object storage, grep at
- query time → ~10x cheaper
- columnar (ClickHouse): SQL
Traces
Traces are the heaviest signal: every request can produce dozens of spans. Head sampling decides at the start, keeping say one percent, which is cheap but misses most rare errors. Tail sampling buffers the whole trace in the collector and decides at the end, keeping every error and every slow request plus a small sample of the rest. Put the trace id in every log line, so you can jump from a log to its trace.
- every request → 20-100 spans
- keep all = more data than
- the logs
- head sampling: keep 1% at
- the start (cheap, misses
- rare errors)
- tail sampling: buffer whole
- trace, keep all errors and
- slow ones + 1% of the rest
- link: trace_id in every log
Alerting
The alert evaluator runs rules against the time-series store every thirty seconds or so. Page on symptoms users feel, like error rate and latency, not on causes like high CPU. The strongest pattern is SLO burn-rate alerts: with a ninety-nine point nine percent target, page when the error rate over the last hour, and also the last five minutes, is more than fourteen times the error budget. That catches real fires fast and ignores brief blips.
alert: CheckoutErrorBudgetBurn
expr: error_ratio[1h] > 14.4
* (1 - 0.999)
and error_ratio[5m] > 14.4
* (1 - 0.999)
for: 2m
severity: page
page on symptoms (errors,
latency); ticket on causes Routing
Between the rule and a human sits an alert manager. It groups four hundred pods firing the same alert into one notification, deduplicates across redundant evaluators, and inhibits noise: if a whole region is down, it suppresses the per-service alerts that follow. It routes by team and severity: page for urgent, chat for warnings, a ticket for the rest. Every page should link a runbook. Too many pages and people stop reading them.
- group: 400 pods firing the
- same alert → 1 notification
- dedupe across evaluator
- replicas
- inhibit: region down silences
- its per-service alerts
- route by team + severity:
- page vs Slack vs ticket
- silence during maintenance
- every page needs a runbook
Watch the watcher
Who monitors the monitor? Run the monitoring stack in its own failure domain, a separate cluster and ideally another region, because if it shares a cluster with production, they fail together. Add a dead man's switch: an alert that always fires, and an outside service pages you when it stops arriving. And run external probes from outside your network, which catch DNS and certificate failures internal checks never see.
- separate failure domain:
- own cluster, own region,
- not on the prod cluster it
- watches
- dead man's switch: an alert
- that always fires; if it
- stops arriving, page
- external probes from outside
- your network
Scaling
To scale metrics, shard series across ingesters by hashing their labels, replicate three ways, and flush blocks to object storage, as systems like Mimir, Thanos and VictoriaMetrics do. Downsample: keep fifteen-second resolution for two weeks and five-minute resolution for over a year. For logs, add ingesters and keep chunks in object storage. Speed up queries by splitting long ranges into parallel pieces and caching results.
- metrics: shard series by
- hash(labels) over ingesters,
- replicate 3x (Cortex/Mimir,
- Thanos, VictoriaMetrics)
- downsample: raw 15 s for
- 2 weeks, 5 min for 13 months
- logs: more ingesters, chunks
- to object storage
- queries: split by time, cache
Sizing
Sizing. Two thousand pods with a thousand series each is two million active series. Scraped every fifteen seconds, that's about a hundred thirty thousand samples a second, roughly seventeen gigabytes a day at a byte and a half each. Logs: two thousand pods at fifty lines a second of three hundred bytes is thirty megabytes a second, two point six terabytes a day raw, maybe two hundred sixty compressed. Logs, not metrics, are the cost problem.
metrics: 2,000 pods × 1,000 series = 2M active series ÷ 15 s = ~133k samples/s × 1.5 B × 86,400 ≈ 17 GB/day logs: 2,000 pods × 50 lines/s × 300 B = 30 MB/s ≈ 2.6 TB/day raw ~10x compression → 260 GB/day 30 days ≈ 8 TB in object store
Say it like this
In the interview, draw agents feeding a Kafka buffer, then a store per signal: a time-series database sharded by series with cardinality limits, a log store with a small label index and chunks in object storage, and traces with tail sampling. Alert on SLO burn rate, grouped and routed with runbooks. Put the whole thing in a separate failure domain with a dead man's switch, and give series counts and gigabytes per day.
- agents → Kafka → stores
- metrics: pull, TSDB sharded
- by series, cardinality caps
- logs: label index + chunks
- in object storage
- traces: tail sampling
- alerts: SLO burn rate,
- group + route + runbooks
- separate failure domain,
- dead man's switch
- numbers: series, GB/day