Start practising

SYSTEM DESIGN · CONCEPTS · RUNNING IT IN PRODUCTION · EPISODE 37

Observability: Logs, Metrics, Traces

How you find out what a production system is doing: metrics and the four golden signals, structured logs with request ids, distributed tracing with spans, alerting on SLO symptoms instead of causes, dashboards, and how to describe monitoring in a system design interview.

  1. 1
  2. 2
  3. 3
  4. 4
  5. 5
  6. 6
  7. 7
  8. 8
  9. 9
  10. 10
  11. 11
  12. 12
  13. 13
  14. 14
  15. 15
  16. 16
  17. 17
  18. 18
  19. 19
  20. 20
  21. 21
  22. 22
  23. 23
  24. 24
  25. 25
  26. 26
  27. 27
  28. 28
  29. 29
  30. 30
  31. 31
  32. 32
  33. 33
  34. 34
  35. 35
  36. 36
  37. 37
  38. 38
  39. 39
  40. 40
  41. 41
1 / 10

Press play to hear the narration, or step through at your own pace. Every picture here is the same one used in the YouTube video.

Notes

The question

Observability answers three questions. Is something wrong? Metrics tell you. Where is it wrong? Traces tell you. Why is it wrong? Logs tell you. Monitoring watches the failures you predicted; observability lets you investigate the ones you didn't.

Three questions
  • is something wrong? → metrics
  • where is it wrong? → traces
  • why is it wrong? → logs
  • monitoring: watch known
  • failure modes
  • observability: answer new
  • questions you did not plan for

Metrics

Metrics are numbers over time: cheap to store and easy to graph and alert on. Start with the four golden signals for every service: latency, using percentiles, traffic in requests per second, error rate, and saturation, meaning how full your CPU, memory, connection pools and queues are. Prometheus and Grafana are the common open-source pair.

Four golden signals
  • latency: p50 / p95 / p99
  • traffic: requests per second
  • errors: % of 5xx / failures
  • saturation: CPU, memory,
  • queue depth, connections
  • counters, gauges, histograms
  • Prometheus + Grafana, Datadog
  • cheap to store, easy to alert on

Logs

Logs record individual events. Make them structured, JSON instead of free text, with the service, level, duration and, most importantly, a trace id that ties all logs for one request together. Ship them to a central store so you can search across services. And never log passwords, tokens or card numbers.

Structured logs
{ "ts": "10:42:07.113",
  "level": "error",
  "service": "payments",
  "trace_id": "4bf9…",
  "user_id": 812,
  "msg": "card declined",
  "ms": 1874 }

JSON, not free text
never log passwords or cards

Traces

Traces follow one request through every service. The gateway creates a trace id and passes it along in headers; each service records a span with its timing. The picture is instant: the order service took forty milliseconds, payments took nineteen hundred, and inside it one database query took eighteen fifty. Found it. OpenTelemetry is the standard way to emit all three signals.

Distributed tracing
trace 4bf9  total 1,950 ms
├ gateway          1,950
├─ order service      40
└─ payment service 1,900
   └─ SELECT …     1,850  ←

trace id passed in headers
(W3C traceparent)
OpenTelemetry → Jaeger, Tempo

Alerting

Alerting is where teams go wrong. Page people on symptoms users actually feel: error rate above one percent for five minutes, or latency burning through the SLO. High CPU on its own belongs on a dashboard, not in someone's pager at three a.m. Every alert should be actionable, with a runbook. Too many alerts, and people start ignoring them.

Alerting
  • alert on symptoms users feel:
  • error rate > 1% for 5 min
  • p99 > 500 ms (SLO burn)
  • not on every cause:
  • CPU 80% → dashboard, not pager
  • every page must be actionable
  • and link to a runbook
  • too many alerts = ignored alerts

Cost

It isn't free. Logs are the expensive signal, so sample noisy ones and keep them for days or weeks, not forever. Sample traces, keeping every error and slow request. And watch metric cardinality: a label like user id creates millions of time series and a huge bill.

Costs and limits
  • logs are the expensive one:
  • sample debug logs
  • short retention (7–30 days)
  • traces: sample 1–10%,
  • but keep all errors / slow ones
  • metrics: watch cardinality
  • (no user_id as a label!)
  • pay for what helps you debug

Practice

Day to day, each service gets a dashboard of its golden signals and dependencies, with deploy markers so you can see when a release changed the graphs. Add an on-call rotation with runbooks, and after incidents, write blameless postmortems focused on what to change, not who to blame.

Dashboards and on-call
  • one dashboard per service:
  • golden signals + dependencies
  • one top-level SLO dashboard
  • deploy markers on graphs
  • on-call rotation, runbooks
  • blameless postmortems:
  • what happened, why,
  • what we change

Say it like this

In an interview: collect the four golden signals for every service, alert on SLO symptoms like error rate and p ninety-nine, use structured logs carrying a trace id, and OpenTelemetry tracing with sampling that keeps every error. Add business metrics too, like orders per minute; a sudden drop there catches bugs that technical metrics miss.

In the interview
  • "How would you monitor this?"
  • golden signals per service
  • SLO alerts: error rate, p99
  • structured logs with trace id
  • OpenTelemetry traces,
  • sampled, errors always kept
  • business metrics too:
  • orders/min, signups/min

Interview questions

What are the three pillars of observability?

Metrics (numeric time series for detecting problems), logs (detailed records of individual events for explaining them) and traces (the path and timing of a request across services for locating them).

What are the four golden signals?

Latency, traffic, errors and saturation. Monitoring these for every service catches most user-facing problems.

What is distributed tracing?

Propagating a trace id through every service a request touches and recording timed spans for each operation, so you can see where time is spent and where errors occur in a multi-service request.

Why use structured logging?

JSON or key-value logs can be indexed, filtered and aggregated reliably, and including a trace or request id lets you join logs from many services for one request.

What should you alert on?

User-facing symptoms tied to SLOs, such as error rate or latency burning the error budget. Causes like high CPU should be on dashboards; every page should be actionable with a runbook.

What is metric cardinality and why does it matter?

The number of unique label combinations in a metric. High-cardinality labels such as user id or request id explode the number of time series, increasing cost and slowing the metrics system.

How do you control the cost of logs and traces?

Sample debug logs and traces (while keeping all errors and slow requests), set retention periods, drop noisy fields, and aggregate into metrics where possible.

What is OpenTelemetry?

An open standard and set of SDKs for producing metrics, logs and traces in a vendor-neutral way, exportable to backends such as Prometheus, Jaeger, Tempo or commercial tools.