Observability & SRE
A metrics store, a log pipeline and a tracer, then Prometheus, Grafana, OpenTelemetry, SLOs and incident response.
Every on-call question is a query: is it broken for users, since when, where, and what changed. The tools that answer it (a Prometheus graph, a LogQL search, a trace waterfall, a burn-rate alert, an incident timeline) are only as good as your model of what they store and how they compute. A rate() that hides a counter reset, a label that explodes cardinality, a log shipper that drops lines on rotation, a trace that breaks at a queue and an alert that pages for nothing all come from mechanisms you can build in an afternoon. This phase builds each one from the standard library first, then runs the real tool on the same data and compares the answers.
How this phase is built
| Lesson | Type | What you build | What you then use |
|---|---|---|---|
| Metrics | Build | minitsdb: the exposition format, a scraper with relabelling and staleness, a TSDB with Gorilla-compressed chunks and a cardinality report | Prometheus, promtool tsdb analyze, node_exporter |
| PromQL | Build | miniql: selectors, rate with resets and extrapolation, aggregation, vector matching, histogram_quantile | the query API, recording rules, promtool test rules |
| Tracing | Build | minitrace: W3C Trace Context across three services, OTLP export, head and tail sampling, critical path | OpenTelemetry Collector, Jaeger |
| SLOs | Build | minislo: error budgets, multiwindow multi-burn-rate alerts, generated recording and alerting rules | Sloth or Pyrra, OpenSLO, promtool |
| Incidents | Build | miniincident: incidents from alert webhooks, roles, on-call schedules, escalation, timeline, MTTA/MTTR and a postmortem draft | Alertmanager webhooks, PagerDuty's event shape |
| Chaos | Build | minichaos: experiments with a steady-state hypothesis, faults, abort conditions and rollbacks | Chaos Mesh, tc netem, game days |
More lessons for this phase are being written; the plan is in ROADMAP.md.
Reference layer
The long-form guides for this phase live under Observability and the SRE learning path. The AIOps phase builds on this one: LLM observability traces agents with the same OpenTelemetry model.