How Things Work

Observability

Understanding a distributed system from the outside — metrics, logs, and traces; distributed tracing with correlation IDs; and the OpenTelemetry standard tying it all together.

How It Works

Observability is your ability to understand what a system is doing — and why — from the telemetry it emits. Its three pillars are metrics (aggregate health), logs (discrete events), and traces (a request's path across services). In microservices, where one request fans out across many services, distributed tracing with a propagated correlation ID is what makes debugging tractable.

The Three Pillars

Observability is the ability to understand a system's internal state from its outputs. It rests on three pillars: metrics (aggregated numbers — request rate, error rate, latency), logs (discrete, ideally structured events), and traces (the path of a single request across services). Together they answer 'is it healthy?', 'what happened?', and 'where?'.

Monitoring vs Observability

Monitoring watches known failure modes with predefined dashboards and alerts ('CPU > 80%'). Observability lets you ask new questions about unanticipated problems after the fact. In distributed systems where failures are emergent and novel, you need observability, not just monitoring.

Distributed Tracing

A single user request can touch a dozen services, so a per-service log is useless on its own. Tracing propagates a trace_id (and span context) through every hop; each service records a span with timing and metadata. Assembled, they form a waterfall that shows exactly where time was spent and which service failed.

Correlation & Standards

The glue is correlation: structured logs tagged with the same trace_id let you pivot from a slow trace straight to the relevant logs. OpenTelemetry standardizes how traces, metrics, and logs are generated and propagated (the W3C traceparent header), feeding tools like Prometheus, Grafana, Jaeger, and Tempo.

Key Concepts

📊Metrics

Aggregated numeric time series (rate, errors, latency). Cheap, great for dashboards and alerting on trends.

📝Logs

Discrete event records. Most useful when structured (JSON) and tagged with a trace/correlation id.

🕸️Traces & Spans

A trace is one request's journey; a span is one operation within it, with timing and parent/child links.

🔗Correlation ID

A trace_id propagated through every service so logs, metrics, and spans for one request tie together.

🔭OpenTelemetry

The vendor-neutral standard for generating and propagating telemetry across languages and tools.

🚦RED / USE

Metric playbooks: RED (Rate, Errors, Duration) for services; USE (Utilization, Saturation, Errors) for resources.

OpenTelemetry tracing + correlated logs

tsx

1// OpenTelemetry: propagate context so spans link into ONE distributed trace.
2const tracer = trace.getTracer("orders-service");
3
4app.post("/orders", async (req, res) => {
5  // a child span continues the trace started upstream (trace_id flows in headers)
6  await tracer.startActiveSpan("place-order", async (span) => {
7    span.setAttribute("order.items", req.body.items.length);
8
9    // structured log carrying the SAME trace_id → logs & traces correlate
10    log.info({ trace_id: span.spanContext().traceId, msg: "placing order" });
11
12    await inventory.reserve(req.body);   // these calls become child spans,
13    await payments.charge(req.body);     // each timed, all under one trace
14
15    span.end();
16  });
17});
18
19// W3C traceparent header carries context across every service hop:
20//   traceparent: 00-<trace_id>-<span_id>-01

💡

Why This Matters

You can't operate or debug a distributed system you can't see into. Observability is core to SRE and on-call work and increasingly expected in interviews. Understanding the three pillars, distributed tracing, and correlation IDs is the difference between resolving an incident in minutes and grepping logs across a dozen services for hours.

Common Pitfalls

⚠Unstructured logs with no trace/correlation ID — impossible to follow one request across services.

⚠Logging everything at full volume — cost explodes and signal drowns in noise (no sampling strategy).

⚠Alerting on raw resource metrics instead of user-facing symptoms (RED) — alert fatigue and missed real issues.

⚠Metrics without traces — you learn something is slow but not where, leaving you guessing.

⚠Not propagating trace context across async/message boundaries, breaking the trace at the queue.

Real-World Use Cases

1The Mystery Latency Spike

Scenario

Users report the checkout page is intermittently slow. The checkout service's own logs look fine, but p99 latency is terrible. With 8 services involved, nobody can tell which one is responsible.

Problem

Each service only sees its own slice. Without a way to follow a single slow request across services, the team is reduced to guessing and grepping logs in eight places with no common thread.

Solution

With distributed tracing, they open a slow trace and see the waterfall: the inventory service's span dominates the request. Drilling in (and pivoting to logs by trace_id) reveals a missing database index. The bottleneck is found in minutes, not days.

💡

Takeaway: Distributed tracing pinpoints where time goes across services. Metrics tell you something is slow; a trace waterfall tells you exactly which span is the culprit — indispensable for debugging microservices.

2Alert Fatigue and Unstructured Logs

Scenario

An on-call engineer gets paged at 2 AM for 'high error rate' but has only plain-text logs scattered across services and no way to correlate them to the specific failing requests. They burn an hour just gathering context.

Problem

Logs were unstructured free text with no trace_id, alerts fired on raw counts without context, and there was no link between an alert, the failing requests, and the relevant logs. Every incident started from zero.

Solution

Adopt structured (JSON) logging with a trace_id on every line, alert on RED metrics (error rate/latency) tied to dashboards, and wire traces↔logs↔metrics together. Now an alert links to the affected traces, which link to exactly the relevant logs — context arrives instantly.

💡

Takeaway: Observability is only useful if the signals connect. Structured logs with correlation IDs, meaningful metric-based alerts, and trace linkage turn a 2 AM scramble into a guided, fast investigation.

Join the discussion

Comments

Loading comments...

Cookie preferences

We use essential cookies for sign-in and preferences. With your permission, we also use basic analytics to improve lessons.