Observability Architecture

Three Signals, One Question

What broke, where, and for whom

Metrics answer "is it bad, and how bad." Traces answer "which hop." Logs answer "what exactly." If those three do not share a correlation id, you have three products, not a system. OpenTelemetry exists to make the shared context boring: one SDK family, one context propagator, exporters you can swap.

Do not start by picking a commercial backend. Start by listing the questions you must answer at 3 a.m.: error rate by route, latency by dependency, the trace for this request id, the log line that named the customer-facing order id. Then instrument until those queries are cheap.

Three braided signal rivers converging into a dark-glass collector prism — Logs, metrics, and traces are different encodings of the same request. The braid is the context. The prism is the collector. The backend is replaceable.

schema · telemetry path

APP / WORKER / EDGE
  traces  (spans)
  metrics (counters, histograms)
  logs    (structured records)
       │  W3C Trace Context + Baggage
       ▼
OTEL SDK  (batch, sample, redact)
       │  OTLP
       ▼
COLLECTOR  (receive → process → export)
       │
       ├── traces  → store / APM
       ├── metrics → TSDB
       └── logs    → log store

  One collector graph per environment.
  Apps do not speak five vendor SDKs.

The Signals

Traces

A span is a timed operation with attributes. A trace is a tree of spans. Use them for request path and dependency time. Sample, or you will drown.

Metrics

Aggregates. Counters and histograms. RED (rate, errors, duration) for services. USE (utilization, saturation, errors) for resources. Low cardinality labels only.

Logs

Discrete events. Structured JSON. Include trace_id. Never a debug wall in production by default. Levels are a filter, not a design.

Profiles (optional)

Continuous profiling explains CPU and memory when traces say "it is slow" and logs say nothing. Attach via the same service name.

RED vs USE

RED (Tom Wilkie) fits request-driven services. USE (Brendan Gregg) fits disks, queues, and thread pools. A service that is "green" on RED can still be saturated on USE (thread pool at 100%, queue depth climbing). Instrument both classes where they apply.

Trace Context

W3C Trace Context is the header

traceparent carries version, trace-id, parent-id, and flags. Every hop that does not forward it breaks the tree. That includes the queue worker that consumed the outbox, the cron that retried the payment, and the edge that terminated TLS. If you only instrument HTTP handlers, you will debug half a system.

mermaid · sequence

sequenceDiagram
  participant B as Browser
  participant E as Edge
  participant A as API
  participant Q as Worker
  B->>E: request
  E->>A: traceparent
  A->>A: span handler
  A->>Q: enqueue + trace context
  Q->>Q: span process-order
  Note over B,Q: One trace-id across hops

HTTP · W3C Trace Context

traceparent: 00-0af7651916cd43dd8448eb211c80319c-b7ad6b7169203331-01
tracestate:  vendorname=opaque

# 00          version
# 0af765...   trace-id (16 bytes hex)
# b7ad6b...   parent-id
# 01          sampled

Sampling is a policy

Head sampling decides at the root span. Tail sampling decides after you have the tree (keep errors and slow traces). Production defaults: sample a fraction of healthy traffic, keep all errors, keep a higher fraction of traces that touch a payment or auth path. Do not sample by "whatever the SDK default was on the day someone copied a snippet."

Cardinality

The bill and the query planner

A metric label with user id, email, or request url unbounded values explodes time series. That is not a pricing debate. It is a storage and query-latency fact. High-cardinality identity belongs on spans and log fields, not on metric labels.

Safe metric labels: service, route template (/orders/:id not the raw path), status class, dependency name, region. Unsafe: user id, raw URL, email, session token, full user-agent.

Field	Metrics	Spans / logs
service.name	Yes	Yes
http.route	Yes (template)	Yes
user.id	No	Yes, if policy allows
order.id	No	Yes
http.status_code	Yes (or class)	Yes

Collectors exist to enforce this

The OpenTelemetry Collector can drop, hash, or flatten attributes before they hit a backend. Put the policy there so application teams cannot invent a label that pages finance. Tail sampling belongs here too.

SLI and SLO

Measure the user journey, not the CPU

Google's SRE workbook is still the clean model: an SLI is a measurement (ratio of successful probes, or the fraction of requests faster than a threshold). An SLO is a target for that SLI over a window. An error budget is the remainder. Alerts fire on budget burn, not on "CPU > 80" unless CPU is the SLI you meant.

Do not invent a fake 99.999% target in a document. Pick a window and a measurement you can actually compute from your metrics. Multi-window, multi-burn-rate alerts (also from that workbook) beat a single threshold that pages on blips and sleeps through slow burns.

Name the journey

"Checkout completes" or "article HTML is served." Not "API is up."

Define the SLI

Good events over valid events. Exclude bots if they are not users. Document the exclusion.

Set the SLO

A target you will defend in a launch review. Tighter is not better if you cannot staff the budget.

Alert on burn

Fast burn pages now. Slow burn opens a ticket. Both use the same SLI.

Instrumentation Rules

Semantic conventions over folklore

OpenTelemetry semantic conventions name HTTP, DB, messaging, and FaaS attributes. Use them. http.route not endpoint. db.system not database_type. Folklore attributes make every dashboard a translation layer.

TypeScript · minimal span

import { trace } from "@opentelemetry/api";

const tracer = trace.getTracer("orders");

export async function placeOrder(input: OrderInput) {
  return tracer.startActiveSpan("orders.place", async (span) => {
    span.setAttribute("http.route", "POST /orders");
    span.setAttribute("order.id", input.id);
    try {
      const result = await writeOrder(input);
      span.setStatus({ code: 1 }); // OK
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: 2, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Auto-instrument, then add the business spans

HTTP and DB auto-instrumentation gives you the skeleton. The span a human wants is orders.place or stk.callback.apply. If the only spans you have are GET and SELECT, you will still not know which business step failed.

What Must Never Land in Telemetry

Redact at the source

No secrets. No session tokens. No passwords. No raw authorization headers. No health identifiers (this site does not process PHI; if you do, keep clinical fields out of logs and spans). Prefer resource ids over payloads. If you must log a payload for a short debug window, gate it behind a flag with an expiry.

Vendor-neutral rule: the commercial observability backend is another attacker surface. Treat it like a replica of production data and minimize what you send.

Trace ids are not secrets, payloads are

Putting a trace id in a user-facing error page is useful. Putting the request body in that same page is a leak. Same for support tools. Share the id. Fetch the span from a system that already has access control.

Key Takeaways

Principles

Instrument for questions, not for dashboards. If on-call cannot ask a new question, you have monitoring, not observability.
One context across hops. W3C traceparent through HTTP, queues, and crons.
Metrics stay low cardinality. Identifiers belong on spans and logs.
Use OpenTelemetry as the export contract. Swap backends. Do not swap the data model.
SLIs measure user journeys. Alert on error-budget burn, not on leftover CPU.
Auto-instrument the skeleton, hand-span the business step.
RED for services, USE for resources. You need both when a queue saturates behind a green request rate.
Redact at emit time. The telemetry store is production data.