Distributed Tracing

In a microservices architecture, one user request can touch dozens of services. When it is slow or fails, logs and metrics on any single service rarely tell the whole story. Distributed Tracing stitches the request back together: it follows a request across every service it visits, recording timing and metadata at each hop, and assembles them into one end-to-end view. The result shows exactly where time went and where a failure originated.

How It Works

A trace represents one request's full journey and is composed of spans — timed units of work, each with a start, duration, status, and attributes. The first span is the root; downstream spans are children, forming a tree.

The mechanism is context propagation. When a service calls another, it passes trace context (a trace ID identifying the whole request, plus the current span ID as parent) in request headers. The W3C Trace Context standard (traceparent/tracestate headers) defines this propagation, so spans created in different services and languages share one trace ID. Services emit spans to a collector, which a backend (Jaeger, Zipkin, Grafana Tempo, or a vendor APM) reconstructs into a timeline. OpenTelemetry is now the de facto standard for instrumentation, providing SDKs and a vendor-neutral collector.

Because tracing every request is expensive, systems often sample — head-based (decide at the start) or tail-based (decide after seeing the whole trace, keeping slow or errored ones).

When to Use It

Use distributed tracing in any system spanning multiple services, async hops, or external calls: microservices, serverless workflows, and event-driven pipelines. It is the primary tool for diagnosing cross-service latency, pinpointing which dependency caused an error, and understanding real request topology.

Trade-offs

Instrumentation has a cost: code or agents must propagate context everywhere, and a single un-instrumented hop breaks the trace. Trace data volume is large, forcing sampling, which can miss the rare event you need. Storage and query infrastructure add operational burden. Tracing shows where time was spent but not always why, so it works best alongside logs and metrics — the three pillars of observability.

Related Patterns

Distributed tracing relies on correlation-id propagation, is frequently implemented transparently via the sidecar-pattern (service mesh), and complements health-check and circuit-breaker observability.

Example

Manual context propagation with OpenTelemetry:

with tracer.start_as_current_span("checkout") as span:
  span.set_attribute("order.id", order_id)
  inject(headers)                 # add traceparent
  resp = http.post(payments_url, headers=headers)

The payments service extracts the same trace ID, so both spans appear under one trace in the backend.