How to Store Traces at Scale with Grafana Tempo

What and why

Grafana Tempo is a trace backend designed for cost efficiency: it stores spans in cheap object storage and indexes only enough to find traces by ID or by TraceQL query. Unlike systems that index every span, Tempo scales to high volume without large index costs.

Prerequisites

• Services instrumented with OpenTelemetry.
• A running Grafana instance.
• Object storage: AWS S3, MinIO, or a compatible bucket.

Steps

1. Provision object storage

For local testing, run MinIO and create a bucket:

docker run -p 9000:9000 -p 9001:9001 minio/minio server /data --console-address ':9001'

Create a tempo bucket through the MinIO console.

2. Configure and run Tempo

Create tempo.yaml:

server:
  http_listen_port: 3200
distributor:
  receivers:
    otlp:
      protocols:
        http:
        grpc:
storage:
  trace:
    backend: s3
    s3:
      endpoint: minio:9000
      bucket: tempo
      insecure: true
      access_key: minioadmin
      secret_key: minioadmin

Run it: docker run -p 3200:3200 -p 4318:4318 -v $(pwd)/tempo.yaml:/etc/tempo.yaml grafana/tempo -config.file=/etc/tempo.yaml.

3. Send OTLP spans to Tempo

Point your services or Collector at Tempo's OTLP endpoint:

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

4. Add Tempo as a Grafana data source

In Grafana, add a Tempo data source with URL http://host.docker.internal:3200.

5. Query traces with TraceQL

Use the Explore view and the Tempo data source. TraceQL filters by span attributes:

{ .service.name = "checkout-service" && duration > 500ms }

This returns slow checkout traces.

6. Link traces to logs and metrics

Configure trace-to-logs and trace-to-metrics in the data source settings so a trace links to its Loki log lines and exemplar metrics, giving you one-click navigation between signals.

Verification

• Tempo's /ready endpoint returns ready.
• New objects appear in the storage bucket as traces are ingested.
• A TraceQL query returns matching traces in Grafana.

Next Steps

Deploy Tempo in microservices mode (separate distributor, ingester, querier) for high availability, set retention via compaction, and add metrics-generator to derive RED metrics directly from spans.