How to set up LLM observability and tracing

Why LLM observability is different

Traditional observability tracks latency and errors. LLM applications add nondeterministic output, token cost, retries, and multi-step chains. Observing them means capturing not just timing but the prompts, responses, and token counts so you can explain why an answer was wrong or expensive. OpenTelemetry provides a vendor-neutral way to do this.

Prerequisites

• A running LLM application
• A tracing backend such as Jaeger, plus Grafana for dashboards
• A way to count tokens for your model

Steps

1. Decide what to capture

Capture per call: model name, prompt, response, input and output tokens, latency, status, and any tool calls. Mask or redact sensitive content per your policy.

2. Instrument LLM calls with spans

Wrap each model call in a span so it appears in a trace. Nest spans for retrieval, the LLM call, and tool execution to see the full chain.

with tracer.start_as_current_span("llm.call") as span:
    resp = llm(messages)
    span.set_attribute("llm.model", model_name)

3. Record prompts, responses, and metadata

Attach the prompt and response as span attributes or events. These let you replay and debug a specific request later.

4. Track tokens, latency, and cost

Record input and output token counts and compute cost from your provider's rates. Emit these as metrics so you can aggregate spend per feature.

span.set_attribute("llm.input_tokens", usage.input)
span.set_attribute("llm.output_tokens", usage.output)

5. Export traces to a backend

Configure the OpenTelemetry exporter to send traces to Jaeger or another collector so they are searchable.

6. Build dashboards and alerts

In Grafana, chart latency percentiles, error rate, token usage, and cost over time. Alert on cost spikes and latency regressions.

Verification

Make a few requests, then open the tracing backend and confirm each request shows a complete trace with retrieval, the LLM call, token counts, and latency. Trigger an error and confirm it appears as a failed span with the captured prompt.

Next Steps

Sample high-volume traffic to control cost, link traces to your eval suite, redact PII before export, and set budgets that alert before spend gets out of control.