LLM Observability Stack on Kubernetes

Overview

LLM observability makes AI applications debuggable and accountable by recording what was sent to a model, what came back, how long it took, what it cost, and how good the output was. Traditional metrics miss the non-deterministic nature of LLMs, so this stack adds prompt/response traces and quality evaluations. Use it when LLM features reach production and you need to diagnose regressions, control spend, and prove quality over time.

Components

• OpenTelemetry Collector: receives traces and metrics from instrumented applications using GenAI semantic conventions.
• Langfuse: captures prompt/response traces, sessions, and runs online and offline evaluations.
• ClickHouse: stores high-volume trace and metric data for fast analytical queries.
• Kafka: buffers the trace stream so spikes never overload storage.
• PostgreSQL: holds evaluation results, datasets, and configuration.
• MinIO: stores large payloads and exported reports.
• Grafana: dashboards for latency, token cost, error rates, and quality scores.

Data Flow

Applications emit spans for each LLM call, retrieval, and tool use through OpenTelemetry. The collector forwards them to Kafka, which feeds Langfuse and ClickHouse. Langfuse links spans into full traces, attaches token and cost data, and runs evaluators (model-graded scoring, heuristics, or human review) against sampled traces. Quality scores, latency, and cost are aggregated in ClickHouse and visualized in Grafana, with alerts on regressions or budget breaches.

Scaling and Resilience

Kafka decouples ingestion from storage so trace bursts are buffered rather than dropped. ClickHouse scales horizontally and uses time-based partitioning with retention to control storage growth. Sample high-volume traces to keep cost manageable while retaining all errors and a representative sample of successes. Run components across nodes and zones, and back up evaluation data in PostgreSQL. Make ingestion tolerant of late or out-of-order spans.

Security

Prompts and responses frequently contain sensitive data; redact PII at the collector before storage and apply strict access controls and retention limits. Encrypt data at rest and in transit. Separate tenants and teams so one team cannot read another's traces. Audit access to stored prompts. Use observability data itself to detect prompt-injection attempts and anomalous usage patterns that may indicate abuse.

Trade-offs and Alternatives

Self-hosting keeps sensitive prompts in your environment and avoids per-trace SaaS pricing, but you operate the stack. Managed LLM observability services reduce that burden at the cost of data residency. Full trace capture is rich but expensive; sampling trades completeness for cost. Model-graded evaluation scales better than human review but is itself an LLM call with its own cost and error; use a mix. Reuse existing OpenTelemetry infrastructure rather than building a separate pipeline where possible.