Observability
Monitoring, logging, and tracing best practices
Best Practices
Google Site Reliability Engineering Practices
Codified principles (error budgets, toil elimination, SLIs/SLOs) for operating large-scale services reliably.
by GoogleOpenTelemetry Instrumentation Guidelines
Best practices for generating consistent traces, metrics, and logs using OpenTelemetry.
by OpenTelemetry ProjectRED & USE Monitoring Methodologies
Standard approaches for selecting golden signals (Rate-Errors-Duration / Utilisation-Saturation-Errors).
by Google SRE & Brendan GreggChaos Engineering Principles
Run controlled experiments to build confidence in system resilience under turbulent conditions.
by PrinciplesOfChaos.orgService Level Objectives (SLOs)
A target reliability level for a service, expressed as a measurable percentage of good events over a window, used to balance reliability against feature velocity.
by Google SREError Budgets
The allowed amount of unreliability derived from an SLO (100% minus the target), spent deliberately to balance new features against reliability work.
by Google SREThe Four Golden Signals
Google SRE's four core metrics for monitoring a user-facing system: latency, traffic, errors, and saturation.
by Google SREOpenTelemetry Semantic Conventions
Standardized names and attributes for telemetry (spans, metrics, logs) so observability data is consistent and portable across tools and languages.
by OpenTelemetry (CNCF)Structured Logging
Emitting logs as machine-parseable key-value records (typically JSON) with consistent fields, so logs can be searched, filtered, and correlated at scale.
by OpenTelemetry (CNCF)Distributed Tracing Best Practices
Techniques for instrumenting and propagating trace context across services so requests can be followed end-to-end, with sampling and span design that aid debugging.
by OpenTelemetry (CNCF)Prometheus Monitoring Best Practices
Guidance for naming metrics, controlling label cardinality, and writing alerting rules in Prometheus, the CNCF metrics and alerting system.
by Prometheus (CNCF)Symptom-Based Alerting
Alerting on user-visible symptoms (errors, latency, SLO burn) rather than internal causes, to reduce noise and page only on things that matter.
by Google SREIncident Management Best Practices
A structured process for detecting, coordinating, and resolving outages with clear roles, communication, and severity levels to restore service quickly.
by PagerDutyBlameless Postmortems
Post-incident reviews focused on systemic causes and learning rather than individual blame, producing concrete action items to prevent recurrence.
by Google SREObservability-Driven Development
Building instrumentation into software as a first-class part of development so engineers can ask new questions of production behavior without shipping new code.
by CNCFService Mesh Best Practices
Guidance for using a service mesh to manage service-to-service traffic, security, and observability through sidecar proxies, keeping that logic out of application code.
by Cloud Native Computing FoundationData Quality Management
The practice of measuring, monitoring, and improving data across dimensions like accuracy, completeness, consistency, timeliness, and validity so it stays fit for use.
by DAMA InternationalData Lineage
The traceable record of data's origin, movement, and transformation across systems, enabling impact analysis, debugging, compliance, and trust.
by Linux Foundation (OpenLineage)ML Model Monitoring and Drift Detection
Continuously tracking deployed ML models for performance decay, data drift, and concept drift so degradation is caught and corrected before it harms outcomes.
by Evidently AILLM Observability
LLM observability is the practice of tracing, logging, and measuring LLM applications in production to monitor quality, cost, latency, and safety and to debug failures.
by Cloud Native Computing Foundation (OpenTelemetry)Patterns
Health Endpoint Monitoring
Expose health-check endpoints that monitoring tools and load balancers probe to verify an application is functioning correctly.
Wire Tap
Copies messages flowing through a channel to a secondary channel for inspection, logging, or analysis without disturbing the primary flow.
Distributed Tracing
Tracks a single request as it flows across many services by propagating context, producing an end-to-end timeline that reveals latency and failure sources.
Correlation ID
Assigns a unique identifier to a request and propagates it through every service and log, so related events across a distributed system can be tied together.
Tutorials
How to Instrument an Application with OpenTelemetry
Add OpenTelemetry traces, metrics, and logs to a service and export them through the OpenTelemetry Collector to your backend.
How to Expose Prometheus Metrics from a Service
Add a metrics endpoint with counters, gauges, and histograms, then scrape it with Prometheus and query the results.
How to Build a Grafana Dashboard from Prometheus Metrics
Connect Grafana to Prometheus and build a service dashboard with RED-method panels, variables, and provisioning as code.
How to Set Up Distributed Tracing with Jaeger
Deploy Jaeger, send OpenTelemetry spans to it, propagate trace context across services, and analyze latency in the UI.
How to Store Traces at Scale with Grafana Tempo
Run Grafana Tempo as an object-storage trace backend, ingest OpenTelemetry spans, and query traces from Grafana.
How to Add Structured Logging with Correlation IDs
Replace plain text logs with structured JSON logs and propagate a correlation ID so all logs for one request can be joined.
How to Define SLOs and Error Budgets for a Service
Pick SLIs, set SLO targets, calculate an error budget, and track burn rate so you know when to slow releases.
How to Create Prometheus Alerting Rules with Alertmanager
Write Prometheus alert rules, route and group them in Alertmanager, and deliver notifications to Slack or email.
How to Centralize Logs with Grafana Loki
Ship structured logs to Loki, label them effectively, and query with LogQL to investigate incidents from one place.
How to set up LLM observability and tracing
Instrument an LLM application to trace prompts, responses, tokens, latency, and cost so you can debug and optimize in production.
Checklists
Incident Response Readiness Checklist
Verify the people, processes, and tooling needed to detect, respond to, and learn from production incidents are in place.
On-Call Handover Checklist
Ensure a clean transfer of on-call responsibility with full context on ongoing issues, risks, and operational state.
Observability & SLO Review Checklist
Assess whether a service is observable enough to operate, with meaningful SLOs, golden-signal metrics, tracing, and actionable alerts.
Technology Stacks
ELK Analytics Stack
Elasticsearch, Logstash, and Kibana for ingesting, indexing, searching, and visualizing logs, metrics, and events at scale.
PLG Stack (Prometheus + Loki + Grafana)
Open-source observability stack: Prometheus collects metrics, Loki aggregates logs, and Grafana unifies visualization, dashboards, and alerting.
Elastic Observability Stack
Observability built on the Elastic Stack: Beats and APM agents feed Elasticsearch, with Kibana unifying logs, metrics, and traces in one platform.
Grafana LGTM Stack
Grafana's full observability suite: Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics at scale, unified under OpenTelemetry.
OpenTelemetry + Tempo + Grafana Tracing Stack
Vendor-neutral distributed tracing: OpenTelemetry instruments and collects traces, Tempo stores them cheaply in object storage, and Grafana visualizes them.
Jaeger Distributed Tracing Stack
CNCF distributed tracing stack: OpenTelemetry instrumentation feeds Jaeger, which stores spans in Elasticsearch or Cassandra and visualizes request flows.
Prometheus + Thanos
A scalable, highly available metrics stack extending Prometheus with Thanos for global query, long-term object-storage retention, and multi-cluster aggregation.
Datadog Full Platform
A unified commercial SaaS observability stack covering metrics, traces, logs, RUM, security, and infrastructure monitoring through Datadog agents and integrations.
New Relic Platform
A consumption-based SaaS observability platform unifying APM, infrastructure, logs, and browser monitoring on a single telemetry database with OpenTelemetry support.
Sentry + OpenTelemetry
An application monitoring stack pairing Sentry's error tracking and performance monitoring with OpenTelemetry instrumentation for vendor-neutral traces and metrics.
Jaeger + OpenTelemetry + Prometheus
An open-source observability stack combining OpenTelemetry instrumentation, Jaeger distributed tracing, and Prometheus metrics, visualized in Grafana.
Loki + Tempo + Mimir
Grafana Labs' scalable, object-storage-backed observability stack for logs (Loki), traces (Tempo), and metrics (Mimir), unified in Grafana.
FAQs
What is an SLO, SLI, and error budget?
An SLI (Service Level Indicator) is a measured metric of service health, such as request success rate or latency. An SLO (Service Level Objective) is the target you set for that indicator, for example 99.9% of requests succeed over 30 days. The error budget is the allowed amount of failure—the gap between 100% and your SLO—that the service may consume before changes are halted to focus on reliability. Together they give teams a data-driven way to balance shipping features against maintaining reliability.
What is the difference between observability and monitoring?
Monitoring is collecting and alerting on predefined metrics and checks—it answers known questions like 'is CPU above 90%?' Observability is the broader property of being able to understand a system's internal state from its external outputs, so you can investigate problems you did not anticipate. Monitoring tells you that something is wrong; observability helps you ask new questions and discover why. In practice observability builds on rich telemetry—metrics, logs, and traces—while monitoring is one consumer of that data.
What are the three pillars of observability?
The three pillars of observability are metrics, logs, and traces. Metrics are numeric time-series data, such as request rate or error count, that are cheap to store and ideal for dashboards and alerts. Logs are timestamped records of discrete events, useful for detailed, contextual debugging. Traces follow a single request as it flows across services, revealing where time is spent and where failures occur in a distributed system. Used together, they let you detect, diagnose, and understand problems.
Benchmarks
k6 Load Testing
Developer-centric, scriptable load-testing tool using JavaScript scenarios to measure API and web performance with rich thresholds and metrics.
Apache JMeter
Mature, GUI-driven Java load-testing tool for simulating complex multi-protocol user scenarios and measuring throughput, latency, and error rates.
Gatling Load Testing
Scala-based, asynchronous load-testing tool with an expressive scenario DSL and detailed HTML reports for high-concurrency performance testing.
Locust Load Testing
Python-based, distributed load-testing tool where user behavior is defined in code, scaling to many workers for high-concurrency scenario testing.
API P99 Latency Benchmark
Measures tail latency, the response time at the 99th percentile, to capture worst-case API responsiveness that averages hide and that users feel most.