Observability
148 items tagged with "observability"
Best Practices22
Google Site Reliability Engineering Practices
Codified principles (error budgets, toil elimination, SLIs/SLOs) for operating large-scale services reliably.
OpenTelemetry Instrumentation Guidelines
Best practices for generating consistent traces, metrics, and logs using OpenTelemetry.
RED & USE Monitoring Methodologies
Standard approaches for selecting golden signals (Rate-Errors-Duration / Utilisation-Saturation-Errors).
Production-Ready Micro-services Checklist
A checklist covering operability, reliability, deployability, and observability of micro-services.
Chaos Engineering Principles
Run controlled experiments to build confidence in system resilience under turbulent conditions.
Service Level Objectives (SLOs)
A target reliability level for a service, expressed as a measurable percentage of good events over a window, used to balance reliability against feature velocity.
Error Budgets
The allowed amount of unreliability derived from an SLO (100% minus the target), spent deliberately to balance new features against reliability work.
The Four Golden Signals
Google SRE's four core metrics for monitoring a user-facing system: latency, traffic, errors, and saturation.
OpenTelemetry Semantic Conventions
Standardized names and attributes for telemetry (spans, metrics, logs) so observability data is consistent and portable across tools and languages.
Structured Logging
Emitting logs as machine-parseable key-value records (typically JSON) with consistent fields, so logs can be searched, filtered, and correlated at scale.
Distributed Tracing Best Practices
Techniques for instrumenting and propagating trace context across services so requests can be followed end-to-end, with sampling and span design that aid debugging.
Prometheus Monitoring Best Practices
Guidance for naming metrics, controlling label cardinality, and writing alerting rules in Prometheus, the CNCF metrics and alerting system.
Symptom-Based Alerting
Alerting on user-visible symptoms (errors, latency, SLO burn) rather than internal causes, to reduce noise and page only on things that matter.
Incident Management Best Practices
A structured process for detecting, coordinating, and resolving outages with clear roles, communication, and severity levels to restore service quickly.
Blameless Postmortems
Post-incident reviews focused on systemic causes and learning rather than individual blame, producing concrete action items to prevent recurrence.
Observability-Driven Development
Building instrumentation into software as a first-class part of development so engineers can ask new questions of production behavior without shipping new code.
Service Mesh Best Practices
Guidance for using a service mesh to manage service-to-service traffic, security, and observability through sidecar proxies, keeping that logic out of application code.
Data Quality Management
The practice of measuring, monitoring, and improving data across dimensions like accuracy, completeness, consistency, timeliness, and validity so it stays fit for use.
Data Lineage
The traceable record of data's origin, movement, and transformation across systems, enabling impact analysis, debugging, compliance, and trust.
ML Model Monitoring and Drift Detection
Continuously tracking deployed ML models for performance decay, data drift, and concept drift so degradation is caught and corrected before it harms outcomes.
Data Catalog and Discovery
A searchable inventory of an organization's data assets with metadata, ownership, and lineage so people can find, understand, and trust the data they need.
LLM Observability
LLM observability is the practice of tracing, logging, and measuring LLM applications in production to monitor quality, cost, latency, and safety and to debug failures.
Patterns10
Sidecar Pattern
Deploy auxiliary components alongside primary services for cross-cutting concerns
Adapter Microservice
A microservice that translates between an application and an external system with an incompatible interface or protocol.
Service Mesh
A dedicated infrastructure layer that manages service-to-service communication via co-located proxies and a central control plane.
Health Endpoint Monitoring
Expose health-check endpoints that monitoring tools and load balancers probe to verify an application is functioning correctly.
Dead Letter Channel
Routes messages that cannot be delivered or processed to a dedicated channel for inspection and recovery instead of discarding or blocking them.
Correlation Identifier
Tags messages with a unique id that links related messages together, so a reply can be matched to its request and parts to their whole.
Wire Tap
Copies messages flowing through a channel to a secondary channel for inspection, logging, or analysis without disturbing the primary flow.
Health Check
Exposes endpoints that report whether a service is alive and ready to serve, enabling orchestrators and load balancers to route traffic only to healthy instances.
Distributed Tracing
Tracks a single request as it flows across many services by propagating context, producing an end-to-end timeline that reveals latency and failure sources.
Correlation ID
Assigns a unique identifier to a request and propagates it through every service and log, so related events across a distributed system can be tied together.
Anti-Patterns9
Exception Swallowing
Catching exceptions and then ignoring them, hiding failures so that errors pass silently and bugs become nearly impossible to diagnose.
Over-Caching
Adding caches everywhere to chase speed, multiplying staleness, invalidation bugs, and operational complexity for marginal gains that profiling never justified.
Deploy and Pray
Pushing releases to production with no automated verification, monitoring, or rollback, then hoping nothing breaks instead of knowing it works.
No Monitoring (Flying Blind)
Running production systems with no metrics, logs, or alerts, so problems are invisible until users complain and incidents cannot be diagnosed.
Alert Fatigue
So many low-value or noisy alerts fire that responders become desensitized and start ignoring them, including the ones that actually matter.
Log Everything (Logging Noise)
Logging indiscriminately at high verbosity, burying useful signal in a flood of low-value messages while driving up storage cost and slowing search.
Verbose Error Leakage
Returning stack traces, SQL errors, internal paths, or version details to clients, handing attackers a map of the system to exploit.
Logging Sensitive Data
Writing passwords, tokens, PII, or payment data into logs, where it spreads to aggregators and backups far beyond its intended access controls.
Vanity Metrics
Tracking impressive-looking numbers that do not inform decisions or correlate with real outcomes, creating an illusion of progress.
Tutorials11
How to add a lightweight service mesh with Linkerd
Install Linkerd, mesh a workload, and gain automatic mTLS plus golden-metrics observability with low overhead.
How to Instrument an Application with OpenTelemetry
Add OpenTelemetry traces, metrics, and logs to a service and export them through the OpenTelemetry Collector to your backend.
How to Expose Prometheus Metrics from a Service
Add a metrics endpoint with counters, gauges, and histograms, then scrape it with Prometheus and query the results.
How to Build a Grafana Dashboard from Prometheus Metrics
Connect Grafana to Prometheus and build a service dashboard with RED-method panels, variables, and provisioning as code.
How to Set Up Distributed Tracing with Jaeger
Deploy Jaeger, send OpenTelemetry spans to it, propagate trace context across services, and analyze latency in the UI.
How to Store Traces at Scale with Grafana Tempo
Run Grafana Tempo as an object-storage trace backend, ingest OpenTelemetry spans, and query traces from Grafana.
How to Add Structured Logging with Correlation IDs
Replace plain text logs with structured JSON logs and propagate a correlation ID so all logs for one request can be joined.
How to Define SLOs and Error Budgets for a Service
Pick SLIs, set SLO targets, calculate an error budget, and track burn rate so you know when to slow releases.
How to Create Prometheus Alerting Rules with Alertmanager
Write Prometheus alert rules, route and group them in Alertmanager, and deliver notifications to Slack or email.
How to Centralize Logs with Grafana Loki
Ship structured logs to Loki, label them effectively, and query with LogQL to investigate incidents from one place.
How to set up LLM observability and tracing
Instrument an LLM application to trace prompts, responses, tokens, latency, and cost so you can debug and optimize in production.
Blueprints2
VM Monitoring to Prometheus and Grafana Blueprint
Replace legacy agent-based VM monitoring with Prometheus metrics, exporters, and Grafana dashboards using the RED and USE methodologies.
Ad-hoc Logging to OpenTelemetry Blueprint
Unify scattered logs, metrics, and traces under OpenTelemetry with the Collector, semantic conventions, and vendor-neutral export.
Products5
Reference Architectures6
Container Platform with Service Mesh
A Kubernetes container platform with an Istio service mesh providing mTLS, traffic management, and uniform observability across services.
LLM Observability Stack on Kubernetes
A reference design for an LLM observability platform on Kubernetes that captures traces, evaluations, cost, and quality signals across AI applications.
SIEM and Security Data Lake on Google Cloud
Scalable security analytics platform that ingests logs, detects threats, and supports investigation over a data lake.
Observability and SLO Platform on Kubernetes
Unified metrics, logs, and traces platform with OpenTelemetry, Prometheus, and SLO-based alerting on Kubernetes.
Service Mesh with mTLS on Kubernetes
Istio-based service mesh providing mutual TLS, traffic management, and observability for microservices.
Chaos Engineering and Resilience Platform on AWS
Controlled fault-injection platform that validates resilience hypotheses against production-like systems safely.
Playbooks14
Performance Optimization Playbook
Systematic approach to improving application performance
Kubernetes Platform Adoption Playbook
A phased program to stand up a production-grade internal Kubernetes platform and onboard application teams onto golden paths.
Service Mesh Adoption Program Playbook
A phased program to roll out a service mesh for mTLS, traffic management, and observability across a Kubernetes microservices estate.
Serverless Backend Migration Program Playbook
A program to migrate suitable backend workloads to serverless functions with cost, cold-start, and observability controls.
Spring Boot 2 to 3 Upgrade Program Playbook
A fleet program to upgrade Spring Boot 2 services to Spring Boot 3, covering the Jakarta namespace move, Java baseline, and observability changes.
Data Quality Program Playbook
Stand up a data quality program with profiling, automated tests, SLAs, anomaly detection, and incident workflows across pipelines.
Data Pipeline Orchestration Program Playbook
Modernize fragile cron and script-based data pipelines into a governed orchestration platform with lineage, retries, and observability.
OpenTelemetry Observability Rollout Playbook
A program to instrument services with OpenTelemetry for unified traces, metrics, and logs, replacing fragmented vendor-specific monitoring.
SRE and SLO Program Playbook
A program to adopt Site Reliability Engineering with service level objectives, error budgets, and toil reduction to balance reliability and velocity.
Incident Management Program Playbook
A program to build structured incident response with clear roles, severity levels, blameless postmortems, and continuous learning.
Service Mesh Adoption Playbook
A program to introduce a service mesh for secure service-to-service communication, traffic control, and observability across microservices.
Chaos Engineering and Resilience Program Playbook
A program to validate system resilience through controlled fault injection, hypothesis-driven experiments, and game days.
gRPC Migration Program Playbook
A program to migrate internal service-to-service communication from REST/JSON to gRPC for lower latency and strong contracts.
Frontend Performance Program Playbook
A program for systematically improving frontend performance and core web vitals across a product through budgets, optimization, and continuous monitoring.
Checklists17
Kubernetes Production Readiness Checklist
Confirm a Kubernetes cluster and its workloads are secure, observable, and resilient before serving production traffic.
Cloud Cost Optimization Checklist
Systematically reduce cloud spend through visibility, rightsizing, commitments, and architectural changes without harming reliability.
Disaster Recovery Test Checklist
Plan and run a disaster-recovery test that proves backups, runbooks, and recovery objectives actually work end to end.
Serverless Production Readiness Checklist
Confirm a serverless application is observable, secure, resilient, and cost-aware before it serves production traffic.
Microservice Production-Readiness Checklist
Confirm a new or extracted microservice meets operational, security, and resilience bars before it serves production traffic.
Go Service Modernization Checklist
Modernize a Go service for production: modules, context propagation, observability, and idiomatic error handling.
Data Pipeline Observability Checklist
Checks for instrumenting data pipelines with freshness, volume, schema, lineage, and quality monitoring to detect issues early.
Production Go-Live Readiness Checklist
End-to-end verification that a service is ready to serve real users in production, covering scaling, monitoring, security, and rollback.
Incident Response Readiness Checklist
Verify the people, processes, and tooling needed to detect, respond to, and learn from production incidents are in place.
On-Call Handover Checklist
Ensure a clean transfer of on-call responsibility with full context on ongoing issues, risks, and operational state.
Observability & SLO Review Checklist
Assess whether a service is observable enough to operate, with meaningful SLOs, golden-signal metrics, tracing, and actionable alerts.
Platform Engineering & Internal Developer Platform Readiness Checklist
Assess readiness to build an internal developer platform that provides self-service, paved-path delivery for product teams.
LLM/RAG Production-Readiness Checklist
Verification items for taking a retrieval-augmented generation (RAG) application from prototype to reliable production service.
ML Model Deployment Checklist
Pre-flight verification for promoting a trained machine learning model into a production serving environment.
MLOps Pipeline Review Checklist
Audit items for assessing the maturity, reproducibility, and automation of an end-to-end machine learning operations pipeline.
AI Agent Deployment Checklist
Pre-flight items for safely deploying an autonomous LLM agent that calls tools and takes actions on behalf of users.
gRPC Rollout Checklist
Pre-flight items for rolling out gRPC services across a system, covering contracts, compatibility, security, and observability.
Stacks17
Cloud Native Stack
Kubernetes, Helm, Istio, Prometheus - CNCF ecosystem
Kubernetes + Istio Service Mesh Stack
Cloud-native platform stack pairing Kubernetes orchestration with the Istio service mesh for traffic management, security, and observability.
gRPC Service Mesh Stack
Microservices stack combining gRPC inter-service communication with a service mesh on Kubernetes for typed, observable, secure service-to-service traffic.
ELK Analytics Stack
Elasticsearch, Logstash, and Kibana for ingesting, indexing, searching, and visualizing logs, metrics, and events at scale.
PLG Stack (Prometheus + Loki + Grafana)
Open-source observability stack: Prometheus collects metrics, Loki aggregates logs, and Grafana unifies visualization, dashboards, and alerting.
Elastic Observability Stack
Observability built on the Elastic Stack: Beats and APM agents feed Elasticsearch, with Kibana unifying logs, metrics, and traces in one platform.
Grafana LGTM Stack
Grafana's full observability suite: Loki for logs, Grafana for visualization, Tempo for traces, and Mimir for metrics at scale, unified under OpenTelemetry.
OpenTelemetry + Tempo + Grafana Tracing Stack
Vendor-neutral distributed tracing: OpenTelemetry instruments and collects traces, Tempo stores them cheaply in object storage, and Grafana visualizes them.
Jaeger Distributed Tracing Stack
CNCF distributed tracing stack: OpenTelemetry instrumentation feeds Jaeger, which stores spans in Elasticsearch or Cassandra and visualizes request flows.
ClickHouse + Grafana
A real-time analytics stack using ClickHouse as a fast columnar OLAP database with Grafana for dashboards over high-volume event data.
Dagster + dbt
An asset-oriented data orchestration stack where Dagster schedules and observes pipelines built around dbt's transformation models.
Prometheus + Thanos
A scalable, highly available metrics stack extending Prometheus with Thanos for global query, long-term object-storage retention, and multi-cluster aggregation.
Datadog Full Platform
A unified commercial SaaS observability stack covering metrics, traces, logs, RUM, security, and infrastructure monitoring through Datadog agents and integrations.
New Relic Platform
A consumption-based SaaS observability platform unifying APM, infrastructure, logs, and browser monitoring on a single telemetry database with OpenTelemetry support.
Sentry + OpenTelemetry
An application monitoring stack pairing Sentry's error tracking and performance monitoring with OpenTelemetry instrumentation for vendor-neutral traces and metrics.
Jaeger + OpenTelemetry + Prometheus
An open-source observability stack combining OpenTelemetry instrumentation, Jaeger distributed tracing, and Prometheus metrics, visualized in Grafana.
Loki + Tempo + Mimir
Grafana Labs' scalable, object-storage-backed observability stack for logs (Loki), traces (Tempo), and metrics (Mimir), unified in Grafana.
Comparisons9
NGINX vs Envoy
NGINX is a battle-tested web server and reverse proxy; Envoy is a modern, dynamically configurable proxy built for cloud-native service mesh and observability.
Istio vs Linkerd
Istio and Linkerd are the leading Kubernetes service meshes. Istio is feature-rich and powerful; Linkerd is lightweight, simple, and fast with a purpose-built Rust proxy.
TimescaleDB vs InfluxDB
A Postgres extension that turns SQL into a time-series database versus InfluxDB, a purpose-built time-series platform.
Prometheus vs Datadog
Prometheus is an open-source, self-hosted metrics and alerting system; Datadog is a hosted, all-in-one observability SaaS spanning metrics, logs, and traces.
Grafana vs Kibana
Grafana is a source-agnostic visualization and dashboarding tool; Kibana is the visualization layer of the Elastic Stack, optimized for log and search analytics.
OpenTelemetry vs Vendor Agents
OpenTelemetry is an open, vendor-neutral standard for collecting telemetry; proprietary vendor agents are tightly integrated SDKs from a single observability provider.
ELK Stack vs Grafana Loki
The ELK Stack indexes full log content for powerful search; Grafana Loki indexes only labels for cheaper, lighter log aggregation.
Datadog vs New Relic
Two leading observability SaaS platforms covering APM, infrastructure, logs, and more. They differ in breadth, pricing model, and product focus.
Jaeger vs Zipkin
Both are open-source distributed tracing systems. Jaeger is a CNCF-graduated, cloud-native tracer; Zipkin is an older, lightweight, simple-to-run tracer.
Benchmarks5
k6 Load Testing
Developer-centric, scriptable load-testing tool using JavaScript scenarios to measure API and web performance with rich thresholds and metrics.
Apache JMeter
Mature, GUI-driven Java load-testing tool for simulating complex multi-protocol user scenarios and measuring throughput, latency, and error rates.
Gatling Load Testing
Scala-based, asynchronous load-testing tool with an expressive scenario DSL and detailed HTML reports for high-concurrency performance testing.
Locust Load Testing
Python-based, distributed load-testing tool where user behavior is defined in code, scaling to many workers for high-concurrency scenario testing.
API P99 Latency Benchmark
Measures tail latency, the response time at the 99th percentile, to capture worst-case API responsiveness that averages hide and that users feel most.
FAQs4
What is a service mesh?
A service mesh is an infrastructure layer that manages communication between microservices, handling traffic routing, load balancing, retries, encrypt...
What is an SLO, SLI, and error budget?
An SLI (Service Level Indicator) is a measured metric of service health, such as request success rate or latency. An SLO (Service Level Objective) is ...
What is the difference between observability and monitoring?
Monitoring is collecting and alerting on predefined metrics and checks—it answers known questions like 'is CPU above 90%?' Observability is the broade...
What are the three pillars of observability?
The three pillars of observability are metrics, logs, and traces. Metrics are numeric time-series data, such as request rate or error count, that are ...
Glossaries16
DaemonSet
A DaemonSet is a Kubernetes controller that ensures a copy of a specified pod runs on every (or a selected subset of) node in the cluster, commonly used for node-level agents.
Kubelet
The kubelet is the primary node agent in Kubernetes that runs on every worker node, ensuring the containers described in the pods assigned to that node are running and healthy.
Observability
The degree to which the internal state of a system can be understood from the external data it produces, typically its metrics, logs, and traces.
Telemetry
The automated collection and transmission of measurement data from a running system to a remote location for monitoring and analysis.
Distributed Tracing
A technique that follows a single request as it propagates across multiple services, recording timing and context at each step to reveal the end-to-end path.
Span
The basic unit of work in distributed tracing, representing a single named, timed operation with a start, an end, and contextual attributes.
Metric
A numeric measurement of some aspect of a system captured over time, such as request rate, error count, or memory usage, used for monitoring and alerting.
Structured Logging
The practice of emitting log entries as machine-readable structured data, typically key-value pairs or JSON, rather than free-form text strings.
Service Level Objective (SLO)
A target value or range for a service level indicator over a period of time, expressing the desired level of reliability for a service.
Service Level Indicator (SLI)
A quantitative measure of a specific aspect of a service's level of service, such as the proportion of successful requests or requests served within a latency threshold.
Error Budget
The maximum amount of unreliability a service is allowed over a period, calculated as the difference between 100% and its service level objective.
Toil
Manual, repetitive, automatable operational work that scales linearly with service size and provides no lasting value, a key target for reduction in site reliability engineering.
Incident Management
The coordinated process for detecting, responding to, mitigating, and resolving unplanned disruptions to a service, then learning from them.
Postmortem
A written, blameless analysis produced after an incident that documents what happened, the impact, the root causes, and the actions to prevent recurrence.
On-Call
An arrangement in which designated engineers are available to respond to alerts and incidents outside normal working hours, usually on a rotating schedule.
Mean Time to Recovery (MTTR)
The average time taken to restore a service after a failure, measured from the start of an incident to its resolution.