The Four Golden Signals

Best Practice: The Four Golden Signals

The Four Golden Signals, from Google's Site Reliability Engineering book, are the minimum metrics you should monitor for any user-facing system: latency (how long requests take), traffic (how much demand the system sees), errors (the rate of failed requests), and saturation (how full the system's most constrained resource is). If you can only measure four things, measure these.

They matter because they cover the symptoms users actually feel while staying small enough to act on. They give on-call engineers a fast, consistent way to answer "is the service healthy?" without drowning in dashboards, which is especially valuable when validating a migrated system.

Each signal answers a distinct question. Latency tells you how fast the service responds and, at the tail, how the slowest users experience it. Traffic tells you how much demand exists, which gives context to the other signals. Errors tell you what fraction of requests are failing, the clearest proxy for user pain. Saturation tells you how close the system is to a limit, which is your early warning of trouble before latency and errors spike. The signals are related to the RED and USE methods: RED (Rate, Errors, Duration) suits request-driven services, while USE (Utilization, Saturation, Errors) suits resources. The four golden signals blend both perspectives into one practical starting set.

Step-by-Step Implementation Guidance

1. For each service, instrument latency as a distribution (p50, p95, p99), not just an average.
2. Measure traffic in a unit meaningful to the service (requests/sec, transactions/sec, messages/sec).
3. Track errors explicitly, including both hard failures and wrong-but-200 responses where possible.
4. Identify the constraining resource and measure saturation against its limit (CPU, memory, connection pool, queue depth).
5. Build one dashboard per service that shows all four signals together.
6. Separate latency of successful requests from failed ones so failures do not skew the picture.
7. Alert on user-visible symptoms in these signals, not on individual machine metrics.

Common Mistakes Teams Make When Ignoring This Practice

• Reporting average latency, which hides tail pain.
• Tracking infrastructure metrics with no link to user impact.
• Ignoring saturation until the system tips over.
• Counting only 5xx as errors and missing logical failures.
• Building hundreds of panels and no clear health view.

Tools and Techniques That Support This Practice

• Prometheus histograms with histogram_quantile for latency percentiles.
• Grafana for combined four-signal dashboards.
• OpenTelemetry metrics for vendor-neutral instrumentation.
• Datadog, New Relic, and Cloud Monitoring service dashboards.
• Alertmanager and PagerDuty for pages derived from the signals.
• Synthetic probes to measure the signals from the user's perspective.

How This Practice Applies to Different Migration Types

• Cloud Migration: Compare all four signals between source and target to confirm parity before traffic shift, and watch saturation closely on smaller right-sized instances.
• Database Migration: Watch query latency and connection-pool saturation closely during cutover.
• SaaS Migration: Use error and latency signals from the new vendor's endpoints to validate the integration.
• Codebase Migration: Keep golden-signal dashboards stable across rewrites to catch regressions early.

Checklist

• Latency measured as percentiles
• Traffic measured in a meaningful unit
• Error rate captured, including logical errors
• Saturation tracked against resource limits
• One combined dashboard per service
• Successful and failed latency separated
• Alerts tied to user-visible symptoms