Reliability
46 items tagged with "reliability"
Best Practices23
Production-Ready Micro-services Checklist
A checklist covering operability, reliability, deployability, and observability of micro-services.
Deployment Rings
A progressive rollout strategy that releases changes to expanding audience groups, or rings, validating each ring before exposing the next to limit blast radius.
Dark Launching
Deploying new functionality to production in a hidden state and exercising it with real traffic before exposing it to users, to validate behavior and capacity safely.
Service Level Objectives (SLOs)
A target reliability level for a service, expressed as a measurable percentage of good events over a window, used to balance reliability against feature velocity.
Error Budgets
The allowed amount of unreliability derived from an SLO (100% minus the target), spent deliberately to balance new features against reliability work.
The Four Golden Signals
Google SRE's four core metrics for monitoring a user-facing system: latency, traffic, errors, and saturation.
Prometheus Monitoring Best Practices
Guidance for naming metrics, controlling label cardinality, and writing alerting rules in Prometheus, the CNCF metrics and alerting system.
Symptom-Based Alerting
Alerting on user-visible symptoms (errors, latency, SLO burn) rather than internal causes, to reduce noise and page only on things that matter.
Incident Management Best Practices
A structured process for detecting, coordinating, and resolving outages with clear roles, communication, and severity levels to restore service quickly.
Blameless Postmortems
Post-incident reviews focused on systemic causes and learning rather than individual blame, producing concrete action items to prevent recurrence.
On-Call Best Practices
Sustainable on-call practices covering rotation design, escalation, actionable alerts, runbooks, and workload limits to keep services reliable without burning out engineers.
Runbook Automation
Codifying operational procedures as automated, repeatable workflows so common incident responses and maintenance tasks run reliably with less manual toil.
Capacity Planning
Forecasting future demand and provisioning resources ahead of need, combining organic growth, launches, and headroom to avoid both outages and waste.
Toil Reduction
Systematically identifying and eliminating repetitive, manual, automatable operational work so engineers can spend time on durable engineering instead.
Saga Pattern
A pattern for managing data consistency across microservices using a sequence of local transactions coordinated by events or a central orchestrator, with compensating actions on failure.
Circuit Breaker Pattern
A resilience pattern that stops calls to a failing dependency once errors cross a threshold, preventing cascading failures and giving the dependency time to recover.
Bulkhead Pattern
A resilience pattern that isolates resources into separate pools so a failure or overload in one part of a system cannot consume the resources others depend on.
Cell-Based Architecture
An architecture that partitions a system into independent, self-contained cells, each serving a subset of traffic, to limit blast radius and scale through replication.
Apache Kafka Streaming Best Practices
Design and operational guidance for building reliable, scalable event streaming on Apache Kafka, covering topics, partitions, delivery semantics, and consumers.
Idempotency Keys
A pattern where clients send a unique key with unsafe requests so the server can safely retry without applying the same operation twice, preventing duplicate charges or records.
API Rate Limiting
Controlling how many requests a client can make in a time window to protect API capacity, ensure fair use, and defend against abuse, using algorithms like token bucket.
Webhook Best Practices
Guidance for sending and receiving reliable webhooks: signature verification, idempotent handlers, retries with backoff, and fast acknowledgement of events.
Flaky Test Management
A disciplined approach to detecting, quarantining, and fixing nondeterministic tests so CI signal stays trustworthy and developers keep merging.
Patterns2
Guaranteed Delivery
Persists messages so they are not lost if the sender, broker, or receiver fails, ensuring each message is eventually delivered despite outages.
Idempotency Key
A client-supplied unique key lets a server detect and dedupe retried requests, so repeated submissions produce the same result exactly once.
Anti-Patterns11
Deadlock-Prone Locking
Acquiring multiple locks in inconsistent orders so two threads can each hold one lock while waiting for the other, freezing both forever.
Broken Double-Checked Locking
A lazy-initialization idiom that checks a field outside a lock, locks, then checks again — but without proper memory barriers it returns partially constructed objects.
Memory Leak
Allocating memory that is never released because references are unintentionally retained, so usage grows without bound until the process slows, thrashes, or crashes.
Unbounded Cache
A cache with no size limit, eviction, or expiry that grows until it consumes all memory and turns a performance optimization into an out-of-memory failure.
Retry Storm
Aggressive, uncoordinated retries during a partial outage that multiply traffic against an already-struggling dependency and turn a blip into a full collapse.
Resource Leak
Acquiring file handles, sockets, connections, or threads without reliably releasing them, so a finite pool is exhausted and the application stops being able to do work.
Over-Caching
Adding caches everywhere to chase speed, multiplying staleness, invalidation bugs, and operational complexity for marginal gains that profiling never justified.
Flaky Pipeline
A CI/CD pipeline that fails intermittently for reasons unrelated to the code change, eroding trust and training teams to ignore red builds.
No Monitoring (Flying Blind)
Running production systems with no metrics, logs, or alerts, so problems are invisible until users complain and incidents cannot be diagnosed.
Alert Fatigue
So many low-value or noisy alerts fire that responders become desensitized and start ignoring them, including the ones that actually matter.
Hero Culture
A team that depends on a few individuals heroically saving the day, rewarding firefighting over the boring, systemic work that prevents fires.
Playbooks3
SRE and SLO Program Playbook
A program to adopt Site Reliability Engineering with service level objectives, error budgets, and toil reduction to balance reliability and velocity.
Incident Management Program Playbook
A program to build structured incident response with clear roles, severity levels, blameless postmortems, and continuous learning.
Chaos Engineering and Resilience Program Playbook
A program to validate system resilience through controlled fault injection, hypothesis-driven experiments, and game days.
Benchmarks2
Flaky Test Rate Benchmark
Measures the prevalence and impact of nondeterministic tests that pass and fail without code changes, eroding trust in the test suite.
CI/CD Pipeline Reliability Benchmark
Measures how dependably a delivery pipeline succeeds, reporting pass rate, infrastructure-failure share, and mean time to recover a broken pipeline.
FAQs4
What is hallucination in LLMs?
Hallucination is when a language model produces text that sounds confident and plausible but is factually wrong or unsupported by its sources. It happ...
What is an SLO, SLI, and error budget?
An SLI (Service Level Indicator) is a measured metric of service health, such as request success rate or latency. An SLO (Service Level Objective) is ...
What is the difference between observability and monitoring?
Monitoring is collecting and alerting on predefined metrics and checks—it answers known questions like 'is CPU above 90%?' Observability is the broade...
What is chaos engineering?
Chaos engineering is the practice of deliberately injecting failures into a system to test its resilience before real incidents expose weaknesses. Tea...