Resilience
167 items tagged with "resilience"
Best Practices9
AWS Well-Architected Framework
A set of cloud design principles and check-lists for building secure, high-performing, resilient, and efficient workloads on AWS.
Saga Pattern
A pattern for managing data consistency across microservices using a sequence of local transactions coordinated by events or a central orchestrator, with compensating actions on failure.
Circuit Breaker Pattern
A resilience pattern that stops calls to a failing dependency once errors cross a threshold, preventing cascading failures and giving the dependency time to recover.
Bulkhead Pattern
A resilience pattern that isolates resources into separate pools so a failure or overload in one part of a system cannot consume the resources others depend on.
Cell-Based Architecture
An architecture that partitions a system into independent, self-contained cells, each serving a subset of traffic, to limit blast radius and scale through replication.
Idempotency Keys
A pattern where clients send a unique key with unsafe requests so the server can safely retry without applying the same operation twice, preventing duplicate charges or records.
API Rate Limiting
Controlling how many requests a client can make in a time window to protect API capacity, ensure fair use, and defend against abuse, using algorithms like token bucket.
Webhook Best Practices
Guidance for sending and receiving reliable webhooks: signature verification, idempotent handlers, retries with backoff, and fast acknowledgement of events.
Progressive Enhancement
A frontend strategy that builds a baseline experience with semantic HTML first, then layers CSS and JavaScript so the site works for every browser and device.
Patterns44
Service Registry
A database of available service instances and their network locations, kept current as instances start, stop, and fail.
Ambassador
An out-of-process helper that proxies network calls on behalf of an application, handling connectivity concerns transparently.
Transactional Outbox
Reliably publishes messages by writing them to an outbox table in the same local transaction as the business data change.
Leader Election
Designates a single instance among many to coordinate work, with automatic failover if the leader becomes unavailable.
Distributed Lock
Coordinates exclusive access to a shared resource across multiple processes or nodes that do not share memory.
Competing Consumers
Enable multiple concurrent consumers to process messages from the same queue to increase throughput and improve resilience.
Queue-Based Load Leveling
Use a queue between tasks and a service to smooth intermittent heavy loads and protect the service from being overwhelmed.
Throttling
Control the consumption of resources by an instance, tenant, or service so a system stays within capacity under load.
Gatekeeper
Protect services by brokering all client requests through a dedicated host that validates and sanitizes them before forwarding.
Compensating Transaction
Undo the completed steps of a multi-step operation when one step fails, restoring consistency without distributed ACID transactions.
Geode
Deploy independent geographically distributed nodes that each serve any request, placing compute close to users worldwide.
Deployment Stamps
Deploy multiple independent copies of a full application stack to scale, isolate tenants, and contain failures.
Health Endpoint Monitoring
Expose health-check endpoints that monitoring tools and load balancers probe to verify an application is functioning correctly.
Rate Limiting
Constrain the rate of operations against a service or resource to stay within quotas and avoid throttling or overload.
Retry
Automatically reattempt a failed operation that is likely transient, using backoff and limits to recover without user impact.
Idempotent Receiver
Makes a consumer safely handle duplicate messages so that processing the same message more than once has the same effect as processing it once.
Dead Letter Channel
Routes messages that cannot be delivered or processed to a dedicated channel for inspection and recovery instead of discarding or blocking them.
Guaranteed Delivery
Persists messages so they are not lost if the sender, broker, or receiver fails, ensuring each message is eventually delivered despite outages.
Idempotent Writer
A pattern that makes repeated writes safe by ensuring duplicate operations produce the same result, essential for at-least-once delivery and retries.
Actor Model
A concurrency model where independent actors encapsulate state and communicate only by asynchronous messages, avoiding shared memory and locks.
Pessimistic Locking
A concurrency strategy that acquires locks on data before use to prevent concurrent modification, ensuring correctness under high contention at the cost of throughput.
Two-Phase Commit (2PC)
A distributed-transaction protocol that coordinates multiple participants to commit or abort atomically through a prepare phase and a commit phase.
Retry with Backoff
Automatically re-attempts a failed operation after progressively longer waits, smoothing over transient faults without overwhelming a struggling dependency.
Exponential Backoff with Jitter
Adds randomness to exponentially growing retry delays so that many clients do not retry in lockstep and overwhelm a recovering service.
Timeout
Bounds how long a caller waits for an operation, freeing resources and surfacing failures fast instead of blocking indefinitely on a slow or hung dependency.
Bulkhead
Isolates resources into independent pools so a failure or overload in one part of a system cannot consume capacity needed by the rest.
Rate Limiter
Caps how many requests a client or system may make in a time window, protecting services from overload, abuse, and runaway cost.
Load Shedding
Deliberately rejects or drops lower-priority work when a system nears capacity, preserving stability and protecting high-priority requests under overload.
Fallback
Provides an alternative response or behavior when a primary operation fails, keeping the system useful instead of returning an error to the user.
Graceful Degradation
Keeps core functionality working by selectively disabling or simplifying non-essential features when parts of a system fail or are overloaded.
Fail Fast
Detects invalid state or unavailable dependencies as early as possible and reports the error immediately, rather than continuing into deeper, costlier failure.
Fail Safe
Designs a system so that when a component fails it falls into a safe, known default state rather than an unsafe or undefined one.
Heartbeat
Has a component emit periodic signals so observers can detect when it has failed or become unreachable within a bounded time.
Health Check
Exposes endpoints that report whether a service is alive and ready to serve, enabling orchestrators and load balancers to route traffic only to healthy instances.
Watchdog
An independent supervisor that monitors a system or process and takes corrective action — restart, alert, or failover — when it stops responding.
Backpressure
Lets a slow consumer signal upstream producers to slow down, preventing unbounded queues and memory exhaustion when demand exceeds processing capacity.
Idempotency Key
Attaches a unique key to a request so the server can detect and de-duplicate retries, making non-idempotent operations safe to repeat.
Dead-Letter Queue
Routes messages that cannot be processed after repeated attempts to a separate queue for inspection and recovery, keeping the main pipeline flowing.
Poison Message Handling
Detects and quarantines messages that repeatedly crash or block a consumer, preventing one bad message from stalling an entire queue.
Hedged Requests
Sends a duplicate request to another replica after a delay, taking whichever response returns first to cut tail latency from slow servers.
Request Coalescing
Merges multiple concurrent identical requests into a single backend call and shares the result, preventing duplicate work and cache-stampede overload.
Idempotency Key
A client-supplied unique key lets a server detect and dedupe retried requests, so repeated submissions produce the same result exactly once.
Defense in Depth
Layers multiple independent security controls so that if one fails, others still protect the system, avoiding reliance on any single defense.
Token Bucket (Rate Limiting)
A rate-limiting algorithm where requests consume tokens refilled at a steady rate, allowing controlled bursts while capping the long-run request rate.
Anti-Patterns25
Vendor Lock-In
Designing a system so deeply around one provider's proprietary services that switching becomes prohibitively expensive, eroding negotiating power and portability.
Exception Swallowing
Catching exceptions and then ignoring them, hiding failures so that errors pass silently and bugs become nearly impossible to diagnose.
Dual Write
Writing the same change to two systems in sequence without a single transaction or log, so a failure between them leaves the stores inconsistent.
No Connection Pooling
Opening a fresh database connection per request or query and closing it after, paying high handshake cost and exhausting connection limits under load.
Busy-Waiting (Spin-Waiting)
Repeatedly polling a condition in a tight loop instead of blocking, burning CPU cycles while waiting for an event that the scheduler could deliver for free.
Lock Contention
Many threads competing for the same lock, serializing work that should run in parallel and turning a multicore machine into an expensive single-core one.
Deadlock-Prone Locking
Acquiring multiple locks in inconsistent orders so two threads can each hold one lock while waiting for the other, freezing both forever.
Thread-Per-Request Overload
Spawning a dedicated OS thread for every incoming request, so concurrency is capped by thread count and memory, collapsing under load instead of degrading gracefully.
Memory Leak
Allocating memory that is never released because references are unintentionally retained, so usage grows without bound until the process slows, thrashes, or crashes.
Cache Stampede (Dogpile Effect)
When a hot cache entry expires, many concurrent requests all miss and recompute it at once, hammering the backing store and amplifying load instead of absorbing it.
Retry Storm
Aggressive, uncoordinated retries during a partial outage that multiply traffic against an already-struggling dependency and turn a blip into a full collapse.
Thundering Herd
Many clients or threads waking and acting at the same instant — on a recovery, a timer, or a wakeup — creating a synchronized load spike that overwhelms the resource they target.
Head-of-Line Blocking
A single slow or stuck item at the front of a strictly ordered queue holds up everything behind it, even when the later items are unrelated and ready to proceed.
Missing Back-Pressure
A producer that pushes work faster than the consumer can handle, with no mechanism to slow it down, so queues grow unbounded until memory or the downstream collapses.
Resource Leak
Acquiring file handles, sockets, connections, or threads without reliably releasing them, so a finite pool is exhausted and the application stops being able to do work.
Snowflake Server
A server hand-configured over time into a unique, irreproducible state that no one can recreate, document, or safely replace.
Pets vs Cattle (Pet Servers)
Treating individual servers as irreplaceable pets that are named, nurtured, and manually healed, instead of disposable cattle that are replaced on failure.
No Rollback Plan
Deploying with no tested, fast way to revert, so a bad release means scrambling under pressure while the outage drags on.
Deploy and Pray
Pushing releases to production with no automated verification, monitoring, or rollback, then hoping nothing breaks instead of knowing it works.
Single Region, No Disaster Recovery
Running an entire system in one region or data center with no disaster-recovery plan, so a regional failure takes everything down with no recovery path.
Bus Factor of One
Critical knowledge or capability concentrated in a single person, so the project halts if that person becomes unavailable.
Happy-Path-Only Testing
Tests that exercise only the expected, valid flow and ignore errors, edge cases, and failures — leaving real-world conditions untested.
Ignoring Idempotency
Designing write operations that cause duplicate effects when retried, so network blips and client retries create double charges or duplicate records.
Synchronous Call Chains
Deep chains of blocking request-response calls between services, so latency compounds and one slow or failed service cascades into widespread failure.
Death Star (Distributed Big Ball of Mud)
A microservice estate where every service calls nearly every other with no clear boundaries, producing a tangled mesh impossible to change or reason about.
Tutorials7
How to configure liveness, readiness, and startup probes in Kubernetes
Add health probes so Kubernetes restarts unhealthy pods and only routes traffic to ready ones.
How to decouple services with Amazon SQS
Use Amazon SQS queues to decouple producers and consumers, with dead-letter queues and Lambda triggers.
How to orchestrate workflows with AWS Step Functions
Build a state machine with AWS Step Functions to orchestrate Lambda tasks with retries, branching, and error handling.
How to build a multi-region active-active app on AWS
Architect a multi-region active-active application on AWS with Route 53, global tables, and health-based failover.
How to do blue-green deployments on AWS with CodeDeploy
Run zero-downtime blue-green deployments for containers on Amazon ECS using AWS CodeDeploy and a load balancer.
How to perform a zero-downtime database schema migration
Use the expand-and-contract pattern to change a live schema without downtime, keeping old and new code compatible during rollout.
How to Add Rate Limiting to an API
Protect an API with token-bucket rate limiting, return standard rate-limit headers, and share limits across instances with Redis.
Blueprints9
Single-Region to Multi-Region Active-Active Blueprint
Evolve a single-region cloud deployment into a multi-region active-active architecture for resilience, low latency, and disaster recovery.
Active-Passive DR to Active-Active Resilience Blueprint
Convert a cold or warm active-passive disaster-recovery setup into an always-on active-active architecture that uses all capacity and removes failover risk.
Synchronous REST to Event-Driven Blueprint
Convert tightly-coupled synchronous REST call chains into an event-driven architecture using a broker, async messaging, and the outbox pattern.
Stateful Monolith to Stateless Services Blueprint
Re-architect a session-bound stateful backend into horizontally scalable stateless services with externalized session and cache state.
Relational to Apache Cassandra Migration Blueprint
Migrate a relational workload to Apache Cassandra with query-first data modeling for write-heavy, globally distributed scale.
Synchronous Calls to Event-Driven (Kafka) Blueprint
Replace brittle synchronous service-to-service calls with asynchronous events on Apache Kafka to decouple services and improve resilience.
Synchronous API to SNS/SQS Fan-Out Blueprint
Decouple synchronous workloads using Amazon SNS topics and SQS queues for durable, buffered fan-out and asynchronous processing.
Webhooks to Event Streaming Blueprint
Evolve unreliable HTTP webhook integrations into a durable event-streaming backbone with replay, ordering, and at-least-once delivery.
Central Orchestration to Choreography (Saga) Blueprint
Replace a central orchestrator coordinating distributed transactions with event-driven choreography using the saga pattern and compensations.
Reference Architectures20
Multi-Region Active-Active
Architecture for globally distributed applications with active-active failover
Multi-Region Active-Active Web Platform
A multi-cloud active-active web platform serving users from multiple regions with global routing and replicated data for high availability.
Autoscaling Web Tier on AWS
A classic three-tier web application on AWS with an autoscaling compute tier behind a load balancer and a managed relational database.
Multi-Region Disaster Recovery on AWS
Cross-region pilot-light and warm-standby design that meets aggressive recovery objectives for critical workloads.
Active-Active Resilience Across Clouds
Multi-cloud design that serves traffic from two providers simultaneously to survive a full provider outage.
Backup and Restore Architecture on Azure
Policy-driven backup design with immutable, geo-redundant recovery points and tested restore for workloads and data.
Global Edge and CDN Architecture
Multi-CDN edge design that caches content, runs logic at the edge, and routes users to the nearest healthy origin.
Hybrid Cloud Connectivity Network
Resilient private connectivity between on-premises data centers and cloud using dedicated links and redundant VPNs.
Chaos Engineering and Resilience Platform on AWS
Controlled fault-injection platform that validates resilience hypotheses against production-like systems safely.
CQRS with Event Sourcing
A command-query separated system where state is derived from an append-only event log and read models are projected for queries.
Saga Orchestration for Distributed Transactions
An orchestrated saga design that coordinates multi-service business transactions with compensating actions instead of two-phase commit.
Reliable Webhook Delivery Platform
A platform that delivers outbound webhooks to customer endpoints with retries, signing, idempotency, and per-tenant rate control.
Pub/Sub Fan-Out for Event Distribution
A publish-subscribe fan-out architecture that broadcasts each event to many independent consumers with per-subscriber queues.
Asynchronous Task Queue with Workers
A durable task queue that offloads slow or unreliable work from request handlers to scalable background workers with retries.
Global Edge API Gateway
A globally distributed edge gateway that authenticates, caches, and routes API traffic close to users across multiple regions.
Transactional Outbox for Reliable Events
A transactional outbox design that guarantees events are published exactly when their database changes commit, avoiding dual-write loss.
Real-Time Collaboration Application
A web app where many users edit shared documents concurrently, synchronized over WebSockets with conflict-free replicated data.
E-Commerce Platform on Microservices
A modular online store splitting catalog, cart, checkout, payments, and orders into independent services with event-driven coordination.
Content Platform with Global CDN
A high-traffic content site delivering articles and media worldwide through a multi-tier CDN cache in front of a publishing backend.
Video Streaming Platform
A platform that ingests, transcodes, packages, and delivers on-demand and live video at scale using adaptive bitrate over a CDN.
Playbooks6
Multi-Region Resilience Program Playbook
A phased program to make a critical application survive a full regional outage through active-active or active-passive multi-region architecture.
Disaster Recovery Program Playbook
A phased program to design, implement, and continuously test disaster recovery for critical systems against defined RTO and RPO targets.
Service Mesh Adoption Program Playbook
A phased program to roll out a service mesh for mTLS, traffic management, and observability across a Kubernetes microservices estate.
Zero-Downtime Database Migration Program Playbook
Migrate a production database engine or version with no downtime using dual-write, CDC replication, shadow reads, and staged cutover.
Chaos Engineering and Resilience Program Playbook
A program to validate system resilience through controlled fault injection, hypothesis-driven experiments, and game days.
Real-Time Inference Program Playbook
A program to deliver low-latency, high-throughput real-time ML inference with autoscaling, feature freshness, and strict SLOs.
Checklists18
Kubernetes Production Readiness Checklist
Confirm a Kubernetes cluster and its workloads are secure, observable, and resilient before serving production traffic.
Multi-Region Failover Readiness Checklist
Verify your application can fail over to a secondary region and meet its recovery objectives under a regional outage.
AWS Well-Architected Review Checklist
Run a structured Well-Architected review across the six pillars to find and prioritize risks in an AWS workload.
Disaster Recovery Test Checklist
Plan and run a disaster-recovery test that proves backups, runbooks, and recovery objectives actually work end to end.
Serverless Production Readiness Checklist
Confirm a serverless application is observable, secure, resilient, and cost-aware before it serves production traffic.
Microservice Production-Readiness Checklist
Confirm a new or extracted microservice meets operational, security, and resilience bars before it serves production traffic.
Zero-Downtime Database Migration Checklist
Checks for migrating a production database with no service interruption using dual-write, CDC, and gradual cutover techniques.
Streaming Pipeline Readiness Checklist
Production-readiness checks for real-time streaming data pipelines built on platforms such as Apache Kafka or Pulsar.
Backup and Restore Verification Checklist
Verification checks confirming database and data backups are complete, secure, and reliably restorable within recovery targets.
Rollback Readiness Checklist
Confirm a service can be reverted to a known-good state quickly and safely, covering artifacts, data, configuration, and traffic.
Incident Response Readiness Checklist
Verify the people, processes, and tooling needed to detect, respond to, and learn from production incidents are in place.
On-Call Handover Checklist
Ensure a clean transfer of on-call responsibility with full context on ongoing issues, risks, and operational state.
Observability & SLO Review Checklist
Assess whether a service is observable enough to operate, with meaningful SLOs, golden-signal metrics, tracing, and actionable alerts.
Disaster Recovery Readiness Checklist
Confirm an organization can recover critical services and data within defined objectives after a major failure.
gRPC Rollout Checklist
Pre-flight items for rolling out gRPC services across a system, covering contracts, compatibility, security, and observability.
Webhook Reliability Checklist
Verification items for delivering and consuming webhooks reliably, covering signing, retries, idempotency, and ordering.
Third-Party Integration Cutover Checklist
Cutover items for switching to or replacing a third-party API or vendor integration with minimal disruption.
Message Broker Migration Checklist
Migration items for moving messaging workloads to a new broker while preserving delivery guarantees and ordering.
Stacks3
Rust Axum/Actix Backend Stack
High-performance, memory-safe Rust backend using Axum or Actix with PostgreSQL for low-latency, resource-efficient services.
Phoenix Elixir Stack
Concurrent Elixir backend using the Phoenix framework on the BEAM VM with PostgreSQL for fault-tolerant, real-time, highly concurrent services.
Event-Driven Microservices Stack
Asynchronous microservices architecture using Kafka as an event backbone with independently deployable services for loose coupling and scalability.
FAQs8
What is the difference between cloud regions and availability zones?
A region is a distinct geographic area where a cloud provider operates data centers, chosen for proximity to users, latency, and data-residency requir...
What is multi-cloud?
Multi-cloud is the practice of using services from more than one cloud provider, such as AWS, Azure, and Google Cloud, within a single organization or...
What is rate limiting and how does it work?
Rate limiting caps how many requests a client may make in a given window to protect a service from overload, abuse, and runaway costs. Common algorith...
What does idempotency mean in APIs?
An operation is idempotent if performing it multiple times has the same effect as performing it once. In HTTP, GET, PUT, and DELETE are defined as ide...
What is an idempotency key?
An idempotency key is a unique client-generated identifier, often a UUID, sent with a request so the server can recognize and safely deduplicate retri...
What is a load balancer?
A load balancer distributes incoming network traffic across multiple servers so no single instance becomes a bottleneck, improving availability and sc...
What is the difference between blue-green and canary deployments?
Both are strategies for releasing new versions with minimal risk. In a blue-green deployment you run two identical environments—one live (blue) and on...
What is chaos engineering?
Chaos engineering is the practice of deliberately injecting failures into a system to test its resilience before real incidents expose weaknesses. Tea...
Glossaries17
Availability Zone
An availability zone is one or more discrete data centers within a cloud region, with independent power, cooling, and networking, designed to be isolated from failures in other zones.
Region
A region is a geographic area where a cloud provider operates a cluster of data centers, organized into availability zones, in which customers deploy and store resources.
ReplicaSet
A ReplicaSet is a Kubernetes controller that ensures a specified number of identical pod replicas are running at all times, recreating pods that fail or are deleted.
Replication
Replication is the process of copying and maintaining database data across multiple servers so that the same data is available on more than one node, improving availability, fault tolerance, and read scalability.
Defense in Depth
Defense in depth is a security strategy that layers multiple, independent controls so that if one defense fails, others continue to protect the system.
Rate Limiting
Rate limiting is a technique that caps how many requests a client may make to a service within a time window, protecting capacity and enforcing fair usage.
Idempotency
Idempotency is the property that performing an operation multiple times produces the same result as performing it once, making safe retries possible in distributed systems.
Canary Deployment
A release strategy in which a new version is rolled out to a small subset of users or servers first, so its behavior can be observed before exposing the whole user base.
Service Level Objective (SLO)
A target value or range for a service level indicator over a period of time, expressing the desired level of reliability for a service.
Service Level Indicator (SLI)
A quantitative measure of a specific aspect of a service's level of service, such as the proportion of successful requests or requests served within a latency threshold.
Error Budget
The maximum amount of unreliability a service is allowed over a period, calculated as the difference between 100% and its service level objective.
Toil
Manual, repetitive, automatable operational work that scales linearly with service size and provides no lasting value, a key target for reduction in site reliability engineering.
Incident Management
The coordinated process for detecting, responding to, mitigating, and resolving unplanned disruptions to a service, then learning from them.
Postmortem
A written, blameless analysis produced after an incident that documents what happened, the impact, the root causes, and the actions to prevent recurrence.
On-Call
An arrangement in which designated engineers are available to respond to alerts and incidents outside normal working hours, usually on a rotating schedule.
Mean Time to Recovery (MTTR)
The average time taken to restore a service after a failure, measured from the start of an incident to its resolution.
Idempotent Operation
An idempotent operation produces the same result whether it is performed once or many times, so repeating it has no additional effect beyond the first successful application.