Heartbeat

In a distributed system, a silent component is ambiguous: it might be busy, slow, partitioned, or dead. The Heartbeat pattern resolves the ambiguity by having each component periodically emit a small "I am alive" signal. If the signal stops arriving for longer than a threshold, observers conclude the component has failed and act — failing over, removing it from a cluster, or triggering an alert.

How It Works

A heartbeat is a lightweight message sent at a fixed interval (say every few seconds). A monitor tracks the last time it heard from each peer. If now - last_seen > timeout, the peer is declared dead. Key design parameters:

• Interval: how often heartbeats are sent. Shorter intervals detect failures faster but cost more traffic.
• Timeout / miss count: how long, or how many missed beats, before declaring failure. Larger margins reduce false positives from transient delays.
• Direction: push (the component sends to a monitor) or pull (the monitor polls, blurring into health checks).

Heartbeats underpin cluster membership protocols, leader election (a leader's heartbeats keep followers from starting an election), and failure detectors in systems like Kubernetes node leases, ZooKeeper/etcd sessions, and database replication.

When to Use It

Use heartbeats wherever you must detect failure of a peer, worker, or leader within a bounded time: distributed coordination, primary/replica failover, job-worker liveness, and session keep-alives. They are the basis of most automatic failover and membership systems.

Trade-offs

Heartbeats cannot perfectly distinguish a crashed node from a slow or network-partitioned one — the classic limitation of failure detection. Aggressive timeouts cause false positives (healthy nodes evicted, needless failovers, split-brain risk); lenient timeouts slow detection. Heartbeat traffic and monitoring state grow with cluster size. Tuning interval and timeout against the network's real latency distribution is essential.

Related Patterns

Heartbeat is closely related to health-check (often pull-based liveness/readiness probing), watchdog (which acts when heartbeats stop), circuit-breaker (which can use liveness signals), and leader-election (which relies on leader heartbeats).

Example

A monitor declaring a worker dead:

workers[id].last_seen = now()   // on each heartbeat

// background sweep
for id, w in workers:
  if now() - w.last_seen > 3 * interval:
    mark_dead(id); reassign_work(id)

Kubernetes uses node Lease objects as heartbeats: a node that stops renewing its lease is eventually marked NotReady and its pods are rescheduled.