Watchdog

Some failures are not crashes but hangs: a process that deadlocks, an event loop that stalls, or a thread stuck in an infinite wait keeps running but does no useful work. Internal error handling cannot fix this, because the very code that would react is the code that is stuck. The Watchdog pattern adds an independent supervisor that observes the system from outside and forces recovery when the system stops making progress.

How It Works

The watchdog requires the monitored component to periodically prove liveness — by resetting a timer, renewing a lease, or responding to a probe. If the proof does not arrive before a deadline, the watchdog assumes the component is stuck and acts:

• Restart: kill and relaunch the hung process or container.
• Failover: promote a standby and route around the failure.
• Alert: page operators when automatic recovery is unsafe.

The defining property is independence: the watchdog runs in a separate process, thread, container, or even hardware so a hang in the monitored system cannot disable it. Hardware watchdog timers in embedded systems reset the whole device if firmware fails to "pet" the timer in time; software supervisors (systemd, Kubernetes liveness probes, Erlang/OTP supervisors, process managers) play the same role for services.

When to Use It

Use a watchdog for long-running processes that can hang without crashing, for unattended and embedded systems that must self-recover, and as a backstop for any critical service where a stall is as bad as a crash. It complements normal error handling by covering the failures error handling cannot reach.

Trade-offs

A poorly tuned watchdog restarts healthy-but-slow components, causing churn and masking the real problem (a service that restarts every few minutes may look "up" while never serving). Forced restarts can interrupt in-flight work and, if not idempotent, corrupt state. The watchdog itself is a critical component whose own failure or misjudgment can cause outages, so it must be simple, well-tested, and conservative.

Related Patterns

Watchdog consumes heartbeat-pattern signals and health-check probes to decide when to act, and relates to supervisor hierarchies (as in Erlang/OTP) and to circuit-breaker as a controlled-failure mechanism.

Example

A timer-reset watchdog:

watchdog: deadline = now + 10s
worker loop:
  do_work()
  watchdog.reset(now + 10s)   // prove progress each iteration

watchdog thread:
  if now > deadline: restart(worker)

Kubernetes provides this declaratively: a failing liveness probe is a watchdog that restarts the container.