Competing Consumers

Competing Consumers has multiple instances of a consumer read from a single message queue. Each message is delivered to exactly one consumer, so the consumers compete for work. This spreads load across instances and lets the system scale horizontally.

The problem is handling a fluctuating volume of asynchronous tasks. A single consumer becomes a bottleneck and a single point of failure. Adding more consumers that pull from the same queue increases throughput without changing the producers.

How It Works

Producers post messages to a queue such as Amazon SQS, Azure Service Bus, or RabbitMQ. A pool of consumer instances reads from the queue. The broker ensures a given message is handed to only one consumer at a time, typically by making it invisible (a visibility timeout or lock) while being processed.

When a consumer finishes, it acknowledges or deletes the message. If it crashes or times out, the message becomes visible again and another consumer retrieves it. Consumers should be idempotent because a message can be delivered more than once under at-least-once semantics.

The pool scales with queue depth: autoscaling rules add instances when the backlog grows and remove them when it shrinks.

When to Use It

Use it when work arrives as discrete, independent messages that can be processed in parallel and in any order: image processing, email sending, order fulfillment, and background jobs. It pairs naturally with queue-based load leveling.

Avoid it when messages must be processed in strict order, unless you partition ordering (see Sequential Convoy), or when tasks share state that makes parallelism unsafe.

Trade-offs

Consumers must be idempotent and tolerate out-of-order delivery. Poison messages that always fail can block progress; a dead-letter queue isolates them. Scaling logic and visibility timeouts need tuning so messages are neither processed twice unnecessarily nor stuck.

The pattern increases throughput and resilience but adds complexity in monitoring, retries, and ensuring exactly-once effects through idempotency keys.

Related Patterns

It is almost always combined with Queue-Based Load Leveling, which absorbs spikes. Priority Queue routes urgent work to dedicated consumers. Sequential Convoy preserves ordering within partitions while still allowing parallelism across them.

Example

An e-commerce platform writes each placed order to an SQS queue. An autoscaling group of worker containers reads from the queue; each worker takes one order, charges payment, reserves inventory, and emits a confirmation, then deletes the message. During a flash sale the queue depth climbs, autoscaling adds workers, and the backlog drains. If a worker crashes mid-order, the message reappears after the visibility timeout and another worker reprocesses it idempotently using the order ID as a dedup key.