Pipes and Filters

The Pipes and Filters pattern breaks a complex processing task into a chain of discrete, independent components (filters) connected by channels (pipes). Each filter performs one well-defined transformation and passes its output to the next, building a processing pipeline.

The problem is a monolithic task that does many things in sequence. Such code is hard to maintain, reuse, and scale, because all steps are coupled and must run together at the same scale.

How It Works

Each filter has a single responsibility and a uniform interface: it consumes input from an inbound pipe, transforms it, and emits output to an outbound pipe. Filters do not share state and are unaware of each other; they only know their input and output contracts. Pipes are often message queues or streams, which decouple filters in time and allow buffering.

Because filters are independent, they can be developed, tested, deployed, replaced, and scaled separately. A slow filter can run more instances (combined with Competing Consumers) without scaling the rest. Filters can be recombined into different pipelines for reuse.

When to Use It

Use it for multi-stage data processing where steps are distinct and may need different scaling or technologies: ETL, media transcoding, log and event processing, and document ingestion pipelines. Use it when you want reusable, composable processing steps.

Avoid it when steps are tightly coupled and share substantial state, or when the per-stage overhead and added latency outweigh the modularity benefit for a simple task.

Trade-offs

The pipeline adds latency from passing data between stages and overhead from the pipes themselves. End-to-end reliability requires handling failures at each stage, often with idempotent filters and retries, since a message may be reprocessed. Maintaining state across filters is awkward by design; if a step needs context, it must travel in the message.

The benefit is modularity, independent scaling, and reuse of processing components.

Related Patterns

It frequently uses Competing Consumers to scale individual filters and Claim Check to pass large payloads between stages efficiently. Sequential Convoy preserves ordering within a partition through a pipeline. It underlies many streaming and ETL architectures.

Example

A document ingestion pipeline has filters for extract-text, detect-language, redact-PII, and index. Each filter reads from its input queue, does one job, and writes to the next queue. The redaction step is CPU-heavy, so it runs ten instances as competing consumers while the others run two each. Large documents are passed by reference using a claim check. Adding a new translate step is as simple as inserting a filter and two queues, without touching the others.