Lambda Architecture

Lambda architecture is a data-processing design that combines two parallel paths to balance latency, throughput, and accuracy. A batch layer processes the complete, immutable dataset to produce accurate, comprehensive views, while a speed layer processes only recent data in near real time to fill the gap before the next batch run. A serving layer merges the two so queries see both historical accuracy and fresh updates.

How It Works

All incoming data is appended to an immutable master dataset. The batch layer (historically Hadoop/Spark) recomputes batch views from scratch or incrementally on a schedule; because it reprocesses raw data, errors are self-correcting on the next run. The speed layer (Storm, Spark Streaming, Flink) computes real-time views over data that has arrived since the last batch, accepting approximation for speed. The serving layer indexes batch views for fast reads, and queries combine the batch view with the speed view to produce a complete answer. Once a batch run absorbs recent data, the corresponding speed-layer state is discarded.

When to Use It

Lambda fits high-volume systems that need both reliable historical analytics and low-latency views of recent activity, and that benefit from the ability to reprocess history (for bug fixes, new metrics, or schema changes). It was the dominant big-data pattern of the 2010s for clickstream, telemetry, and IoT analytics.

Trade-offs

The defining drawback is duplicated logic: the same computation must be implemented and maintained twice, once for batch and once for streaming, in different frameworks. Keeping the two in sync is error-prone, and the serving-layer merge adds complexity. Operational cost is high. These pains motivated kappa architecture, which uses a single streaming path and reprocesses by replaying the log. Modern unified engines (Flink, Spark Structured Streaming) and lakehouse table formats have further reduced lambda's appeal.

Related Patterns

Kappa-architecture is the streaming-only successor; medallion-architecture organizes the layered refinement of data on a lakehouse; and an event-driven-architecture typically feeds the ingestion log.

Example

A telemetry platform appends device events to a durable log. A nightly Spark batch job recomputes accurate daily aggregates, while a Flink job maintains approximate live counters for the current day. Dashboards merge yesterday's exact figures with today's streaming estimates, and each night's batch run replaces the estimates with corrected values.