Kappa Architecture

Kappa architecture simplifies big-data processing by eliminating the batch layer of lambda architecture. There is one processing path: a stream processor that consumes an immutable, replayable log. "Batch" results are obtained not by a separate system but by replaying the log from the beginning through the same streaming code. One codebase serves both real-time and historical computation.

How It Works

All data is ingested as an append-only event log, typically Kafka or Pulsar with long retention. A stream processor (Flink, Kafka Streams, Spark Structured Streaming) consumes the log to maintain materialized views or output topics. When logic changes, when a bug is fixed, or when a new view is needed, a new processor instance replays the relevant log history from the desired offset, builds a fresh view, and then switches consumers over to it. The old view is retired. Because there is no batch layer, there is no second implementation to keep in sync.

When to Use It

Kappa suits systems that are naturally stream-oriented and where the source log can be retained or reconstructed: real-time analytics, fraud detection, monitoring, and event-sourced applications. It is the right choice when you want a single mental model and codebase, and when most processing is incremental with occasional full reprocessing.

Trade-offs

Kappa depends on a durable, replayable log; retaining enough history can be costly, and reprocessing very large histories through a stream engine can be slow compared with an optimized batch job. Some heavy analytical workloads (large joins, complex ML training) remain easier in batch. Exactly-once semantics, state management, and reprocessing orchestration push complexity into the stream layer. Kappa removes lambda's duplicated logic but raises the bar on streaming-engine maturity and log retention strategy.

Related Patterns

Kappa is the streaming-only evolution of lambda-architecture, built on an event-driven-architecture log, and often refines data through medallion-architecture layers on a lakehouse.

Example

A fraud-detection platform ingests transactions into a long-retention Kafka topic. A Flink job scores transactions in real time. When the data team improves the scoring model, they deploy a new Flink job that replays months of transactions from the log to rebuild aggregate features, then promote it to live traffic — all without a separate batch pipeline.