Streaming Lakehouse with Flink and Iceberg

Streaming Lakehouse with Flink and Iceberg

A streaming lakehouse unifies real-time and batch analytics by writing stream-processed data directly into open table-format tables that batch engines can also query. Apache Flink provides low-latency stateful stream processing, and Apache Iceberg provides ACID tables with snapshots over object storage. This platform on AWS lets one set of tables serve both fresh streaming queries and historical batch analysis. Use it when you need continuously updated tables without maintaining separate streaming and batch stores.

Components

• Kafka: the ingest backbone carrying source event streams.
• Apache Flink: stateful stream processor performing enrichment, joins, and aggregations with exactly-once semantics.
• Apache Iceberg: the open table format providing ACID commits, snapshots, and schema evolution on S3.
• AWS S3: durable object storage holding Iceberg data and metadata files.
• AWS Glue / Nessie: the catalog managing Iceberg table metadata and, with Nessie, git-like branching.
• Trino: the query engine for ad-hoc and batch SQL over Iceberg tables.

Data Flow

Events flow from sources into Kafka. Flink jobs consume topics, apply windowed aggregations and joins against state, and commit results into Iceberg tables on a short interval, producing new snapshots. Because Iceberg writes are atomic, readers always see a consistent table version. Trino and other engines query the same tables for dashboards and ad-hoc analysis, automatically seeing the latest committed snapshot. Compaction jobs periodically merge small files written by streaming.

Scaling and Resilience

Flink scales by parallelism across task slots and checkpoints state to S3 so it recovers exactly-once after failures. Iceberg's metadata layer handles large tables with hidden partitioning and partition pruning, while snapshot expiration and compaction keep performance healthy. S3 provides eleven nines of durability, and the catalog tracks every snapshot for time travel and rollback after a bad write. Streaming and query workloads scale independently.

Security

Kafka and Flink use TLS and authenticated access. S3 data is encrypted with KMS, and IAM roles scope which jobs read and write each table. The catalog enforces table-level permissions, and Nessie branches allow isolated, reviewable changes before merging to the main branch. Network isolation via VPC endpoints keeps traffic private.

Trade-offs and Alternatives

Streaming into Iceberg unifies pipelines and avoids the dual-system Lambda architecture, but small-file management, compaction, and exactly-once tuning add operational work. Latency is seconds, not milliseconds, so true sub-second use cases need a serving store too. Alternatives include Delta Lake with Spark Structured Streaming, Apache Hudi for upsert-heavy streams, or a managed service like Confluent plus Tableflow. Choose Flink plus Iceberg when you want open formats and one set of tables for both streaming and batch.