Flink + Apache Paimon
A streaming lakehouse stack combining Apache Flink's unified stream-batch processing with Apache Paimon's update-optimized lake table format. It excels at CDC and near-real-time mutable tables over object storage.
Flink + Apache Paimon
This stack builds a streaming lakehouse with Apache Flink as the unified stream-and-batch engine and Apache Paimon as a lake table format designed for high-frequency updates. Paimon (formerly Flink Table Store) combines a log-structured merge tree with lake storage, so streaming changes and batch queries share one consistent table. It targets teams that want near-real-time lakehouse tables without separate streaming and batch systems.
Components
- Apache Flink: A stateful stream-processing engine with event-time semantics, exactly-once guarantees, and a unified API for streaming and batch. It powers both ingestion and continuous transformation.
- Apache Paimon: A lake table format using LSM trees for efficient upserts and changelog generation. It supports streaming reads and writes, time travel, and compaction over object storage.
- Object storage: S3, OSS, or HDFS holds Paimon's data and metadata files.
- CDC sources: Flink CDC connectors stream database changes directly into Paimon tables.
Strengths
- True streaming lakehouse. Paimon accepts continuous upserts and emits changelogs, enabling incremental downstream processing.
- Unified batch and stream. One Flink job model serves both, reducing duplicated pipelines.
- Efficient updates. The LSM design handles mutation-heavy workloads better than append-oriented formats.
- Open and portable. Both projects are open source and run on commodity object storage.
Trade-offs
- Maturity. Paimon is newer than Iceberg or Delta Lake, with a smaller ecosystem.
- Operational depth. Flink state, checkpoints, and Paimon compaction require tuning.
- Tooling breadth. Fewer turnkey integrations than established formats.
- Learning curve. Streaming semantics and LSM internals demand expertise.
When to Use It
Choose this stack when you need a lakehouse that ingests change data continuously and serves fresh, mutable tables, especially for CDC-driven pipelines and real-time analytics. It fits teams already invested in Flink. If your workload is append-only or batch-dominant, Iceberg or Delta Lake with Spark may be safer. For update-heavy streaming lakehouse use cases, Flink plus Paimon is a leading-edge fit.