Lakehouse Migration Program Playbook
A program for consolidating a separate data lake and warehouse into a unified lakehouse on open table formats. It covers format selection, medallion design, validated migration, and the file-layout optimization lakehouses require.
A lakehouse unifies the low-cost, open storage of a data lake with the reliability and performance of a warehouse, using open table formats such as Apache Iceberg, Delta Lake, or Apache Hudi. These formats add ACID transactions, schema evolution, and time travel on top of columnar files in object storage. This playbook consolidates separate lake and warehouse silos into one governed lakehouse, reducing duplication and cost.
The format decision is consequential. It affects engine compatibility, governance, and how easily you can avoid lock-in later, so make it deliberately with the workloads you actually run.
Phase-by-Phase
Architecture and Format Selection. Assess the current lake and warehouse estate, select an open table format based on engine support and feature needs, and design the medallion layering.
Platform Foundation. Provision object storage and compute, implement the table format layer with governance, and set up least-privilege access from the start.
Data and Workload Migration. Migrate lake data and warehouse workloads onto the unified platform, then validate consistency between old and new systems before consumers switch.
Optimization and Cutover. Open table formats suffer from the small-files problem; implement compaction and clustering, tune query performance, and decommission the legacy silos.
Team and Roles
A data architect owns format selection and layering. Data engineers migrate data and workloads. A DBA supports performance tuning. DevOps provisions infrastructure. Security defines the access and governance model.
Risks and Mitigations
Data consistency during migration is managed with validation and dual-run. The small-files problem is addressed with compaction strategies. Format lock-in is reduced by choosing an open, multi-engine format. Governance gaps are closed with least-privilege access and lineage.
Success Criteria
Target strong query latency, reduced storage cost from consolidation, verified data parity, and full ACID compliance on critical tables.
Tooling
Use object storage such as S3 or R2, an open table format for ACID and schema evolution, Terraform for provisioning, and Datadog for observability. Compaction and maintenance jobs keep file layout healthy.