Data Lakehouse Architecture

Best Practice: Data Lakehouse Architecture

A data lakehouse unifies the cheap, open, scalable storage of a data lake with the reliability features of a data warehouse: ACID transactions, schema enforcement, and fast SQL. It rests on open table formats that add a transactional metadata layer over files in object storage. The concept was formalized by Databricks researchers in 2021. It matters because it removes the need to copy data between separate lake and warehouse systems, reducing cost, staleness, and duplication. The lakehouse pattern emerged to end the long-standing split where raw data sat in a cheap lake while curated data was copied into an expensive warehouse, leaving two systems to keep in sync. By adding a transactional metadata layer over open file formats, the lakehouse offers warehouse-grade reliability on lake-grade economics. A growing ecosystem of engines can read the same open tables, which reduces lock-in and lets teams pick the right engine per workload.

Step-by-Step Implementation Guidance

1. Choose an open table format (Delta Lake, Apache Iceberg, or Apache Hudi) and standardize on it.
2. Store data in open file formats such as Parquet on object storage.
3. Enable ACID transactions and schema enforcement at the table layer.
4. Define a layered model (for example medallion) for raw, refined, and curated data.
5. Connect query engines and BI tools directly to the lakehouse tables.
6. Manage table maintenance: compaction, partitioning, and snapshot expiration.
7. Apply governance and fine-grained access control at the catalog level.

Common Mistakes Teams Make When Ignoring This Practice

• Maintaining parallel lake and warehouse copies that drift apart.
• Writing raw files without a transactional table layer, losing consistency.
• Ignoring small-file and compaction problems that degrade query speed.
• Locking data into a proprietary format that blocks future engines.
• Bolting governance on after the fact rather than at the catalog.
• Treating the lakehouse as just a file dump and skipping table maintenance, so query performance silently degrades.

Tools and Techniques That Support This Practice

• Open table formats: Delta Lake, Apache Iceberg, Apache Hudi.
• Query engines: Spark, Trino, Presto, Snowflake, Databricks SQL.
• Catalogs such as Unity Catalog and the Iceberg REST catalog.
• Object storage like Amazon S3, Azure Data Lake Storage, and Google Cloud Storage.
• Catalog services such as Unity Catalog or the Iceberg REST catalog for governance and multi-engine access.

How This Practice Applies to Different Migration Types

• Cloud Migration: Consolidate on-prem lakes and warehouses into one cloud lakehouse.
• Database Migration: Offload heavy analytical workloads from operational databases to the lakehouse.
• SaaS Migration: Centralize exported SaaS data into open tables for unified analytics.
• Codebase Migration: Point analytics services at lakehouse tables instead of bespoke extracts.
• Consolidation: Use a migration as the moment to collapse separate lake and warehouse copies into a single governed lakehouse.

Checklist

• Open table format selected and standardized
• Data stored in open file formats on object storage
• ACID transactions and schema enforcement enabled
• Layered data model defined
• Query and BI engines connected directly
• Table maintenance (compaction, partitioning) scheduled
• Catalog-level governance and access control applied