Medallion Architecture

Best Practice: Medallion Architecture

Medallion architecture is a data design pattern that organizes a lakehouse into progressively refined layers: Bronze for raw ingested data, Silver for cleaned and conformed data, and Gold for curated, business-ready aggregates. Popularized by Databricks, it gives each layer a clear purpose and contract, so quality and structure improve as data flows through. It matters because it makes pipelines easier to reason about, debug, and reuse, while preserving raw data for reprocessing. The layers are sometimes described as multi-hop: each hop adds value and trust while keeping the previous layer intact. Because Bronze is immutable raw history, you can always rebuild Silver and Gold from scratch after fixing a transformation bug, which is invaluable during audits or incident response. The pattern pairs naturally with the lakehouse, where open table formats give each layer ACID guarantees and time travel.

Step-by-Step Implementation Guidance

1. Land source data unchanged in the Bronze layer, keeping full history and source metadata.
2. In Silver, deduplicate, standardize types, apply quality rules, and conform to shared keys.
3. In Gold, model business-level aggregates and metrics for analytics and reporting consumers.
4. Make each layer reproducible from the one below so you can rebuild after logic changes.
5. Apply schema enforcement and expectations at the Bronze-to-Silver boundary.
6. Document the contract and grain of each Gold table for its consumers.
7. Use incremental processing to keep refresh costs and latency manageable.

Common Mistakes Teams Make When Ignoring This Practice

• Transforming raw data in place, leaving no source of truth to reprocess.
• Mixing cleaning and business logic in a single opaque job.
• Letting Gold tables proliferate without clear ownership or grain.
• Skipping quality checks between layers.
• Treating Bronze as a dumping ground with no metadata.
• Reprocessing from a mutated layer, so a fix in logic cannot be applied to historical data.

Tools and Techniques That Support This Practice

• Open table formats such as Delta Lake, Apache Iceberg, and Apache Hudi.
• Spark, Databricks, and SQL warehouses for layer transformations.
• Orchestration via Airflow, Dagster, or Databricks Workflows.
• dbt for modeling Silver and Gold layers.
• Time travel and table versioning features in Delta Lake and Iceberg for safe reprocessing.

How This Practice Applies to Different Migration Types

• Cloud Migration: Re-land legacy extracts into Bronze, then rebuild Silver and Gold in the cloud.
• Database Migration: Use Bronze as a faithful copy to validate against the legacy source.
• SaaS Migration: Ingest vendor exports to Bronze and conform them in Silver before consumption.
• Codebase Migration: Refactor transformation logic layer by layer with reproducible outputs.
• Validation: Keep the legacy system's raw extract in Bronze so reconciliation against the old source is always possible during and after migration.

Checklist

• Raw data landed unchanged in Bronze with metadata
• Silver applies cleaning, typing, and conformance
• Gold models business-ready aggregates with clear grain
• Each layer reproducible from the one below
• Quality checks enforced between layers
• Gold table contracts documented for consumers
• Incremental processing used where possible