Data Lineage

Best Practice: Data Lineage

Data lineage is the documented, traceable path of data from its source through every transformation to its final use. It answers where a value came from, what changed it, and what depends on it downstream. It matters because without lineage, teams cannot do reliable impact analysis, debug bad numbers, satisfy auditors, or safely retire systems. OpenLineage, a Linux Foundation project, provides an open standard for collecting lineage metadata across tools. Lineage becomes most powerful when it is column-level and automatically harvested, because that combination answers the precise questions auditors and engineers ask: which exact field fed this regulated report, and what breaks if I change it. Regulations such as GDPR and BCBS 239 effectively require lineage to demonstrate where personal or risk data flows, and manual diagrams cannot keep pace with daily pipeline changes. Treated as an operational signal rather than documentation, lineage drives automated impact analysis in CI before a change ships, so engineers see the downstream blast radius of a schema edit while they are still writing the pull request.

Step-by-Step Implementation Guidance

1. Decide the granularity you need: dataset-level, column-level, or both.
2. Capture lineage automatically from pipelines rather than maintaining it by hand.
3. Adopt an open standard such as OpenLineage to avoid tool lock-in.
4. Integrate lineage collection into orchestration and transformation tools.
5. Surface lineage in a catalog so consumers can explore upstream and downstream dependencies.
6. Use lineage for impact analysis before any schema or pipeline change.
7. Keep lineage current as part of CI so it reflects the live system.

Common Mistakes Teams Make When Ignoring This Practice

• Maintaining lineage diagrams by hand that quickly go stale.
• Capturing only table-level lineage when column-level is needed for compliance.
• Treating lineage as documentation rather than an operational signal.
• Skipping lineage, then guessing at the blast radius of a change.
• Locking lineage into one vendor's proprietary format.
• Building lineage only for compliance reporting and never wiring it into change review.

Tools and Techniques That Support This Practice

• OpenLineage and Marquez for open lineage collection.
• Catalogs such as DataHub, OpenMetadata, Atlan, and Collibra.
• dbt for built-in model lineage graphs.
• Orchestrators (Airflow, Dagster) that emit lineage events.
• Marquez as a reference OpenLineage backend for storing and exploring lineage events.

How This Practice Applies to Different Migration Types

• Cloud Migration: Use lineage to map dependencies before moving pipelines to the cloud.
• Database Migration: Trace which reports and jobs depend on tables slated to move.
• SaaS Migration: Track how exported data flows so nothing breaks during cutover.
• Codebase Migration: Identify downstream consumers before refactoring a producer.
• Decommissioning: Lineage confirms a system has no remaining downstream consumers before you safely retire it.

Checklist

• Required lineage granularity decided
• Lineage captured automatically from pipelines
• Open standard (OpenLineage) adopted
• Lineage integrated with orchestration and transforms
• Lineage surfaced in a catalog for consumers
• Impact analysis performed before changes
• Lineage kept current via CI