Hadoop to Databricks (Spark) Blueprint
A blueprint to migrate on-prem Hadoop/MapReduce to Databricks on cloud object storage, converting Hive tables to Delta Lake, rewriting jobs as Spark, and adopting the medallion lakehouse architecture.
Overview
On-prem Hadoop clusters (HDFS, MapReduce, Hive, YARN) are expensive to operate, couple storage and compute, and slow to scale. Databricks runs Apache Spark on elastic cloud compute over object storage, with Delta Lake adding ACID transactions and time travel to the data lake. Migrating decouples storage and compute and modernizes the processing layer. This blueprint covers the move.
Phases
Assessment. Inventory HDFS data, Hive tables, and jobs (MapReduce, Pig, Hive QL, Spark, Oozie workflows). Classify jobs by migration difficulty. Map Hive metastore to Unity Catalog. Size the data set for transfer.
Platform setup. Provision Databricks workspaces and cloud object storage (S3/ADLS/GCS). Configure Unity Catalog for governance and Delta Lake as the table format (the lakehouse foundation).
Data migration. Copy HDFS data to object storage (DistCp, AWS DataSync, or Azure Data Box for large volumes). Convert Hive tables to Delta. Use the medallion architecture (bronze/silver/gold) for the curated layers.
Job migration. Rewrite MapReduce and Pig jobs as Spark (PySpark/Scala/SQL). Port Hive QL to Spark SQL. Replace Oozie with Databricks Workflows or Airflow. Reuse existing Spark jobs with minimal change.
Validation. Reconcile record counts and aggregates; compare job outputs against Hadoop results.
Cutover. Switch downstream consumers and schedulers to Databricks, run in parallel, then decommission the Hadoop cluster.
Key Risks and Mitigations
- Data consistency: Reconcile counts and aggregates per dataset; Delta Lake transactions reduce partial-write issues.
- Job rewrite effort: MapReduce/Pig have no automatic translation. Prioritize high-value jobs and retire dead ones.
- Skills gap: Train teams on Spark, Delta, and Unity Catalog; MapReduce skills do not transfer directly.
- Cost overrun: Use autoscaling clusters, spot/preemptible instances, and auto-termination to control compute spend.
Recommended Tooling
DistCp/DataSync for bulk transfer; Delta Lake table format; Databricks Workflows or Airflow for orchestration; Unity Catalog for governance; Spark UI and cluster metrics for tuning.
Success Metrics
Faster job runtimes on elastic compute, cost reduction from decoupled storage/compute and spot instances, and reduced operational toil from a managed platform.
Prerequisites
Cloud object storage, network bandwidth (or offline transfer) for HDFS data, a job inventory with priorities, governance via Unity Catalog, and a parallel-run validation plan.