Batch ETL on Apache Spark

Batch ETL on Apache Spark

Apache Spark is a distributed engine that processes large datasets in parallel across a cluster, making it a workhorse for batch ETL (extract, transform, load). Running Spark on Kubernetes gives elastic, on-demand compute without dedicated clusters. This platform reads raw data from object storage, transforms it, and writes curated tables for analytics. Use it when datasets are too large for a single machine and transformations are complex enough to need a full programming model.

Components

• Object storage: S3-compatible storage holding raw input and curated output as Parquet files.
• Spark: the distributed processing engine running ETL jobs in Scala, Python, or SQL.
• Spark Operator: a Kubernetes operator that submits and manages Spark applications as native resources.
• Apache Airflow: orchestrates the DAG of dependent ETL jobs on schedules.
• Apache Iceberg: open table format providing ACID tables, schema evolution, and partition management.
• Hive Metastore: central metadata catalog mapping table names to storage locations and schemas.
• Parquet: the columnar file format for efficient compressed storage and scans.

Data Flow

Airflow triggers Spark jobs in dependency order. Each Spark application launches an executor pool on Kubernetes, reads raw Parquet from object storage, applies cleansing, joins, and aggregation, and writes results to Iceberg tables registered in the metastore. Data-quality checks run as job steps and can fail the pipeline on violation. Curated Iceberg tables are then queried by warehouses or BI engines. Jobs are idempotent so reruns overwrite partitions safely.

Scaling and Resilience

Spark scales horizontally by adding executors; Kubernetes provisions them on demand and reclaims them when jobs finish, with dynamic allocation matching resources to workload. Spark's lineage of resilient distributed datasets recomputes lost partitions on executor failure. Airflow retries failed tasks with backoff, and Iceberg snapshots allow rollback after a bad write. Spot or preemptible nodes cut cost for fault-tolerant batch work.

Security

Spark pods run with least-privilege service accounts and access object storage through scoped IAM roles or credentials in Kubernetes secrets. Network policies isolate the driver and executors, and pod security standards constrain the workloads. Data is encrypted at rest in object storage and in transit with TLS. The metastore enforces table-level access where integrated with a governance layer.

Trade-offs and Alternatives

Spark on Kubernetes is flexible and cloud-portable but requires tuning of partitions, memory, and shuffles, and operating the cluster. Managed alternatives like Databricks, EMR, or Dataproc reduce that effort. For simpler SQL transforms, an ELT approach in a warehouse with dbt may be cheaper and easier. Choose Spark when transformations are large-scale and complex, or when you need an open, portable engine across clouds.