Spark

10 items tagged with "spark"

Filter by type:

Tutorials1

Tutorial

How to write and run a PySpark batch job

Build a PySpark job that reads, transforms, and writes data using the DataFrame API, then run it with spark-submit.

Reference Architectures1

Reference Architecture

Batch ETL on Apache Spark

A scalable batch ETL platform on Apache Spark running on Kubernetes, transforming large datasets from object storage into curated tables.

Stacks5

Stack

Databricks Lakehouse

Unified lakehouse architecture combining data-lake storage with warehouse reliability via Delta Lake, Spark compute, Unity Catalog governance, and ML.

Stack

Airflow + Spark Batch Pipeline Stack

Classic big-data batch pattern: Apache Airflow orchestrates scheduled DAGs that run distributed Apache Spark jobs over data-lake storage.

Stack

Databricks + Unity Catalog

A governed lakehouse stack centering Databricks compute and Delta Lake storage under Unity Catalog for unified data and AI governance.

Stack

Spark + Delta Lake

A lakehouse foundation pairing Apache Spark for distributed processing with Delta Lake's ACID table format over cloud object storage.

Stack

Prefect + Spark

A Python-native orchestration stack where Prefect schedules and monitors large-scale data processing jobs running on Apache Spark.

Comparisons1

Comparison

Spark vs Flink

The unified batch-and-streaming engine with micro-batch roots versus Apache Flink's true stream-first processing engine.

Benchmarks2

Benchmark

TPCx-BB (BigBench)

An end-to-end big-data analytics benchmark mixing SQL, machine learning, and natural-language processing over a retail data set on Hadoop/Spark-style platforms.

Benchmark

TeraSort

A distributed sort benchmark that orders one terabyte (or more) of data on a cluster, measuring big-data engine throughput and shuffle efficiency.