How to write and run a PySpark batch job

What and why

Apache Spark processes large datasets in parallel across a cluster. PySpark is its Python API, and the DataFrame interface offers SQL-like transformations that Spark optimizes and distributes. This tutorial builds a typical read-transform-write batch job and runs it with spark-submit.

Prerequisites

• Python and Java installed (Spark runs on the JVM).
• A Spark installation or access to a cluster.
• Input data, for example CSV or Parquet files.

Steps

1. Install Spark

pip install pyspark

This includes a local mode suitable for development.

2. Create a SparkSession

The session is the entry point. In job.py:

from pyspark.sql import SparkSession, functions as F
spark = SparkSession.builder.appName("daily-rollup").getOrCreate()

3. Read source data

orders = spark.read.parquet("s3a://bucket/raw/orders/")

Spark reads many formats; Parquet is preferred for its columnar layout and schema.

4. Transform with DataFrames

Use the DataFrame API rather than Python loops so Spark can parallelize:

daily = (orders
  .withColumn("day", F.to_date("created_at"))
  .groupBy("day")
  .agg(F.sum("amount").alias("revenue")))

5. Write the output

(daily.write
  .mode("overwrite")
  .partitionBy("day")
  .parquet("s3a://bucket/curated/daily_revenue/"))

Partitioning the output speeds up downstream reads.

6. Submit the job

Run locally or on a cluster:

spark-submit --master local[4] job.py

On a cluster, set --master yarn or a Spark URL and tune executor memory and cores.

Verification

The job should complete without errors and write Parquet files partitioned by day. Read the output back with spark.read.parquet(...).show() and confirm the aggregated revenue is correct. The Spark UI shows stages and task timing.

Next Steps

Cache reused DataFrames, broadcast small lookup tables to avoid shuffles, tune partition counts to match data size, and schedule the job through Airflow or a workflow tool.