How to convert data to Apache Parquet for analytics

What and why

Apache Parquet is a columnar file format built for analytics. Because it stores each column together, queries read only the columns they need and benefit from strong per-column compression and encoding. Converting row-based CSV or JSON to Parquet typically shrinks files and speeds up warehouse and Spark queries dramatically. This tutorial converts and reads Parquet.

Prerequisites

• Python with pip.
• Source data in CSV or JSON.
• Optional cloud storage such as S3 for the output.

Steps

1. Understand columnar storage

Row formats store whole records together, so reading one column still loads all of them. Columnar formats store each column contiguously, enabling column pruning, predicate pushdown, and high compression ratios.

2. Install the tooling

pip install pyarrow pandas

PyArrow provides fast Parquet reading and writing.

3. Read the source data

import pandas as pd
df = pd.read_csv("orders.csv", parse_dates=["created_at"])

Ensure types are correct, especially dates and numerics, since Parquet stores types explicitly.

4. Write Parquet with compression

df.to_parquet("orders.parquet", engine="pyarrow", compression="zstd")

ZSTD and Snappy are common; ZSTD compresses more, Snappy decompresses faster.

5. Partition the output

For large datasets, write a partitioned directory so queries skip irrelevant files:

import pyarrow.dataset as ds
df["day"] = df["created_at"].dt.date
ds.write_dataset(
    df.to_arrow() if hasattr(df, "to_arrow") else __import__("pyarrow").Table.from_pandas(df),
    "orders/", format="parquet", partitioning=["day"])

Each partition value becomes a subdirectory.

6. Read columns selectively

Read only needed columns to exploit the format:

sub = pd.read_parquet("orders.parquet", columns=["id", "amount"])

Engines also push filters down to skip row groups.

Verification

Compare file sizes: the Parquet output should be much smaller than the CSV. Reading a subset of columns should be faster and use less memory than loading the full CSV. Partitioned reads with a filter should touch only matching subdirectories.

Next Steps

Standardize on Parquet for your data lake, choose a sensible row-group size, and pair it with a table format like Apache Iceberg or Delta Lake for schema evolution and ACID guarantees.