How to validate data quality with Great Expectations

What and why

Great Expectations validates data against declarative rules called expectations, such as "this column is never null" or "values fall within a range." Catching violations early stops bad data from corrupting reports and models. This tutorial sets up checks for a table and runs them.

Prerequisites

• Python with pip.
• A dataset, for example a database table or a CSV.
• Basic familiarity with pandas or SQL.

Steps

1. Install Great Expectations

pip install great_expectations
great_expectations init

This scaffolds a project directory.

2. Connect a data source

Add a data source pointing at your database or files. For pandas, you can validate a DataFrame directly via the Python API; for a warehouse, configure a SQL data source with a connection string.

3. Create an expectation suite

A suite groups expectations for a dataset:

import great_expectations as gx
context = gx.get_context()
suite = context.add_expectation_suite("orders.warning")

4. Add expectations

Declare rules the data must satisfy:

validator.expect_column_values_to_not_be_null("order_id")
validator.expect_column_values_to_be_between("amount", min_value=0, max_value=100000)
validator.expect_column_values_to_be_unique("order_id")
validator.save_expectation_suite()

5. Run a checkpoint

A checkpoint runs a suite against a batch of data and records results:

checkpoint = context.add_or_update_checkpoint(
    name="orders_checkpoint", validator=validator)
result = checkpoint.run()
print(result["success"])

Great Expectations can build Data Docs, an HTML report of pass/fail results.

6. Wire into a pipeline

Run the checkpoint as a step in Airflow or your ETL job. If result['success'] is false, fail the task so downstream steps do not consume bad data.

Verification

Run the checkpoint against clean data and confirm it reports success. Introduce a null order_id or a negative amount and confirm the run fails and the Data Docs flag the exact failing expectation and rows.

Next Steps

Profile data to bootstrap expectations automatically, version suites in git, and alert on failures so data quality regressions are caught the moment they appear.