How to author a data pipeline DAG in Apache Airflow

What and why

Apache Airflow orchestrates data pipelines as directed acyclic graphs (DAGs) of tasks written in Python. It schedules runs, manages dependencies and retries, and provides a UI to monitor and re-run work. This tutorial builds a small DAG and runs it.

Prerequisites

• Python familiarity.
• An Airflow environment; the official Docker Compose is the simplest start.
• A folder mapped to Airflow's dags/ directory.

Steps

1. Start Airflow

Use the official Compose file:

docker compose up airflow-init && docker compose up -d

The webserver runs at http://localhost:8080.

2. Create a DAG file

Add dags/etl_pipeline.py:

from airflow import DAG
from airflow.operators.python import PythonOperator
import pendulum

with DAG(
    dag_id="etl_pipeline",
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
    schedule="@daily",
    catchup=False,
) as dag:
    ...

3. Define tasks

def extract():
    return {"rows": 100}

def load(**ctx):
    data = ctx["ti"].xcom_pull(task_ids="extract")
    print("loading", data)

extract_t = PythonOperator(task_id="extract", python_callable=extract)
load_t = PythonOperator(task_id="load", python_callable=load)

Tasks pass small values through XCom.

4. Set dependencies

extract_t >> load_t

The >> operator declares that load runs only after extract succeeds.

5. Configure the schedule

The schedule="@daily" and catchup=False settings run the DAG once per day without backfilling missed intervals. Use a cron expression for finer control.

6. Trigger and monitor

In the UI, unpause the DAG and trigger a run. Each task shows green on success; click a task to view logs and retry if it fails.

Verification

The DAG should appear in the UI without import errors. A triggered run should show both tasks succeeding in order, and the load task's log should print the rows passed from extract.

Next Steps

Use the TaskFlow API for cleaner Python tasks, add sensors to wait on external data, set retries and SLAs, and move connections and secrets into Airflow's connection store.