What is a data pipeline?

Answer

A data pipeline is a series of automated steps that move data from sources to destinations, transforming and validating it along the way so it is ready for analytics, machine learning, or applications. Pipelines can run in batch (processing data on a schedule) or streaming mode (processing events continuously as they arrive). Orchestration tools such as Apache Airflow, Dagster, or Prefect schedule and monitor the steps, handling dependencies, retries, and failures. Reliable pipelines emphasize idempotency, observability, and data quality checks so downstream consumers can trust the output.