Data Pipeline
12 items tagged with "data-pipeline"
Best Practices3
ELT vs ETL Best Practices
Guidance on when to transform data before loading (ETL) versus loading raw and transforming in the warehouse (ELT), and how to run each pattern well.
Reverse ETL
The practice of moving modeled data from the warehouse back into operational tools like CRM and marketing platforms so business teams act on it directly.
Apache Kafka Streaming Best Practices
Design and operational guidance for building reliable, scalable event streaming on Apache Kafka, covering topics, partitions, delivery semantics, and consumers.
Benchmarks2
dbt Pipeline Performance Benchmark
A category of benchmarks measuring transformation pipeline performance in dbt — model build time, warehouse compute cost, and incremental run efficiency.
ETL/ELT Pipeline Throughput Benchmark
A benchmark category measuring data-integration pipeline performance — extraction and load throughput, transformation latency, and end-to-end freshness.
FAQs4
What is the difference between ETL and ELT?
ETL (Extract, Transform, Load) transforms data before loading it into the target system, which suited traditional warehouses with limited compute and ...
What is change data capture (CDC)?
Change data capture is a technique that detects and streams row-level changes (inserts, updates, deletes) from a source database so downstream systems...
What is a data pipeline?
A data pipeline is a series of automated steps that move data from sources to destinations, transforming and validating it along the way so it is read...
What is the difference between batch and stream processing?
Batch processing handles large volumes of data in scheduled chunks, such as a nightly job that aggregates the previous day's sales; it is simple and e...
Glossaries2
ETL
Extract, Transform, Load - a process for moving and transforming data between systems
Change Data Capture (CDC)
Change Data Capture (CDC) is a technique for identifying and capturing changes made to data in a source database — inserts, updates, and deletes — and streaming them to downstream systems in near real time.