Data Engineering

Data Mesh Principles

Domain-oriented, self-serve data infrastructure principles promoting product thinking for data.

dbt Style Guide

Community conventions for naming, structuring, and documenting dbt transformation projects.

Data Governance Framework

A structured set of roles, policies, and processes that make an organization accountable for the quality, security, and proper use of its data assets.

Data Quality Management

The practice of measuring, monitoring, and improving data across dimensions like accuracy, completeness, consistency, timeliness, and validity so it stays fit for use.

Data Contracts

Explicit, version-controlled agreements between data producers and consumers that define schema, semantics, quality, and SLAs to prevent breaking changes.

Medallion Architecture

A layered data design that refines data through Bronze (raw), Silver (cleaned and conformed), and Gold (business-ready) tables to improve quality and reuse.

Data Lakehouse Architecture

An architecture that combines the low-cost, open storage of a data lake with the transactions, schema, and performance of a data warehouse using open table formats.

ELT vs ETL Best Practices

Guidance on when to transform data before loading (ETL) versus loading raw and transforming in the warehouse (ELT), and how to run each pattern well.

Data Lineage

The traceable record of data's origin, movement, and transformation across systems, enabling impact analysis, debugging, compliance, and trust.

Reverse ETL

The practice of moving modeled data from the warehouse back into operational tools like CRM and marketing platforms so business teams act on it directly.

Feature Store Best Practices

A centralized system for defining, storing, and serving machine learning features consistently for training and inference, avoiding skew and duplicated work.

Data Catalog and Discovery

A searchable inventory of an organization's data assets with metadata, ownership, and lineage so people can find, understand, and trust the data they need.

Apache Kafka Streaming Best Practices

Design and operational guidance for building reliable, scalable event streaming on Apache Kafka, covering topics, partitions, delivery semantics, and consumers.

Patterns17

Database per Service

Each microservice owns and manages its own database, enabling loose coupling and independent deployability

Saga Pattern

Manage data consistency across microservices using a sequence of local transactions with compensating actions

Event Sourcing

Store state changes as a sequence of events rather than just the current state

Change Data Capture (CDC)

Captures row-level changes from a database's transaction log and streams them as events to downstream consumers.

Pipes and Filters

Decompose complex processing into a sequence of independent components connected by channels so each step can scale and evolve.

Splitter

Breaks a composite message into a series of individual messages so each element can be processed independently downstream.

Content Enricher

Augments a message with additional data from an external source when the original lacks information the receiver requires.

Normalizer

Translates messages arriving in many different formats into a single common format so downstream components handle one canonical representation.

Materialized View

A precomputed, stored result set that trades storage and refresh cost for fast reads of expensive queries, aggregations, or denormalized projections.

Polyglot Persistence

An architecture that uses multiple, purpose-fit data stores within one system, matching each store's strengths to each data access pattern.

Command Query Responsibility Segregation (CQRS)

Separates the write model that handles commands from the read model that serves queries, letting each be optimized, scaled, and evolved independently.

Lambda Architecture

A big-data design that runs a batch layer for accurate historical views and a speed layer for low-latency recent data, merging both at query time.

Kappa Architecture

A streaming-first data design that uses a single processing path over an immutable log, reprocessing history by replay instead of a separate batch layer.

Medallion Architecture

A lakehouse data-organization pattern that refines data through bronze (raw), silver (cleaned), and gold (curated) layers for progressive quality and reuse.

Data Lakehouse

A data architecture that adds warehouse-style ACID tables, schema, and governance directly on low-cost data-lake storage, unifying analytics and ML on one platform.

Star Schema

A dimensional data-modeling pattern with a central fact table linked to denormalized dimension tables, optimized for fast, intuitive analytical queries.

Slowly Changing Dimension (SCD)

Techniques for handling changes to dimension attributes over time in a data warehouse, ranging from overwriting to preserving full historical versions.

Anti-Patterns5

Polling the Database

Repeatedly querying a table on a tight loop to detect changes instead of using events or notifications, wasting resources and adding latency.

Chatty Data Access

Making many fine-grained round-trips to the database for one logical operation, so network latency dominates and throughput collapses under load.

Dual Write

Writing the same change to two systems in sequence without a single transaction or log, so a failure between them leaves the stores inconsistent.

Schemaless Sprawl

Treating a schemaless store as license to skip data design, so documents drift into inconsistent shapes that no consumer can reliably read.

Hot Partition (Hot Shard)

A partitioning key that sends a disproportionate share of traffic to one shard, overloading it while the rest sit idle and capping the system at one node's throughput.

Tutorials10

How to stream database changes with Debezium CDC

Capture row-level changes from PostgreSQL using Debezium and Kafka Connect, producing an ordered stream of insert, update, and delete events.

How to build a Kafka producer and consumer

Create a Kafka topic, produce messages with keys, and consume them in a group, covering offsets, partitions, and delivery basics.

How to load Kafka topics into a database with Kafka Connect

Use a Kafka Connect sink connector to stream topic data into a relational database with no custom consumer code.

How to build transformation models and tests with dbt

Structure dbt staging and mart models, add tests and documentation, and run a build that materializes them in a warehouse.

How to author a data pipeline DAG in Apache Airflow

Write an Airflow DAG with scheduled tasks and dependencies, run it locally, and monitor execution in the UI.

How to write and run a PySpark batch job

Build a PySpark job that reads, transforms, and writes data using the DataFrame API, then run it with spark-submit.

How to bulk load data into Snowflake with COPY INTO

Stage files, define a file format, and use COPY INTO to load data into Snowflake tables efficiently with error handling.

How to validate data quality with Great Expectations

Define expectations for a dataset, run validations, and surface failures so bad data is caught before it spreads downstream.

How to convert data to Apache Parquet for analytics

Convert CSV or JSON to columnar Parquet with partitioning and compression to speed up and shrink analytic queries.

How to use Avro and Schema Registry with Kafka

Define an Avro schema, register it, and produce and consume typed Kafka records with schema evolution and compatibility checks.

Blueprints20

SQL to NoSQL Migration Blueprint

Blueprint for migrating relational databases to NoSQL document stores

Batch to Streaming Blueprint

Migration blueprint for converting batch processing to real-time streaming

MySQL to PostgreSQL Blueprint

Complete migration guide from MySQL to PostgreSQL

Mainframe-Adjacent Infrastructure to Azure Blueprint

Migrate the distributed mid-tier and batch infrastructure surrounding a mainframe to Azure, decoupling it while integrating back to the core system.

Relational to MongoDB Migration Blueprint

Re-model a normalized relational schema into MongoDB document collections for flexible schemas and scale-out reads.

Relational to Apache Cassandra Migration Blueprint

Migrate a relational workload to Apache Cassandra with query-first data modeling for write-heavy, globally distributed scale.

Single Database to Database-per-Service Blueprint

Decompose a shared monolithic database into per-service data stores to enable independent microservice deployment.

Batch to Streaming with Apache Kafka Blueprint

Replace nightly batch ETL with real-time event streaming on Apache Kafka for low-latency data movement.

Batch to Streaming with Amazon Kinesis Blueprint

Convert scheduled batch pipelines to real-time streaming on Amazon Kinesis Data Streams for managed, low-latency ingestion.

ETL to ELT with dbt Blueprint

Move from transform-before-load ETL tools to in-warehouse ELT using dbt for version-controlled, testable transformations.

On-Prem Warehouse to Snowflake Blueprint

Migrate an on-premises data warehouse to Snowflake for elastic, separated storage and compute with pay-per-use scaling.

On-Prem Warehouse to BigQuery Blueprint

Migrate an on-premises data warehouse to Google BigQuery for serverless, petabyte-scale analytics with separated storage and compute.

On-Prem Warehouse to Amazon Redshift Blueprint

Migrate an on-premises data warehouse to Amazon Redshift for managed, columnar MPP analytics integrated with the AWS data stack.

Hadoop to Databricks (Spark) Blueprint

Migrate on-prem Hadoop and MapReduce workloads to Databricks on cloud object storage with Apache Spark and Delta Lake.

Data Lake to Lakehouse Blueprint

Upgrade a raw data lake to a lakehouse using open table formats (Delta, Iceberg, or Hudi) for ACID transactions and governance.

Teradata to Cloud Warehouse Blueprint

Migrate a Teradata enterprise warehouse to a cloud warehouse (Snowflake, BigQuery, or Redshift) to retire appliance costs.

Build a Change Data Capture (CDC) Pipeline Blueprint

Stand up a CDC pipeline with Debezium and Kafka to stream database changes in real time to downstream consumers.

Notebooks to MLOps Pipeline Blueprint

Productionize ad-hoc data science notebooks into reproducible, versioned MLOps pipelines with CI/CD, tracking, and automated retraining.

Classic ML to Feature Store and Serving Blueprint

Migrate ad-hoc feature engineering for classic ML models to a central feature store with consistent offline training and online serving.

Batch File Transfer to Streaming CDC Blueprint

Replace nightly batch file transfers between systems with real-time change data capture streamed onto an event backbone.

Reference Architectures23

Modern Data Lake Architecture

Lakehouse architecture combining data lake flexibility with data warehouse capabilities

Batch Jobs Platform on Kubernetes

A Kubernetes-native batch platform using job queues and Argo Workflows to run scheduled and on-demand container jobs at scale.

Lakehouse on Databricks

A unified lakehouse on Databricks combining Delta Lake storage, Unity Catalog governance, and SQL plus ML compute on AWS.

BigQuery Analytics Platform

A serverless analytics platform on Google BigQuery with streaming ingestion, dbt transforms, and Looker for enterprise BI.

Real-Time Streaming Platform with Kafka

A real-time streaming platform on Kafka with stream processing, a schema registry, and exactly-once pipelines on Kubernetes.

Change Data Capture Pipeline with Debezium

A CDC pipeline streaming row-level changes from operational databases into a warehouse using Debezium and Kafka on AWS.

Medallion Lakehouse on Azure

A bronze-silver-gold medallion lakehouse on Azure with Data Lake Storage, Delta tables, and Synapse for serving.

Data Mesh Platform

A decentralized data mesh where domain teams own data products on a self-serve platform with federated governance across multi-cloud.

Batch ETL on Apache Spark

A scalable batch ETL platform on Apache Spark running on Kubernetes, transforming large datasets from object storage into curated tables.

Event Streaming on Amazon Kinesis

A managed event streaming platform on Amazon Kinesis with serverless processing and real-time analytics on AWS.

Feature Store Platform

A feature store on GCP providing consistent online and offline ML features with point-in-time correctness via Feast.

Reverse ETL Data Activation Platform

A reverse ETL platform that syncs modeled warehouse data back into operational SaaS tools for marketing and sales activation on GCP.

Streaming Lakehouse with Flink and Iceberg

A streaming lakehouse on AWS where Apache Flink writes continuously into Apache Iceberg tables for unified streaming and batch analytics.

IoT Time-Series Analytics Platform

An IoT analytics platform on Azure ingesting device telemetry into a time-series store for real-time monitoring and forecasting.

Data Observability and Quality Platform

A data observability platform on Snowflake that monitors freshness, volume, and schema with automated tests and lineage on AWS.

Batch Inference Pipeline on Azure

A reference design for scheduled, large-scale offline scoring on Azure using Synapse, Azure ML batch endpoints, and a data lake for inputs and results.

Feature Store and Online Serving on AWS

A reference design for a dual offline/online feature store on AWS that keeps training and serving features consistent and serves them at low latency.

Streaming ML Feature Pipeline on GCP

A reference design for a real-time feature engineering pipeline on GCP that computes streaming aggregates and serves them to online models consistently.

Data Labeling and Training Platform on AWS

A reference design for a closed-loop data labeling and training platform on AWS that turns raw data into labeled datasets and continuously retrains models.

SIEM and Security Data Lake on Google Cloud

Scalable security analytics platform that ingests logs, detects threats, and supports investigation over a data lake.

Event-Driven Backbone with Kafka

An organization-wide event streaming backbone on Kafka that decouples producers and consumers through durable, replayable topics.

High-Volume Streaming Ingestion Pipeline

A streaming ingestion pipeline that captures high-velocity event data, buffers it durably, and lands it for real-time and batch use.

Change Data Capture Streaming Pipeline

A change-data-capture pipeline that streams database changes as events to keep downstream systems and caches in sync in near real time.

Playbooks22

Database Modernization Playbook

Comprehensive guide for database upgrades and migrations

Oracle to PostgreSQL Migration Program Playbook

Run an enterprise program to migrate Oracle databases to PostgreSQL, covering schema conversion, PL/SQL rewrite, data movement, and cutover.

On-Prem Warehouse to Snowflake Program Playbook

Migrate a legacy on-prem data warehouse to Snowflake, including workload analysis, ELT redesign, governance, and decommissioning.

Data Warehouse to BigQuery Program Playbook

Migrate analytics workloads to Google BigQuery with serverless design, partitioning strategy, governance, and cost controls.

Batch to Streaming Program Playbook

Transform nightly batch data pipelines into real-time streaming pipelines using event-driven architecture and stream processing.

Data Mesh Rollout Program Playbook

Roll out a data mesh operating model with domain-owned data products, a self-serve platform, and federated computational governance.

Data Governance Program Playbook

Establish an enterprise data governance program covering ownership, cataloging, lineage, quality, privacy, and policy enforcement.

dbt and ELT Adoption Program Playbook

Adopt dbt-style ELT to modernize transformation logic with version control, testing, documentation, and a medallion architecture.

Lakehouse Migration Program Playbook

Consolidate a data lake and warehouse into a unified lakehouse using open table formats, medallion architecture, and ACID transactions.

Master Data Management Program Playbook

Establish a master data management program to create golden records, resolve duplicates, and govern shared reference data across systems.

CDC and Real-Time Pipeline Program Playbook

Build change data capture pipelines to stream database changes into analytics and event systems with low latency and strong consistency.

Data Quality Program Playbook

Stand up a data quality program with profiling, automated tests, SLAs, anomaly detection, and incident workflows across pipelines.

Zero-Downtime Database Migration Program Playbook

Migrate a production database engine or version with no downtime using dual-write, CDC replication, shadow reads, and staged cutover.

SQL Server to PostgreSQL Migration Program Playbook

Migrate SQL Server databases to PostgreSQL, converting T-SQL, schema, and data with validated cutover to cut licensing costs.

MongoDB to PostgreSQL Migration Program Playbook

Migrate document data from MongoDB to PostgreSQL, modeling collections into relational and JSONB schemas with validated cutover.

Data Pipeline Orchestration Program Playbook

Modernize fragile cron and script-based data pipelines into a governed orchestration platform with lineage, retries, and observability.

Schema Evolution and Versioning Program Playbook

Adopt a schema registry and safe evolution practices for event and data schemas to prevent breaking changes across producers and consumers.

Feature Store Program Playbook

Build a feature store to unify ML feature engineering, ensure online and offline consistency, and enable feature reuse across teams.

ETL to ELT Modernization Program Playbook

Modernize legacy ETL tools into a cloud-native ELT architecture, pushing transformation into the warehouse with version control and tests.

Data Catalog and Discovery Program Playbook

Deploy a data catalog to make datasets discoverable with metadata, lineage, ownership, and a business glossary across the organization.

Vector Database Adoption Playbook

A program to adopt a vector database for semantic search and RAG, covering embeddings, indexing, scaling, and operations.

Feature Store Rollout Playbook

A program to roll out a feature store that unifies feature engineering for training and serving with consistency and reuse.

Checklists10

Data Warehouse Migration Readiness Checklist

Readiness checks for migrating an enterprise data warehouse to a cloud platform such as Snowflake, BigQuery, or Databricks.

Data Quality Validation Checklist

Checks for validating accuracy, completeness, consistency, and timeliness of data after a migration or pipeline change.

ETL to ELT Migration Checklist

Checks for converting traditional ETL pipelines to an ELT model that loads raw data first and transforms inside the warehouse.

Streaming Pipeline Readiness Checklist

Production-readiness checks for real-time streaming data pipelines built on platforms such as Apache Kafka or Pulsar.

Data Governance Review Checklist

A review checklist for assessing data ownership, quality, lineage, access control, and policy compliance across data assets.

Data Reconciliation Checklist

Checks for reconciling data between source and target systems after a migration to prove completeness and accuracy.

Data Lake to Lakehouse Migration Checklist

Readiness checks for evolving a raw data lake into a governed lakehouse with table formats, schema enforcement, and ACID guarantees.

Data Pipeline Observability Checklist

Checks for instrumenting data pipelines with freshness, volume, schema, lineage, and quality monitoring to detect issues early.

Data Format Migration Checklist

Checks for migrating data files between serialization formats such as CSV, JSON, Avro, Parquet, and ORC without losing fidelity.

MLOps Pipeline Review Checklist

Audit items for assessing the maturity, reproducibility, and automation of an end-to-end machine learning operations pipeline.

Regulations4

EU Data Act

EU regulation governing access to and sharing of data from connected products and related services, including cloud-switching rules.

EU Data Governance Act

EU regulation establishing trusted mechanisms for data sharing, data intermediaries, and data altruism across sectors.

EU Digital Markets Act

EU regulation imposing fairness and contestability obligations on large digital gatekeeper platforms providing core platform services.

EU Corporate Sustainability Reporting Directive (Directive (EU) 2022/2464)

EU directive requiring large and listed companies to report standardized, audited sustainability information including environmental, social, and governance data.

Stacks23

Modern Data Stack

Snowflake/BigQuery, dbt, Airflow, Looker - Analytics pipeline

Kafka + Flink Streaming Stack

Real-time stream processing stack pairing Apache Kafka for durable event streams with Apache Flink for stateful, low-latency computation.

Modern Data Stack (ELT)

Cloud-native ELT pattern: managed ingestion loads raw data into a warehouse, where dbt transforms it and a BI tool serves analytics.

dbt + BigQuery Analytics Stack

Serverless analytics engineering on Google BigQuery with dbt managing SQL transformations, tests, and lineage on top of warehouse-native compute.

Databricks Lakehouse

Unified lakehouse architecture combining data-lake storage with warehouse reliability via Delta Lake, Spark compute, Unity Catalog governance, and ML.

Kafka + Flink + Iceberg Streaming Stack

Real-time streaming architecture: Kafka transports events, Flink processes them with stateful low-latency compute, and Iceberg lands them in an open lakehouse.

Airflow + Spark Batch Pipeline Stack

Classic big-data batch pattern: Apache Airflow orchestrates scheduled DAGs that run distributed Apache Spark jobs over data-lake storage.

ClickHouse Real-Time Analytics Stack

High-performance analytics stack: ClickHouse ingests event streams from Kafka and serves sub-second OLAP queries powering user-facing dashboards.

Open Lakehouse Stack (Trino + Iceberg + dbt)

Open, engine-agnostic lakehouse: Iceberg tables on object storage, queried by Trino across sources, with dbt managing transformations in SQL.

Snowflake + Fivetran + Looker

A fully managed cloud analytics stack: Fivetran ingests data, Snowflake stores and transforms it, and Looker serves governed BI on top.

dbt + Amazon Redshift

An ELT analytics stack where data lands in Amazon Redshift and dbt transforms it with version-controlled, tested SQL models.

Databricks + Unity Catalog

A governed lakehouse stack centering Databricks compute and Delta Lake storage under Unity Catalog for unified data and AI governance.

Kafka + ksqlDB

A stream-processing stack using Apache Kafka for event transport and ksqlDB for SQL-based streaming transformations and materialized views.

Flink + Apache Paimon

A streaming lakehouse stack using Apache Flink for unified stream and batch processing over Apache Paimon's high-update table format.

Spark + Delta Lake

A lakehouse foundation pairing Apache Spark for distributed processing with Delta Lake's ACID table format over cloud object storage.

Trino + Iceberg + S3

An open lakehouse query stack: Trino runs fast SQL across Apache Iceberg tables stored as open files on Amazon S3.

ClickHouse + Grafana

A real-time analytics stack using ClickHouse as a fast columnar OLAP database with Grafana for dashboards over high-volume event data.

Apache Druid + Superset

A real-time OLAP stack using Apache Druid for sub-second slice-and-dice queries and Apache Superset for self-service exploration.

Apache Pinot Real-Time Analytics

A user-facing analytics stack built on Apache Pinot for ultra-low-latency, high-throughput queries over fresh streaming data.

Airbyte + dbt + Snowflake

An open-core ELT stack: Airbyte ingests data, Snowflake stores it, and dbt transforms it with tested, version-controlled SQL models.

Dagster + dbt

An asset-oriented data orchestration stack where Dagster schedules and observes pipelines built around dbt's transformation models.

Prefect + Spark

A Python-native orchestration stack where Prefect schedules and monitors large-scale data processing jobs running on Apache Spark.

Kafka + Flink Streaming Lakehouse

An end-to-end streaming lakehouse stack: Kafka ingests events, Flink processes them in real time, and Iceberg tables on S3 serve analytics.

Comparisons20

Azure vs GCP

Microsoft Azure and Google Cloud Platform are the second and third largest public clouds, with different strengths in enterprise integration versus data and AI.

S3 vs EBS

Amazon S3 is object storage accessed over HTTP; Amazon EBS is block storage attached to EC2 instances. They serve fundamentally different storage needs.

Snowflake vs BigQuery

Cloud-agnostic data warehouse with decoupled compute versus Google's fully serverless analytics warehouse.

Snowflake vs Redshift

Cloud-agnostic warehouse with auto-scaling compute versus Amazon Redshift's AWS-native MPP data warehouse.

BigQuery vs Redshift

Google's fully serverless analytics warehouse versus Amazon Redshift's AWS-native MPP warehouse.

Databricks vs Snowflake

Lakehouse platform built on Spark and Delta Lake versus Snowflake's cloud data warehouse, now converging on shared use cases.

Spark vs Flink

The unified batch-and-streaming engine with micro-batch roots versus Apache Flink's true stream-first processing engine.

Airflow vs Dagster

The established task-based workflow orchestrator versus Dagster's asset-oriented, developer-friendly data orchestrator.

dbt vs Dataform

The widely adopted SQL transformation framework versus Google's warehouse-native transformation tool integrated with BigQuery.

Iceberg vs Delta Lake

Two open table formats that bring ACID transactions and schema evolution to data lakes, with different engine ecosystems.

Parquet vs ORC

Two columnar storage file formats for big-data analytics with different ecosystem strengths and compression characteristics.

Python vs R

Two leading languages for data work: Python is a general-purpose language strong across the data and ML pipeline, while R is purpose-built for statistics and visualization.

Pinecone vs Weaviate

Pinecone is a fully managed vector database; Weaviate is open source with managed and self-hosted options. The split shapes control, operations, and feature flexibility.

Weaviate vs Qdrant

Weaviate and Qdrant are open-source vector databases. Weaviate emphasizes a modular, feature-rich platform; Qdrant emphasizes a lean, performant Rust-based engine.

pgvector vs Dedicated Vector Database

pgvector adds vector search to PostgreSQL; a dedicated vector database is purpose-built. The choice trades operational simplicity against scale and specialized features.

Embeddings vs Keyword Search

Embedding-based semantic search matches by meaning; keyword search matches by terms. Each handles different query types, and hybrid search often beats either alone.

Feature Store: Build vs Buy

Teams can build a custom feature store or adopt a managed or open-source one. The choice trades control and fit against time-to-value and maintenance burden.

DVC vs Git LFS

DVC and Git LFS both version large files alongside Git. DVC targets ML data and pipelines; Git LFS is a general-purpose large-file extension.

Data Lake vs Data Warehouse

A data lake stores raw data of any type cheaply; a data warehouse stores structured, modeled data for fast analytics. They serve different stages and users.

ETL vs ELT

ETL transforms data before loading it; ELT loads raw data first and transforms inside the destination. Cloud warehouses have made ELT increasingly common.

Benchmarks10

Database Schema Mapping Accuracy

Tests accuracy of AI models in mapping database schemas between different systems

TPCx-BB (BigBench)

An end-to-end big-data analytics benchmark mixing SQL, machine learning, and natural-language processing over a retail data set on Hadoop/Spark-style platforms.

TPCx-AI

An end-to-end machine-learning benchmark measuring the full data-science pipeline — ingestion, training, and serving — across multiple AI use cases.

JMH (Java Microbenchmark Harness)

The standard harness for writing reliable JVM microbenchmarks, widely used to measure data-processing and serialization library performance on the JVM.

TeraSort

A distributed sort benchmark that orders one terabyte (or more) of data on a cluster, measuring big-data engine throughput and shuffle efficiency.

Sort Benchmark

A long-running family of competitive sorting benchmarks (GraySort, MinuteSort, JouleSort) that rank systems on speed, cost, and energy for large-scale sorting.

dbt Pipeline Performance Benchmark

A category of benchmarks measuring transformation pipeline performance in dbt — model build time, warehouse compute cost, and incremental run efficiency.

Streaming Throughput Benchmark

A benchmark category for event-streaming platforms measuring producer/consumer throughput and end-to-end latency under sustained load and varying durability settings.

ETL/ELT Pipeline Throughput Benchmark

A benchmark category measuring data-integration pipeline performance — extraction and load throughput, transformation latency, and end-to-end freshness.

fio (Flexible I/O Tester)

The standard tool for benchmarking storage I/O, measuring IOPS, bandwidth, and latency across configurable read/write patterns, block sizes, and queue depths.

FAQs9

What is the difference between OLTP and OLAP?

OLTP (Online Transaction Processing) handles many short, concurrent transactions like orders, payments, and updates, optimized for fast writes and row...

What is the difference between a data warehouse, a data lake, and a lakehouse?

A data warehouse stores structured, modeled data optimized for fast SQL analytics, typically loaded through schema-on-write pipelines; examples includ...

What is the difference between ETL and ELT?

ETL (Extract, Transform, Load) transforms data before loading it into the target system, which suited traditional warehouses with limited compute and ...

What is change data capture (CDC)?

Change data capture is a technique that detects and streams row-level changes (inserts, updates, deletes) from a source database so downstream systems...

What is a data pipeline?

A data pipeline is a series of automated steps that move data from sources to destinations, transforming and validating it along the way so it is read...

What is the difference between batch and stream processing?

Batch processing handles large volumes of data in scheduled chunks, such as a nightly job that aggregates the previous day's sales; it is simple and e...

What is Apache Parquet?

Apache Parquet is an open, columnar file format designed for efficient storage and querying of large analytical datasets. It stores data by column, wh...

What is data partitioning?

Partitioning divides a large table or dataset into smaller, manageable pieces based on a key such as date, region, or category. In databases it improv...