Data Engineering
Data pipelines and analytics best practices
Best Practices
Data Mesh Principles
Domain-oriented, self-serve data infrastructure principles promoting product thinking for data.
by ThoughtWorks (Zhamak Dehghani)dbt Style Guide
Community conventions for naming, structuring, and documenting dbt transformation projects.
by dbt LabsData Governance Framework
A structured set of roles, policies, and processes that make an organization accountable for the quality, security, and proper use of its data assets.
by DAMA InternationalData Quality Management
The practice of measuring, monitoring, and improving data across dimensions like accuracy, completeness, consistency, timeliness, and validity so it stays fit for use.
by DAMA InternationalData Contracts
Explicit, version-controlled agreements between data producers and consumers that define schema, semantics, quality, and SLAs to prevent breaking changes.
by dbt LabsMedallion Architecture
A layered data design that refines data through Bronze (raw), Silver (cleaned and conformed), and Gold (business-ready) tables to improve quality and reuse.
by DatabricksData Lakehouse Architecture
An architecture that combines the low-cost, open storage of a data lake with the transactions, schema, and performance of a data warehouse using open table formats.
by DatabricksELT vs ETL Best Practices
Guidance on when to transform data before loading (ETL) versus loading raw and transforming in the warehouse (ELT), and how to run each pattern well.
by dbt LabsData Lineage
The traceable record of data's origin, movement, and transformation across systems, enabling impact analysis, debugging, compliance, and trust.
by Linux Foundation (OpenLineage)Reverse ETL
The practice of moving modeled data from the warehouse back into operational tools like CRM and marketing platforms so business teams act on it directly.
by dbt LabsData Catalog and Discovery
A searchable inventory of an organization's data assets with metadata, ownership, and lineage so people can find, understand, and trust the data they need.
by Linux Foundation (DataHub)Apache Kafka Streaming Best Practices
Design and operational guidance for building reliable, scalable event streaming on Apache Kafka, covering topics, partitions, delivery semantics, and consumers.
by ConfluentPatterns
Database per Service
Each microservice owns and manages its own database, enabling loose coupling and independent deployability
Saga Pattern
Manage data consistency across microservices using a sequence of local transactions with compensating actions
Event Sourcing
Store state changes as a sequence of events rather than just the current state
Splitter
Breaks a composite message into a series of individual messages so each element can be processed independently downstream.
Content Enricher
Augments a message with additional data from an external source when the original lacks information the receiver requires.
Normalizer
Translates messages arriving in many different formats into a single common format so downstream components handle one canonical representation.
Polyglot Persistence
An architecture that uses multiple, purpose-fit data stores within one system, matching each store's strengths to each data access pattern.
Lambda Architecture
A big-data design that runs a batch layer for accurate historical views and a speed layer for low-latency recent data, merging both at query time.
Kappa Architecture
A streaming-first data design that uses a single processing path over an immutable log, reprocessing history by replay instead of a separate batch layer.
Medallion Architecture
A lakehouse data-organization pattern that refines data through bronze (raw), silver (cleaned), and gold (curated) layers for progressive quality and reuse.
Data Lakehouse
A data architecture that adds warehouse-style ACID tables, schema, and governance directly on low-cost data-lake storage, unifying analytics and ML on one platform.
Slowly Changing Dimension (SCD)
Techniques for handling changes to dimension attributes over time in a data warehouse, ranging from overwriting to preserving full historical versions.
Tutorials
How to stream database changes with Debezium CDC
Capture row-level changes from PostgreSQL using Debezium and Kafka Connect, producing an ordered stream of insert, update, and delete events.
How to build a Kafka producer and consumer
Create a Kafka topic, produce messages with keys, and consume them in a group, covering offsets, partitions, and delivery basics.
How to load Kafka topics into a database with Kafka Connect
Use a Kafka Connect sink connector to stream topic data into a relational database with no custom consumer code.
How to build transformation models and tests with dbt
Structure dbt staging and mart models, add tests and documentation, and run a build that materializes them in a warehouse.
How to author a data pipeline DAG in Apache Airflow
Write an Airflow DAG with scheduled tasks and dependencies, run it locally, and monitor execution in the UI.
How to write and run a PySpark batch job
Build a PySpark job that reads, transforms, and writes data using the DataFrame API, then run it with spark-submit.
How to bulk load data into Snowflake with COPY INTO
Stage files, define a file format, and use COPY INTO to load data into Snowflake tables efficiently with error handling.
How to validate data quality with Great Expectations
Define expectations for a dataset, run validations, and surface failures so bad data is caught before it spreads downstream.
Checklists
Data Warehouse Migration Readiness Checklist
Readiness checks for migrating an enterprise data warehouse to a cloud platform such as Snowflake, BigQuery, or Databricks.
Data Quality Validation Checklist
Checks for validating accuracy, completeness, consistency, and timeliness of data after a migration or pipeline change.
ETL to ELT Migration Checklist
Checks for converting traditional ETL pipelines to an ELT model that loads raw data first and transforms inside the warehouse.
Streaming Pipeline Readiness Checklist
Production-readiness checks for real-time streaming data pipelines built on platforms such as Apache Kafka or Pulsar.
Data Reconciliation Checklist
Checks for reconciling data between source and target systems after a migration to prove completeness and accuracy.
Data Lake to Lakehouse Migration Checklist
Readiness checks for evolving a raw data lake into a governed lakehouse with table formats, schema enforcement, and ACID guarantees.
Data Pipeline Observability Checklist
Checks for instrumenting data pipelines with freshness, volume, schema, lineage, and quality monitoring to detect issues early.
Technology Stacks
Modern Data Stack
Snowflake/BigQuery, dbt, Airflow, Looker - Analytics pipeline
Kafka + Flink Streaming Stack
Real-time stream processing stack pairing Apache Kafka for durable event streams with Apache Flink for stateful, low-latency computation.
Modern Data Stack (ELT)
Cloud-native ELT pattern: managed ingestion loads raw data into a warehouse, where dbt transforms it and a BI tool serves analytics.
dbt + BigQuery Analytics Stack
Serverless analytics engineering on Google BigQuery with dbt managing SQL transformations, tests, and lineage on top of warehouse-native compute.
Databricks Lakehouse
Unified lakehouse architecture combining data-lake storage with warehouse reliability via Delta Lake, Spark compute, Unity Catalog governance, and ML.
Kafka + Flink + Iceberg Streaming Stack
Real-time streaming architecture: Kafka transports events, Flink processes them with stateful low-latency compute, and Iceberg lands them in an open lakehouse.
Airflow + Spark Batch Pipeline Stack
Classic big-data batch pattern: Apache Airflow orchestrates scheduled DAGs that run distributed Apache Spark jobs over data-lake storage.
ClickHouse Real-Time Analytics Stack
High-performance analytics stack: ClickHouse ingests event streams from Kafka and serves sub-second OLAP queries powering user-facing dashboards.
Open Lakehouse Stack (Trino + Iceberg + dbt)
Open, engine-agnostic lakehouse: Iceberg tables on object storage, queried by Trino across sources, with dbt managing transformations in SQL.
Snowflake + Fivetran + Looker
A fully managed cloud analytics stack: Fivetran ingests data, Snowflake stores and transforms it, and Looker serves governed BI on top.
dbt + Amazon Redshift
An ELT analytics stack where data lands in Amazon Redshift and dbt transforms it with version-controlled, tested SQL models.
Databricks + Unity Catalog
A governed lakehouse stack centering Databricks compute and Delta Lake storage under Unity Catalog for unified data and AI governance.
Kafka + ksqlDB
A stream-processing stack using Apache Kafka for event transport and ksqlDB for SQL-based streaming transformations and materialized views.
Flink + Apache Paimon
A streaming lakehouse stack using Apache Flink for unified stream and batch processing over Apache Paimon's high-update table format.
Spark + Delta Lake
A lakehouse foundation pairing Apache Spark for distributed processing with Delta Lake's ACID table format over cloud object storage.
Trino + Iceberg + S3
An open lakehouse query stack: Trino runs fast SQL across Apache Iceberg tables stored as open files on Amazon S3.
ClickHouse + Grafana
A real-time analytics stack using ClickHouse as a fast columnar OLAP database with Grafana for dashboards over high-volume event data.
Apache Druid + Superset
A real-time OLAP stack using Apache Druid for sub-second slice-and-dice queries and Apache Superset for self-service exploration.
Apache Pinot Real-Time Analytics
A user-facing analytics stack built on Apache Pinot for ultra-low-latency, high-throughput queries over fresh streaming data.
Airbyte + dbt + Snowflake
An open-core ELT stack: Airbyte ingests data, Snowflake stores it, and dbt transforms it with tested, version-controlled SQL models.
Dagster + dbt
An asset-oriented data orchestration stack where Dagster schedules and observes pipelines built around dbt's transformation models.
Prefect + Spark
A Python-native orchestration stack where Prefect schedules and monitors large-scale data processing jobs running on Apache Spark.
Kafka + Flink Streaming Lakehouse
An end-to-end streaming lakehouse stack: Kafka ingests events, Flink processes them in real time, and Iceberg tables on S3 serve analytics.
FAQs
What is the difference between OLTP and OLAP?
OLTP (Online Transaction Processing) handles many short, concurrent transactions like orders, payments, and updates, optimized for fast writes and row-level access with normalized schemas. OLAP (Online Analytical Processing) supports complex analytical queries over large historical datasets, optimized for aggregation and reads, often using columnar storage and denormalized or star schemas. In short, OLTP runs the business in real time while OLAP analyzes it. Data is typically moved from OLTP systems into OLAP systems such as data warehouses through ETL or ELT pipelines.
What is the difference between a data warehouse, a data lake, and a lakehouse?
A data warehouse stores structured, modeled data optimized for fast SQL analytics, typically loaded through schema-on-write pipelines; examples include Snowflake, BigQuery, and Redshift. A data lake stores raw data of any type (structured, semi-structured, or unstructured) cheaply in object storage with schema-on-read, which is flexible but can become a disorganized 'data swamp.' A lakehouse combines both: it keeps data in open formats on cheap object storage while adding warehouse-style features such as ACID transactions, schema enforcement, and performance via table formats like Delta Lake, Apache Iceberg, or Hudi. The lakehouse aims to serve both data science and BI from one copy of the data.
What is the difference between ETL and ELT?
ETL (Extract, Transform, Load) transforms data before loading it into the target system, which suited traditional warehouses with limited compute and enforced quality up front. ELT (Extract, Load, Transform) loads raw data first and transforms it inside the target using its compute, which fits modern cloud warehouses and lakes that scale elastically. ELT keeps raw data available for reprocessing and lets analysts transform with SQL using tools like dbt, while ETL can reduce storage and apply governance before landing. The right choice depends on your platform's compute model, governance needs, and data volume.
What is change data capture (CDC)?
Change data capture is a technique that detects and streams row-level changes (inserts, updates, deletes) from a source database so downstream systems can stay in sync in near real time. The most efficient approach reads the database transaction log (such as the MySQL binlog or PostgreSQL WAL) rather than repeatedly polling tables, which avoids extra load and captures every change in order. CDC powers data replication, cache invalidation, search indexing, and feeding data lakes and streaming pipelines. Common tools include Debezium, and many managed services offer built-in CDC.
What is a data pipeline?
A data pipeline is a series of automated steps that move data from sources to destinations, transforming and validating it along the way so it is ready for analytics, machine learning, or applications. Pipelines can run in batch (processing data on a schedule) or streaming mode (processing events continuously as they arrive). Orchestration tools such as Apache Airflow, Dagster, or Prefect schedule and monitor the steps, handling dependencies, retries, and failures. Reliable pipelines emphasize idempotency, observability, and data quality checks so downstream consumers can trust the output.
What is the difference between batch and stream processing?
Batch processing handles large volumes of data in scheduled chunks, such as a nightly job that aggregates the previous day's sales; it is simple and efficient but introduces latency. Stream processing handles data continuously as individual events arrive, enabling near real-time results for use cases like fraud detection, monitoring, and live dashboards. Streaming frameworks such as Apache Flink, Kafka Streams, and Spark Structured Streaming manage state, windowing, and exactly-once semantics. Many architectures combine both, using streaming for low-latency needs and batch for heavy historical reprocessing.
What is data partitioning?
Partitioning divides a large table or dataset into smaller, manageable pieces based on a key such as date, region, or category. In databases it improves query performance through partition pruning, where the engine skips partitions that cannot match a query, and it makes maintenance like dropping old data efficient. In data lakes, partitioning by columns (for example, year/month/day directories) lets engines read only relevant files. Partitioning differs from sharding: partitioning typically splits data within a single system, while sharding spreads it across separate database instances.
Benchmarks
Database Schema Mapping Accuracy
Tests accuracy of AI models in mapping database schemas between different systems
TPCx-BB (BigBench)
An end-to-end big-data analytics benchmark mixing SQL, machine learning, and natural-language processing over a retail data set on Hadoop/Spark-style platforms.
TPCx-AI
An end-to-end machine-learning benchmark measuring the full data-science pipeline — ingestion, training, and serving — across multiple AI use cases.
JMH (Java Microbenchmark Harness)
The standard harness for writing reliable JVM microbenchmarks, widely used to measure data-processing and serialization library performance on the JVM.
TeraSort
A distributed sort benchmark that orders one terabyte (or more) of data on a cluster, measuring big-data engine throughput and shuffle efficiency.
Sort Benchmark
A long-running family of competitive sorting benchmarks (GraySort, MinuteSort, JouleSort) that rank systems on speed, cost, and energy for large-scale sorting.
dbt Pipeline Performance Benchmark
A category of benchmarks measuring transformation pipeline performance in dbt — model build time, warehouse compute cost, and incremental run efficiency.
Streaming Throughput Benchmark
A benchmark category for event-streaming platforms measuring producer/consumer throughput and end-to-end latency under sustained load and varying durability settings.
ETL/ELT Pipeline Throughput Benchmark
A benchmark category measuring data-integration pipeline performance — extraction and load throughput, transformation latency, and end-to-end freshness.
fio (Flexible I/O Tester)
The standard tool for benchmarking storage I/O, measuring IOPS, bandwidth, and latency across configurable read/write patterns, block sizes, and queue depths.
See a real scan run
A replay of the actual CLI running against our test repositories — live progress, real findings, a genuine DriftScore. Nothing executes in your browser.