ETL
35 items tagged with "etl"
Best Practices2
Medallion Architecture
A layered data design that refines data through Bronze (raw), Silver (cleaned and conformed), and Gold (business-ready) tables to improve quality and reuse.
ELT vs ETL Best Practices
Guidance on when to transform data before loading (ETL) versus loading raw and transforming in the warehouse (ELT), and how to run each pattern well.
Patterns2
Medallion Architecture
A lakehouse data-organization pattern that refines data through bronze (raw), silver (cleaned), and gold (curated) layers for progressive quality and reuse.
Slowly Changing Dimension (SCD)
Techniques for handling changes to dimension attributes over time in a data warehouse, ranging from overwriting to preserving full historical versions.
Tutorials2
How to load Kafka topics into a database with Kafka Connect
Use a Kafka Connect sink connector to stream topic data into a relational database with no custom consumer code.
How to bulk load data into Snowflake with COPY INTO
Stage files, define a file format, and use COPY INTO to load data into Snowflake tables efficiently with error handling.
Blueprints7
Batch to Streaming with Apache Kafka Blueprint
Replace nightly batch ETL with real-time event streaming on Apache Kafka for low-latency data movement.
Batch to Streaming with Amazon Kinesis Blueprint
Convert scheduled batch pipelines to real-time streaming on Amazon Kinesis Data Streams for managed, low-latency ingestion.
ETL to ELT with dbt Blueprint
Move from transform-before-load ETL tools to in-warehouse ELT using dbt for version-controlled, testable transformations.
On-Prem Warehouse to Snowflake Blueprint
Migrate an on-premises data warehouse to Snowflake for elastic, separated storage and compute with pay-per-use scaling.
Hadoop to Databricks (Spark) Blueprint
Migrate on-prem Hadoop and MapReduce workloads to Databricks on cloud object storage with Apache Spark and Delta Lake.
Teradata to Cloud Warehouse Blueprint
Migrate a Teradata enterprise warehouse to a cloud warehouse (Snowflake, BigQuery, or Redshift) to retire appliance costs.
Batch File Transfer to Streaming CDC Blueprint
Replace nightly batch file transfers between systems with real-time change data capture streamed onto an event backbone.
Reference Architectures5
Medallion Lakehouse on Azure
A bronze-silver-gold medallion lakehouse on Azure with Data Lake Storage, Delta tables, and Synapse for serving.
Batch ETL on Apache Spark
A scalable batch ETL platform on Apache Spark running on Kubernetes, transforming large datasets from object storage into curated tables.
Batch Inference Pipeline on Azure
A reference design for scheduled, large-scale offline scoring on Azure using Synapse, Azure ML batch endpoints, and a data lake for inputs and results.
iPaaS Integration Hub for SaaS Connectivity
An integration platform-as-a-service hub that connects SaaS and on-prem systems with connectors, transformation, and orchestration.
High-Volume Streaming Ingestion Pipeline
A streaming ingestion pipeline that captures high-velocity event data, buffers it durably, and lands it for real-time and batch use.
Playbooks6
On-Prem Warehouse to Snowflake Program Playbook
Migrate a legacy on-prem data warehouse to Snowflake, including workload analysis, ELT redesign, governance, and decommissioning.
Data Warehouse to BigQuery Program Playbook
Migrate analytics workloads to Google BigQuery with serverless design, partitioning strategy, governance, and cost controls.
dbt and ELT Adoption Program Playbook
Adopt dbt-style ELT to modernize transformation logic with version control, testing, documentation, and a medallion architecture.
Lakehouse Migration Program Playbook
Consolidate a data lake and warehouse into a unified lakehouse using open table formats, medallion architecture, and ACID transactions.
Data Pipeline Orchestration Program Playbook
Modernize fragile cron and script-based data pipelines into a governed orchestration platform with lineage, retries, and observability.
ETL to ELT Modernization Program Playbook
Modernize legacy ETL tools into a cloud-native ELT architecture, pushing transformation into the warehouse with version control and tests.
Checklists4
Data Warehouse Migration Readiness Checklist
Readiness checks for migrating an enterprise data warehouse to a cloud platform such as Snowflake, BigQuery, or Databricks.
Data Quality Validation Checklist
Checks for validating accuracy, completeness, consistency, and timeliness of data after a migration or pipeline change.
ETL to ELT Migration Checklist
Checks for converting traditional ETL pipelines to an ELT model that loads raw data first and transforms inside the warehouse.
Data Pipeline Observability Checklist
Checks for instrumenting data pipelines with freshness, volume, schema, lineage, and quality monitoring to detect issues early.
Stacks2
FAQs2
What is the difference between ETL and ELT?
ETL (Extract, Transform, Load) transforms data before loading it into the target system, which suited traditional warehouses with limited compute and ...
What is a data pipeline?
A data pipeline is a series of automated steps that move data from sources to destinations, transforming and validating it along the way so it is read...