Data Format
22 items tagged with "data-format"
Patterns2
Message Translator
Converts a message from one data format or schema to another so systems with incompatible representations can communicate.
Normalizer
Translates messages arriving in many different formats into a single common format so downstream components handle one canonical representation.
Tutorials3
How to bulk load data into Snowflake with COPY INTO
Stage files, define a file format, and use COPY INTO to load data into Snowflake tables efficiently with error handling.
How to convert data to Apache Parquet for analytics
Convert CSV or JSON to columnar Parquet with partitioning and compression to speed up and shrink analytic queries.
How to use Avro and Schema Registry with Kafka
Define an Avro schema, register it, and produce and consume typed Kafka records with schema evolution and compatibility checks.
Blueprints6
ETL to ELT with dbt Blueprint
Move from transform-before-load ETL tools to in-warehouse ELT using dbt for version-controlled, testable transformations.
On-Prem Warehouse to Snowflake Blueprint
Migrate an on-premises data warehouse to Snowflake for elastic, separated storage and compute with pay-per-use scaling.
On-Prem Warehouse to BigQuery Blueprint
Migrate an on-premises data warehouse to Google BigQuery for serverless, petabyte-scale analytics with separated storage and compute.
On-Prem Warehouse to Amazon Redshift Blueprint
Migrate an on-premises data warehouse to Amazon Redshift for managed, columnar MPP analytics integrated with the AWS data stack.
Hadoop to Databricks (Spark) Blueprint
Migrate on-prem Hadoop and MapReduce workloads to Databricks on cloud object storage with Apache Spark and Delta Lake.
Data Lake to Lakehouse Blueprint
Upgrade a raw data lake to a lakehouse using open table formats (Delta, Iceberg, or Hudi) for ACID transactions and governance.
Reference Architectures1
Checklists2
Data Lake to Lakehouse Migration Checklist
Readiness checks for evolving a raw data lake into a governed lakehouse with table formats, schema enforcement, and ACID guarantees.
Data Format Migration Checklist
Checks for migrating data files between serialization formats such as CSV, JSON, Avro, Parquet, and ORC without losing fidelity.
Comparisons2
Iceberg vs Delta Lake
Two open table formats that bring ACID transactions and schema evolution to data lakes, with different engine ecosystems.
Parquet vs ORC
Two columnar storage file formats for big-data analytics with different ecosystem strengths and compression characteristics.
FAQs2
What is columnar storage and why is it faster for analytics?
Columnar storage organizes data on disk by column rather than by row, so all values of a single column are stored together. Analytical queries that sc...
What is Apache Parquet?
Apache Parquet is an open, columnar file format designed for efficient storage and querying of large analytical datasets. It stores data by column, wh...
Glossaries2
JSON
JSON (JavaScript Object Notation) is a lightweight, text-based data interchange format that represents structured data as key-value objects and arrays.
Protocol Buffers
Protocol Buffers is a language-neutral, binary serialization format from Google that uses a schema definition to encode structured data compactly and efficiently.