Data Format Migration Checklist
A checklist for migrating data between serialization formats like CSV, JSON, Avro, Parquet, and ORC without losing fidelity. It focuses on type mapping, null and timestamp handling, sample round-trip validation, and reconciliation. Use it whenever changing how data is stored on disk.
When to Use This Checklist
Use this checklist when migrating data between serialization formats: CSV to Parquet, JSON to Avro, or similar. Format choice affects storage cost, query speed, and schema enforcement. Row-based text formats like CSV and JSON are simple but verbose; columnar formats like Parquet and ORC compress well and accelerate analytics; Avro suits streaming with strong schema support. Converting between them risks subtle fidelity loss.
How to Use This Checklist
Document the source format, encoding, and schema precisely, then choose a target that fits how the data is read. The type-mapping and value-representation items are where fidelity is won or lost: nulls, timestamps, timezones, and decimal precision are the usual casualties. Validate a sample round-trip before converting everything, then reconcile counts and key fields. Keep source files until downstream readers confirm they handle the new format.
What Good Looks Like
A clean format migration preserves every value. The target format matches the access pattern, type mappings are explicit with no unresolved lossy conversions, and null, timestamp, timezone, and precision handling are verified. A sample round-trips without loss before the full conversion runs, counts reconcile afterward, and downstream readers are updated and tested. Source files are retained until adoption is confirmed, giving a clean rollback if a reader cannot parse the new format.
Common Pitfalls
The quiet killers are type and value conversions: floats losing precision, timestamps shifting timezone, and empty strings becoming nulls or vice versa. Choosing a columnar format for a row-by-row access pattern, or the reverse, hurts performance. Teams convert everything before validating a sample, then discover loss at scale. Forgetting to update downstream readers breaks consumers at cutover. Deleting source files too early removes the only recovery path.
Related Resources
Use schema-evolution-schema-registry and data-contracts to govern format and schema changes, and verify fidelity with data-quality-management. Trace conversions through data-lineage so consumers can follow the change.