What is data partitioning?

Answer

Partitioning divides a large table or dataset into smaller, manageable pieces based on a key such as date, region, or category. In databases it improves query performance through partition pruning, where the engine skips partitions that cannot match a query, and it makes maintenance like dropping old data efficient. In data lakes, partitioning by columns (for example, year/month/day directories) lets engines read only relevant files. Partitioning differs from sharding: partitioning typically splits data within a single system, while sharding spreads it across separate database instances.