Sharding

Sharding is a form of horizontal partitioning in which rows of a table are distributed across multiple independent database servers, called shards. Each shard holds a subset of the data and runs on its own hardware, allowing a system to scale beyond the limits of a single machine.

How It Works

A shard key determines which shard a given row belongs to. Common strategies include:

• Hash-based sharding: A hash of the key spreads rows evenly, avoiding hotspots but making range queries expensive.
• Range-based sharding: Contiguous key ranges go to each shard, which is efficient for range scans but risks uneven load.
• Directory/lookup sharding: A lookup table maps keys to shards, offering flexibility at the cost of an extra hop.

A routing layer or proxy directs queries to the correct shard. Cross-shard queries and transactions are harder, often requiring scatter-gather or distributed-transaction coordination. Databases such as MongoDB, Vitess (MySQL), Citus (PostgreSQL), and Cassandra provide built-in sharding.

Why It Matters

Sharding is the primary way to scale write throughput and total storage past what one server can handle. It is essential for very large datasets and high-traffic applications.

The trade-offs are significant: choosing a poor shard key causes uneven hotspots, resharding is operationally painful, and cross-shard joins and transactions add complexity. Many teams delay sharding until simpler options such as replication and caching are exhausted.

Related Terms

Sharding is a specialized form of partitioning, often combined with replication for fault tolerance.