Sharding

Sharding splits a large data store horizontally into multiple smaller stores called shards, each holding a subset of the data on its own node. When a single database can no longer hold the data or handle the write throughput, sharding spreads both across many machines, allowing the system to scale horizontally rather than by buying ever-larger hardware.

How It Works

Data is divided by a shard key. A sharding strategy maps each record to a shard: range-based (key ranges per shard), hash-based (hash of the key selects the shard, often via consistent hashing), or lookup-based (an explicit directory maps keys to shards). Each shard is an independent database that stores and serves its own slice. Queries that target a single shard go directly to it; queries spanning shards must fan out and combine results.

Choosing a good shard key is the central decision: it should spread data and load evenly and match common access patterns to keep most queries on a single shard.

Many systems keep a routing layer that knows the shard map, so application code addresses a logical store and the router forwards each operation to the right shard. Rebalancing, moving key ranges or virtual buckets between shards, is the hardest operational task; designing the scheme up front to move data in small, online increments avoids long maintenance windows later.

When to Use It

Shard when the dataset or write load exceeds what one node can handle, when you need to scale capacity horizontally, or when data can be partitioned naturally (for example, by tenant or region). It is common in large-scale OLTP systems and key-value stores.

Avoid premature sharding; it adds significant complexity, and vertical scaling, read replicas, or caching may suffice for smaller workloads.

It also suits multi-tenant systems where each tenant's data can live on a specific shard, simplifying isolation, capacity planning, and even per-tenant data residency.

Trade-offs

Sharding unlocks near-linear horizontal scale but complicates the system substantially. Cross-shard queries, joins, and transactions are hard and slow. A poor shard key causes hotspots and uneven load. Resharding to add capacity can be disruptive unless consistent hashing is used. Operational burden grows with the number of shards. The payoff is scale beyond a single machine. The most consequential and least reversible decision is the shard key, so teams model real access patterns first and prefer a key that keeps the common queries on a single shard while still spreading load evenly.

Related Patterns

Consistent Hashing is a common sharding strategy that eases resharding. Sharding pairs with CQRS and read replicas for read scale, and underlies the Database per Service approach. Scatter-Gather implements queries that span shards.

Example

A messaging app stores billions of messages. A single PostgreSQL instance cannot keep up. The team shards by user ID hash across 16 PostgreSQL instances; each user's messages live on one shard, so loading a conversation hits a single shard. Analytics that span all users use scatter-gather across the 16 shards. As load grows, more shards are added.