Sharding (partitioning)
Split one dataset across many databases by a shard key when a single primary can no longer keep up.
When one database isn't enough
Replicas add read capacity, but every write still lands on one primary, and one machine also has a storage limit.Sharding splits the data itself: each shard is an independent database holding a slice of the rows, with its own primary and replicas. Ten shards give you roughly ten times the write capacity and storage.
Choosing the shard key
The shard key decides which shard a row lives on. It's the most important decision, and hard to change later.
- Range-based: rows with keys in a range share a shard. Range queries are cheap, but new or popular ranges become hot (all of today's orders on one shard).
- Hash-based: a hash of the key picks the shard. Load spreads evenly, but range queries must ask every shard. Use consistent hashing so adding shards doesn't reshuffle everything.
- Directory-based: a lookup table maps each key to a shard. Flexible, but the directory is one more thing to keep fast and available.
Pick a key that spreads load and matches how you read: shard a chat app by conversation_id so a conversation's messages live together; shard a social app by user_id.
What gets harder
- Cross-shard queries and joins must fan out to many shards and merge results.
- Transactions across shards need two-phase commit or a saga, both slower and more complex.
- Hot keys: one celebrity's data can overload a shard no matter how you split. Cache it or split that key further.
- Resharding when you grow means moving data while the system stays up.
Shard last, not first. Caching, read replicas and a bigger primary go a long way, and each is far simpler to run.
In the interview
- Justify sharding with numbers: write QPS or storage beyond what one primary can take.
- Name the shard key and why it matches the main access pattern.
- Call out one cost: cross-shard queries, hot keys, or resharding.