02.4 Database sharding (horizontal partitioning)

Mid-level · Component

Core idea

When your data is too large for one machine, you split it across multiple machines. Each machine (shard) holds a subset of data. Sharding solves write scalability and storage limits — replication alone cannot.

Mental model

Imagine a library with 10 million books. One building can’t hold them. So you build 10 buildings: Building A–C, Building D–F, etc. Each query goes to the right building based on the first letter. Sharding does this for data rows.

Why it matters

  • A single node can hold maybe 5TB comfortably. At 50TB you must shard
  • Sharding also distributes write load — each shard handles a fraction
  • But sharding adds enormous complexity — use it only when necessary

What to know

  • Hash sharding: shard = hash(user_id) % N — even distribution, no range queries across shards
  • Range sharding: shard by time or ID range — allows range queries, risk of hotspots
  • Directory sharding: lookup table maps key → shard — flexible, lookup table is a bottleneck
  • Hotspot problem: celebrity users or trending items land all load on one shard
  • Re-sharding: adding a shard requires moving data — use consistent hashing to minimize this

Common traps

  • Sharding too early — enormous operational complexity for no gain
  • Cross-shard joins — become application-level, slow and complex
  • Choosing a bad shard key — everything lands on one shard (hotspot)

Recognise it

Recognition trigger: Write QPS > what one machine can handle OR storage > what one machine can hold → time to shard. First question: what is the shard key? It must distribute load evenly AND avoid cross-shard queries for your most common operations.

Practice question

Design the sharding strategy for a social network’s user posts. What’s your shard key and why?

Apply at work

Answer these for the system you work on.

Go deeper