Skip to main content

Scaling out: replication, partitioning, sharding

When one machine is no longer enough, you spread the data across several. There are three techniques, and the order you reach for them matters: replicate first, partition next, shard last. Each adds capability and complexity.

Replication: copy the data​

Replication keeps copies of the database on more than one machine. One node (the primary) takes all writes; replicas receive a stream of those changes and serve reads. This buys two things: read scaling (spread reads across replicas) and high availability (if the primary dies, a replica is promoted).

The catch is replication lag: with asynchronous replication, a replica may be a moment behind the primary, so a read just after a write can return stale data. Synchronous replication removes that gap but makes writes wait for replicas to confirm.

Partitioning: split a big table​

Partitioning breaks one large table into smaller pieces - by range (dates), hash, or list - within a single database. Queries that target one partition scan less data, and indexes stay smaller. It is a management and performance win for huge tables, but the data still lives on one server.

Sharding: split across many databases​

Sharding splits the data across many independent databases, each holding a slice keyed by a shard key (customer id, region). This is the only one of the three that scales writes horizontally - each shard takes its own.

The cost is steep: queries that span shards are hard, joins across shards may be impossible, a badly chosen key creates hotspots (one shard overloaded), and rebalancing as you add shards is painful. That is why you shard last - after replication and partitioning have been exhausted.

The right order​

  • Replicate for read scaling and failover - cheap, do it early.
  • Partition when a single table grows unwieldy.
  • Shard only when write volume genuinely exceeds what one primary can take - it adds the most complexity for the most scale.

Quick quiz​

Scaling out

4 questions

1Which technique scales WRITES across machines?

2What does replication primarily buy you?

3What is replication lag?

4Why is sharding the last resort?

Next up

Spreading data across machines forces trade-offs. Consistency and concurrency covers CAP, how databases stay correct under concurrency, and how NewSQL keeps SQL and ACID at scale.