The building blocks of scale

#Why it matters

Most system designs are combinations of a few building blocks. Knowing what each one solves, and what new problem it creates, is what lets you explain a design rather than recite it.

#The concept

Consistency trade-offs. When data is copied (replicas, caches, multiple regions), you choose between always reading the latest value and always being available and fast. Say which one each feature needs: a bank balance needs the latest value; a like count can be a few seconds old.

Cache hit rate matters more than it looks. With 3,900 reads per second, a 90% hit rate leaves about 390 for the database, and 99% leaves about 39. Going from 90% to 99% cuts database load by ten times.

Sharding needs a good key. The key decides which shard holds each row. A key such as user_id spreads load evenly if users are similar; a key such as country creates a hot shard for the largest country. And when you add a shard, you want to move as little data as possible.

#Worked example: adding a fifth shard

Compare simple modulo placement (hash(key) % n) with consistent hashing, which places nodes and keys on a ring and assigns each key to the next node clockwise. Virtual nodes (100 points per server here) spread the load evenly.

import bisect
import hashlib


def h(value: str) -> int:
    return int(hashlib.md5(value.encode()).hexdigest(), 16)


def modulo_shard(key: str, n: int) -> int:
    return h(key) % n


class Ring:
    """Consistent hashing with virtual nodes."""

    def __init__(self, nodes: list[str], vnodes: int = 100):
        self.points = sorted((h(f"{node}#{i}"), node) for node in nodes for i in range(vnodes))
        self.hashes = [p for p, _ in self.points]

    def node_for(self, key: str) -> str:
        i = bisect.bisect(self.hashes, h(key)) % len(self.points)
        return self.points[i][1]


keys = [f"user:{i}" for i in range(100_000)]

before = [modulo_shard(k, 4) for k in keys]
after = [modulo_shard(k, 5) for k in keys]
moved = sum(a != b for a, b in zip(before, after))
print(f"modulo, 4 -> 5 shards: {moved / len(keys):.1%} of keys move")

ring4 = Ring(["db1", "db2", "db3", "db4"])
ring5 = Ring(["db1", "db2", "db3", "db4", "db5"])
moved = sum(ring4.node_for(k) != ring5.node_for(k) for k in keys)
print(f"consistent hashing, 4 -> 5 nodes: {moved / len(keys):.1%} of keys move")

load = {}
for k in keys:
    node = ring5.node_for(k)
    load[node] = load.get(node, 0) + 1
print("keys per node:", dict(sorted(load.items())))

$ python3 sharding.py
modulo, 4 -> 5 shards: 80.1% of keys move
consistent hashing, 4 -> 5 nodes: 19.8% of keys move
keys per node: {'db1': 20632, 'db2': 21415, 'db3': 19933, 'db4': 18222, 'db5': 19798}

With modulo placement, going from 4 to 5 shards moves about 80% of all keys, which means copying most of the data while the system is live. With consistent hashing only about 20% move, roughly the one fifth that the new node should own, and the load stays close to even across nodes. This is why systems such as Cassandra and DynamoDB, and many caches, use consistent hashing or a close relative of it.

#Common mistakes

• Adding caching before measuring. Know the read pattern and the hit rate you expect.

• Read replicas for a write problem. Replicas add read capacity, not write capacity.

• Sharding too early. A single well-tuned database with replicas goes a long way; sharding adds permanent complexity.

• Queues without monitoring. A queue hides problems until the backlog is hours long. Alert on age of the oldest message.

#Interview angle

For each box you draw, be ready for "what happens when this fails?" and "what happens at ten times the load?". Good answers mention the trade-off: "a replica adds read capacity, but a user who just posted might not see their post for a second, so we read their own posts from the primary."

#Practice task

1. Change vnodes to 1 and to 1,000 and record how even the load is. Explain why.

2. For the URL shortener, choose a shard key and describe what a hot key looks like (one viral link) and how the cache protects the shard.

#Further reading

• The System Design Primer (open source)

• Amazon DynamoDB paper (2007)