System Design Fundamentals: Load Balancing, Caching, Sharding & Message Queues

System design questions look endlessly varied, but 90% of them are assembled from the same set of building blocks. Master these four fundamentals and you have the vocabulary for the vast majority of problems:

• Load Balancing: spread traffic across many machines — the start of horizontal scaling.
• Caching: keep hot data closer — cut latency, shield the backend.
• Sharding: split data across machines — break the single-node capacity ceiling.
• Message Queue: turn synchronous into asynchronous — decouple, smooth, buffer.

This post covers each block — "what it solves, how it works, what the trade-offs are" — with the follow-ups interviewers ask. Treat it as the system design "dictionary"; any later problem can be assembled from it.

🧱 Core idea: system design isn't inventing new things — it's assembling these standard blocks per requirements and explaining each choice's trade-off. What the interviewer wants is "why this block, and what's the cost."

① Load Balancing

What it solves: a single machine has limits (CPU, connections). When traffic exceeds one machine, you need many + a "dispatcher" spreading requests — the load balancer (LB). It also provides high availability (one dies, traffic auto-routes to healthy ones).

L4 vs L7:

• L4 (transport): forwards by IP/port only — fast, blind to application content.
• L7 (application): understands HTTP (path, headers, cookies), enabling content-based routing (e.g. /api to cluster A, /img to B), SSL termination — but heavier.

Common algorithms:

• Round Robin: rotate — simplest.
• Least Connections: send to the one with fewest active connections — good for uneven request durations.
• Consistent Hashing / IP Hash: pin a source to the same server (needed for session stickiness or cache affinity).

🎤 Interviewer follow-up: "Can the LB itself be a single point of failure?" (Yes — so run multiple LBs + failover, or DNS/Anycast for multiple LBs.) "What about session stickiness?" (Hash stickiness, or better, move sessions to shared storage (Redis) and make servers stateless — stateless scales horizontally.)

② Caching

What it solves: the database is slow and expensive. Put high-frequency-read hot data in memory (Redis) or somewhere closer, shielding the DB from read traffic — cut latency, raise throughput, protect the DB.

Where (layered): browser → CDN (static/edge) → app-layer cache → distributed cache (Redis) → the DB's own cache.

Write strategies:

• Cache-aside (most common): on read, check cache, on miss read DB and backfill; on write, update DB and invalidate (delete) the cache.
• Write-through: write cache and DB together (better consistency, slower writes).
• Write-back: write cache first, batch-flush to DB later (fast, but risks loss).

Eviction: memory is finite — evict with LRU / LFU / TTL.

Two classic problems:

• Invalidation consistency: DB changed but cache didn't → stale reads. Common fix: "update DB then delete cache" + a short TTL as backstop.
• Avalanche / penetration / breakdown: many keys expiring at once (avalanche), repeatedly querying nonexistent data (penetration), a hot key expiring and hammering the DB (breakdown). Fixes: jittered TTLs, cache null values, lock-and-rebuild for hot keys.

🎤 Interviewer follow-up: "How do DB and cache stay consistent?" (Cache-aside "update DB then delete cache," and why strong consistency is hard — usually accept eventual consistency + short TTL.) "What data should you not cache?" (Rarely read, strongly consistent, or frequently changing data.)

③ Sharding / Partitioning

What it solves: when data volume or write load exceeds what a single DB can hold/handle, split data horizontally into shards across machines. (Contrast: replicas solve read scaling and HA; sharding solves write and capacity scaling.)

Sharding strategies:

• Hash sharding: shard = hash(key) % N. Even distribution, but range queries are hard, and adding/removing machines reshuffles a lot of data.
• Range sharding: split by key range (A–M, N–Z). Range-query friendly, but prone to hotspots (one range runs hot).
• Consistent Hashing: adding/removing nodes moves only a little data — the common answer for distributed caches/DBs, easing the "change N → massive reshuffle" pain.

Core challenges:

• Hotspot: one shard is extra busy (e.g. a celebrity's data). Fixes: finer shard keys, salting, extra replication of hot data.
• Rebalancing: minimize data movement when adding machines — consistent hashing or virtual nodes.
• Cross-shard operations get hard: joins, transactions, aggregations across shards become complex — so choose the shard key well (so common queries land on one shard).

🎤 Interviewer follow-up: "How do you choose the shard key?" (One that lands most queries on a single shard, distributes evenly, and avoids hotspots — e.g. user_id rather than time.) "Why not hash % N?" (Adding/removing machines re-hashes nearly all data — use consistent hashing.) "How do you handle cross-shard transactions?" (Avoid if possible; if needed, 2PC or saga, but costly.)

④ Message Queue

What it solves: synchronous calls couple services — a slow downstream stalls the whole chain; a dead downstream drops requests. Make communication asynchronous: the producer drops a message in the queue and returns; the consumer processes at its own pace.

Three values:

• Decoupling: the producer needn't know who or how many consumers exist; each deploys/scales independently.
• Buffering (smoothing): on a traffic spike, the queue buffers and consumers digest at their own pace, not getting crushed.
• Reliability: if a consumer dies, the message stays in the queue and resumes after recovery (no lost tasks).

Typical uses: async tasks (email, transcoding), inter-service events, logging/streaming (Kafka), spike-smoothing orders.

Delivery semantics (guaranteed question):

• At-most-once: maybe lost (acceptable for unimportant data).
• At-least-once: maybe duplicated — so consumers must be idempotent.
• Exactly-once: hardest and costliest (usually approximated via dedup + idempotency).

🎤 Interviewer follow-up: "Messages get consumed twice — what do you do?" (Make the consumer idempotent — dedup by unique ID so reprocessing is safe. Almost always asked.) "How do you guarantee ordering when it matters?" (Ordered within a partition/key, unordered across partitions — route by key.) "What if consumption can't keep up with production?" (Scale consumers, watch backlog, route to a dead-letter queue if needed.)

Assembling the Blocks: a Read-Heavy, Write-Heavy Service

These four rarely appear alone — they usually combine into a chain:

user → DNS → load balancer (L7) → app servers (stateless, many)
                                      │  read
                                      ├─► cache (Redis) ──miss──► DB (sharded + replicas)
                                      │  write (async)
                                      └─► message queue ──► consumers ──► DB / other services

• Stateless app layer + LB → horizontal scaling, HA.
• Cache → absorbs most reads, protects the DB.
• Sharding + replicas → writes/capacity via sharding, reads/HA via replicas.
• Message queue → make slow writes/side-effects async, smooth and decouple.

🎤 Interviewer follow-up: "Read-heavy vs write-heavy — how does the architecture differ?" (Read-heavy → cache + read replicas; write-heavy → sharding + queue smoothing.) "Where might the bottleneck be?" (Walk the chain: LB → app → cache hit rate → DB shard hotspot → queue backlog.)

One Table to Close the Four Blocks

The Answer Framework Cheat

Any system design question can use these four as the skeleton:

1. Estimate scale (QPS, data volume, read/write ratio) → decide which blocks
2. Stateless app layer + load balancing → lay down horizontal scaling first
3. Add caching → absorb reads, cut latency (explain cache-aside and invalidation)
4. Shard + replicate the data layer → writes/capacity via sharding, reads/HA via replicas
5. Use a message queue → make slow work/side-effects async, smooth and decouple
6. State the trade-off at every step → the interviewer wants "why this, at what cost"

Remember one line: system design isn't memorizing architecture diagrams — it's recognizing what each of these four blocks solves, then assembling them per requirements and explaining the trade-offs. Know the blocks, and any large problem is just a permutation of them.