03.2 Caching — the highest leverage optimization

Junior · Component

Core idea

A cache stores copies of expensive-to-compute or expensive-to-fetch data close to where it’s needed. The core insight: most reads access the same 20% of data (Pareto). Cache that 20%, eliminate 80% of DB load.

Mental model

A chef who prepares 10 dishes memorises the recipes for the 3 most-ordered dishes (cache). For rare dishes, they look it up in the cookbook (database). The memorised recipes are served instantly. The cookbook ones take a moment. If the recipe changes, they must update their memory (cache invalidation).

Why it matters

  • Database I/O is the most common bottleneck in web systems
  • Cache can serve 10x–1000x more QPS than a database for the same hardware
  • Understanding cache patterns separates junior from senior engineers

What to know

  • Cache-aside (lazy loading): app checks cache first, on miss fetches from DB and populates cache — most common
  • Write-through: write to cache and DB synchronously — consistent, slower writes
  • Write-behind (write-back): write to cache, async flush to DB — fast writes, risk of data loss
  • Read-through: cache fetches from DB on miss automatically — simpler app code
  • Eviction: LRU (least recently used), LFU (least frequently used), TTL (time-based expiry)
  • Cache stampede: all cache keys expire simultaneously → thundering herd → use jitter on TTL

Common traps

  • Cache invalidation — ‘one of the two hard problems in CS’. When data changes, stale cache must be updated or expired
  • Caching at the wrong layer — caching a slow query that’s slow because of missing index, not I/O
  • Not handling cache miss gracefully — cache miss under load can cascade to DB overload

Recognise it

Recognition trigger: High read QPS → cache. Check: is data read more than written? Is it okay if it’s slightly stale? Is the hot set small enough to fit in RAM? If yes to all three → cache is the right call.

Practice question

How does Twitter serve a celebrity’s tweet to 10 million followers without hitting the database 10 million times?

Apply at work

Answer these for the system you work on.

Go deeper