Caching

Why cache

Reading from memory takes well under a millisecond; a database query takes several, and the database has a hard limit on queries per second. Most real traffic is skewed: a small set of items (popular posts, product pages, short links) gets most of the reads. Keep those hot items in an in-memory cache such as Redis or Memcached and most reads never reach the database.

The key number is the hit rate: the share of reads the cache answers. Going from 80% to 90% hits halves the traffic that reaches the database (20% of reads becomes 10%).

Caching patterns

• Cache-aside (most common): the app checks the cache; on a miss it reads the database and puts the result in the cache.
• Read-through: the cache itself loads from the database on a miss.
• Write-through: writes go to the cache and the database together. Fresher, but every write pays twice.
• Write-behind: writes go to the cache and are flushed to the database later. Fast, but data can be lost if the cache dies.

The hard part: staleness and eviction

• Invalidation: when data changes, delete or update the cached copy, or give entries a TTL (time to live) so they expire.
• Eviction: memory is limited, so something must go. LRU (least recently used) is the usual default.
• Thundering herd: when a hot key expires, thousands of requests miss at once and hit the database together. Fixes: let one request refill while others wait, or refresh before expiry.
• Don't cache data that must never be stale, such as account balances.

Try it

A read-heavy app with no cache: the database is overloaded. Turn the cache on and try the hit rates. At 50% the database is still overloaded; at 80% it has room to spare. That jump is why hit rate is the number to watch.