Caching
A cache is a derived, disposable copy optimized for a particular access path. It can reduce latency, backend load, and cost, but creates coherence, invalidation, capacity, and failure problems. Senior candidates define the cache key, authority, freshness contract, fill path, eviction, stampede protection, hot-key strategy, and behavior when the cache is empty or unavailable.
How to use this chapter
- Read the mental model and mechanics without taking notes.
- Close the page and explain the topic aloud in five minutes.
- Work the quantitative example on paper.
- Answer the question ladder without revealing the answer signals.
- Complete one design drill and score yourself with the senior checklist.
Learning objectives
- Explain locality, hit ratio, working set, cache keys, TTLs, and eviction policies.
- Compare cache-aside, read-through, write-through, write-behind, refresh-ahead, and materialized views.
- Design invalidation and consistency by operation and failure mode.
- Prevent stampedes, penetration, hot keys, oversized values, and cache pollution.
- Size multi-level caches and quantify hit-ratio impact on backend load and latency.
- Operate cache failures, warmup, resharding, and tenant isolation at senior level.
Mental model
A cache answers a question using a copy that may be missing or stale. The source of truth remains elsewhere unless the “cache” is actually an authoritative store. Every cache design must answer: what exact query does the key represent, how fresh may the answer be, how is it populated, and what happens when the copy disagrees or disappears?
Core mechanics
Cache layers and locality
Caches exist in CPU, operating-system page cache, process memory, distributed cache, database buffer pool, reverse proxy/CDN, and client. A request may cross several layers. The closer cache is faster but smaller and less shareable; a shared cache improves reuse but adds network latency and failure dependency.
Temporal locality means recently used data is likely reused; spatial locality means nearby data may be accessed together. Workload skew often makes a small hot set responsible for most requests, which is why caching works. Measure popularity distribution and object sizes rather than assuming a uniform keyspace.
Cache keys and values
The key must encode every input that changes the answer: resource ID, tenant, locale, authorization class, query/filter, version, and sometimes experiment cohort. Missing a dimension leaks or corrupts data; including noisy dimensions destroys hit ratio. Normalize keys so semantically equivalent requests share entries.
Decide whether to cache full objects, fragments, query results, aggregates, or negative results. Large values cause network and eviction amplification. Compression may save capacity but adds CPU. Version values/schema so old readers do not misinterpret a rolling deployment.
Population patterns
Cache-aside: application checks cache, loads source on miss, and writes cache. It is simple but can race and stampede. Read-through: cache library/service loads source. Write-through: writes update cache and store synchronously. Write-behind: cache acknowledges then persists later, improving write latency at durability/ordering risk. Refresh-ahead: hot entries refresh before expiry.
A materialized view is a durable derived representation and may be rebuilt from source/log; it differs operationally from an ephemeral cache. Name the authority and loss behavior accurately.
Freshness and invalidation
TTL bounds how long an entry may remain without refresh, but actual staleness depends on write timing and replication. Event-driven invalidation can reduce staleness, yet events can be delayed or lost. Versioned keys avoid deletion races: write new data under version N+1 and update a small pointer or have readers request the current version.
Common patterns include delete-on-write, update-on-write, leases, generation numbers, and publish invalidation. In distributed systems, a writer can update the database then crash before invalidating the cache. Use outbox/change-data-capture or accept a bounded TTL and document the anomaly.
Eviction and admission
LRU approximates recency; LFU favors frequently used items; FIFO/random can be cheaper. Production caches often use approximate policies and admission filters to avoid one-time scans evicting valuable hot data. TTL expiration and memory eviction are different: an unexpired item can be evicted under pressure.
Capacity should consider bytes, item overhead, fragmentation, replication, and headroom. One huge key or tenant can evict everyone else, so apply max-value size, per-tenant quotas, and cost-aware admission.
Stampede, penetration, and hot keys
A stampede occurs when many requests miss or expire together and all load the source. Use request coalescing/single-flight, leases, probabilistic early refresh, TTL jitter, stale-while-revalidate, and backend concurrency limits. Keep a fallback when the loader fails so one poisoned key does not block forever.
Cache penetration is repeated lookup of absent data. Negative caching, Bloom filters, validation, and rate limits help. Hot keys can overwhelm one cache shard even with a high global hit ratio; replicate hot values, use local L1 caches, split counters, or push results closer to clients.
Consistency patterns
Cache-aside reads can return stale data after writes, and concurrent misses can overwrite a newer cache value with older source data if replication lag is involved. Use version numbers and only install a value if its version is not older. For read-your-writes, bypass cache temporarily, populate synchronously, or route to an authoritative read.
Do not cache security decisions longer than revocation requirements. Permissions, balances, inventory, and feature flags need domain-specific freshness and invalidation. Sometimes the right cache is a short-lived compiled policy with explicit epoch rather than arbitrary TTL.
Observability and failure behavior
Monitor hit ratio by operation/tenant/key class, miss reason, fill latency, evictions, expired versus absent, memory fragmentation, hot-key distribution, loader concurrency, error rate, and backend load. A global hit ratio can hide a critical endpoint with poor caching.
Decide whether cache failure causes fail-open to the source, fail-closed, stale serving, or rejection. Uncontrolled fail-open can collapse the database. Use circuit breakers, miss budgets, request shedding, and gradual warmup after restart or resharding.
Decision table
| Decision | Prefer the first option when… | Prefer the second option when… | Senior caveat |
|---|---|---|---|
| Local L1 vs distributed cache | Ultra-low latency and per-instance duplication are acceptable. | Shared reuse, larger capacity, and centralized invalidation dominate. | A two-level cache needs coherent versioning and prevents L1 from hiding L2 hot spots. |
| TTL-only vs event invalidation | Bounded staleness is acceptable and simplicity wins. | Writes must become visible quickly and event infrastructure is reliable. | Use both: event for promptness, TTL/generation for repair. |
| Cache-aside vs write-through | Reads dominate and occasional cold miss is acceptable. | Immediate cache freshness and predictable reads justify write-path coupling. | Write-through can make cache availability part of write availability unless designed carefully. |
| Serve stale vs fail | Availability and bounded stale data are safer for the product. | Wrong/stale data violates correctness or security. | Define max staleness and expose age; not every endpoint gets the same policy. |
| LRU vs LFU/admission | Recent reuse predicts popularity and workload shifts quickly. | Long-lived popularity and scans would pollute recency. | Approximate policies are normal; evaluate hit ratio per byte, not only per item. |
| Replicate hot key vs shard value | The value is read-only/small and replication is simple. | Writes or large value need partitioned aggregation/chunks. | Replication increases invalidation fan-out; sharding complicates reads/atomicity. |
Quantitative reasoning
Failure modes and production signals
| Failure mode | What users see | Likely cause | Mitigation / design response | Useful signals |
|---|---|---|---|---|
| Cache stampede | Origin latency/errors spike at expiry | Synchronized TTL or cold miss | Single-flight, jitter, refresh-ahead, stale serve, origin limit | Concurrent fills/key, miss burst, origin QPS |
| Stale overwrite | Old value returns after a newer write | Slow miss fill races invalidation/new value | Versioned values, CAS install, generation keys | Version regressions, fill duration, write/read mismatch |
| Hot shard/key | One cache node saturated; global metrics look fine | Skewed popularity or hash imbalance | Replicate/local-cache hot key, vnode balance, split workload | Per-node QPS/CPU, top keys, max/mean load |
| Fail-open cascade | Database overload after cache outage | Every miss bypasses cache | Miss budget, circuit/load shed, stale fallback, staged warmup | Cache error, source QPS, rejected misses, warmup rate |
| Cache poisoning/leak | Incorrect or cross-user data served | Bad key, untrusted populate, partial write | Key completeness, validation, tenant namespace, atomic set | Mismatch canaries, key dimensions, suspicious values |
| Eviction churn | Low hit ratio despite enough nominal memory | Large objects, fragmentation, scan pollution, TTL burst | Admission policy, size classes, quotas, increase/partition capacity | Evictions/s, bytes evicted, item size, fragmentation |
Senior-level lenses
Define the authority and repair path
If every cache node is lost, how is the value reconstructed and how fast? A cache backed by an overloaded database may have an unacceptable cold-start RTO. Prewarm critical keys, rate-limit fills, retain stale snapshots, or use a durable materialized view.
Freshness is an SLO
Express freshness as a user requirement: 99.9% of profile updates visible within 5 seconds; permission revocation within 30 seconds; product price never older than one minute. Then choose invalidation, TTL, and fallback. “Eventually consistent cache” is not specific enough.
Use versions to defeat races
Attach source version/epoch to cache values, invalidation events, and requests. A delayed event or fill should not replace a newer value. Version comparison is often more robust than trying to perfectly order distributed invalidations.
Cache hit ratio is a distribution
Measure per endpoint, object size, tenant, region, and cache layer. A high count hit ratio can coexist with low byte hit ratio if large objects miss. A high global ratio can hide low ratio on expensive queries. Capacity decisions should weight miss cost.
Protect source during recovery
Cache fleet restarts, resharding, deployment, or popular-key expiry can all create cold load. Warm gradually, prioritize critical keys, use admission, and cap fills per source shard. A cache is a reliability feature only if its failure is controlled.
Tenant isolation applies to cache
Namespaces prevent data mixing; quotas and fair admission prevent eviction attacks; per-tenant encryption or dedicated pools may be required. Include cache memory and miss cost in tenant billing/limits.
Interview question ladder
What is cache-aside?
Show strong-answer signals
Application reads cache, loads source on miss, writes cache; on writes it updates/invalidates separately. Mention stale windows and stampede.
What is the difference between expiration and eviction?
Show strong-answer signals
Expiration is freshness policy/TTL; eviction removes entries for capacity/policy even if not expired.
Why can a 99% hit ratio still be dangerous?
Show strong-answer signals
At huge traffic 1% is large, expensive misses may dominate, distribution/hot shards matter, and a drop to 95% multiplies source load.
How do you prevent a cache stampede?
Show strong-answer signals
Single-flight/leases, TTL jitter, early refresh, stale-while-revalidate, negative caching, source concurrency limit, and bounded failure behavior.
How do you invalidate a cache after a database write?
Show strong-answer signals
Delete/update plus outbox/CDC event, version/generation, TTL fallback; discuss crash between DB commit and invalidation and race with fills.
When is negative caching useful?
Show strong-answer signals
Repeated absent/invalid lookups; short TTL, distinguish authoritative not-found from transient error, avoid hiding newly created data too long.
Design caching for a product catalog with price updates and viral products.
Show strong-answer signals
CDN/app/distributed layers, keys by locale/price context, version/TTL, invalidation via CDC, hot-key replication, stale policy, source protection, price correctness boundary.
A cache cluster is lost. How do you avoid taking down the database?
Show strong-answer signals
Failover/stale snapshots, admission and miss budget, prioritize requests, rate-limited single-flight fills, prewarm top keys, autoscale source, degraded responses, monitor warm curve.
Explain and fix a race where an old cache fill overwrites a newer value.
Show strong-answer signals
Timeline: miss reads old replica, write commits/invalidates, slow fill sets old. Include version in source/value and compare/CAS; route authoritative read or generation key.
How do you cache authorization decisions?
Show strong-answer signals
Key principal/resource/action/policy epoch, short TTL or event invalidation, revocation SLO, deny/allow fail policy, audit, avoid cross-tenant leakage, bypass for high-risk actions.
Design a multi-region cache hierarchy for a global API with regional writes.
Show strong-answer signals
Client/edge, regional L1/L2, source ownership, versioned data/invalidation, local stale reads, read-your-write routing, failover, cold-start protection, and cost/egress.
Create a fairness policy for a shared cache where one tenant runs scans and another has latency-critical hot keys.
Show strong-answer signals
Per-tenant quotas/admission, workload classes, protected pools or frequency policy, max item size, cost-weighted eviction, borrowing rules, telemetry and billing.
Design drills
Design caching for home timelines with celebrity fan-out, edits/deletes, per-user ranking, and read-your-own-post. Choose cached objects versus pages and invalidation/freshness.
What the interviewer is testing
Cache key cardinality, hot keys, materialized views, consistency and source protection.
Design client, process, and regional caches for feature flags. Revocations must reach 99.99% of requests in 10 seconds, and the control plane may be unavailable.
What the interviewer is testing
Freshness SLO, epochs/stream updates, last-known-good, secure fail behavior, and observability.
A marketing launch causes one key to receive 2M RPS and its 60-second TTL expires synchronously. Walk through immediate mitigation and permanent architecture.
What the interviewer is testing
Hot-key replication, single-flight, jitter/refresh, edge/local layers, origin overload control.
Common weak answers and how to improve them
“Use Redis.”
Show the stronger answer
First define key, value, freshness, population, eviction, authority, and failure behavior; product name comes later.
“Invalidate on every write.”
Show the stronger answer
Explain crash/race paths, event durability, versions, TTL fallback, and multi-layer caches.
“LRU is best.”
Show the stronger answer
Match policy to workload, object sizes, scans, and admission; measure hit ratio per byte and miss cost.
“If cache is down, query the database.”
Show the stronger answer
Bound fail-open with miss budgets, stale fallback, shedding, and gradual warmup.
“TTL guarantees data is at most N seconds stale.”
Show the stronger answer
Writes can occur just after fill, clocks/replication/event paths matter, and serving stale beyond TTL may be configured; define exact age semantics.