Computer Architecture for System Designers
System design interviews do not require chip-design expertise, but senior candidates must reason from physical limits. CPU time, memory locality, kernel work, network interrupts, storage queues, and serialization all shape the latency and cost of a service. The useful abstraction is not “a server has 16 cores”; it is “a request consumes measurable CPU, memory bandwidth, allocations, I/O operations, and queueing time under a particular concurrency pattern.”
How to use this chapter
- Read the mental model and mechanics without taking notes.
- Close the page and explain the topic aloud in five minutes.
- Work the quantitative example on paper.
- Answer the question ladder without revealing the answer signals.
- Complete one design drill and score yourself with the senior checklist.
Learning objectives
- Explain the CPU–cache–memory–storage hierarchy and why locality dominates performance.
- Translate per-request resource cost into cores, memory, network bandwidth, and device IOPS.
- Distinguish concurrency from parallelism and identify contention, context switching, and false sharing.
- Recognize when page cache, virtual memory, NUMA, syscalls, and kernel networking affect a design.
- Use profiling and saturation signals instead of guessing which resource is the bottleneck.
- Discuss tail latency, queueing, overload, and hardware-aware deployment at a senior level.
Mental model
Think of a request as a pipeline of work moving through progressively slower resources. Registers and CPU caches are tiny and fast; DRAM is larger and slower; local SSD is slower again; remote storage and networks add more latency and variance. The exact nanoseconds change by generation, but the ordering and several-orders-of-magnitude gaps remain. A design that repeatedly crosses those boundaries pays for each crossing.
Core mechanics
CPU execution, cores, and utilization
CPU demand is commonly approximated in core-seconds per second. If a request uses 2 ms of CPU and the service receives 50,000 requests/s, it needs 100 core-seconds/s before headroom. At a 60% target utilization, that is roughly 167 cores, excluding background work and imbalance.
Average utilization alone is not enough. A process can show moderate CPU while one hot shard, one lock owner, one garbage-collection thread, or one network receive queue is saturated. Inspect per-core utilization, run-queue depth, throttling, steal time in virtualized environments, instructions per cycle, and time in user versus kernel mode. Senior candidates explicitly reserve headroom for bursts, failover, and noisy neighbors.
Memory hierarchy and locality
CPUs fetch data in cache lines rather than individual fields. Sequential scans, compact representations, batching, and structure-of-arrays layouts can exploit spatial locality; pointer-heavy graphs and random hash-table access often miss caches. Reusing recently touched data exploits temporal locality. The same algorithmic complexity can have radically different wall-clock behavior depending on locality.
Memory bandwidth can become the bottleneck even when CPU arithmetic is light. Large in-memory scans, compression, encryption, and copying between buffers may saturate channels. Zero-copy APIs, scatter/gather I/O, buffer reuse, and avoiding unnecessary serialization can reduce memory traffic, but they increase implementation complexity and must be justified by profiles.
Virtual memory, page cache, and TLBs
Processes use virtual addresses that are translated through page tables. CPUs cache translations in the translation lookaside buffer (TLB); large working sets and random access can generate TLB misses. Huge pages may reduce translation pressure, but can increase memory waste and operational complexity.
File reads and writes often interact with the operating system page cache. A database benchmark that “reads from disk” may actually read warm pages from memory; a write acknowledged by an application may still reside in volatile caches unless durability barriers and fsync-like semantics are involved. Interview answers should separate application acknowledgment, kernel buffering, device persistence, and replicated durability.
Concurrency, locks, and scheduling
Threads are useful for parallel CPU work and blocking APIs, but too many runnable threads cause scheduler overhead. Event loops reduce thread count for I/O-heavy workloads, yet any blocking handler can stall many connections. Thread pools introduce a queue whose bound, rejection policy, and per-task cost must be designed.
Locks serialize critical sections. Contention grows nonlinearly when hold time or arrival rate rises. Alternatives include partitioning state, immutable snapshots, read-copy-update patterns, actor ownership, optimistic concurrency, and lock-free structures. These alternatives trade simplicity for memory use, retries, or weaker instantaneous visibility. False sharing—independent counters on the same cache line—can also create coherence traffic that looks like unexplained CPU cost.
I/O devices, interrupts, and batching
Storage and network devices expose throughput, operation-rate, queue-depth, and latency constraints. Tiny random I/O may be IOPS-bound; large sequential transfer may be bandwidth-bound. Network packet processing can be packet-rate-bound before link bandwidth is full. Batching improves amortization but increases waiting time and can worsen tail latency.
Modern systems use DMA so devices move data without the CPU copying every byte, but the CPU still handles descriptors, interrupts or polling, protocol stacks, checksums, encryption, and application parsing. Kernel-bypass or busy-polling techniques can reduce latency for specialized workloads while consuming dedicated cores and operational simplicity.
NUMA and placement
On non-uniform memory access machines, a CPU accesses local memory faster than remote-node memory. A process whose threads migrate across sockets or whose memory is allocated on the wrong node can lose performance despite ample total capacity. Databases, caches, and packet-processing services may benefit from CPU affinity, memory policy, and topology-aware sharding.
The system-design implication is broader than NUMA tuning: placement matters at every scale. Keep computation near its state, avoid cross-zone chatty calls, and understand that “one large machine” may itself contain multiple latency domains.
Decision table
| Decision | Prefer the first option when… | Prefer the second option when… | Senior caveat |
|---|---|---|---|
| Threads vs event loop | Handlers perform blocking work or CPU tasks that parallelize cleanly. | The workload has many mostly-idle connections and nonblocking libraries. | A hybrid often wins: event loops for sockets, bounded pools for CPU or blocking calls. |
| Scale up vs scale out | The workload benefits from shared memory, low coordination, or strong single-node transactions. | Failure-domain isolation and horizontal capacity matter more than shared-memory speed. | Large nodes create larger blast radius and slower recovery; many tiny nodes create coordination overhead. |
| Batching vs immediate processing | Per-operation overhead dominates and latency budget has slack. | Interactive tail latency or fairness dominates. | Use bounded batch size and bounded wait time; monitor queue age, not only throughput. |
| Copying vs zero-copy | Simplicity, safety, or payload size makes copies cheap. | Profiling proves memory bandwidth and copies dominate. | Zero-copy can pin buffers, complicate ownership, and move bottlenecks rather than remove them. |
| Shared mutable state vs partitioned ownership | The state is small and contention is demonstrably low. | Write contention or cache-line bouncing limits scale. | Partitioning requires routing, rebalancing, and handling cross-partition operations. |
| Warm page cache vs direct/device-oriented I/O | General-purpose workloads benefit from OS caching and readahead. | The application already manages caching or needs predictable durability/control. | Do not infer persistence or device performance from a warm-cache benchmark. |
Quantitative reasoning
Failure modes and production signals
| Failure mode | What users see | Likely cause | Mitigation / design response | Useful signals |
|---|---|---|---|---|
| CPU saturation | Rising latency and timeouts, often without errors initially | Insufficient cores, hot partition, expensive code path, retries | Admission control, profile hot paths, add capacity, shed optional work | Per-core CPU, run queue, throttling, flame graphs, p99 CPU time |
| Memory pressure | Long pauses, OOM kills, swapping, unstable latency | Leaks, oversized caches, unbounded queues, fragmentation | Bound queues/caches, memory limits, streaming, leak analysis | RSS, working set, page faults, reclaim, GC pause, OOM events |
| Lock contention | Throughput plateaus while CPU may stay below 100% | Large critical section or shared hot key | Shard ownership, shorten lock, optimistic design, batch updates | Lock wait, blocked threads, off-CPU profiles, context switches |
| I/O queue saturation | Tail latency spikes and write stalls | Random IOPS limit, compaction, fsync bursts | Shape writes, separate workloads, provision IOPS, backpressure | Device queue depth, await, utilization, fsync latency |
| NUMA imbalance | One deployment underperforms identical hardware | Remote memory, thread migration, asymmetric IRQ placement | Topology-aware pinning and allocation; validate with counters | Remote access counters, per-node memory, CPU migrations |
| Cache/TLB thrash | Higher CPU per request after dataset or concurrency growth | Working set exceeds caches, random access, oversized code/data | Improve locality, compact data, partition working sets | Cache misses, TLB misses, IPC, memory bandwidth |
Senior-level lenses
Tail latency is a queueing problem
Service time and response time are different. A 2 ms handler can produce 200 ms responses when it waits behind other work. As utilization approaches saturation, small traffic changes create large queueing changes. Senior designs use bounded queues, concurrency limits, deadlines, load shedding, and per-priority isolation rather than assuming autoscaling reacts instantly.
Measure the whole request cost
CPU profiles can miss off-CPU waiting; application metrics can miss kernel cost; node averages can hide one hot core. Combine traces, on-CPU and off-CPU profiles, allocator metrics, device latency, network counters, and per-shard measurements. State what unit drives capacity: requests, bytes, records, connections, or a weighted cost model.
Failure changes the hardware math
During a zone loss, surviving nodes receive more traffic while caches are cold and replicas rebuild. Recovery traffic competes with foreground work for CPU, disk, and network. Capacity should be tested in the degraded state, not inferred from steady-state averages.
Efficiency is an architecture feature
At senior scope, a 20% reduction in CPU or egress can be worth more than a complex new subsystem. Data layout, compression choice, batching, cache hit ratio, and avoiding fan-out all affect fleet cost. Explain the engineering and operational cost of optimization, then define the measurement that justifies it.
Containers do not erase the machine
CPU quotas, cgroup throttling, memory limits, shared last-level caches, virtual-machine steal time, and noisy neighbors can distort performance. Request and limit settings, pod placement, and topology-aware scheduling are part of the design when latency is sensitive.
Correctness boundaries include durability
“The write returned 200” is not a durability statement. Clarify whether data reached a process buffer, kernel page cache, device stable media, a quorum of replicas, or a remote region. Tie the boundary to RPO and the failure being tolerated.
Interview question ladder
Why is memory access often more important than arithmetic complexity in backend performance?
Show strong-answer signals
Mention the memory hierarchy, cache lines, locality, random access, stalls, and why two O(n) algorithms can differ greatly in wall-clock time.
What is the difference between concurrency and parallelism?
Show strong-answer signals
Concurrency is multiple in-flight tasks; parallelism is simultaneous execution. Explain I/O waiting, core count, event loops, and why excessive concurrency creates queueing and context switches.
What does it mean for a server to be CPU-bound, memory-bound, IOPS-bound, or bandwidth-bound?
Show strong-answer signals
Define the saturated resource and show that mitigation differs: optimize compute, locality/capacity, operation pattern, or bytes transferred.
A service has low average CPU but high p99 latency. What would you investigate?
Show strong-answer signals
Look for hot cores/shards, lock waits, run queues, GC, I/O queues, downstream waits, burstiness, throttling, and retries. Avoid equating node-average CPU with spare capacity.
When would an event-driven server be worse than a thread-per-request model?
Show strong-answer signals
Blocking code on the event loop, CPU-heavy handlers, difficult library integration, fairness problems, and operational complexity. Propose bounded worker pools or multiple loops.
How can batching improve and hurt a system?
Show strong-answer signals
Amortizes syscalls, network headers, compression, and storage operations; adds wait time, memory, head-of-line blocking, and fairness issues. Bound by size and time.
Estimate the compute fleet for 100k RPS when each request uses 0.8 ms CPU, with 60% target utilization and tolerance for losing one of four equal zones.
Show strong-answer signals
Compute 80 core-s/s; 134 cores at 60%; divide by 0.75 for zone loss ≈ 178 cores, then discuss imbalance, background work, burst/growth margin, and benchmark validity.
Your database becomes slower after moving to a machine with more sockets and memory. Explain plausible causes and a diagnostic plan.
Show strong-answer signals
NUMA remote memory, thread migration, cross-socket locks, cache-coherence traffic, IRQ placement, different power settings, and larger working set. Compare per-node counters and pin an experiment.
Design overload protection for a CPU-heavy API.
Show strong-answer signals
Use deadlines, bounded concurrency, cost-aware admission, per-tenant quotas, queue limits, load shedding, retry budgets, autoscaling, and degraded responses. Explain where rejection occurs and how clients back off.
How would you prove that zero-copy networking is worth adopting?
Show strong-answer signals
Baseline profiles; quantify copy CPU and memory bandwidth; prototype under realistic payload/concurrency; include tail latency, pinned memory, buffer ownership, safety, portability, and operational complexity.
A multi-tenant service has excellent average efficiency but one tenant can double everyone’s p99. Propose resource isolation across the stack.
Show strong-answer signals
Tenant-aware admission and queues, CPU/memory/network quotas, shard isolation, workload classes, fair scheduling, cache partitioning, per-tenant observability, and a policy for unused capacity borrowing.
Explain how recovery traffic can turn a single-node failure into a cascading fleet incident.
Show strong-answer signals
Failover raises foreground load; cold caches and replica rebuild add disk/network/CPU; latency triggers retries; queues grow. Use repair throttles, spare capacity, retry budgets, gradual warmup, priority separation, and chaos tests.
Design drills
An image-resize API handles 30k requests/s. Images average 400 KiB input and 120 KiB output. p99 rises sharply above 55% fleet CPU. Sketch a measurement plan, estimate CPU and bandwidth capacity, and choose between synchronous resizing, precomputation, and a job queue.
What the interviewer is testing
Separating CPU, memory bandwidth, storage, and network costs; workload segmentation; tail-latency protection; evidence-based optimization.
Design the process model for a gateway maintaining one million mostly idle client connections per region. Estimate connection-state memory, file descriptors, heartbeat cost, and how many event-loop processes or machines are needed. Explain what happens during reconnect storms.
What the interviewer is testing
Concurrency versus parallelism, kernel limits, bounded work, memory accounting, and recovery behavior.
A product manager says every accepted payment event must survive a host loss and a zone loss. Draw the acknowledgment path from application memory to replicated storage and identify where latency is paid.
What the interviewer is testing
Precise durability guarantees, device and replica boundaries, synchronous versus asynchronous replication, and RPO/RTO language.
Common weak answers and how to improve them
“Add more threads to use the CPU.”
Show the stronger answer
First identify whether work is runnable or waiting, whether cores are saturated, and whether shared state or memory bandwidth will scale.
“The server has 64 GB, so 50 GB of cache is safe.”
Show the stronger answer
Account for process overhead, page cache, connection buffers, fragmentation, replicas, memory limits, and failure-time growth.
“SSD is fast.”
Show the stronger answer
Specify sequential bandwidth, random IOPS, queue depth, durability latency, and foreground/background contention.
“Average latency is 20 ms, so we need RPS × 20 ms workers.”
Show the stronger answer
Little’s Law estimates in-flight work, not necessarily thread count; use nonblocking concurrency and inspect the latency distribution.
“Containers make hardware differences irrelevant.”
Show the stronger answer
Discuss quotas, topology, noisy neighbors, kernel behavior, and why placement still matters.