Proxies and Load Balancing
Proxies mediate traffic; load balancers choose destinations. They can operate at transport or application layers, centrally or in clients/sidecars, globally or within a region. Senior candidates design health, algorithms, connection reuse, draining, affinity, retries, outlier detection, overload protection, and observability—and recognize that the load balancer itself can create correlated failure.
How to use this chapter
- Read the mental model and mechanics without taking notes.
- Close the page and explain the topic aloud in five minutes.
- Work the quantitative example on paper.
- Answer the question ladder without revealing the answer signals.
- Complete one design drill and score yourself with the senior checklist.
Learning objectives
- Distinguish forward, reverse, transparent, tunneling, sidecar, and gateway proxies.
- Compare L4 and L7 load balancing and global versus local traffic steering.
- Choose round-robin, weighted, least-request, random, hash, and locality-aware policies.
- Design active/passive health, outlier detection, draining, connection pools, and session affinity.
- Coordinate retries, timeouts, circuit breaking, load shedding, and failover.
- Operate proxies as security and reliability control planes at senior level.
Mental model
A proxy terminates or forwards one connection and creates another path to an upstream. A load balancer decides which upstream should receive work. At L4 the decision is based mainly on addresses, ports, and connection state; at L7 it can use HTTP/RPC metadata and make per-request decisions. Client-side load balancing moves selection into the caller; proxy-based balancing centralizes policy.
Core mechanics
Proxy roles
A forward proxy acts for clients, controlling egress, privacy, or caching. A reverse proxy acts for servers, exposing one endpoint while routing to origins. A tunnel such as CONNECT forwards bytes without understanding the application after setup. A transparent proxy intercepts flows through routing changes. Service-mesh sidecars or node proxies apply service-to-service policy.
Proxies can terminate TLS, authenticate, normalize requests, enforce quotas, cache, transform, and observe. Each function adds trust and failure impact. Keep a clear chain of client identity and use authenticated proxy-to-upstream metadata rather than trusting arbitrary forwarded headers.
L4 versus L7 balancing
L4 load balancers distribute connections based on transport tuples and can handle arbitrary TCP/UDP protocols with low per-request insight. L7 load balancers parse application protocols, route by host/path/header/method, retry selected requests, and emit richer metrics. They consume more CPU and must be protocol-correct.
L4 connection hashing means one long-lived connection can carry disproportionate load. L7 HTTP/2/gRPC balancing can choose upstream per stream/request if it terminates the connection. For encrypted pass-through traffic, L7 features are limited unless metadata such as SNI is inspected.
Placement: global, regional, and client-side
Global balancing selects a region or edge using DNS, anycast, or a global proxy. Regional balancing distributes among zones and instances. Client-side balancing gets an endpoint list and chooses locally, avoiding a central data hop and enabling RPC-aware policy. It also distributes configuration and can create inconsistent behavior across client versions.
A common architecture uses global steering → regional L4/L7 ingress → service proxy/client-side balancing. State placement and data consistency constrain where traffic may go; routing to the nearest healthy compute is wrong if the authoritative data or tenant residency is elsewhere.
Algorithms
Round robin is simple and works when requests and backends are similar. Weighted round robin handles capacity differences. Least-request/least-connections uses current load proxies but can be misled by long-lived idle connections or stale distributed state. Power-of-two-choices samples two backends and chooses the less loaded, approaching good balance with low overhead. Random can be robust at scale.
Hash policies provide affinity by client, session, or key. They improve cache locality but can create hotspots and uneven failure behavior. Use consistent/rendezvous hashing or bounded-load variants when membership changes, and include weights/virtual nodes for heterogeneous capacity.
Health and outlier detection
Active health checks probe a known path; passive checks observe real failures. A shallow TCP check can pass while the application is wedged; a deep check can overload dependencies and create correlated false failures. Health endpoints should verify ability to serve without requiring every optional dependency.
Outlier detection ejects endpoints with unusual errors/latency. Use minimum volume, statistical thresholds, max ejection percentage, and recovery probing to avoid ejecting the whole fleet during a shared dependency outage. Distinguish endpoint failure from request-specific errors.
Connections, pools, and draining
Proxies maintain downstream and upstream connections. Connection pooling amortizes handshakes but creates limits, queues, and load pinning. HTTP/2/gRPC multiplexing may send many streams over one upstream connection; multiple connections may be needed to avoid stream limits and improve distribution.
During deployment, mark an endpoint unhealthy for new work, allow in-flight work to finish, send protocol drain signals where available, and enforce a maximum drain time. Killing endpoints before the balancer converges creates resets and retries; draining too long blocks releases and keeps old code indefinitely.
Retries, hedging, and circuit breaking
Proxies can retry connect failures, resets, or selected 5xx responses. Only retry operations safe under the API’s idempotency contract, within an end-to-end deadline and retry budget. Avoid retrying overload into another overloaded endpoint. Include attempt count and original request ID.
Hedging sends a backup attempt after a delay to reduce tail latency, but increases load and can duplicate side effects. Use for idempotent reads with strict budgets and cancel the loser. Circuit breakers cap pending requests, connections, or retries to contain dependency failure.
Overload and fairness
A load balancer cannot create capacity. When all endpoints are saturated, spreading traffic can turn a localized issue into a fleet-wide collapse. Reject early using concurrency limits, queues, priority classes, and per-tenant quotas. Preserve capacity for health, control, and critical operations.
Load metrics should reflect work: active requests weighted by expected cost, CPU, queue delay, or endpoint-reported capacity. Feedback loops need smoothing and bounds; aggressively chasing instantaneous load can oscillate traffic.
Security and observability
Proxies are choke points for TLS, authentication, WAF, policy, and logging, making them valuable and dangerous. Harden admin APIs, sign configuration, isolate tenants, rotate certificates, and have a rollback path. Decide where original client IP and identity are trusted and scrub inbound spoofed headers.
Track downstream/upstream latency separately, queue wait, retries, ejections, health state, connection counts/age, response flags, routing decision, config version, and per-zone capacity. Distributed traces should show each attempt, not collapse retries into one span.
Decision table
| Decision | Prefer the first option when… | Prefer the second option when… | Senior caveat |
|---|---|---|---|
| L4 vs L7 | Protocol agnosticism, low overhead, and pass-through encryption dominate. | Request-aware routing, auth, retries, caching, and rich telemetry dominate. | Many systems combine L4 outer distribution with L7 proxies behind it. |
| Proxy vs client-side balancing | Central policy, simple clients, and heterogeneous languages matter. | One less hop and RPC-aware local decisions matter. | Service mesh can centralize implementation while distributing data plane; control-plane availability still matters. |
| Round robin vs least request | Requests/backends are homogeneous and simplicity is preferred. | Request durations vary and live load signal is reliable. | Use weights, power-of-two choices, and bounded queues; stale state can create herding. |
| Affinity vs stateless routing | Session/cache locality materially improves cost or correctness. | Fast failover and even balance dominate. | Keep durable state external and make affinity soft; hot clients/keys need special handling. |
| Active vs passive health | Failures must be detected before user traffic and probes are representative. | Real request outcomes better capture partial application failures. | Use both and prevent shared-dependency failures from mass ejection. |
| Retry vs fail fast | A transient endpoint-local failure and safe operation justify another attempt. | Overload, non-idempotency, or little deadline remains. | Retry budgets are fleet-wide capacity policy, not a per-call convenience. |
Quantitative reasoning
Failure modes and production signals
| Failure mode | What users see | Likely cause | Mitigation / design response | Useful signals |
|---|---|---|---|---|
| Mass ejection | Sudden 503 despite healthy capacity | Shared dependency makes all endpoints fail health/outlier threshold | Limit ejection, distinguish local/shared errors, panic mode with shedding | Ejected %, error correlation, health reason |
| Connection imbalance | One backend hot; connection counts seem balanced | Unequal long-lived stream traffic or multiplexing | L7/request balancing, multiple upstream conns, load-aware policy | Requests/bytes per connection/backend, active streams |
| Retry storm | Backend load rises as success falls | Multiple retry layers/no budget | Single owner, deadline, jitter, retry budget, no retry on overload | Attempts/original, retry reasons, remaining deadline |
| Drain failure | Resets during deploy or old instances never exit | Health/connection lifetime mismatch | Pre-stop drain, GOAWAY/close, max age, bounded timeout | Connection age, draining endpoints, reset reason |
| Proxy config outage | Broad routing/auth failure | Bad global rule/certificate/config | Canary, versioned config, validation, fast rollback and last-known-good | Config version, reject reason, control-plane errors |
| Overload spread | All backends become slow instead of one failing | Balancer continues sending to saturated pool | Admission, priority, endpoint concurrency, fail fast/degrade | Queue wait, saturation, shed rate, max/mean load |
Senior-level lenses
Health is ability to serve the requested class
A backend can be healthy for reads but not writes, healthy for cached paths but not expensive searches, or healthy only within a zone. Binary global health loses information. Use endpoint metadata, priorities, degraded states, and route classes where complexity is justified.
Load balancing and overload control are one design
The algorithm determines where queues form. Pair balancing with endpoint and proxy concurrency limits, queue bounds, and rejection. Least-request without admission simply chooses the least-bad overloaded endpoint.
Retries change routing feedback
Retried traffic is not independent demand and can bias outlier metrics. Track original versus attempts and avoid treating a backend as healthy merely because another endpoint rescued its failures. Include retry success and cost in SLO/error-budget analysis.
Locality is a constrained optimization
Prefer same-zone/region to reduce latency and egress, but keep enough cross-zone capability for failure. Locality weights should account for data ownership, capacity, and fault-domain risk. Strict locality can strand capacity; no locality can increase correlated dependency and cost.
Control plane should fail static
Existing data-plane proxies should continue with last-known-good routes during control-plane outage. Version config, use leases carefully, validate before activation, and avoid synchronous dependency on control plane per request. Staleness still needs an expiry/emergency policy.
Test connection-age distributions
Long-lived HTTP/2, gRPC, database, and WebSocket connections can keep traffic on old endpoints or certificates. Use max connection age with jitter, graceful drain, and metrics by connection cohort so changes actually converge.
Interview question ladder
What is the difference between a forward and reverse proxy?
Show strong-answer signals
Forward proxy represents clients toward servers; reverse proxy represents servers toward clients. Discuss egress vs ingress use cases.
L4 versus L7 load balancing?
Show strong-answer signals
L4 balances flows using transport metadata; L7 parses application protocol and can route/retry/auth/cache per request.
Why can round robin be poor even with equal servers?
Show strong-answer signals
Requests differ in cost/duration, long connections pin, slow endpoints retain work, and weights/health may differ.
What is connection draining?
Show strong-answer signals
Remove endpoint from new work, allow in-flight/long-lived sessions to finish or receive close signal, enforce max drain, then terminate.
How does least-connections differ from least-request?
Show strong-answer signals
Connections may be idle or multiplex many requests; active request/work is often a better load signal, though distributed state can be stale.
When is session affinity appropriate?
Show strong-answer signals
Cache/local session optimization or protocol state; make soft, externalize durable state, handle hot clients and failover.
Design load balancing for gRPC services with long streams and unary calls.
Show strong-answer signals
L7 termination/client-side policy, separate pools/classes, multiple HTTP/2 conns, active-stream/cost metrics, max connection age/drain, health, retries only unary/idempotent, slow-stream isolation.
How do you avoid a health-check-induced outage?
Show strong-answer signals
Shallow representative check, dependency classification, thresholds, multi-vantage, max ejection/panic, stagger/jitter, capacity-aware failover, last-known-good.
A zone fails. How should traffic shift?
Show strong-answer signals
Reserve N+1 capacity, local-first with weighted cross-zone, gradual capacity-aware shift, warm caches, protect databases, retry budget, load shed/degrade, monitor per-zone saturation.
Where should TLS terminate in a zero-trust service architecture?
Show strong-answer signals
External edge plus authenticated encrypted hop to workload; mTLS via proxy/app, identity/authorization, cert rotation, SNI, observability, and avoiding trust solely from network location.
Design a global traffic-management system for hundreds of services and regions.
Show strong-answer signals
Hierarchical global/regional/client balancing, capacity and health signals, data-residency constraints, control/data-plane separation, last-known-good, gradual shifts, overload, simulation/game days, audit.
A proxy retry policy reduced errors but doubled cost and worsened p99. Redesign governance.
Show strong-answer signals
Attempt telemetry, single owner, idempotency registry, per-route retry classes, deadline/budget, overload signals, max amplification SLO, canary and lint, charge attempts to tenant/service.
Design drills
Design service discovery, load balancing, mTLS, retries, and outlier detection for 1,000 microservices. State which policies belong centrally and how proxies behave when control plane is unavailable.
What the interviewer is testing
Data/control-plane design, safe defaults, and avoiding mesh-wide correlated failure.
Design public ingress for checkout across three regions with a home-region database and optional read-only browse fallback. Include DDoS/WAF, global routing, draining, retries, and residency.
What the interviewer is testing
Routing constrained by state/correctness, not nearest-server simplification.
An L4 balancer shows equal connections across 50 WebSocket gateways, but one gateway has 4× CPU. Diagnose and redesign.
What the interviewer is testing
Connection traffic inequality, rooms/hot users, L7/consistent routing, per-connection metrics, and fan-out isolation.
Common weak answers and how to improve them
“Use least connections.”
Show the stronger answer
Choose a load signal appropriate to multiplexing and request cost; address stale state, weights, and overload.
“Health check the database from every backend.”
Show the stronger answer
Deep checks can amplify dependency failure; distinguish readiness and optional dependencies with safe thresholds.
“The load balancer retries failed requests.”
Show the stronger answer
Specify failure classes, idempotency, deadlines, one retry owner, budgets, and overload behavior.
“Sticky sessions solve state.”
Show the stronger answer
They do not survive endpoint loss or rebalance; externalize durable state and plan reconnect.
“Active-active doubles capacity.”
Show the stronger answer
Failover only works if each side has reserved degraded capacity and state/dependencies can serve shifted traffic.