Replication and Sharding
Replication creates multiple copies for availability, durability, locality, or read scale. Sharding divides a dataset for capacity and throughput. Both introduce coordination: replicas can disagree, leaders can fail, shard maps can become stale, and moving data competes with foreground work. Senior interview answers describe exact read/write paths, acknowledged durability, failover and fencing, partition-key behavior, online resharding, and the operational budgets for catch-up and repair.
How to use this chapter
- Read the mental model and mechanics without taking notes.
- Close the page and explain the topic aloud in five minutes.
- Work the quantitative example on paper.
- Answer the question ladder without revealing the answer signals.
- Complete one design drill and score yourself with the senior checklist.
Learning objectives
- Compare leader–follower, multi-leader, and leaderless replication with precise failure semantics.
- Explain synchronous/asynchronous replication, log positions, quorum reasoning, and read guarantees.
- Design range, hash, and directory sharding from access patterns and skew.
- Handle failover, split brain, stale routing, resharding, backfill, and cross-shard operations.
- Quantify replication lag, recovery bandwidth, shard fan-out, and failure headroom.
- Choose regional topology from latency, sovereignty, consistency, RPO, and RTO rather than fashion.
Mental model
Separate two questions. Placement: which nodes should hold each logical item? Protocol: how do those copies agree on accepted writes and expose reads? Sharding answers placement across independent partitions; replication creates copies within each partition; a consensus, leader, quorum, or conflict-resolution protocol determines behavior.
Core mechanics
Why replicate
Replication can improve fault tolerance, data durability, read capacity, and geographic latency. These are related but not identical. Three copies in one rack do not tolerate a rack loss. An asynchronous remote replica may improve disaster recovery while still losing the latest acknowledged writes. A read replica may add capacity but not safely serve monotonic or read-after-write flows.
Define failure domains—process, host, rack, zone, region, operator account—and place replicas accordingly. More replicas increase storage and write/network cost, repair time, and protocol complexity. Durability claims depend on independent failure modes, checksums/scrubbing, backups, and recovery procedures, not just replica count.
Leader–follower replication
One leader orders writes and streams a log to followers. Reads from the leader are simplest for freshness; follower reads may be stale. Synchronous acknowledgment to one or more followers improves acknowledged-write survival but adds the slowest required replica to latency and can reduce availability. Asynchronous replication favors latency/availability but exposes a nonzero recovery point on leader loss.
Failover requires failure detection, candidate selection, promotion, routing change, and fencing of the old leader. A lease, term/epoch, or consensus-backed role prevents an isolated old leader from continuing authoritative writes. Promotion should consider durable/applied log position, not only health. Clients need idempotent retry because a timeout can hide a committed write.
Multi-leader replication
Multiple sites accept writes and exchange them. This supports local writes and disconnected operation but creates concurrent updates and uniqueness conflicts. Conflict avoidance—routing each record to a home leader—can be simpler than generic merge. Where conflicts are possible, define field/domain merges, operation logs, causal metadata, or user resolution.
Multi-leader is not a free multi-region transaction system. Cross-leader ordering, sequences, foreign keys, and globally unique names need additional coordination. Clock-based last-write-wins can discard intent and is sensitive to clock behavior. Use it only when loss semantics are acceptable and auditable.
Leaderless and quorum-style replication
A coordinator sends writes to multiple replicas and reads from one or more. N, W, and R describe replica count and response thresholds. Overlap such as R + W > N can help a read see a completed write under assumptions, but sloppy quorums, concurrent versions, network delays, failed writes, and stale replica sets require version resolution and repair.
Leaderless systems often use hinted handoff, read repair, and anti-entropy. They can remain writable during some failures, yet clients may observe conflicting versions or stale reads. Explain version metadata, reconciliation, tombstones, and what “success” means when only a subset acknowledges.
Read consistency and session guarantees
Useful guarantees include linearizable reads, sequential consistency, causal consistency, read-your-writes, monotonic reads, and bounded staleness. A follower can satisfy read-your-writes by waiting until it applies the client’s commit position, routing the session to an adequate replica, or falling back to the leader. Time-based “lag under one second” is weaker than position-based completeness and can be misleading during stalled apply.
Some workloads accept stale product descriptions but not stale authorization or payment state. Route by operation criticality rather than declaring one global consistency mode. State behavior after failover and across devices, where session tokens may not travel.
Sharding strategies
Range sharding keeps adjacent keys together and supports ordered scans, but rising ranges and popular regions can hotspot. Hash sharding spreads point access but scatters range scans. Directory sharding maps tenants/keys through explicit metadata, enabling custom placement and tenant moves at the cost of a highly available routing service. Consistent hashing is a hash-placement technique optimized for membership changes, not a complete shard protocol.
Composite keys often combine locality and spread: tenant plus bucket, geography plus hash, or time range plus suffix. Define shard size, QPS, storage, working set, and growth thresholds. A key should make common operations single-shard where possible and make exceptional large tenants movable.
Routing and metadata
Clients can compute placement, call a stateless router, or query a directory/cache. Client-side routing removes a hop but complicates upgrades and stale-map handling. Routers centralize policy and connection pools but add a tier. Version shard maps with monotonic epochs; servers must validate ownership and redirect or reject stale writes.
Cache metadata with bounded staleness, single-flight refresh, and loop limits. During control-plane outage, a stable old map may be safer than accepting uncoordinated changes. Separate serving availability from reconfiguration availability.
Online resharding
A safe move usually has: choose source/target range; create target; copy a consistent snapshot; capture concurrent changes through a log/dual route; catch up; verify counts/checksums; atomically advance ownership epoch; drain old traffic; retain rollback/read fallback; then delete old data later. The exact mechanism varies, but every phase needs idempotency and observability.
Avoid naïve application dual writes: partial success creates divergence. Prefer source-log replay, storage-level migration, or a transactional outbox. Throttle on foreground latency, source disk, replica lag, target apply, and network. Moving many shards simultaneously can turn a routine expansion into a cluster-wide incident.
Cross-shard operations
Scatter-gather reads pay the slowest shard and increase partial-failure probability. Global secondary indexes are themselves partitioned replicated systems with freshness and transaction questions. Cross-shard transactions use a coordinator/consensus protocol, can block or abort under failures, and expand the blast radius.
Alternatives include colocating aggregates, sagas, reservations, escrow, append-only ledgers, asynchronous materialized views, and dedicated global services for rare invariants. Do not weaken a business invariant accidentally: state the substitute semantics and reconciliation process.
Multi-region topology
Common topologies include single writable region with remote replicas, per-record home regions, active-active conflict-tolerant writes, and globally coordinated replication. Choose from write latency, failover objectives, partition behavior, data residency, egress, and operational maturity. “Active-active” can describe traffic routing while data remains single-leader; clarify each layer.
Region evacuation requires spare capacity, routing, dependency readiness, secrets/keys, data position, and tested runbooks. A remote copy that takes days to restore may satisfy durability but not availability. Practice failover and failback; the latter often exposes more data reconciliation risk.
Decision table
| Decision | Prefer the first option when… | Prefer the second option when… | Senior caveat |
|---|---|---|---|
| Synchronous vs asynchronous replica acknowledgment | Near-zero acknowledged-write loss is required and latency/availability budget permits coordination. | Low write latency and continued operation during replica isolation dominate. | State exactly how many failure domains have the write at acknowledgment and the resulting RPO. |
| Leader vs leaderless | Ordered writes, transactions, and simple conflict avoidance dominate. | Some partition-time write availability and tunable reads matter, with acceptable reconciliation. | Both need repair, membership, and client retry; neither label alone gives linearizability. |
| Hash vs range sharding | Point access and even distribution dominate. | Ordered scans, locality, and range lifecycle dominate. | Hybrid/bucketed designs often manage hotspots while retaining limited range access. |
| Computed vs directory routing | Keys/nodes are uniform and simple deterministic placement is enough. | Tenants need custom isolation, moves, weights, or residency. | Directory metadata becomes critical infrastructure and needs versioning/cache behavior. |
| Follower reads vs leader reads | Staleness is explicit and read scale/locality matters. | Freshness, monotonicity, or correctness-sensitive authorization matters. | Position-aware routing can provide session guarantees without sending every read to the leader. |
| Cross-shard transaction vs workflow | Invariant truly spans shards and coordination rate/availability is acceptable. | Steps can be reserved, retried, compensated, and reconciled asynchronously. | A saga is not equivalent to isolation; define intermediate visibility and irreversible effects. |
Quantitative reasoning
Failure modes and production signals
| Failure mode | What users see | Likely cause | Mitigation / design response | Useful signals |
|---|---|---|---|---|
| Split brain | Conflicting accepted writes after network isolation | Two leaders or stale owner lacks fencing | Terms/epochs, quorum lease, server-side ownership validation, reconciliation | Concurrent leaders, epoch rejects, divergent logs |
| Promotion of stale replica | Acknowledged data missing after failover | Candidate selected by liveness not log durability | Require eligible log position/quorum, document RPO, restore missing writes | Commit/applied positions, lost-write audit |
| Replication lag | Stale reads and long recovery | Apply below generation, long query/transaction, network or disk saturation | Route critical reads, remove blockers, tune/apply capacity, reseed | Lag bytes/time, apply rate, oldest transaction |
| Hot shard | One shard throttles while cluster averages are low | Skewed key or large tenant | Split/bucket/move tenant, cache, rate limit, adaptive isolation | Per-shard QPS/CPU/disk, top tenants, max/mean |
| Stale shard map | Redirects, wrong-owner errors, duplicate writes | Client/router missed ownership epoch | Version validation, bounded redirect/refresh, fencing, compatible rollout | Map age, redirects, epoch mismatch |
| Resharding overload | Cluster latency and lag rise during expansion | Copy/catch-up consumes shared resources | Throttle, stage moves, reserve capacity, pause on SLOs | Move rate, disk/network queue, foreground p99 |
| Cross-shard coordinator failure | Transactions remain pending or locks held | Coordinator state unavailable mid-protocol | Durable decision log, timeout/recovery protocol, idempotent participants | In-doubt transactions, lock age, coordinator recovery |
| Replica divergence/corruption | Different reads or checksum failures | Missed writes, bit rot, faulty repair | Checksums, anti-entropy, authoritative version, rebuild and audit | Merkle/checksum mismatch, repair outcomes |
Senior-level lenses
Durability is an acknowledgment contract
“Replicated three times” is insufficient. Say whether success means leader memory, leader disk, one remote-zone disk, a quorum, or asynchronous enqueue. Then describe the exact loss scenario and recovery point. Include correlated software/operator errors and backups.
Failure detection is never perfect
A timeout cannot distinguish a dead node from a slow or isolated node. Promotion therefore needs quorum/leases/terms and fencing, not confidence. Tune detection against false failovers and recovery speed, and design clients for ambiguous outcomes.
Resharding is a distributed transaction with a long data plane
Ownership change is small metadata, but safe activation depends on terabytes of copied and caught-up state. Model phases, authority, epochs, validation, rollback, and delayed deletion. Treat migrations as normal continuous operations, not one-time heroics.
Use per-shard distributions, not cluster averages
Report max and percentiles for QPS, bytes, storage, compaction, and lag. A cluster at 35% average may have a shard at 100%. Capacity and SLO should consider the hottest shard and largest failure domain.
Geography couples data and dependency architecture
A database can fail over while identity, queues, object storage, KMS, configuration, or third-party callbacks remain regional. Draw the entire write path and test that the alternate region has data, capacity, credentials, and routable dependencies.
A global invariant needs a global serialization point or a new invariant
Globally unique usernames, account balance, and single active lease cannot be guaranteed by independent regional writes plus hope. Coordinate, assign ownership, preallocate/escrow, or allow conflict and resolution. Make the trade explicit.
Interview question ladder
What is the difference between replication and sharding?
Show strong-answer signals
Replication copies data for availability/durability/read locality; sharding partitions data for capacity/throughput. Real systems combine both.
Why can a read replica return stale data?
Show strong-answer signals
The write log is shipped/applied asynchronously or the read snapshot trails the leader.
What is split brain?
Show strong-answer signals
Multiple nodes believe they are authoritative and accept conflicting work; prevent with quorum/terms/leases and fencing.
How do you provide read-your-writes from replicas?
Show strong-answer signals
Carry commit/log position or session token; route/wait until a replica has applied it, otherwise use leader.
Choose a shard key for orders.
Show strong-answer signals
Tenant/customer/order locality, write distribution, time queries, largest tenant, cross-tenant admin analytics, retention, and online tenant moves.
Describe a safe shard split.
Show strong-answer signals
Snapshot, change capture, catch-up, verify, epoch cutover, fallback/drain, delayed delete, throttling and idempotency.
Design failover for a primary database across zones.
Show strong-answer signals
Replication acknowledgment, failure detection, election/eligibility, fencing, promotion, routing/pool refresh, ambiguous retries, validation, failback, RPO/RTO metrics.
Design a multi-region account service.
Show strong-answer signals
Identify global invariants, choose home-region/global consensus/escrow, read routing/session guarantees, partition behavior, residency, region loss and reconciliation.
Your shard count doubled and p99 worsened. Why?
Show strong-answer signals
More fan-out, smaller cache locality, connection overhead, metadata, background movement, skew, cross-shard coordination; inspect per-query partitions and slowest shard.
How do you move a large tenant with continuous writes?
Show strong-answer signals
Directory epoch, snapshot and ordered delta, dual-read/fallback, target readiness, fenced cutover, reconciliation, throttles, rollback, tenant communication.
Design a fleet-wide resharding control plane.
Show strong-answer signals
Inventory/telemetry, planner with constraints, versioned plans, admission budgets, data movers, checksums, fencing, pause/rollback, audit, SLO guardrails and human controls.
Set replication tiers for a multi-product company.
Show strong-answer signals
Classify RPO/RTO/latency/residency, standard topologies, cost, backup/restore, chaos tests, client contracts, observability and exception governance.
Design drills
Design profile storage with local reads worldwide, one home region per user, privacy deletion, and region evacuation. Include routing and move protocol.
What the interviewer is testing
Replication topology, data residency, and directory-based migration.
A tenant holds 20% of a shard and 45% of its QPS. Split or move it without downtime and quantify spare capacity.
What the interviewer is testing
Skew detection, change capture, fencing, throttling, and rollback.
Create a timeline for leader-zone isolation, including detection, client symptoms, promotion, stale writes, duplicate retries, verification, and failback.
What the interviewer is testing
Concrete failure semantics rather than a generic ‘automatic failover’ claim.
Common weak answers and how to improve them
“Three replicas means no data loss.”
Show the stronger answer
State acknowledgment placement, correlated failures, repair, and backups; async replicas can lose recent writes.
“Use quorum reads and writes for strong consistency.”
Show the stronger answer
Quorum overlap is only one condition; include versioning, replica sets, concurrent writes, failures, and protocol.
“Hash the user ID to shard evenly.”
Show the stronger answer
Test tenant popularity, data size, access locality, scans, and exceptional-tenant migration.
“Fail over to the replica.”
Show the stronger answer
Explain eligibility, fencing, routing, client retry, RPO/RTO, and failback.
“Dual-write during migration.”
Show the stronger answer
Use an ordered change source or transaction and define partial-failure repair and cutover authority.