Message Queues, Logs, and Event-Driven Systems
Messaging decouples producers from consumers in time and rate, but it does not make distributed work automatic or exactly once. Queues, publish/subscribe brokers, and partitioned logs differ in retention, replay, ordering, and consumption. Senior candidates define event ownership and schema, delivery and acknowledgment boundaries, idempotency, retry/DLQ policy, backpressure, partitioning, consumer recovery, observability, and the transaction between database state and publication.
How to use this chapter
- Read the mental model and mechanics without taking notes.
- Close the page and explain the topic aloud in five minutes.
- Work the quantitative example on paper.
- Answer the question ladder without revealing the answer signals.
- Complete one design drill and score yourself with the senior checklist.
Learning objectives
- Compare work queues, pub/sub, and durable logs by consumption and retention semantics.
- Explain at-most-once, at-least-once, effectively-once, and scoped exactly-once claims.
- Design ordering keys, partitions, consumer groups, acknowledgments, visibility/leases, and replay.
- Use idempotent consumers, transactional outbox/inbox, deduplication, and state-machine transitions.
- Handle retries, poison messages, DLQs, backpressure, overload, rebalances, and recovery.
- Govern event schemas, privacy, multi-tenancy, observability, and disaster recovery.
Mental model
A message system is a durable handoff plus a scheduling policy. The producer writes a record; the broker stores and replicates it; one or more consumers claim or read it; acknowledgment advances a queue item or consumer offset. Every boundary can time out after success, so duplicates are normal unless a larger transaction encloses both state and progress.
Core mechanics
Queue, pub/sub, and log models
A work queue distributes each task to one worker (with possible redelivery) and removes/ages it after acknowledgment. Publish/subscribe delivers an event to multiple independent subscriptions. A partitioned log retains an ordered sequence for a period; consumers track offsets and can replay independently. Products often blend these models, so describe semantics rather than relying on the label.
Use queues for commands/jobs where one completion matters. Use pub/sub for multiple reactions to a fact. Use logs for event streams, CDC, rebuildable projections, and high-throughput ordered partitions. A queue with destructive consumption is a poor audit/replay source; a log can be awkward for per-message scheduling, long delays, and arbitrary priorities.
Commands, events, and ownership
A command asks one owner to perform an action: GenerateInvoice. An event states a completed fact: InvoiceGenerated. Events should be past-tense, immutable, and meaningful to consumers without requiring private producer internals. Avoid generic “entity updated” envelopes that force every consumer to fetch and infer changes, unless change-data-capture is explicitly the abstraction.
Define the authoritative producer, event ID, aggregate/entity key, schema/version, occurred time, publication time, causation/correlation IDs, tenant, and privacy classification. Do not place secrets or unnecessary personal data in widely retained streams. Consumers must not reinterpret an event as proof of a business fact the producer did not actually commit.
Delivery semantics
At-most-once may lose work but does not retry after delivery. At-least-once retries until acknowledgment and can duplicate. “Exactly once” is always scoped: a broker may atomically write output records and offsets within its ecosystem, but an email, card charge, HTTP call, or independent database still needs idempotency or reconciliation. Prefer “effectively once” for business effects achieved through unique operation IDs and atomic state transitions.
Acknowledging before the side effect risks loss; acknowledging after risks duplicate effects if the consumer dies between effect and ack. Store a message/event ID in the same transaction as the local state change, or make the destination operation idempotent. Deduplication windows must cover maximum replay delay, not just normal retry time.
Producer atomicity and the outbox
The dual-write problem occurs when an application commits database state and publishes a message separately. Either can succeed alone. A transactional outbox writes the domain change and an outbox row in one database transaction. A relay/CDC publisher sends rows to the broker and marks/progresses them; duplicates are expected. Consumers use event IDs and versions.
An inbox pattern records consumed IDs or applies a conditional aggregate version in the same transaction as local effects. Do not let an ever-growing inbox become unbounded; partition/expire only after the replay horizon and audit needs. For databases with native transactional message integration, state the exact scope and failure behavior.
Ordering and partitioning
Global ordering is expensive and usually unnecessary. Partition by the entity whose events require order—for example, account or order ID—so one partition has an ordered sequence. Different keys can process concurrently. Ordering is typically guaranteed only within a partition and only for accepted records; retries and multiple downstream systems may still reorder observed effects.
Too few partitions cap parallelism; too many add metadata, file, connection, replication, and rebalance overhead. A hot key remains on one partition. If operations commute, process concurrently and reconcile by version; otherwise isolate/split the domain or accept the per-key throughput limit. Include producer sequence/idempotence when retrying sends could reorder records.
Consumer groups and rebalancing
In a consumer group, each partition is assigned to one active consumer at a time (model-dependent), providing scale and per-partition order. Membership changes trigger reassignment. A consumer should stop processing revoked partitions, flush/commit safe progress, and fence stale workers. Long processing can look like failure and cause duplicate concurrent work.
Separate fetch from bounded processing, track in-flight records, and commit offsets only after effects are durable. Cooperative/sticky assignment can reduce movement, but correctness still needs idempotency. For long tasks, store job state externally or extend visibility/heartbeat carefully rather than blocking one partition indefinitely.
Acknowledgments, visibility, and leases
Queue systems may hide a claimed message for a visibility timeout. If work exceeds the timeout, another worker receives it; extend the lease with a heartbeat or choose a realistic bound. If a worker crashes, expiry enables retry. A receipt/lease token should fence an old worker from deleting a newer delivery where supported.
Acknowledge batches only when every included effect is safe, or track per-record completion. Large batches improve throughput but increase duplicate replay after a late failure. Bound prefetch so one worker does not hoard tasks and so shutdown/rebalance is manageable.
Retries, poison messages, and DLQs
Classify errors: transient dependency failures merit bounded exponential backoff with jitter; rate limits should honor retry hints; validation/permanent errors should not spin; unknown failures need a capped policy and inspection. A delayed retry queue prevents one bad task from blocking the head of a FIFO stream, but can change ordering.
A dead-letter queue is quarantine, not a solution. Include original payload/reference, event ID, failure classification, attempts, timestamps, consumer version, and safe redrive tooling. Alert on age/rate and assign ownership. Before redrive, fix idempotency and the cause; dumping millions of messages back at full speed can recreate the incident.
Backpressure and overload
Messaging absorbs temporary bursts by converting them into backlog, but a sustained arrival rate above service rate creates unbounded delay. Measure oldest-message age, not only depth, because message sizes and rates vary. Autoscale on lag/age plus resource saturation, while respecting downstream capacity. More consumers can amplify database/API overload.
Apply admission control at producers, tenant quotas, priority classes, bounded retention, and load shedding for expired/obsolete work. Propagate deadlines: sending a notification after its event has expired may be worse than dropping it with audit. Keep critical control streams isolated from bulk analytics.
Schema evolution and replay
Events are long-lived APIs. Use a schema registry or compatibility checks, additive changes, defaults, and tolerant readers. Do not change the meaning of an existing field silently. New consumers may replay years of records, so producer code history is irrelevant unless the event is self-describing and documented.
For breaking semantic change, create a new event type/version and run migration/dual publication carefully. Backfills should be distinguishable from live facts and carry stable IDs/versions so they do not overwrite newer state. Test consumers against mixed versions and unknown fields.
Operations and disaster recovery
Observe publish success/latency, replication health, partition throughput/skew, consumer lag and oldest age, retries, DLQ, processing latency, offset commits, duplicate/idempotency hits, and end-to-end trace correlation. A healthy broker with a six-hour consumer backlog is not a healthy business workflow.
Replicate broker metadata/data across failure domains, define retention and replay, back up schemas/configuration where needed, and rehearse region loss. Mirrored streams can duplicate/reorder during failover; consumer IDs and idempotent projections must survive. Define whether producers block, buffer locally, or drop when the broker is unavailable.
Decision table
| Decision | Prefer the first option when… | Prefer the second option when… | Senior caveat |
|---|---|---|---|
| Queue vs partitioned log | One worker completes each task and arbitrary retry/delay scheduling matters. | Many consumers need independent replay and ordered streams. | Use both when a durable event log feeds a job queue for expensive task execution. |
| At-most-once vs at-least-once | Loss is acceptable and duplicate side effects are more harmful or impossible to dedupe. | Work must not be silently lost and consumers can be idempotent. | Most business workflows choose at-least-once plus effectively-once effects. |
| One global partition vs keyed partitions | Volume is tiny and a total order is a true invariant. | Per-entity order is enough and throughput/availability matter. | Global order creates a single coordination and recovery bottleneck. |
| Synchronous API vs asynchronous command | Caller needs immediate authoritative result and work is bounded/reliable. | Work is slow, bursty, retryable, or can complete later. | Async APIs need accepted/job state, idempotency, cancellation, expiry, and completion delivery. |
| Retry in place vs delayed retry queue | Delay is short and preserving strict order matters. | Backoff is long or poison messages would block useful work. | Moving records can reorder effects; use entity version guards. |
| Payload vs reference | Events are small, immutable, and consumers need self-contained replay. | Data is large/sensitive or changes in a controlled authoritative store. | References can break replay and create temporal coupling; include immutable version and retention. |
Quantitative reasoning
Failure modes and production signals
| Failure mode | What users see | Likely cause | Mitigation / design response | Useful signals |
|---|---|---|---|---|
| Duplicate side effect | Double email, charge, or state transition | Crash after effect before ack; producer retry | Idempotency key, atomic inbox/state, destination dedupe/reconciliation | Duplicate IDs, idempotency hits, side-effect audit |
| Lost publication | Database state changes but downstream never reacts | Dual write commits DB but broker publish fails | Transactional outbox/CDC and relay monitoring | Oldest outbox row, publish attempts, DB/event reconciliation |
| Poison-message loop | Partition/queue makes no progress | Permanent parse/business error retried forever | Classify, cap attempts, quarantine DLQ, alert and repair | Same ID attempts, partition lag, DLQ rate |
| Consumer rebalance storm | Throughput collapses and duplicates rise | Unstable membership, long processing, bad timeouts | Static/cooperative membership, bounded work, heartbeats, idempotency | Rebalance count/time, assignment churn, poll gaps |
| Hot partition | One partition lags while others idle | Skewed ordering key/large tenant | Key redesign/bucketing, dedicated stream, commutative processing | Lag/bytes by partition, top keys |
| Retry storm | Broker and dependency overload after outage | Immediate unbounded redelivery | Exponential backoff+jitter, retry budget, circuit breaker, admission | Attempts/original, retry queue age, downstream 429/5xx |
| DLQ graveyard | Business work silently accumulates | No ownership/redrive or poor diagnostics | SLO/alerts, metadata, runbook, safe sampled/redrive tools | Oldest DLQ age, unresolved count, owner |
| Offset/data mismatch | Projection skips or repeats records | Offset committed separately from state | Atomic state+offset or idempotent versioned apply | Source vs projection version, gaps, replay differences |
| Broker outage | Producers time out or local buffers grow | Quorum/storage/network/control failure | Defined block/buffer/drop policy, failover, quotas, recovery | Publish errors, local spool, replica health, ISR/quorum |
Senior-level lenses
Exactly once is a boundary claim
Ask “exactly once where?” Broker transactions can atomically consume and produce within one platform, but an external database or email service is outside unless explicitly integrated. Design idempotent business operations and reconciliation, then describe any narrower broker guarantee accurately.
Backlog is stored latency
Queue depth is not success; it is deferred user work. Convert depth into oldest age and completion SLO by class/tenant. Decide when work becomes obsolete, and shed expired low-value messages before they consume recovery capacity.
The outbox is a log with lifecycle
The outbox needs ordering, partitioning, relay concurrency, retention, cleanup, monitoring, and disaster recovery. A polling query without an index can become the database bottleneck. CDC reduces polling but adds connector offsets and schema/DDL behavior.
Ordering scope should follow the invariant
Total order is rarely required. Order account events by account, order order-state transitions by order, and let unrelated entities process concurrently. If one entity is too hot, revisit whether operations commute, can be aggregated, or need a dedicated owner.
Consumers are materialized-view maintainers
Store source event ID/version and make updates monotonic/conditional. Support replay into a fresh target, compare old/new views, switch atomically, and delete only after verification. This is stronger than patching a corrupt projection in place.
Messaging shifts, rather than removes, coupling
Producers and consumers are decoupled in time/deployment, but coupled through event meaning, retention, ordering, throughput, privacy, and operational ownership. Schema governance and end-to-end SLOs prevent the broker from becoming an organizational dumping ground.
Interview question ladder
Why use a message queue?
Show strong-answer signals
Decouple timing/rate, buffer bursts, retry work, and isolate producers from consumer availability—while accepting asynchronous state and operational backlog.
What is at-least-once delivery?
Show strong-answer signals
A message may be redelivered until acknowledged, so consumers must tolerate duplicates.
What is a dead-letter queue?
Show strong-answer signals
A quarantine for messages that exceed retry/permanent-failure policy; it requires alerts, diagnostics, ownership, and safe redrive.
How do you publish an event atomically with a database update?
Show strong-answer signals
Transactional outbox/CDC; relay can duplicate; consumer uses stable event ID/version.
How do you preserve order?
Show strong-answer signals
Choose an entity ordering key mapped to one partition, one active partition owner, safe offset commit, and version guards; avoid unnecessary global order.
How do you choose a visibility timeout?
Show strong-answer signals
Above expected high-percentile work or extend by heartbeat; finite for crash retry; account for duplicates and split long jobs.
Design an email notification pipeline.
Show strong-answer signals
Preference/consent snapshot, outbox event, template/version, idempotent send key, priority, rate/provider limits, retries/DLQ, expiry, delivery callbacks, audit/privacy.
Design a payment-event consumer.
Show strong-answer signals
Immutable ledger authority, unique operation ID, ordered account/payment state, atomic inbox+posting, ambiguous processor response, reconciliation and alerting.
A consumer is six hours behind after an outage. Recover safely.
Show strong-answer signals
Calculate net drain, protect dependencies, add bounded parallelism/partitions, prioritize/expire, pause retries, monitor age, rebuild projection if faster, communicate SLO.
Migrate an event schema with hundreds of consumers.
Show strong-answer signals
Compatibility policy/registry, additive rollout, consumer inventory, dual/new type for semantic break, mixed-version tests, backfill IDs, deprecation telemetry.
Design an internal event platform.
Show strong-answer signals
Tenancy, schemas/catalog, auth/privacy, topic/partition quotas, SLOs, lineage, outbox/CDC, replay, DR, cost, self-service, guardrails and incident ownership.
Define exactly-once for a cross-system workflow.
Show strong-answer signals
Scope each boundary, unique operation IDs, atomic local state/inbox/outbox, idempotent external API or reconciliation, state machine, audit and proofs/limits.
Design drills
From checkout commit through inventory, payment, warehouse, shipment, and notifications, design commands/events, state, retries, compensation, and reconciliation.
What the interviewer is testing
Distributed workflow semantics and business idempotency.
Ingest click events at millions/s, provide near-real-time counters and replayable raw history, handle late/duplicate data and privacy deletion.
What the interviewer is testing
Partitioned log, event time, derived views, object storage, and governance.
A dependency outage creates 500 million retries and growing DLQ. Produce containment, drain, prioritization, and prevention steps.
What the interviewer is testing
Backpressure, retry budgets, net drain math, and operational ownership.
Common weak answers and how to improve them
“Kafka guarantees exactly once.”
Show the stronger answer
Scope the guarantee and design external effects with idempotency/reconciliation.
“Queues decouple services.”
Show the stronger answer
They also create schema, ordering, retention, backlog, and operational coupling; describe it.
“Put failures in a DLQ.”
Show the stronger answer
Classify, alert, own, diagnose, and redrive safely; otherwise it is silent data loss.
“Use more consumers.”
Show the stronger answer
Check partitions, hot keys, downstream capacity, rebalance, and net backlog drain first.
“Retry three times.”
Show the stronger answer
Choose error classes, backoff/jitter, deadlines, retry budget, idempotency, and permanent handling.