Saga, Outbox & CDC for Payments
A distributed transaction can coordinate multiple compatible resource managers, but independently deployed services rarely share that contract and pay a high availability and latency cost when they do. A payment therefore often becomes a saga: local transactions connected by durable messages, with explicit recovery or compensation when a later step fails.
Prerequisites: Local ACID transactions, message brokers, and retry semantics.
Cover these firstDatabases 101Queues & Async MessagingTwo-Phase Commit Protocol
After this: Choose between 2PC and a saga, build a transactional outbox, and design compensation or forward recovery.
Suggested first pass: Read sections 1–5, answer each section in your own words, then use the remaining failure modes and exercises as the advanced pass.
Content reviewed 21 June 2026 · References: Debezium: Outbox Event Router · Transactional Outbox pattern
1. A multi-service payment can be modeled as a saga: a chain of local ACID transactions connected by durable messages. Reversible steps define compensations; irreversible steps require forward recovery and careful ordering.
2. To update a database and publish an event without a distributed transaction, write the event to an outbox table in the same local transaction, then ship it out separately.
3. A relay (CDC / Debezium tailing the DB log, or a poller) reads the outbox and publishes to the broker. Delivery is at-least-once, so every consumer must be idempotent.
Two-phase commit (2PC) can atomically coordinate participants that implement its prepare and recovery contract, but it holds resources while the decision is resolved and can block when the coordinator is unavailable. A saga is a different trade-off: steps commit independently and at different times, intermediate states are visible, and business invariants are restored through retries, compensation, or forward recovery rather than database rollback.
That reframes everything as two sub-problems. (a) Reliable event emission: "update the DB and tell the world" must be atomic. This is the dual-write problem, solved by the outbox. (b) Reliable orchestration of undo: if step 4 fails, run compensations for 3, 2, 1, in order and idempotently, even though a charge cannot truly be "rolled back." Those two sub-problems frame the entire design. (This is the pattern Idempotency & Exactly-Once Payment Processing pivots toward once one service becomes many.)
Consider the bug this pattern exists to prevent. A service needs to do two things when it processes a payment step: commit to its database and publish an event so the next service acts. These are two different systems (Postgres and Kafka) with no shared transaction, so both cannot happen atomically. This is the dual-write problem.
- DB first, then publish: a crash in between leaves the row present but no event ever sent. Downstream never fulfils; the money is silently stuck.
- Publish first, then DB: a crash in between leaves consumers acting on a payment the source DB rolled back. The result is a phantom charge.
- Retrying the publish: retrying after a crash requires durable knowledge that the publish is owed, which is precisely a record in the database. That reinvents the outbox.
- 2PC across DB and broker: technically possible (XA), but it reintroduces the coordinator, the locks, and the blocking that microservices were adopted to escape. Most brokers do not support it well in any case.
Without a shared transaction protocol, two different systems cannot be updated as one atomic operation, but two rows can be written atomically to the same database in one local transaction. So instead of publishing to the broker directly, the event is written as a row in an outbox table in the very same transaction as the business change. The business row and the "please publish this" row then commit or roll back together. A separate process delivers the outbox row to the broker afterward. The dual write becomes a single local write plus an independent, retryable delivery step.
The outbox is one extra table that lives in the same database as the business data, so a single local transaction can write both.
- 1BEGIN a local DB transaction.
- 2Write the business change, e.g.
UPDATE payment SET status='authorized'. - 3Insert the event into
outboxin the SAME transaction, keyed byaggregate_id. - ↻COMMIT. Both rows land atomically, or neither does. The dual-write is gone.
- ✓A relay (CDC or poller) later reads new outbox rows and publishes them to the broker, independently and with retries.
Polling publisher: a worker runs SELECT … WHERE published_at IS NULL ORDER BY created_at, publishes, and marks rows done. It is simple, needs no extra infrastructure, and works on any DB, but it adds query load and polling latency, and concurrent pollers must be handled (SELECT … FOR UPDATE SKIP LOCKED).
Log tailing (change data capture, or CDC): a connector reads the database's write-ahead log and emits a message for every outbox insert. It avoids polling queries and can provide low delivery latency, but adds connector, replication-slot, and schema-evolution operations. Choose it when those trade-offs beat a simpler poller.
This is a common question, because it is the real alternative to the outbox for the dual-write problem. Kafka's transactional producer and exactly-once semantics (see Kafka Internals & Production Operations) do allow events to be produced and input offsets committed atomically, so for a Kafka→process→Kafka pipeline, EOS is the right tool and no outbox is needed.
But it does not span Postgres→Kafka: a Kafka transaction cannot atomically include the relational business write. So when the source of truth is a database (the usual payments case), the choices are (a) outbox, where the DB write and the event commit in one local transaction and then ship via CDC, or (b) write to the DB and separately produce to Kafka with EOS, which reintroduces the exact dual-write being avoided (the DB commit and the Kafka produce are still two systems). The outbox therefore wins when the DB is authoritative, and EOS wins when Kafka is the system of record. The boundary is the point: it is not the case that "Kafka EOS is exactly-once, so it solves everything."
One outbox row per emitted event means the table grows with the entire event volume; without retention it can become one of the largest tables and add vacuum and write-ahead-log pressure. Define retention from the maximum recovery and replay window. A poller can reap rows after publication is confirmed; a CDC design commonly uses time-based partitions and drops an old partition only after monitoring shows the connector has advanced beyond it. Do not assume that "the WAL has it" is enough if the replication slot, connector offsets, or downstream broker can still fail.
Events for the same aggregate (same order) must stay ordered downstream. Use aggregate_id as the partition/message key so all events for one order go to the same partition and preserve order; global ordering across the whole topic is neither available nor required. Ordering across different aggregates generally does not matter.
Change data capture (CDC) turns the database's own replication log into an event stream. Every committed change is already recorded in the write-ahead log (WAL) in Postgres or the binlog in MySQL; a connector tracks its position, decodes relevant changes, and emits records downstream. Debezium is a widely used open-source implementation that commonly runs on Kafka Connect.
- 1Your txn commits the business row + outbox row. The commit is written to the WAL.
- 2Debezium tails the WAL (via a Postgres logical replication slot), sees the
INSERTintooutbox. - 3Outbox Event Router (Debezium SMT) reshapes the row: routes by
aggregate_type→ topic, keys byaggregate_id, unwrapspayloadas the message body. - 4Publishes to Kafka. Debezium periodically commits its WAL offset back, marking how far it has read.
- ✓Consumers subscribe to the topic. Delivery is at-least-once → they dedup on event id.
- No dual write: the event source is the committed log, so it cannot disagree with the DB.
- At-least-once by design: if Debezium crashes, it resumes from its last committed offset and re-reads; nothing is lost (some things re-sent).
- Low overhead: reading the WAL is cheap and does not compete with application queries the way a poller does.
- Decoupled: the application only commits; it has no broker client and no publish to fail.
- Replication slot bloat: a Postgres logical slot pins WAL until the consumer advances. If Debezium is down, WAL accumulates and can fill the disk and take the DB down. Monitor slot lag and alert aggressively.
- Design for duplicates: connector, broker, and consumer configuration determine the exact delivery boundary, and recovery can replay records. Consumers should remain idempotent even when a narrower component advertises exactly-once processing.
- Ordering is per-key only: guaranteed within a partition (one
aggregate_id), not across the topic. - Schema evolution: the event payload is a contract; changing columns or JSON shape can break consumers. The event schema must be versioned.
- Snapshot on startup: a new connector may first snapshot the whole table, so the initial load on large outboxes must be planned for (mitigated by aggressive outbox cleanup).
CDC can point straight at the business tables and skip the outbox, but then the internal table schema leaks as the public event contract, and only raw row diffs can be emitted rather than rich domain events. The outbox provides an explicit, versioned event with exactly the fields consumers need, decoupled from the storage schema. For payments the outbox is preferable: the event "PaymentAuthorized {amount, currency, …}" is a deliberate contract, not "row 4837 changed."
There are two ways to wire the saga's steps together. The choice depends on saga complexity, for the reasons below.
| Aspect | Choreography | Orchestration |
|---|---|---|
| Control | Decentralized; each service reacts to events | Central orchestrator issues commands |
| Coupling | Loose; services don't know each other | Orchestrator knows all steps |
| Flow visibility | Hard; logic spread across services | Easy; one place to read the flow |
| Best for | Short sagas (2–4 steps) | Complex sagas, many branches |
| Risk | Cyclic event spaghetti; emergent behavior | Orchestrator = bottleneck / SPOF |
Each service listens for events and emits its own. OrderCreated → Payment service charges → emits PaymentAuthorized → Inventory reserves → emits StockReserved → Shipping ships. There is no central coordinator. This works well for 2 to 4 steps; beyond that, answering "what happens when X fails?" requires tracing event chains across many codebases. No single place shows the whole flow, and accidental cycles become a risk.
A saga orchestrator (its own service) holds the flow as a state machine: it sends ChargePayment, waits for the reply, then sends ReserveStock, and so on, and on failure it drives the compensations in reverse. The flow lives in one readable place, which is why complex payment sagas (with branches, timeouts, manual review) almost always use orchestration. The cost is that the orchestrator is critical infrastructure and must itself be durable (next callout).
The orchestrator's current position in the saga ("charged, awaiting stock reservation") is critical state. If it crashes mid-saga and forgets, the result is a half-finished payment that is never completed or compensated. So the orchestrator persists each state transition (often using its own outbox for the commands it sends), and on restart resumes from the last durable state. Frameworks like Temporal and Cadence center on this: durable execution so the workflow survives process death. A hand-built version requires a saga-instance table, idempotent step handlers, and a recovery sweep for stuck instances.
A local transaction can ROLLBACK. A committed saga step cannot: the money has already moved, the email already sent. Each step therefore has a compensating transaction: a new action that semantically undoes the effect.
- 1Authorize payment ✓ (compensation: void/refund authorization)
- 2Reserve inventory ✓ (compensation: release reservation)
- ✕Charge fails: card declined on capture. Saga enters compensation mode.
- ↩Compensate step 2: release the inventory reservation.
- ↩Compensate step 1: void the payment authorization. Saga ends "cleanly aborted."
A compensation is a business action, not a database rollback. A card is not "un-charged"; instead a refund is issued, a new, auditable transaction that leaves both the charge and the refund visible in history. A "shipped" record is not deleted; a return is created. This matters because (a) it is the only thing physically possible once money or goods have moved, and (b) regulators and accounting require the full trail, not a disappeared charge. Compensation is best understood as "forward recovery with a reversing entry," not "undo."
The orchestrator retries compensations until they succeed, since giving up and leaving a charge un-refunded is unacceptable, so each compensation must be idempotent: "refund this payment" run twice issues one refund. Each compensation uses an idempotency key (which ties directly to Idempotency & Exactly-Once Payment Processing). Compensations that do not depend on each other's order are also preferable where possible, because exact reverse ordering is not always achievable under retries.
The textbook prescription is to compensate in strict reverse order (undo N, then N−1, and so on). In practice steps may have dependencies that make reverse order necessary (release the hold before closing the account), or independence that makes order irrelevant. The choice must be made per saga and encoded. Getting it wrong produces subtle corruption, such as closing a wallet before refunding into it. The dependency graph should be mapped rather than assuming "reverse order" is always safe or always required.
"Send the funds to an external bank" or "send the shipment" cannot be undone. The design rule is to structure the saga so irreversible or hard-to-compensate steps come last (the "pivot transaction"). Everything before the pivot is retriable or compensatable; once past the pivot, the saga can only roll forward to completion, never back. An irreversible step that must run early should be gated behind a confirmation or two-phase reservation.
Sagas have no isolation: intermediate states are visible to the rest of the system. During compensation the system can be caught half-undone: the refund has posted but the inventory hold is still active, so a customer sees their money back and the item still locked. The fixes all make the in-between observable and bounded: model an explicit saga status (compensating) so nothing treats the order as final mid-unwind; use semantic locks (mark the row PENDING so other operations know not to trust it yet); and ensure every compensation eventually completes via retries so the window closes. The window cannot be eliminated, only made short, labeled, and self-healing.
This is the recursion to avoid: a compensation itself fails, or worse, the forward step actually succeeded after the decision to compensate (a late ack), so something real has now been undone. The guard rails: compensations are idempotent and retried indefinitely (never "compensate the compensation"; keep retrying the same compensation until it sticks); a definitive saga state machine keeps a step in exactly one state so it is never both completed and compensated; and the "succeeded-late" race is resolved with idempotency keys so the duplicate forward action is a no-op. If a compensation truly cannot succeed, it escalates to a human or dead-letter; it does not spawn an inverse saga.
One malformed or perpetually-failing event sits at the head of a partition: the consumer throws, the broker redelivers, it throws again, indefinitely. Because Kafka preserves per-partition order, that one message can block every event behind it for the same key. Defenses: a retry limit then a route to a dead-letter queue (DLQ) so the partition keeps flowing; alerting on DLQ depth; handlers that are resilient to bad payloads (validate rather than crash). For ordered flows, decide deliberately whether skipping a poison message (and routing it to the DLQ) is safer than halting the partition. For payments it usually is, with the DLQ triaged urgently.
Outbox plus CDC is at-least-once, so every consumer will see duplicates eventually (relay restart, offset re-read). A non-idempotent consumer double-charges or double-ships. Every handler dedups on the event id (a processed-events table or upsert keyed by id). This is mandatory and is the direct link to the idempotency note: the saga is where "make the consumer idempotent" becomes concrete.
If the orchestrator dies between steps without persisting its position, the saga is orphaned: never completed, never compensated, with money in limbo. The fix: persist every state transition to durable storage, run a recovery sweep that finds sagas stuck in a non-terminal state past a timeout and resumes or compensates them, and make step handlers idempotent so re-driving a step that already ran is safe.
(Operational, but it takes payments down.) A stopped Debezium connector pins the Postgres WAL via its replication slot; WAL grows until the disk is full and the primary database stops accepting writes, a total outage triggered by a downstream component. Monitor slot and replication lag with hard alerts, set WAL size limits, and keep a runbook to drop the slot in extremis (accepting event loss) rather than lose the DB.
- A business operation spans multiple services or databases and must be all-or-nothing eventually (checkout, payment, order fulfilment).
- Reliable event publishing is needed from a service that also owns data; the outbox is almost always the right choice here, even without a full saga.
- 2PC is off the table (scale, heterogeneous stores, a broker that cannot do XA) and brief inconsistency with compensation is tolerable.
- Every step has a sensible compensating business action (or can be ordered after the irreversible pivot).
Alternatives: choosing the right consistency tool
| Pattern | Consistency | Best for | Gotcha |
|---|---|---|---|
| Saga + Outbox | Eventual, via compensation | Multi-service business flows | No isolation; must write compensations |
| 2PC / XA | Strong (atomic commit) | Few nodes, same trust domain, short txns | Coordinator blocks; doesn't scale; locks held |
| Single-service ACID | Strong | Operations that fit in one DB | Not always possible across domains |
| Event sourcing | Eventual; full history | Audit-heavy domains, rebuildable state | Steeper model; projections lag |
| Listen-to-yourself / CDC only | Eventual | Pure "publish events from writes" | Leaks table schema as contract |
A saga does not provide distributed ACID. It trades isolation (intermediate states are visible) and immediate consistency (the system is eventually consistent) for availability and scale. The outbox cleanly solves the dual-write sub-problem; the saga manages the multi-step undo. The complexity, however, remains: compensations, idempotency, state persistence, and reasoning about partial states. Sagas are not "just distributed transactions"; treating them that way ignores the operational reality.
| System / tool | Role | Style | Notable detail |
|---|---|---|---|
| Debezium | CDC connector | Log tailing | Has a built-in Outbox Event Router SMT |
| Temporal / Cadence | Saga orchestrator | Orchestration | Durable execution; workflow survives crashes |
| AWS Step Functions | Saga orchestrator | Orchestration | Managed state machine; explicit compensation branches |
| Axon / Eventuate | Saga + event sourcing | Either | Framework support for sagas + outbox |
| Kafka | Event backbone | — | Per-partition ordering; at-least-once; DLQ topics |
Common traps
Update Postgres, then call kafka.publish(). → This is the dual-write problem: a crash between the two leaves the DB and the broker disagreeing. The outbox makes the event commit atomically with the data.
If step 4 fails, roll back 1–3. → There is no rollback; those steps committed. The saga runs compensating business actions (refund, release, return), which are new transactions, idempotent, and visible in history.
Debezium is exactly-once. → It is at-least-once; crash recovery re-emits from the last offset. Every consumer must dedup on event id, or charges are duplicated. Reliability does not mean no duplicates.
No central orchestrator = cleaner. → For a 6-step payment saga with branches and compensations, decentralized event chains become unreadable and cycle-prone. Past roughly 4 steps, orchestration's single readable state machine is preferable.
Append-only, no maintenance. → Unreaped, it becomes the largest table and bloats the WAL that CDC reads. TTL-delete or partition-and-drop, and monitor the replication slot, or a stalled relay will fill the disk.
- Chris Richardson — Microservices Patterns (Saga, Transactional Outbox chapters) & microservices.io
- Hector Garcia-Molina & Kenneth Salem — "Sagas" (1987, the original paper)
- Debezium docs — "Outbox Event Router" & "Reliable Microservices Data Exchange with the Outbox Pattern"
- Temporal / Cadence docs — durable execution for sagas
- Pat Helland — "Life beyond Distributed Transactions: an Apostate's Opinion"
The questions that arise after the happy path is drawn. Click each to expand the answer.
2PC requires compatible participants to retain prepared state and locks until they learn the decision. A coordinator outage or partition can therefore block affected resources, and independently deployed HTTP services usually do not implement the XA resource-manager contract. It remains useful inside a controlled domain of compatible participants; a saga is generally easier to operate across autonomous services.
The saga gives up cross-service atomicity and isolation in exchange for availability and scale: each step commits locally and immediately, releasing its locks, and correctness is restored by compensation if a later step fails. The system is eventually consistent with visible intermediate states, but there are no global locks, no coordinator SPOF, and it scales. It is the right trade for long-running, multi-service business flows; 2PC is reasonable only for a few nodes in one trust domain with short transactions.
A service must commit to its DB and publish an event: two systems, no shared transaction. Whichever runs first, a crash in between leaves them disagreeing: DB-then-publish loses the event (the DB says paid, nobody was told); publish-then-DB emits a falsehood (consumers act on a rolled-back change). Two different systems cannot be written atomically.
The outbox observes that two rows can be written atomically to the same DB. So in the same local transaction as the business change, an event row is inserted into an outbox table; they commit together or not at all. A separate relay then delivers the outbox row to the broker, independently and with retries. The atomic part is local; the delivery part is at-least-once and retryable. The event source becomes the committed DB state, which cannot disagree with itself.
Polling is simplest: a worker selects unpublished rows, publishes, and marks them done. It needs no extra infrastructure, works on any DB, and with FOR UPDATE SKIP LOCKED several pollers can run. The downsides are query load on the DB, polling latency, and managing the publisher directly.
CDC (Debezium tailing the WAL) usually offers lower polling load and latency, and the application needs no broker client. Its costs are operational: a logical replication slot that can retain WAL, possible redelivery on restart, and schema and versioning discipline. Choose it when event volume and latency justify that machinery; a transactional poller is often the clearer choice for a smaller system.
This is partial visibility: sagas have no isolation, so during compensation the system can be observed half-undone. The refund compensation succeeded; the release-hold compensation failed or has not run yet, leaving an inconsistent visible state.
Prevention bounds and labels the window rather than eliminating it. Model an explicit saga status so nothing treats the order as final while it is compensating. Use semantic locks: mark affected rows PENDING so other operations do not trust them mid-unwind. Make every compensation idempotent and retried until it succeeds, so the release-hold eventually completes and the window closes; if it truly cannot, escalate to a DLQ or human rather than leaving it silently broken. And order compensations by their real dependencies, so the wallet is not, say, closed before a refund is paid into it.
No; that way lies infinite recursion. Compensations are designed to be idempotent and retried indefinitely: re-run the same compensation until it succeeds, never invert it. A definitive saga state machine ensures each step is in exactly one state, so the same step is never both completed and compensated.
The harder variant is the forward step that succeeded after the decision to compensate (a late ack). Idempotency keys resolve that: the duplicate forward action becomes a no-op, and the compensation still applies to the one real effect. If a compensation truly cannot succeed after bounded retries (for example, an external system rejects it), it goes to a dead-letter queue for human resolution; it does not spawn an inverse saga. The principle is "retry, do not reverse."
A technical undo is a DB rollback: it makes it as if the change never happened. That cannot be done to a committed saga step: the card was charged, the funds moved. The compensation is a business action (issue a refund, release a reservation, create a return), a brand-new transaction that reverses the effect while leaving both the original and the reversal in the record.
For payments this is essential, not pedantic: accounting and regulators require the full audit trail, so a charge and its refund must both be visible, not a vanished row. It is also the only physically possible option once money has left. So compensation is described as "forward recovery with a reversing entry," and each one is made idempotent with its own key so retries do not issue two refunds.
Because Kafka guarantees order within a partition, a message that always fails is redelivered indefinitely and blocks every event behind it for that key. For a payment flow keyed by order, that order (and others in the partition) stalls. A retry loop with no exit is a self-inflicted outage.
Fix: bounded retries, then route the message to a dead-letter queue so the partition keeps flowing, and alert on DLQ depth so it gets triaged fast. Handlers should validate payloads and fail gracefully rather than crash. The judgment call is ordering: skipping a poison message means a gap in the ordered stream, so the choice between DLQ-and-continue and halt is made deliberately. For payments DLQ-and-continue usually wins, because one stuck message should not freeze unrelated orders, and the DLQ'd item gets urgent manual attention.
The orchestrator's position in the flow ("charged, awaiting stock reservation") is critical, durable state. It must persist every transition, typically a saga-instance row updated as each step completes, often using its own outbox to emit the next command atomically with recording the transition. State held only in memory and then lost to a crash produces an orphaned saga: a payment neither completed nor compensated.
On restart it resumes from the last durable state. A recovery sweep also scans for saga instances stuck in a non-terminal state past a timeout and re-drives or compensates them, and every step handler is idempotent so re-issuing a command that already ran is a no-op. This is exactly the "durable execution" guarantee Temporal and Cadence productize; building without one means rebuilding a slice of it.
The outbox grows by one row per emitted event. At, say, 1k events/sec that is ~86M rows/day if never reaped, and that bloat slows the table and the WAL Debezium reads, which shows up as CDC lag. Two things are usually wrong: no cleanup, and/or row-by-row DELETEs leaving dead tuples that vacuum cannot keep up with.
Sizing and fix: keep the outbox tiny. Delete rows seconds to minutes after they are confirmed published (in CDC mode deletion can be almost immediate, since the WAL already captured the insert), and use time-based partitioning to DROP whole old partitions instantly instead of deleting rows. Separately, watch the replication slot: if the connector stalls, the slot pins WAL and grows until the disk fills and the primary stops accepting writes, so hard-alert on slot lag and keep a runbook. In steady state the outbox should hold only minutes of events, not history.
Outbox plus CDC is commonly operated with at-least-once delivery, so duplicates are possible during retries and recovery even when the normal path emits one copy. That means every consumer and compensation that cannot safely repeat must be idempotent, typically deduping on the event id with a processed-events table or an upsert keyed by id. This connects the saga to the idempotency note: there, idempotency protected one API from client retries; here it protects internal hops from broker redelivery.
Concretely: the event carries a stable id; the consumer checks "have I processed this id?" inside the same transaction as its effect, so check-and-act is atomic (no TOCTOU race between two redeliveries). For compensations, the idempotency key ensures "refund X" runs once no matter how many times it is retried. Dedup lives at the consumer, keyed by the producer-assigned id; the broker should never be trusted to deliver exactly once.
Create a payments table and an outbox table in Postgres. Write a service that inserts a payment row and an outbox event row in a single transaction. Write a polling publisher that selects unpublished outbox rows, publishes them to a Kafka topic, and marks them published. Kill the service between the DB commit and the publish step. Restart it. Verify the event was not lost and was not emitted twice.
Then kill the service between the publish and the mark-published step. Restart. Observe that the event is re-published. Add a deduplication key (the outbox row ID) to the Kafka message so downstream consumers can detect and drop the duplicate. This is the transactional outbox and the dual-write problem made directly observable — the two crash points you test here are the only two gaps the outbox pattern closes.
Design a choreography saga for a payment flow: (1) authorize hold, (2) reserve inventory, (3) charge card, (4) ship. Draw the event sequence for the happy path. Then: step 3 (charge) fails. Map every compensation that must run in reverse order: reverse the inventory reserve, release the hold. For each compensation, show the event it emits and the event that triggers the next compensation step.
For each compensation, prove it is idempotent: if the compensation event is delivered twice, does the service charge the card twice or release the hold twice? Design the deduplication key (saga ID + step name) that prevents double-application. Finally: step 2 (inventory reserve) succeeds and emits its event, but the service crashes before step 3 sees it. When the service restarts and the event is redelivered, does the inventory get double-reserved? Show how the idempotency key prevents it. Covers the dual-write problem, choreography vs orchestration, and compensation at interview depth.
The Debezium outbox poller has been down for 2 hours. The outbox table has 200k unprocessed rows. Two instances of the downstream payment service are consuming the same Kafka topic. Describe the CDC catch-up process: how does Debezium know where in the Postgres WAL to resume, how fast can it drain 200k rows, and what guarantees does Debezium provide about the order of events for a given payment ID?
During catch-up, both consumer instances receive the same event at the same time. One processes it and marks the saga step done; the other also processes it a split-second later. How do you prevent both consumers from advancing the saga simultaneously, and how do you make the saga step handler idempotent so the second delivery is a no-op rather than a double-charge? Then: one consumer crashes mid-processing at offset 950. The consumer group rebalances. How many messages get redelivered, and how does your idempotency scheme handle them? Covers the transactional outbox, CDC, and failure modes with exactly-once semantics at staff-engineer depth.