Idempotency & Exactly-Once Effects in Payments
Networks lose responses. Clients retry. Without protection, that retry charges the card twice. An idempotency key and a small lookup table prevent this: the second request finds the work already done and replays the first reply. Same answer, no double charge.
Prerequisites: HTTP retries, database transactions, and unique constraints.
Cover these firstAPI Design BasicsDatabases 101Failures, Timeouts & Retries
After this: Design an idempotent write path and explain exactly-once effects without overclaiming delivery guarantees.
Suggested first pass: Read sections 1–5, answer each section in your own words, then use the remaining failure modes and exercises as the advanced pass.
Content reviewed 21 June 2026 · References: Stripe: Idempotent requests · RFC 9110: idempotent HTTP methods
1. The client generates a unique Idempotency-Key for each action it intends to perform once (for example, one charge).
2. The server stores that key together with the response it returned, in a small lookup table.
3. If the same key arrives again, the server skips the work and returns the stored response.
An unreliable network cannot give a sender perfect knowledge that a response arrived. That makes an end-to-end exactly-once delivery claim unsafe without a clearly bounded transactional system.
Payment APIs instead aim for exactly-once effects: retry delivery at least once, identify repeated attempts with the same key, and make the receiver replay or deduplicate them. State the boundary explicitly—an internal log may offer exactly-once processing while an external payment call still requires idempotency.
A common approach (used by Stripe and described in an IETF Internet-Draft): the client generates a random UUID and places it in a header called Idempotency-Key. The server uses that value as the lookup key for "has this request already been processed?"
The lookup table stores enough to replay the previous response and to detect whether the client reused the key with a different body:
- 1Client sends POST with
Idempotency-Key: K+ body B. - 2Server INSERTs
(K, hash(B), 'in_progress')viaINSERT ... ON CONFLICT DO NOTHING. Insert succeeds → this request owns the work. - 3Server processes the charge: it calls the payment provider and writes a ledger row.
- 4Server UPDATEs the row:
status='completed', stores response_body + status code. - ✓Returns 200 with the response. Subsequent retries with the same key replay this response.
Three reasons: (1) it is small, 32 bytes for SHA-256 versus perhaps 4KB for the body; (2) it has a fixed size, which makes it easy to index; (3) it allows retries to be compared quickly to detect "same key, different body" without retaining the original payload.
The simple case: the client retries 5 seconds later, the server finds the work already done, and replays. The difficult case is when two retries arrive at the same time. The client timed out and fired a second request before the first one finished, so both reach the server within milliseconds.
The dedup row passes through three states: (nothing) → in_progress → completed. Only one worker may take the work. The database's unique-key constraint acts as the referee: both workers attempt to INSERT at the same time, exactly one succeeds, and the other receives a duplicate-key error.
- 1Request A arrives. INSERT with key K, status=in_progress. Wins.
- 2Request B arrives 5ms later with same key K. INSERT fails on unique constraint.
- 3B's options: (a) return 409 Conflict immediately ("retry pending, try again later"), or (b) poll the row until status=completed and replay the response.
- 4One reasonable choice: 409 with retry-after. It avoids server-side blocking and lets the client back off.
- ⚠Trap: do not poll inside the request handler with a long timeout. Under retry storms it exhausts the connection pool.
Consider the case where the first worker crashes after the INSERT but before finishing the work. The row is now stuck at in_progress indefinitely, and every retry sees it and returns 409. The charge can then never be retried.
Fix: a lease. Store locked_by and locked_at on the row. If another worker sees a row whose lease is older than, say, 30 seconds, it is permitted to take over and finish the job. This behaves like a stale lock that can be reclaimed.
The system relies on one rule: the same key means the same action. If a client sends key K with {amount: 50} and then retries K with {amount: 5000}, that is almost always a bug in the client's code. A robust API rejects the request; HTTP 422 is one reasonable status.
Scope keys per user; keys should never be global. If they are global, user A could guess user B's key and read user B's charge response, a serious data leak. Make the lookup key (user_id, key) in the database, or include the user ID within the key itself.
- 1First request: key K, body B1. Server stores
hash(B1), processes, returns 200. - 2Retry: same key K, body B2 (different amount). Server computes
hash(B2). - 3Hash mismatch detected:
hash(B1) ≠ hash(B2). - 4Server returns 422 with error: "Idempotency-Key reused with different body."
- ⚠Do not silently process B2 as new work. That breaks the contract and can double-charge if the client meant to retry B1.
- Global keys: collisions and cross-user response leaks. Always scope by user.
- Hashing the wrong fields: exclude values the client legitimately changes on retry, such as client-side timestamps or request IDs. Otherwise every retry appears to be a body mismatch.
- Skipping JSON canonicalisation:
{"a":1,"b":2}and{"b":2,"a":1}are the same logical body but hash differently. Sort keys and strip whitespace before hashing.
Stripe keeps keys for 24 hours. The trade-off is straightforward: a longer window catches slower retries but consumes more disk.
At 10k TPS, 24h is roughly 864M rows. At ~500 B/row that is ~430 GB of raw payload, but the actual on-disk size is closer to ~700 GB–1 TB once Postgres per-tuple overhead (~23 B header plus alignment) and the indexes (the primary key on key and the expires_at index) are included. The loaded figure is the meaningful one, not the raw payload product. Most retries arrive within minutes, so 24h is a comfortable upper bound that covers almost every legitimate retry without maintaining a very large table.
The most damaging failure: the charge actually succeeded, but the network dropped the 200 response on the way back to the client. The client cannot tell whether the charge was made. It times out and retries. Without idempotency this charges the card twice. With idempotency, the server replays the stored 200 from the first attempt, so the client sees the same successful response it should have seen the first time.
- 1Client → Server: POST with key K. Server processes, captures payment, writes
status='completed', stores response. - 2Network drops the response on the way back. Client times out.
- 3Client retries the same request with key K.
- 4Server sees status=completed for K + matching request_hash → skip processing entirely.
- ✓Server replays the stored response. Client gets the same 200, same charge_id. No double charge.
5f7c…1a09 INSERT … ON CONFLICT referee: one wins, the other gets 409.- Idempotency-Key: the request header convention used by Stripe and described in the archived IETF Internet-Draft
draft-ietf-httpapi-idempotency-key-header. - Idempotent producer (Kafka,
enable.idempotence=true): the producer is assigned an ID (PID) and a per-partition sequence number. The broker rejects messages with a sequence number it has already seen, so retries do not double-write. - Conditional request:
If-Match: "<etag>"on a PUT means "apply only if the resource version is still this." It pairs with idempotency keys: keys protect creates, conditional requests protect updates. - Fencing token: a monotonically increasing number. After a lock holder loses the lock, its old writes are still in flight toward storage; storage rejects any write whose token is lower than the highest token it has seen. This prevents zombie workers from corrupting data.
- Effectively-once: the precise term. At-least-once delivery plus an idempotent handler produces exactly-once behavior as observed by the user. Prefer this over the marketing term "exactly-once."
Everything above protects the API from duplicate requests. The harder problem in payments is that "charge the external processor" and "record that the charge happened" are two different systems with no shared transaction, the same dual-write problem that sagas exist to solve. Calling the PSP first and then crashing before writing the result moves money with no record; writing "completed" first and then having the PSP call fail records a charge that never happened.
The mature shape: write an intent row (in_progress plus a stable downstream idempotency key generated now) and commit it before calling the PSP; pass that key on the PSP call so the processor dedups as well; then commit the completed status and response. A crash at any point leaves a recoverable in_progress row whose lease lets another worker re-drive the same PSP call safely (it is a no-op downstream because the key matches). This is why the concurrent-retry lease takeover and the stored downstream idempotency key must be the same mechanism: takeover is safe only because the downstream key fences the retry.
Idempotency keys and a TTL are a fast-path optimization, not the final word. Production payments back them with an out-of-band reconciliation sweep against the processor's record of truth (a settlement file, or GET /charges?metadata=order_id): anything the ledger considers pending-but-old, or any PSP charge with no local record, is reconciled. This is the safeguard when a key expires before a very late retry — arriving after the dedup row has already been cleaned up — or a crash loses an in-flight write. The system does not depend on the dedup row surviving; it depends on being able to ask the PSP whether the charge happened.
What is persisted for a non-2xx response matters. A deterministic decline (card declined, 4xx) should be cached and replayed: retrying will not change the answer, so replaying it is correct. A transient failure (timeout, 5xx, PSP unreachable) must not be stored as a final result. Caching "failed" permanently wedges a charge that would have succeeded on retry. Leave the row in_progress (lease-recoverable) for transient errors, and write a terminal cached response only for deterministic outcomes. Conflating the two can either lose money or block legitimate retries.
The Idempotency-Key mechanism protects the front door (the public API). Inside the system, the same problem recurs: every time a worker reads a message and writes elsewhere, it could read or write twice. Kafka has its own built-in mechanism for this.
Producer side: enable enable.idempotence=true. Kafka then assigns the producer a unique ID (PID) and a sequence number for every message it sends. The broker remembers the highest sequence number it has seen from each PID; if the producer retries and sends the same number again, the broker discards it. Retries become safe automatically.
Kafka transactions go further. They allow writes to several topics and the input-offset commit to be a single atomic unit. The consumer must set isolation.level=read_committed so it ignores messages from transactions that were rolled back.
- 1initTransactions(): producer registers transactional.id with the broker (zombie fencing).
- 2beginTransaction() + read input record + process.
- 3send() output record to topic-out, sendOffsetsToTransaction() for the input offset.
- 4commitTransaction(): broker writes the COMMIT marker. Both the produce AND the offset commit are atomic.
- ✓Downstream consumer with
read_committedonly sees committed records. Effectively-once across the topic boundary.
Kafka exactly-once applies inside Kafka only: producer → broker → broker → consumer. As soon as the consumer writes to an external database, calls an external API, or sends an email, the guarantee reverts to at-least-once unless that step is also idempotent.
The combination that works: Kafka EOS for the messaging path, idempotent DB writes (upsert by key), and idempotent external calls (using the provider's own idempotency key). Each layer turns at-least-once into effectively-once for its own hop.
If the dedup table is lost (eviction, restart, accidental delete), every retry appears new and cards are double-charged. Treat this table like the ledger: same durability, same backups. Never place it in cache-only storage (Redis without disk persistence).
If keys are global and user A retries with key K while user B happens to use the same K, user B's request replays user A's response (or exposes A's data). Always store the key as (user_id, key) so two users can never collide.
Without comparing body hashes, a buggy client that retries with a different body will succeed with the first body's response, causing silent data corruption. Always hash the body, compare on retry, and return 422 if they differ.
A client retries 8 days later but the TTL was 7 days: the key is gone, the server treats it as a new request, and the card is double-charged. Fixes: (a) document the window publicly, (b) ask clients to stop retrying before the window ends, (c) keep a smaller long-lived index of settled charges by external ID even after the full dedup row is cleaned up.
The API is safe to retry, but the payment processor it calls is not. On a retry, the processor is called twice and charges twice. Fix: when the work first starts, generate (or accept from the client) a stable request ID for the downstream call, save it in the dedup row, and reuse it on every retry. The downstream then sees the same ID and dedups as well.
Checking whether the key exists first and inserting only if it does not allows two simultaneous retries to both see "not present" and both INSERT, so both run the work. Use INSERT ... ON CONFLICT DO NOTHING (or the database's upsert) so the unique constraint resolves the race. One succeeds, the other receives a clean error.
in_progress after worker crashA worker writes "in_progress" and then crashes. All future retries see "in_progress" and return 409 indefinitely, so the charge can never complete. Fix: add a lease (a locked_at timestamp) and run a janitor that marks rows as failed when their lease is older than, say, 60 seconds, so retries can take over.
- Operation is a POST that creates / charges / sends (non-idempotent verb on a stateful side effect).
- Client retries are routine (mobile networks, HTTP timeouts, queues).
- The cost of a duplicate is real: money, email, SMS, push, inventory.
- Clients can be required to generate a unique key and accept a TTL window.
Alternatives, and where each applies:
| Pattern | Where dedup lives | TTL | Best for | Gotcha |
|---|---|---|---|---|
| Idempotency-Key | API edge (DB table) | Hours – days | External-facing POSTs (payments, sends) | Client must generate key |
| Transactional Outbox | Producer DB (outbox tbl) | Until shipped | Reliable event publish from a write | Outbox bloat; needs a relay/CDC |
| Queue dedup (SQS FIFO, Kafka idempotent producer) | Broker | 5 min – hours | In-system message dedup | Window is fixed; not API-edge |
| Optimistic concurrency (If-Match / version) | Resource version | None (per-version) | PUT updates of existing resources | Doesn't help create-style POST |
| Let the user retry manually | Human | — | Cheap, low-volume, reversible ops | Poor UX; unsuitable for payments |
Real systems layer these. Edge: Idempotency-Key. Internal queue: idempotent producer + read_committed consumer. Cross-service write: outbox + CDC. Downstream call: provider's own idempotency key. Each layer turns at-least-once into effectively-once one hop at a time.
| System | Mechanism | Window | Scope | Notable detail |
|---|---|---|---|---|
| Stripe | Idempotency-Key header | 24h | Per API key (account) | Returns 422 on body mismatch; 409 while in-flight |
| AWS SQS FIFO | MessageDeduplicationId or content hash | 5 min (fixed) | Per queue + group | Window is hard-coded; not configurable |
| Kafka idempotent producer | PID + per-partition seq# | Producer session | (PID, partition) | Auto-enabled with EOS; needs acks=all |
| Kafka transactions | transactional.id + commit marker | Per txn | Across partitions | Consumer must use read_committed |
| Square | idempotency_key in body | Indefinite (per docs) | Per merchant | Key in JSON body, not header |
| PayPal | PayPal-Request-Id header | ~72h (varies) | Per merchant | Replays cached response on retry |
Common traps and misconceptions
EOS gives end-to-end exactly-once. → The Two Generals Problem proves it impossible. What ships in practice is at-least-once delivery plus idempotent processing, which is effectively-once. This is the accurate framing.
MD5 collisions are too rare to matter. → Use SHA-256 (or BLAKE2/3). MD5 has known collision attacks, and there is no reason to use a weak hash for a security-adjacent dedup primitive.
Redis is fast enough for the dedup check. → Speed is not the issue; durability is. If Redis is lost (eviction, restart without persistence, failover), every retry appears new and cards are double-charged. Either persist it (AOF plus replicas) and treat it as authoritative, or pair the Redis cache with a durable database for the source of truth.
A week, a month, it makes no difference. → It is a real trade-off: replay-window coverage versus storage growth (~430GB/day at 10k TPS). 24h suits most APIs. State the window in the API docs so clients can cap their retry budget below it.
SHA-256 of the raw bytes works. → Strip non-deterministic fields (client timestamps, request IDs, free-form metadata the client might change on retry). Canonicalise JSON (sorted keys, no whitespace) before hashing; otherwise a different serialisation order looks like a body mismatch.
- Brandur Leach — "Designing robust and predictable APIs with idempotency" (Stripe blog)
- IETF Internet-Draft (archived) —
draft-ietf-httpapi-idempotency-key-header - Confluent — "Exactly-Once Semantics Are Possible: Here's How Kafka Does It"
- Pat Helland — "Idempotence is not a medical condition"
Follow-up questions that probe beyond the happy path. Click each to expand the answer.
Both workers attempt INSERT ... ON CONFLICT DO NOTHING with the same key. The database's unique constraint on the key column allows only one of those inserts to succeed. That worker owns the work; the other gets back zero rows affected.
The losing worker has two reasonable options. (a) Return 409 Conflict immediately: "a request with this key is already in progress, retry in a moment." An API can do this. (b) Poll the row until it becomes completed and replay the response. This is cheaper for the client but expensive for the connection pool. Returning 409 is a simple default when holding the connection open is undesirable.
This is the partial success failure mode, and it is the reason the response is saved before the 200 is returned to the client. Order matters: (1) finish the work, (2) UPDATE the row to completed and save the response, (3) return 200. If the crash happens between (1) and (2), the row is still in_progress, the lease expires, and another worker takes over and re-runs the work. This is safe because the underlying call to the payment processor is also idempotent (the processor's request ID was stored in step 1).
The key principle: the dedup table is the source of truth, not the network reply. If the reply is lost, the table still records what happened.
10,000 × 86,400 ≈ 864M rows per day. With a 24h TTL the steady state is ~864M rows. The raw 500 B/row product is ~430 GB, but the loaded figure is ~700 GB–1 TB once per-tuple overhead and the key and expires_at indexes are included, which is still manageable on a single large Postgres instance or as one shard. A year without expiry would be 315B rows and hundreds of TB, which is not viable without a TTL.
Operationally: (a) Partition by day (Postgres PARTITION BY RANGE(created_at)) and drop yesterday's partition once it is outside the window, giving instant TB-scale cleanup. (b) Index expires_at for the cleanup sweep. (c) If reads dominate (charge replays), keep a hot in-memory cache in front (Redis), but the durable Postgres table remains the source of truth.
Speed is not the problem; durability is. If Redis evicts a key (memory pressure), restarts without persistence, or fails over before AOF flushes, that key is gone, and the next retry of that charge appears new. Cards are then double-charged.
Two valid options: (a) Use Redis but configure it like a real database (AOF on every write, replicated, never eviction-based). This is slower than commonly expected. (b) Use Postgres for durability and Redis as a read-through cache. The dedup write must reach Postgres before the work is done; Redis only speeds up read replays.
"SELECT then INSERT" is a TOCTOU (time-of-check to time-of-use) race. Worker A SELECTs and sees no row. Worker B SELECTs and sees no row. Both INSERT, both succeed, and both run the work. The card is double-charged.
The fix is to push the race down to the database. INSERT ... ON CONFLICT DO NOTHING is atomic: the database picks one winner and tells the other "already present." There is no window in which both can consider themselves first. The same idea applies to Mongo's insertOne with a unique index and DynamoDB's conditional PutItem with attribute_not_exists.
Reject it with 422 Unprocessable Entity and a clear error message. This is a defensible API behavior: the contract is that the same key means the same operation. If the bodies differ, the client has a bug. Silently treating it as a new operation could double-charge; silently treating it as a replay could return a wrong answer.
The check is the SHA-256 of the canonicalised request body (sorted JSON keys, no whitespace, with non-deterministic fields such as client timestamps excluded), computed on the first request and compared on every retry.
Different layer, same idea.
Idempotency-Key is HTTP-level: it protects an external API from duplicate requests across retries. It lives at the API edge.
Kafka EOS is broker-level: it protects messages inside Kafka from duplicate produces and lost commits. It lives inside the message bus.
Most systems need both. A typical architecture is: client → public API (Idempotency-Key) → DB write → emit event with the outbox pattern → Kafka (EOS within Kafka) → downstream consumer (idempotent DB writes by key) → external service (provider's own idempotency key). Each hop has its own dedup mechanism.
This is the failure case: the key is gone, the server treats the retry as a new charge, and the customer pays twice. There are three layers of defense.
(1) Document the window in the API docs. Clients should cap their retry budget below the TTL.
(2) Keep a smaller long-lived index alongside the dedup table, for example an external_charge_id column on the ledger that can be looked up even after the full dedup row is cleaned up. The retry then replays the original ledger entry instead of creating a new charge.
(3) Choose a generous TTL for high-value operations: 7 days for refunds, 30 days for bank transfers. The marginal storage cost is worth avoiding a double charge.
The client, always. The purpose of the key is that retries from the client carry the same value. If the server generated it, every retry would be a new request from the server's perspective. The client must remember and resend the same key on every retry of the same logical operation.
Practical pattern: the client generates a UUID v4 once when the user clicks "Pay," stores it in local state, and resends it on every retry until the operation either succeeds or permanently fails (a 4xx other than 409).
The client retries with the same key. The server sees a row in in_progress and returns 409 Conflict. The client backs off and retries again. Eventually the first worker either completes (the status flips to completed and a replay happens) or its lease expires and a new worker takes over.
The retry-after header tells the client how long to wait before the next attempt. A typical schedule is 1s, then exponential backoff with jitter, capped at a few minutes.
By HTTP semantics, PUT and DELETE are already idempotent: repeating them produces the same result. PATCH is technically not (it depends on the patch format). For these, a different tool is usually appropriate: conditional requests with ETags or version fields.
Pattern: If-Match: "abc123" on a PUT means "apply only if the current resource version is abc123, otherwise reject with 412 Precondition Failed." This protects against the lost-update problem (two clients editing the same record at the same time), a different concern from the same request arriving twice. Many real systems use both: Idempotency-Key for safety against duplicate requests, and ETag for safety against lost updates.
This is the fencing token pattern from Martin Kleppmann's "How to do distributed locking" critique. When a worker takes the lease, it is given a monotonically increasing number. The worker passes that token to every downstream write. The downstream system remembers the highest token it has accepted and rejects writes with a lower token.
Sequence: Worker A takes the lease and gets token 7. A pauses (long GC, network). The lease expires. Worker B takes the lease and gets token 8. B writes with token 8. A wakes up and tries to write with token 7; the downstream rejects it because it has already seen 8. A's stale write cannot corrupt anything.
A simple lock plus TTL cannot solve this on its own. The fencing token is required at the storage layer.
One table per service that exposes a non-idempotent operation, owned by that service's database. The Charges service has its own dedup table, the Refunds service has its own, and the Email service has its own. They should not be shared: coupling services through a shared dedup table produces the kind of distributed monolith this design aims to avoid.
If service A calls service B, A passes its idempotency key to B (often as a different downstream key, generated deterministically from the original). B has its own dedup logic for the call A made to it. The keys are independent, and the safety composes.
Add an idempotency_keys table (key, status, response_body, created_at) with a unique constraint on the key. On first call: insert with status=PROCESSING (the unique constraint rejects a concurrent duplicate). Process the payment, then update to COMPLETED with the stored response. On any retry: look up the key — if PROCESSING, return 202; if COMPLETED, replay the stored response.
Test it by firing the same idempotency key from two goroutines simultaneously and verifying exactly one payment is created and both callers receive identical responses. This is the core idempotency protocol — the unique constraint is what closes the concurrent-retry race window that a simple "check then insert" leaves open.
A payment flows through three steps: authorize (hold funds), capture (charge), settle (transfer). Each step is a separate service call. Walk through idempotency key scoping: should one key cover the whole payment, or each step independently? What happens when capture succeeds but the response is lost and the caller retries with the same key but a different amount — how do you detect and reject a body mismatch?
Then: authorize succeeds, capture fails after 3 retries. The hold on the user's card is now stuck. Design the partial-success state machine: how does the system detect it's in a terminal-failure state, what compensation is required (release the hold), and how do you guarantee the compensation eventually runs even if the service crashes mid-saga? Covers concurrent retries, body mismatch and scope, and partial success together.
Your payment service calls an external bank API that has no idempotency support: calling it twice charges the user twice. Design a wrapper that makes it effectively-once. Walk through: why you must record the intent to call (write to your DB) before making the external call — what happens if you crash between recording and calling vs between calling and recording the result? What does "at-least-once external call with idempotent result storage" mean?
Then: the bank API returns HTTP 200 but also charges twice due to a bug on their side. Your wrapper records a single successful call. From your system's view, what guarantee does your idempotency layer provide and what does it not? How would you detect the double-charge (reconciliation), and what is your recourse? This is partial success with external non-idempotency, and the scenarios where idempotency breaks, at production depth.