Missing Idempotency

Introduction

Missing idempotency is the failure to give a retryable operation a stable identity. When a client, queue, or job runner repeats a request after an ambiguous failure, the server treats the retry as a new operation and may repeat the side effect.

Mental model: a timeout tells you that the outcome is unknown, not that nothing happened. An idempotency key lets the server turn that unknown outcome into a safe lookup instead of a second side effect.

TL;DR

• An operation is idempotent if applying it multiple times produces the same result as applying it once. PUT /users/123 { name: "Alice" } is idempotent. POST /payments { amount: 100 } without an idempotency key is not.
• Without idempotency, retries cause duplicate side effects: money charged twice, emails sent three times, orders created in duplicate.
• The solution is an idempotency key: the client generates a unique token for each logical operation. The server uses it to detect and short-circuit duplicate requests.
• This pattern is critical anywhere retries happen: payment APIs, notification pipelines, job submissions, and any message consumer processing a queue.
• Store idempotency keys in PostgreSQL for financial operations (durability). Use Redis with TTL for lower-stakes deduplication (speed).

What It Is

Before: retrying without an operation identity

It's 2:47 a.m. on Black Friday. Your mobile app calls POST /payments to charge a user $99. The server processes the payment, charges the credit card, and starts sending the response. The network drops. The mobile app never receives the 200 response.

Should the app retry? If the payment succeeded, retrying will charge the user $198. If the payment failed (the request never reached the server), not retrying means the user never completes their purchase. The app can't tell which happened. It has to guess.

A common incident pattern is a flood of "duplicate charge" support tickets. The fix requires correlating payment logs, issuing manual refunds, and sending apology emails because the mobile client retried a request the server had already processed.

If it retries without idempotency, it can double-charge users. If it does not retry, it can lose sales on valid network drops. At 100K payments per day with a 1% network failure rate, that is 1,000 ambiguous requests daily. Even if only 10% of those cause duplicates, that is 100 double-charges per day.

The fundamental problem: HTTP doesn't tell you whether the server processed your request. A timeout means "I don't know what happened," not "the request failed." Without an explicit deduplication mechanism, the client must choose between data loss and data duplication.

The failure math

Consider a payment API processing 100,000 requests per day with 3 retries per failure:

• Network failure rate (mobile): ~1% = 1,000 ambiguous requests/day
• Probability that the request actually succeeded before the drop: ~50%
• Expected duplicates without idempotency: ~500 double-charges per day
• With 3 retries, worst case: up to 1,500 extra charges per day

At a $50 average transaction value, this illustrative workload represents $25,000-$75,000 in erroneous charges daily. Even with immediate refunds, the customer support cost, trust damage, and chargeback fees can make it a high-severity incident.

Payment APIs such as Stripe's expose an Idempotency-Key header. The client generates a UUID for each logical payment attempt. If the same key arrives twice, the service can return the stored result instead of charging again.

Why It Happens

Five individually reasonable decisions compound into this anti-pattern. Each decision makes sense in isolation, but the combination creates a recurring duplicate-side-effect risk.

1. "We'll add retry logic later." The payment endpoint ships without idempotency. When timeouts appear in production, someone wraps the call in a retry loop. Now the system retries, but the server can't tell a retry from a new request.

2. "The network is reliable enough." In development, requests succeed 99.99% of the time. In production with mobile clients on cellular networks, 1-2% of requests fail at the TCP level. At 100K payments per day, that's 1,000-2,000 potential double-charges daily.

3. "The database has unique constraints." True for some columns, but a unique constraint on (user_id, amount) doesn't prevent a legitimate second purchase for the same amount. You need a unique constraint on a client-generated operation identifier.

4. "HTTP is request-response, so it's safe." Developers assume that if the client did not get a response, the server did not process the request. But the failure can happen after processing, before the response reaches the client. This assumption is a common cause of double-charge incidents.

5. "We'll deduplicate on the consumer side." Queue consumers assume the queue delivers each message exactly once. Kafka and SQS both deliver at-least-once. Without consumer-level idempotency, every redelivery is a duplicate side effect.

How to Detect It

Code smell: Any retryable POST endpoint that creates a side effect (charge, email, job submission) without accepting or deriving a stable operation key.

Architecture smell: Retry middleware wrapping calls to services that do not support idempotency. A retry decorator can silently repeat a payment or other side effect for weeks before the duplicate is noticed.

Quick audit: Search your codebase for retry loops (retry, backoff, attempt) and check whether the target endpoint accepts an idempotency key. Any retry without an idempotency key is a duplicate-side-effect bug waiting to happen.

// BAD: retry wrapper on non-idempotent endpoint
const result = await retry(3, () => api.post('/payments', { amount: 99 }));

// GOOD: retry with idempotency key
const key = generateUUID();
const result = await retry(3, () =>
  api.post('/payments', { amount: 99, idempotencyKey: key })
);

The Fix

After: retrying with a stable key

The core idea: give every operation a unique identity so the server can tell "new request" from "retry of the same request."

Three fixes, depending on where in the stack you're working: client-generated keys for API endpoints, server-side dedup stores for persistence, and ON CONFLICT for message consumers.

Fix 1: Client-generated idempotency key

Generate a UUID for each logical operation before sending the request. Use the same UUID on all retries of that same logical operation. Generate a new UUID for a new operation.

The key insight: the client, not the server, generates the key. The client knows whether it's retrying the same operation or starting a new one. The server can't make that distinction without the key.

async function chargePayment(userId: string, amount: number): Promise<Receipt> {
  const idempotencyKey = generateUUID(); // generate ONCE per user intent

  for (let attempt = 0; attempt < 3; attempt++) {
    try {
      return await paymentApi.post('/payments', {
        userId,
        amount,
        idempotencyKey // same key on every retry
      });
    } catch (e) {
      if (e.status === 400) throw e; // bad request, don't retry
      await sleep(exponentialBackoff(attempt));
    }
  }
}

Fix 2: Server-side deduplication store

The server checks the idempotency key against a persistent store before processing. If the key exists, return the cached response. If not, process the request and store the key + response atomically.

The idempotency table schema:

CREATE TABLE idempotency_keys (
  key         TEXT PRIMARY KEY,
  response    JSONB NOT NULL,
  status      TEXT NOT NULL DEFAULT 'completed', -- 'completed' or 'failed'
  created_at  TIMESTAMPTZ NOT NULL DEFAULT NOW(),
  expires_at  TIMESTAMPTZ NOT NULL DEFAULT NOW() + INTERVAL '7 days'
);

-- Clean up expired keys daily
CREATE INDEX idx_idempotency_expires ON idempotency_keys (expires_at);

async function processPayment(req: PaymentRequest): Promise<Receipt> {
  const existing = await db.idempotency.findByKey(req.idempotencyKey);

  if (existing) {
    // Duplicate request: return cached response
    return existing.response;
  }

  // First time: process and store result
  const receipt = await chargeCard(req);

  await db.idempotency.save({
    key: req.idempotencyKey,
    response: receipt,
    createdAt: new Date(),
    expiresAt: addDays(new Date(), 7) // clean up after 7 days
  });

  return receipt;
}

The idempotency store requires a unique constraint on key. For an internal business write, reserve or store the key and update the business record in the same database transaction. An external card charge cannot participate in that transaction, so use the provider's idempotency facility or an outbox plus reconciliation workflow to cover the crash window.

Choosing a storage backend

For payment systems, co-locate the idempotency record with the business data when it protects an internal state change, and also pass a stable key to the payment provider when supported. The database transaction can make the local order state atomic, but it cannot make a remote network call atomic; failures still need provider-side deduplication or reconciliation.

Fix 3: Kafka consumer idempotency

For message consumers, use INSERT ... ON CONFLICT to deduplicate at the processing level:

async function handleOrderEvent(event: OrderEvent): Promise<void> {
  // Use event ID as natural idempotency key
  const result = await db.query(
    `INSERT INTO processed_events (event_id, processed_at)
     VALUES ($1, NOW())
     ON CONFLICT (event_id) DO NOTHING
     RETURNING event_id`,
    [event.id]
  );

  if (result.rowCount === 0) {
    // Already processed, skip
    return;
  }

  // First time processing this event
  await sendConfirmationEmail(event.orderId);
  await updateOrderStatus(event.orderId, 'confirmed');
}

This helps when auto.commit or a rebalance causes redelivery. The database marker is the source of truth for processing attempts, not the consumer offset; the marker and the side effect still need an atomic handoff or an explicit state machine.

For high-throughput consumers, batch the inserts: check a batch of event IDs against the processed_events table in one query, then only process the ones that haven't been seen.

Here's what the fixed flow looks like:

Where idempotency matters

Anywhere retries exist, idempotency matters:

The Kafka/message queue case is especially important. Consumer group rebalancing causes messages to be redelivered. Consumers must be idempotent at the processing level, not just at the queue level.

Choosing your idempotency strategy

Natural vs engineered idempotency

Some operations are naturally idempotent:

• PUT /resources/\{id\} replaces the full resource with the same data. Same result regardless of repeat count.
• DELETE /resources/\{id\} removes the resource. Deleting twice is the same as deleting once.

Some require engineering to be idempotent:

• POST /payments needs an idempotency key.
• POST /emails needs a check for prior sends by message ID.
• Increment counter operations need conditional updates or CRDTs.

Severity and Blast Radius

Severity: High. Duplicate side effects directly impact users and are difficult to reverse. A double charge requires a manual refund, customer support contact, and trust erosion. At scale, this becomes a compliance and legal risk.

Blast radius: Every retry path without idempotency is a potential duplicate. The impact varies by domain:

Recovery difficulty: Moderate to hard. Detecting duplicates after the fact requires log correlation across multiple systems. Reversing side effects (refunds, notification retractions) is operationally expensive. Some side effects (sent emails, triggered webhooks, shipped packages) cannot be reversed at all.

The worst part: without idempotency instrumentation, you may not even know duplicates are happening until a customer complains.

Trade-offs and Legitimate Exceptions

• Read-only operations. GET requests are naturally idempotent. No key needed.
• Naturally idempotent writes. PUT /users/123 { name: "Alice" } sets the resource to the same state regardless of how many times you call it. DELETE /resources/123 is also naturally idempotent (the second delete is a no-op).
• Low-stakes fire-and-forget operations. Logging, analytics events, and telemetry data can tolerate some duplication without business impact. Your analytics pipeline probably already deduplicates.
• Internal services with exactly-once producer guarantees. Kafka with enable.idempotence=true handles producer retries. But consumer processing still needs idempotency since rebalancing triggers redelivery.
• Prototyping and early-stage products. If you have 100 users and no payment processing, idempotency keys are premature optimization. Add them when you add retries or when the stakes increase.

The bottom line: if the operation has side effects, retries can happen, and the stakes are non-trivial, design an idempotency strategy. The cost is more than a UUID and a lookup: you also need key retention, request validation, concurrency control, and a policy for expired or failed attempts.

30-Second Explanation

Missing idempotency means a retry is indistinguishable from a new request. Give each logical operation a stable key, store the key with the result or processing state, and return the prior result when the key is seen again. Apply the same idea to queue consumers using a durable event ID.

5-Minute Explanation

Start at the retry boundary. The client or producer creates one key per user intent and reuses it for every retry; it must not generate a new key for each attempt. The server validates that a repeated key has the same request parameters, atomically claims or records the operation, and returns a completed result or an explicit in-progress/failed state.

For internal database work, the claim and business update can share a transaction. For remote providers, use their idempotency key or a durable outbox and reconciliation process because a local transaction cannot roll back a remote charge. For consumers, use the event ID with a unique constraint, and make the handoff to the side effect observable and retryable.

Common Mistakes and Misconceptions

• "A timeout means the operation failed." It means the caller does not know the outcome. The server may have committed the side effect before the response was lost.
• "A business unique constraint is enough." A unique (user_id, product_id) constraint can block a legitimate second purchase; it does not identify one logical request.
• "Generate a new key for every retry." That defeats deduplication. Generate the key once per logical operation and preserve it across retries and client restarts when the operation is still pending.
• "Check, then insert is atomic." Two concurrent requests can both pass a read-before-write check. Use a unique constraint, atomic claim, or an explicit in-progress record.
• "An idempotency key guarantees exactly-once delivery." It can suppress duplicate handling within the retention window; downstream providers, email systems, and partial failures still need their own guarantees or reconciliation.
• "Consumer offsets are deduplication." Offsets control delivery position, not whether a side effect already happened. Redeliveries remain possible.

Test Your Understanding

Recap

1. Non-idempotent operations + retries = duplicate side effects. This is a design bug, not a runtime edge case.
2. The idempotency key pattern: the client generates a UUID per logical operation, the server stores the key and returns the cached response on duplicates.
3. The idempotency store needs a unique constraint on the key and should be atomically updated with the operation it protects.
4. Kafka consumers must be idempotent because messages are redelivered on rebalance. Use INSERT ... ON CONFLICT DO NOTHING at the processing layer.
5. Naturally idempotent operations (PUT, DELETE) often don't need extra keys. POST endpoints that create side effects usually need an explicit idempotency strategy.
6. Store idempotency keys in PostgreSQL for financial operations (durability) and Redis for lower-stakes deduplication (speed).
7. The key is generated by the client because only the client knows whether this is a new operation or a retry of a previous one.

Related Concepts

• Outbox Pattern — Persists work durably when a database update must trigger a message or remote side effect.
• Saga Pattern — Coordinates multi-step workflows when one transaction cannot cover every participant.
• Message Queues — Explains at-least-once delivery and why consumers need deduplication.
• Consistency Models — Provides the vocabulary for ambiguous outcomes and asynchronous state.
• Stripe Idempotency Case Study — A concrete payment-provider example of stable request identity.