How payment systems handle retries without double-charging

The scenario

A customer taps Pay and the connection times out. The processor may have declined the request, may still be working, or may already have charged the card while the response was lost. Retrying blindly can create two charges; never retrying can lose a legitimate order.

The system needs a durable local state machine, a stable identity for the logical payment attempt, and a reconciliation path for ambiguous outcomes. Retries are a recovery mechanism, not proof that the first attempt failed.

30-second mental model

Give one logical payment attempt one stable idempotency key and one durable internal payment ID. Store the request before calling the processor, pass the same key on safe retries, and treat timeouts as unknown until a provider lookup or webhook resolves them. Keep capture, refund, inventory, and fulfillment as separate idempotent transitions.

5-minute end-to-end flow

1. Create a pending payment record with an authenticated, stable operation key and the amount/currency snapshot.
2. Send the authorization or payment request to the processor with that key, using bounded exponential backoff only for retryable failures.
3. Persist the provider reference and outcome when a response arrives; never infer success from a client timeout.
4. Use provider webhooks as a fast signal and a periodic reconciliation job as the independent source of recovery for stuck payments.
5. Advance the local state machine exactly once per transition, then publish fulfillment work through an outbox or equivalent durable handoff.
6. Expose a stable status to the customer and support retries, refunds, and manual review without creating a second logical payment.

The Architecture

Here is the full picture of how a payment moves from the customer's browser through your server to Stripe and back. The critical detail is that every retry carries the same idempotency key, so Stripe can detect and deduplicate at the API gateway level.

Walk through what happens on a retry. The customer clicks "Pay Now," the request times out, and the client retries with the same idempotency key:

1. First attempt: The client generates a UUID (e.g., pay_abc123) and sends it with the charge request. Your server creates a "pending" payment record with this key. Stripe receives the charge, processes it, and stores the result keyed to pay_abc123.
2. Network timeout: Your server never receives Stripe's response. From your perspective, the payment is in an unknown state.
3. Retry (same key): The client retries with the same pay_abc123. Your server sees the pending record and forwards to Stripe with the same key. Stripe's gateway looks up pay_abc123, finds the cached result, and returns it without processing a new charge.
4. Resolution: Your server receives the original result, updates the state to "captured," and confirms to the client.

The customer is charged exactly once. The retry is safe because both your server and Stripe use the idempotency key to detect the duplicate.

The important detail is something that trips up a lot of engineers: the idempotency key and the payment intent ID are different things. The idempotency key is a deduplication token that you generate before the request. The payment intent ID is Stripe's internal identifier for the charge, which you only get back in the response. During a timeout, you have the idempotency key but you might not have the payment intent ID. This is why the idempotency key is the primary lookup mechanism for retries, not the payment intent ID.

Retry Timing: Exponential Backoff with Jitter

When a payment request fails, you do not retry immediately. Immediate retries during an outage create a thundering herd that makes the outage worse. The standard approach is exponential backoff with jitter.

The formula: delay = min(base * 2^attempt + random_jitter, max_delay)

For payment retries specifically:

• Base delay: 1 second
• Max delay: 30 seconds
• Max attempts: 3 (for synchronous retries to the user) or 5 (for background reconciliation)
• Jitter: Random value between 0 and the current delay (full jitter)

import random
import time

def retry_with_backoff(func, max_attempts=3):
    for attempt in range(max_attempts):
        try:
            return func()
        except TimeoutError:
            if attempt == max_attempts - 1:
                raise  # Last attempt, propagate the error
            delay = min(1 * (2 ** attempt), 30)
            jitter = random.uniform(0, delay)
            time.sleep(delay + jitter)

The jitter is critical. Without it, if 1,000 requests fail at the same time (Stripe momentarily unavailable), all 1,000 retry at exactly 1 second, then exactly 2 seconds, then exactly 4 seconds. With jitter, the retries are spread across the entire window, reducing peak load by 50-80%.

For payment retries, A practical default is to a maximum of 3 synchronous retries (the user is waiting). If all 3 fail, create the payment in "pending_confirmation" state and let the background reconciliation job handle it. The user sees "Payment processing" instead of waiting for more retries.

The Two-Phase Payment State Machine

This is the backbone of reliable payment processing. Every payment must move through explicit states, and each transition must be atomic and auditable. Without a state machine, you cannot reason about what happened during a failure.

The key insight here is the distinction between authorization and capture. In a two-phase payment flow, the bank "holds" the funds during authorization but does not transfer them until capture. This separation gives you a safety window.

If the authorization succeeds but your server crashes before recording it, you can query Stripe's API using the idempotency key to recover the state. If the capture request fails, you can retry it safely because capture is idempotent by nature (you cannot capture the same authorization twice for different amounts).

Here is the practical difference between single-phase and two-phase flows:

A practical default is to two-phase flows for any payment above $100 or any multi-step checkout. The ability to void an authorization without the customer seeing a charge-and-refund on their statement is worth the added complexity.

The Reconciliation Loop in Detail

The reconciliation loop is the unsung hero of payment reliability. It is a background job that continuously sweeps for payments in ambiguous states and resolves them by querying the source of truth (Stripe).

Here is the exact logic:

# Runs every 30 seconds via cron or task scheduler
def reconcile_stuck_payments():
    # Find payments stuck in transitional states
    stuck = db.query("""
        SELECT * FROM payments
        WHERE status IN ('pending', 'pending_confirmation', 'authorized')
        AND updated_at < NOW() - INTERVAL '60 seconds'
        LIMIT 100
    """)

    for payment in stuck:
        try:
            # Query Stripe for the actual state
            intent = stripe.PaymentIntent.retrieve(
                payment.stripe_intent_id
            )
            # Update local state to match Stripe's truth
            if intent.status == 'succeeded':
                transition(payment, to='captured')
            elif intent.status == 'canceled':
                transition(payment, to='failed')
            elif intent.status == 'requires_payment_method':
                transition(payment, to='failed')
            # If still processing at Stripe, leave it and check next cycle
        except stripe.NotFoundError:
            # Stripe has no record. The charge was never created.
            transition(payment, to='failed')

The key design decisions in the reconciliation loop:

• Batch size limit (100): Prevents the job from overwhelming Stripe's API with queries during a mass failure event.
• 60-second threshold: Gives normal requests enough time to complete before flagging them as stuck. Too short (5 seconds) and you get false positives during slow network conditions.
• Idempotent transitions: Calling transition(payment, to='captured') when the payment is already captured is a no-op. This makes the reconciliation job itself safe to run multiple times.

The key point: mention that the state machine is not just for tracking payments. It is the mechanism that makes reconciliation possible. Without explicit states and transitions, you cannot build a reconciliation loop that resolves ambiguous payments.

Idempotency Key Storage and Deduplication

The idempotency key is a client-generated UUID that travels with the payment request through every layer. Understanding where and how it is stored is what separates a surface-level answer from a production-level one.

The deduplication happens at two independent layers:

Layer 1: Your server. When a request arrives, your server checks the payment database for the idempotency key. If it finds a completed payment, it returns the cached result immediately without calling Stripe. If it finds a pending payment, it knows a previous attempt is in progress and can either retry the Stripe call (safe, because Stripe also deduplicates) or wait for the reconciliation job.

Layer 2: Stripe's API gateway. Stripe stores every idempotency key for 24 hours. If the same key arrives twice, Stripe returns the cached response from the first request without processing a new charge. This is the last line of defense.

Handling Partial Failures in Multi-Step Payments

This is where the real complexity lives, and where most engineers lose the thread. A payment is rarely a single API call. In production, a typical checkout involves: creating a payment intent, authorizing the card, capturing funds, creating an order record, sending a confirmation email, and updating inventory. When step 3 succeeds but step 4 fails, you have a consistency problem.

The concrete issue is why this is hard. You have two databases: Stripe's (where the charge lives) and yours (where the order lives). There is no distributed transaction that spans both. You cannot wrap stripe.charge() and db.create_order() in the same ACID transaction. They are independent systems connected by a network that can fail at any point.

The failure modes are asymmetric. If your database write fails, you can retry it (idempotent INSERT with a unique order key). But if the Stripe charge fails, you cannot "un-fail" your database write, you have to roll it back or mark it as "payment_pending." The order of operations matters: charge first means you might take money with no order, and create order first means you might have an order with no payment.

Neither order is perfect, which is why the outbox pattern exists. It decouples the two operations by making the local database write the atomic commit point and handling the external API call asynchronously.

The fundamental tension: the payment processor (Stripe) and your application database are two separate systems. There is no distributed transaction that atomically commits to both. You must design for the case where one succeeds and the other fails.

Here is a table showing the failure permutations and how to handle each one:

Bottlenecks, failure modes, and operations

1. Clock skew and deduplication windows. Stripe's 24-hour idempotency window is measured by Stripe's clock, not yours. If your server's clock is ahead by 30 minutes, you might think you are within the window when Stripe has already expired the key. Always reconcile ambiguous payments within a few minutes, not hours.

2. Concurrent retries from the same client. A user double-clicks the "Pay Now" button. Two identical requests hit your server simultaneously with the same idempotency key. Your database's unique constraint handles this: one INSERT succeeds, the other fails. But you need to handle the constraint violation gracefully (return a 409 Conflict or wait and return the result of the first request), not crash.

3. Idempotency key reuse across different operations. If the client reuses the same idempotency key for a different amount or different customer, Stripe will return the cached response from the first request. The amounts will not match. Your server must validate that the cached response matches the current request parameters, or reject the retry.

4. Authorization holds and expiry. When you authorize a card, the bank places a hold on the funds. This hold expires after a period (typically 7 days). If you do not capture within that window, the authorization expires and you must re-authorize, which might fail if the customer's balance has changed. Your state machine must handle the authorized → expired transition.

5. Bank-level deduplication. Beyond your server and Stripe, the issuing bank has its own dedup logic based on authorization codes. If two charges arrive with the same amount, merchant, and card within a short window, some banks will flag or decline the second as a potential duplicate. This is a third layer of protection, but it is not reliable or consistent across banks.

6. Currency and amount precision. Payment amounts must be stored in the smallest currency unit (cents for USD, pence for GBP). Storing $50.00 as a floating-point number introduces rounding errors that can cause mismatches between your records and Stripe's records. Stripe uses integer cents: amount: 5000 means $50.00. Always mirror this representation in your database.

7. Webhook vs polling for status updates. Stripe sends webhooks for payment status changes, but webhooks can be delayed, duplicated, or lost. Your reconciliation job should not depend solely on webhooks. It should also poll Stripe's API for payments stuck in ambiguous states. Treat webhooks as an optimization (faster notification), not a guarantee.

Common mistakes and misconceptions

Practical checklist

• Create one durable payment record before the provider call and bind it to the authenticated customer/order.
• Use a stable, scoped idempotency key for the logical operation; capture, refund, and fulfillment need their own state and keys.
• Classify errors into retryable, terminal, and unknown; never treat a timeout as proof of decline.
• Bound synchronous retries with exponential backoff and jitter, then hand ambiguous work to reconciliation.
• Persist provider references and raw status transitions needed for support and audit without logging sensitive payment data.
• Use webhooks for prompt updates and polling/reconciliation for recovery from delayed or missing notifications.
• Advance fulfillment only after the payment state is authoritative enough for the business rule.
• Test lost responses, duplicate submissions, worker crashes, provider outages, late webhooks, and manual review.

Test Your Understanding

Quick Recap

1. Network timeouts are ambiguous: you do not know if the payment processor received your charge request, so you must design every operation to be safely retryable.
2. The client generates the idempotency key (a UUID) before sending the request, ensuring retries always carry the same identifier regardless of which server handles them.
3. Your server stores the key in the payment database with a UNIQUE constraint, providing the first layer of deduplication that prevents concurrent duplicates.
4. Stripe (and other processors) store the key for 24 hours and return cached responses on duplicate requests, providing the second layer of deduplication.
5. Every payment moves through a state machine (pending, authorized, captured, settled, failed), and each transition is recorded so you can always determine the current state.
6. A reconciliation job runs every 30-60 seconds, finding stuck payments and querying the payment processor to resolve ambiguous states, so no payment stays unknown for long.
7. For multi-step flows (charge + order + email + inventory), the transactional outbox pattern ensures atomicity: commit the order and outbox events in one database transaction, and process downstream steps asynchronously with independent idempotency.
8. The fundamental principle: make one system the source of truth (your database for business logic, Stripe for charge state), and treat everything else as an eventually-consistent projection that converges through retries and reconciliation.

Related Concepts

• Idempotency keys (covered in detail in the companion article) are the mechanism that makes retries safe at the API level. Understanding the key lifecycle, storage, and race conditions is essential for implementing payment retries correctly. The idempotency article covers the general pattern applicable to any API, while this article focuses on the payment-specific nuances.
• The saga pattern coordinates multi-step transactions across services using compensating actions. Payment flows with authorization, capture, order creation, and fulfillment are classic saga engineers. If any step fails, the saga runs compensating transactions (void the authorization, cancel the order) to restore consistency.
• The outbox pattern ensures that local database writes and external side effects (like charging a card) happen reliably by writing events to an outbox table inside the same transaction. A background worker reads the outbox and processes the external calls with retry logic. This decouples your database commit from the Stripe API call.
• Exponential backoff with jitter is the retry strategy that prevents thundering herd problems when multiple clients retry simultaneously after a payment processor outage. The jitter component (random delay added to each retry) spreads the load and reduces the probability of synchronized retry storms.
• Circuit breakers protect your payment service from cascading failures when the upstream processor (Stripe, bank network) is degraded. When Stripe starts returning errors above a threshold, the circuit breaker opens and immediately returns errors to clients instead of queueing up requests that will fail, giving Stripe time to recover.
• Event sourcing is an alternative to the state machine approach where every payment state transition is stored as an immutable event. The current state is derived by replaying events. This gives you a complete audit trail and the ability to reconstruct the payment's history at any point, which is valuable for compliance and debugging.