Payment Processing

What is a payment processing system?

A payment processor sits between a merchant's checkout page and the card networks (Visa, Mastercard, Amex). It routes authorization requests to the right network, records the result, and handles the inevitable failures: timeouts, partial captures, refunds, and retries. The happy path is straightforward; the design is defined by what happens when the network accepts a request but the response is delayed or lost.

The interesting engineering challenges live off the happy path: ensuring a charge fires exactly once when the client retries a timed-out request, handling the limbo state where the card network accepted a charge but your service never received the response, and scaling the synchronous authorization pipeline to 10K TPS during a flash sale. This design builds a direct card network integration, not a wrapper around Stripe or Braintree; the interesting distributed systems problems only surface when you own the network call yourself.

TL;DR

Require a client-generated Idempotency-Key for every charge and refund. Record the intent transactionally in PostgreSQL, then process it through a durable queue. A worker writes AUTHORIZING before the external call, uses a stable network reference, and records AUTHORIZED, FAILED, or UNKNOWN with an immutable PaymentEvent.

Never treat a network timeout as a decline and never blindly re-authorize an UNKNOWN payment. Reconciliation queries the card network and resolves the state. Keep status reads strongly consistent, isolate card-network slowness with queue backpressure and bulkheads, keep raw card numbers out of the system through tokenization, and make refunds separate idempotent financial operations.

Scope and Assumptions

This design assumes:

• The processor integrates directly with a card network or bank rail using an opaque card token; raw PAN storage and tokenization-vault internals are out of scope.
• Charges, automatic capture, full or partial refunds, and authoritative payment-status reads are in scope. Settlement, reconciliation against merchant ledgers, disputes, and chargebacks are extension points beyond the core flow.
• The illustrative workload is 5M merchants, 50M cardholders, and bursts up to 10K transactions per second. The system accepts requests quickly and processes external authorization through workers with a bounded concurrency/rate limit.
• Payment state transitions and money amounts require strong consistency and an immutable audit trail. External network outcomes can still be temporarily unknown and must be reconciled rather than guessed.
• Merchant authentication, fraud scoring, and PCI controls are represented at their integration boundaries; the processor never trusts a client-supplied merchant identity in the request body.

Functional Requirements

Core Requirements

1. Merchants can initiate a charge with a card token, amount, and currency.
2. The system authorizes and captures the charge via the card network.
3. Merchants can issue full or partial refunds on a captured payment.
4. Merchants can query the current status of any payment.

Below the Line (out of scope)

• PCI-DSS card data storage. We do not store raw card numbers; a tokenization vault (like Stripe's vault or Braintree) converts card details to an opaque token before they reach our system.
• Fraud detection ML model internals.
• End-to-end settlement and ledger reconciliation.
• Dispute and chargeback management.

The hardest part in scope: Exactly-once charge execution. A client that retries after a network timeout must not trigger a second charge. A charge response that gets dropped in transit must not leave the payment in an ambiguous permanent state. The idempotency key mechanism and the payment state machine together solve both problems, and each gets a full deep dive.

PCI-DSS storage is below the line because storing raw card numbers expands the compliance scope far beyond the distributed systems challenge. Integrate a tokenization vault so the processor never sees the actual card number. The opaque card token received by this service is useless without the vault.

Fraud detection is below the line because it runs as a scoring service, not a core payment-path component. To add it, call a fraud score API synchronously before sending the auth request to the card network and reject any charge above a configured risk threshold.

Settlement and reconciliation are below the line because they run as a daily batch pipeline against completed transactions, not a real-time flow. They do not affect the charge or refund paths we are designing.

Dispute and chargeback management is below the line because it is a human-assisted process triggered by customer disputes through their bank, not something our payment API initiates directly.

Non-Functional Requirements

Core Requirements

• Exactly-once delivery: A charge must complete exactly once regardless of how many times the client retries due to network failures or timeouts.
• Strong consistency: The payment state stored in our database must agree with what the card network recorded. Stale state is not acceptable for financial data.
• Availability: 99.99% uptime for the charge and refund endpoints, under 52 minutes of downtime per year.
• Latency: Charge API returns in under 2 seconds p99. Card network authorization adds 100-300ms of unavoidable latency; our infrastructure must not contribute more than an additional 200ms on top of that.
• Scale: 5M active merchants globally, 50M active cardholders. Peak 10K transactions per second during flash sales (Black Friday, holiday surges). That peak implies roughly 1B transactions per day during sustained burst periods.
• Auditability: Every payment state transition is written to an immutable event log and must be queryable indefinitely.

Below the Line

• Sub-100ms charge response time (card networks impose unavoidable latency)
• Multi-region active-active with synchronous cross-region consistency guarantees

Read/write ratio: Payments are write-skewed at the state machine level. Each transaction produces 3-4 state transitions (PENDING to AUTHORIZING to AUTHORIZED to CAPTURED), and each transition produces an immutable PaymentEvent record. The write-to-read ratio on the payments table is roughly 4:1. The strong consistency requirement means we cannot serve reads from an eventually-consistent read replica; every status query must reflect the current authoritative state. We do not apply aggressive caching to payment records.

The 2-second p99 latency target defines the timeout strategy. A 1-second card-network timeout leaves enough budget to write UNKNOWN and return before breaching the 2-second SLA. That is why the "unknown state" problem in Deep Dive 2 exists: the hard timeout is non-negotiable.

The 99.99% availability on a synchronous path with an external dependency (the card network) means we cannot let card network slowness cascade to our uptime SLA. Bulkheads and fallback logic must isolate card network outages from the payment recording path.

Core Entities

• Payment: The transaction record. Carries the payment ID, merchant ID, amount, currency, card token, current state, idempotency key, network reference ID (set before calling the card network), and timestamps.
• PaymentEvent: An immutable record of a single state transition on a Payment. Every time a payment changes state, we append one PaymentEvent. This is the audit trail and the source of truth for what happened and when.
• Refund: A credit linked to a captured Payment. Carries a refund ID, parent payment ID, amount, reason, and current state (PENDING, SUCCEEDED, FAILED).

The full schema, indexes, and constraints are deferred to the data model deep dive. Keep the Payment entity flat here; the audit trail lives in PaymentEvents, not in versioned columns on the payment row. The three entities above are sufficient to drive the API and the High-Level Design.

API Design

FR 1 - Initiate a charge:

POST /payments
Headers: Idempotency-Key: <client-generated uuid>
Body: { card_token, amount, currency, description? }
Response: { payment_id, status }

The Idempotency-Key header is required, not optional. Without it, retrying a timed-out request creates a duplicate charge. The key is a UUID the client generates before sending; any retry uses the exact same key. This is the pattern Stripe uses and the one we replicate here.

FR 2 - Get payment status:

GET /payments/{payment_id}
Response: { payment_id, status, amount, currency, created_at, events: [...] }

FR 3 - Issue a refund:

POST /payments/{payment_id}/refunds
Headers: Idempotency-Key: <client-generated uuid>
Body: { amount?, reason? }
Response: { refund_id, status }

POST /payments/{payment_id}/refunds uses POST rather than DELETE because a refund is a new financial operation, not a reversal of the charge at the resource level. A full refund does not delete the payment; it creates a Refund entity linked to the original. Partial refunds make this unambiguous: you cannot DELETE /payments/{id} by 30%.

The amount field is optional. If omitted, the system defaults to a full refund of the captured amount. The service validates that the sum of all prior refunds plus this refund does not exceed the original captured amount before calling the card network.

If authentication were in scope, add a merchant_id claim to the request context from the auth token and scope all payment lookups to that merchant. Do not add a merchant_id field to the request body because that is a privilege-escalation risk.

30-Second Answer

1. Require an idempotency key, validate the token/amount/currency, and insert one PENDING Payment plus its initial event under a unique merchant-scoped key.
2. Enqueue the work durably and return the payment_id; a worker transitions to AUTHORIZING before calling the card network.
3. Record AUTHORIZED and capture, FAILED on a definitive decline, or UNKNOWN when the network outcome cannot be determined within the timeout.
4. Reconcile UNKNOWN payments through the network status API. A retry with the same key reads the existing payment instead of creating another charge.
5. Model refunds as separate idempotent Refund records, keep the database as the status source of truth, and use rate limits, bulkheads, and a DLQ to absorb bursts and failures.

5-Minute Explanation

The central invariant is one merchant intent maps to one Payment record and one externally addressed charge attempt. The API therefore requires an idempotency key and uses a database uniqueness constraint plus a transaction. Concurrent retries return the existing record or wait for its in-progress state; they do not start another charge.

The accept phase is short: validate the request, write PENDING and an audit event, enqueue a charge job, and return the payment ID. The worker owns the process phase. It writes AUTHORIZING before making the network call, sends a stable network reference, then records the definitive result. A timeout is information loss, not a decline, so the worker marks UNKNOWN and a reconciliation job queries the network before resolving it.

Capture and refunds are explicit financial operations. Automatic capture can follow authorization for the simple checkout case; delayed capture can be added as another state transition. A refund creates a linked Refund record, validates the cumulative amount under a transaction, and runs through its own idempotent queue and network call.

At 10K TPS, the queue provides backpressure and the worker pool is rate-limited to what the card network and credential set can sustain. PostgreSQL remains authoritative for Payment, Refund, and PaymentEvent state. Reads do not use a stale cache, and operational tooling focuses on unknown-state age, queue age, reconciliation, auditability, and safe handling of tokenized payment data.

45-Minute Interview Approach

Use this agenda to answer the design question; spend most of the time on idempotency, unknown outcomes, and bounded external calls:

• 0–5 minutes — Clarify the contract: Confirm direct network integration versus a provider, authorization/capture timing, refund semantics, payment methods, merchant authentication, synchronous versus asynchronous responses, and the SLA.
• 5–10 minutes — Establish scale and invariants: Use the 10K TPS burst, network latency, 99.99% availability, strong consistency, immutable audit, and the invariant that a retry must not create a second charge.
• 10–15 minutes — Define entities and APIs: Introduce Payment, PaymentEvent, Refund, POST /payments with Idempotency-Key, status reads, and refund creation. State the merchant scoping rule.
• 15–22 minutes — Draw the accept path: Show validation, idempotency lookup/unique insert, PENDING, transactional event recording, durable enqueue, and the fast response.
• 22–30 minutes — Draw the process path: Show AUTHORIZING before the network call, stable network reference, success/decline transitions, capture, retries, and the UNKNOWN branch.
• 30–35 minutes — Deep dive on correctness: Explain concurrent retries, timeout ambiguity, reconciliation, state-machine guards, and why a timeout must never trigger blind re-authorization.
• 35–41 minutes — Cover refunds and operations: Add partial-refund limits, queue backpressure, network bulkheads, DLQs, audit retention, metrics, tokenization, secret handling, and failure runbooks.
• 41–45 minutes — Close with trade-offs: Compare synchronous and queued processing, direct network and provider integration, and state storage options. Recap the invariants and invite extensions such as fraud, settlement, and chargebacks.

High-Level Design

Critical flows

The critical flows are idempotent charge acceptance, authorization/capture with an external network, reconciliation of uncertain outcomes, and idempotent refunds. The numbered designs below separate intent recording from the external side effect so each failure window is visible.

FR 1 - Accept a charge and record it

The simplest starting point: record the intent to charge before touching any external system. The client sends a charge request; the Payment Service writes the payment to the database in PENDING state and returns a payment_id. No card network interaction yet.

Components:

• Merchant App: Sends the charge request with a card token, amount, and Idempotency-Key header.
• Payment Service: Validates the request, checks the idempotency key, writes the payment record in PENDING state.
• Payments DB (PostgreSQL): Stores the payment with a UNIQUE constraint on idempotency_key.

Request walkthrough:

1. Client sends POST /payments with card token, amount, currency, and Idempotency-Key.
2. Payment Service validates the request (positive amount, supported currency, non-empty card token).
3. Payment Service executes INSERT ... ON CONFLICT DO NOTHING on the idempotency key.
4. Payment Service returns { payment_id, status: "PENDING" }.

This diagram covers only the intent-recording step. The card network is not involved yet; we just know a merchant wants to charge a customer.

FR 1 (continued) - Authorize the charge with the card network