A/B Testing Platform

What is an A/B testing system?

An A/B testing platform assigns users to experiment variants (control versus one or more treatments) and measures whether a treatment changes a business metric. The interesting engineering challenges are not limited to statistics: the platform must keep assignment stable without a database lookup on every request, isolate concurrent experiments from avoidable interaction, and stream exposure and conversion events into per-variant aggregates.

The design therefore treats assignment and configuration distribution as a hot read path, while treating metrics as an asynchronous data path. Correct assignment comes first; statistical analysis consumes the resulting exposure and outcome aggregates.

TL;DR

Use a control plane backed by a relational database for experiment definitions, variants, targeting, and lifecycle state. Publish a versioned configuration snapshot to a CDN and local SDK caches. Once a client has the snapshot, a stable hash of the canonical user ID, experiment ID, and (when needed) layer ID maps the user to a bucket without a network or database call.

Record the actual exposure and later goal events with client-generated IDs, send them through Kafka, and let a metrics worker perform an idempotent temporal join. Use layers to make potentially interacting experiments mutually exclusive while keeping independent layers orthogonal. Lock the configuration version for a browser session when visual consistency matters.

Scope and assumptions

This article uses the following illustrative interview baseline; the values are capacity-planning inputs, not guarantees about a particular vendor or deployment:

• Up to 500 active experiments, 50 million daily active users, and approximately 50,000 assignment lookups per second at peak.
• Browser, mobile, and server SDKs have a stable canonical user ID after identity resolution. Anonymous users use a stable device or session identifier until they are linked to an account.
• Experiment configuration changes are infrequent compared with assignment reads. A few seconds of configuration propagation and up to five minutes of metric freshness are acceptable except for an explicit emergency kill switch.
• The platform returns raw exposures, conversions, and value aggregates. Statistical tests, bandit allocation, bot classification, and a full real-time dashboard are outside the primary design.
• A client should prefer its last known valid configuration during a temporary config outage; an explicit control fallback is used when no valid configuration is available.

Functional Requirements

Core Requirements

1. Experiment owners define variants (control and one or more treatments) with traffic allocation percentages and targeting rules.
2. The system assigns a user to exactly one variant per experiment, and that assignment never changes for the duration of the experiment.
3. Client SDKs retrieve all active experiment assignments for a user in under 10ms.
4. Success metrics (impressions, conversions, revenue) are collected and computed per variant.

Below the Line (out of scope)

• Statistical significance calculation and p-value computation (delegate to a stats library such as SciPy or statsmodels).
• Multi-armed bandit and Bayesian optimization.
• Fraud and bot filtering for experiment traffic.
• Real-time metric dashboards (accept up to 5-minute lag in aggregates).

The hardest part in scope: Consistent user assignment without a per-request database lookup. This is the constraint that shapes the entire read path. If assignment is slow, every page load in your product is slow.

Statistical significance calculation is below the line because the platform only needs to produce the raw aggregates (exposures and conversions per variant). Calling scipy.stats.chi2_contingency on those aggregates is a one-line operation any experiment analyst can run outside the platform. Designing the significance engine adds months of complexity for a feature that any analyst can replicate locally in 30 seconds.

Multi-armed bandit optimization is below the line because it requires changing variant weights mid-experiment based on incoming results, which breaks the assumption that assignment probabilities are stable for the analysis window. Designing that safely (avoiding peeking problems and inflated false-positive rates) is a research-level problem that deserves its own article.

Fraud and bot filtering is below the line because it requires ML-based classification that sits orthogonal to the assignment and metrics pipeline. An extension could enrich incoming exposure events with a bot score from a separately evaluated classifier and filter or segment those events during analytics aggregation.

Non-Functional Requirements

Core Requirements

• Assignment consistency: Once a user is assigned to a variant, they see the same variant for the experiment lifetime. No flipping mid-experiment.
• Availability: 99.99% uptime for the assignment path. A failure in the assignment logic must fall back to the control variant, never crash the client.
• Latency: SDK retrieves all active experiment assignments in under 10ms p99. This is the NFR that drives every architectural decision on the read path.
• Scale: 500 concurrent active experiments, 50 million DAU, peak assignment lookup rate of approximately 50,000 requests per second. Each experiment write (create, update, launch) happens at most a few times per day per experiment.

Below the Line

• Sub-5ms assignment latency via pure in-process computation (covered in the deep dive but not a primary target).
• Sub-second metric freshness (5-minute lag is acceptable).

Read/write ratio: For every experiment created or updated (roughly 100 writes per day across all experiments), there are approximately 50,000 assignment lookups per second. That is a read/write ratio of roughly 40 million to 1. This extreme imbalance means the assignment path must never touch the primary database. Every design decision on the read path exists to eliminate that database call.

Under 10ms assignment latency means a round trip to a remote cache is already risky. A Redis lookup adds 1ms under ideal conditions, but p99 latency on a busy cluster can spike to 5-10ms, consuming the entire budget.

Call out this latency budget early because it eliminates most caching architectures before the diagram is drawn. The safe approach serves experiment configs from an in-process SDK cache seeded by a CDN-backed config endpoint. Assignment computation then becomes a pure in-memory operation measured in microseconds.

30-second answer / outline

1. Put experiment definitions, variants, targeting, layers, and lifecycle state in a relational control-plane database.
2. Publish an immutable, versioned active-config snapshot through a CDN; SDKs keep it in memory and use conditional requests to refresh it.
3. Compute assignment with a stable, versioned hash of the canonical user identity and experiment inputs. Lock the config version for a browser session when visual flicker is unacceptable.
4. Emit exposure and goal events to Kafka. A metrics worker joins them by user and experiment in event time, then writes idempotent aggregates to an analytics store.
5. Use layer policy for experiments that can interact, and keep the assignment service stateless. State the failure behavior: use a last-known config, or the configured control fallback when no valid config exists.

5-minute explanation

Start by separating the control plane from the serving and measurement paths. Experiment creation and launch are low-volume writes, so a relational database is a good source of truth. Assignment is a high-volume read: the client must obtain a coherent config snapshot, then do a deterministic in-process lookup rather than query storage on every request.

The critical flow is config snapshot → stable bucket → exposure event. The snapshot carries allocation ranges, targeting rules, layer membership, and a version. The SDK evaluates the same canonical inputs every time, records what was actually exposed, and keeps visual experiences on one config version for the session. This avoids both per-request storage and mid-session variant flips.

Measurement is deliberately asynchronous. The ingestion endpoint acknowledges after durable event-bus publication; a worker deduplicates event IDs, joins post-exposure goals to the relevant exposure, and updates per-variant rollups. The results API reads those rollups and can report their freshness rather than pretending that ingestion and analysis are synchronous.

At scale, layers make the statistical assumption explicit: experiments that can interfere share a mutually exclusive layer; experiments that are independent can run in separate layers. The remaining operational work is config propagation, cache fallback, consumer lag, late events, and an emergency kill switch. The detailed flows and alternatives below justify each choice.

45-minute interview approach

This is a time-boxed plan for answering the design question in an interview, not a claim that the article should be read in 45 minutes.

• 0–5 minutes — Clarify the contract: Confirm assignment identity, whether assignments must survive login/device changes, browser versus server SDKs, visual flicker tolerance, metric freshness, and whether statistical testing is in scope.
• 5–10 minutes — Establish scale: Use the illustrative active-experiment, DAU, lookup-rate, write-rate, and freshness assumptions. Separate the low-volume control plane from the read-heavy assignment path.
• 10–15 minutes — Define APIs and invariants: Walk through experiment creation/launch, config retrieval, assignment evaluation, event ingestion, and result queries. State that allocations validate to 100% and assignment inputs are versioned.
• 15–23 minutes — Draw the assignment path: Show the database, config publisher, CDN, SDK cache, deterministic hash, layer policy, and session lock. Explain cache miss and unavailable-config behavior.
• 23–31 minutes — Draw measurement: Show exposure/goal events, Kafka, deduplication, the temporal join, late arrivals, and analytics rollups. Explain why the results API is eventually fresh.
• 31–37 minutes — Deep dive on correctness: Choose deterministic hashing and layer-based isolation; discuss config changes, anonymous-to-known identity, and why pre-exposure conversions are excluded.
• 37–42 minutes — Reliability, security, and operations: Cover durable config writes, replayable events, cache fallbacks, access control, event privacy, consumer lag, config age, and the kill switch.
• 42–45 minutes — Trade-offs and close: Compare CDN versus Redis, streaming versus query-time aggregation, and layers versus manual exclusions. Recap the bottleneck and invite follow-up questions.

Core Entities

• Experiment: The container for a test. Carries a unique key, status (draft, active, paused, concluded), targeting rules, and the date range for the analysis window.
• Variant: One arm of an experiment (control or a named treatment). Carries a variant key, a traffic allocation percentage, and an arbitrary JSON config payload that the client SDK uses to alter the experience.
• Assignment: A durable record of which variant a specific user was placed into and when. Written at first exposure. Serves as the ground truth for metric attribution.
• Event: A user action used as a success metric. Carries a user ID, event name, optional numeric value (e.g. order amount), and a timestamp. Events are linked to assignments at aggregation time, not at tracking time.
• Metric: A named aggregation definition tied to an experiment variant. Stores the exposure count, conversion count, and optional sum (for revenue-type metrics) over the analysis window.

The full schema and column types will be revisited during the data model deep dive if scope expands to include it; the entities above are sufficient to drive the API design and High-Level Design.

API Design

FR 1 and FR 4 - Create and launch an experiment:

# Create a new experiment in draft status
POST /experiments
Body: {
  key: "signup_button_color",
  variants: [
    { key: "control",     allocation: 50, config: {} },
    { key: "treatment_a", allocation: 50, config: { button_color: "green" } }
  ],
  targeting: { platforms: ["web"], user_segments: ["new_users"] },
  metrics: ["signup_conversion", "revenue_30d"]
}
Response: { experiment_id, status: "draft" }

# Transition experiment from draft to active (launches it)
PATCH /experiments/{experiment_id}
Body: { status: "active" }
Response: { experiment_id, status: "active" }

PATCH over PUT for status transitions because we are modifying one field on an existing resource. Separating create from launch lets teams configure an experiment in draft before exposing it to users.

FR 2 and FR 3 - Retrieve assignments for a user:

# Fetch all active experiment assignments for a user in one call
GET /assignments?user_id={user_id}
Response: {
  assignments: {
    "signup_button_color": "treatment_a",
    "homepage_layout":     "control"
  }
}

The SDK calls this endpoint once per session (or polls periodically) and caches the result in memory. Returning all active experiment assignments in one payload avoids per-experiment round trips. An SDK making 500 separate calls for a user enrolled in 500 experiments would be unusable.

FR 4 - Track an event:

# Track batched user events; server returns after Kafka publish, not after aggregation
POST /events
Body: {
  user_id: "u123",
  events: [
    { name: "signup_conversion", timestamp: "2026-04-02T10:00:00Z" },
    { name: "purchase",          value: 49.99, timestamp: "2026-04-02T10:01:00Z" }
  ]
}
Response: { received: 2 }

Events are batched on the client SDK and flushed in bulk to reduce request overhead. The server validates schema and publishes to the event pipeline without waiting for downstream aggregation to complete. A 201 response confirms receipt, not processing.

FR 4 - Retrieve metric results for an experiment:

# Retrieve per-variant metric aggregates for an experiment
GET /experiments/{experiment_id}/results
Response: {
  variants: [
    {
      key: "control",
      exposures: 250000,
      metrics: {
        "signup_conversion": { conversions: 12500, rate: 0.050 },
        "revenue_30d":       { sum: 875000.00, mean_per_user: 3.50 }
      }
    },
    {
      key: "treatment_a",
      exposures: 250000,
      metrics: {
        "signup_conversion": { conversions: 15000, rate: 0.060 },
        "revenue_30d":       { sum: 1050000.00, mean_per_user: 4.20 }
      }
    }
  ]
}

This endpoint is read-only and expensive; cache its response for 60 seconds keyed on experiment ID to prevent analysts from triggering repeated full-table scans when refreshing the results page.

High-Level Design

Critical flows

Keep the design readable by following four flows in order: experiment write and launch, config distribution and assignment, event ingestion and metric aggregation, and cross-experiment isolation/config-version changes. The first flow establishes truth; the next two serve and measure it; the last protects interpretation and user experience.

1. Experiment owners define variants and targeting rules

The write path for experiment configuration. Admins create and launch experiments through a management API that writes to a relational database.

Components:

• Admin Client: The product team's web UI or CI/CD tooling sending experiment definitions.
• Experiment API: Validates variant allocation percentages sum to 100%, persists the experiment and variant records, and invalidates the config cache on any change.
• Experiment DB: The source of truth for all experiment definitions. Relational storage suits this well: experiments and variants are small structured records with clear relationships.

Request walkthrough:

1. Admin sends POST /experiments with variant definitions and targeting rules.
2. API validates that allocation percentages sum to exactly 100%.
3. API writes one row to the experiments table and one row per variant to the variants table.
4. API publishes a config-invalidation event so the cache tier reflects the new experiment.
5. API returns { experiment_id, status: "draft" }.
6. Admin sends PATCH /experiments/{id} with status: "active" to launch.

This is the write path only. The read path that distributes these configs to SDKs comes next.

2. Users are assigned to variants consistently

Every user request needs to know which variant to show. Calling the Experiment API on every request is the naive approach: at 50K assignment lookups per second, even a small increase in API latency directly degrades every page in the product. If the assignment path touches a database on every request, the design will not meet the stated scale or latency target.

The key insight is that assignment does not require a network call. If the experiment config is available locally (which variant holds which bucket range), assignment is a deterministic hash computation: hash(user_id + experiment_id) modulo 100, compared against the variant allocation ranges.

Components:

• Client SDK: An in-process library (exists in every service that needs assignments). Holds a local copy of the active experiment configs, refreshed periodically from the Config endpoint.
• Assignment Logic: Pure in-memory computation inside the SDK. No network call needed once configs are loaded.
• Config Cache (Redis): Serves as the intermediary between the Experiment DB and SDKs. The Experiment API invalidates this cache on every experiment change.

Request walkthrough: