Design a prompt management platform

30-second answer

• The core architectural insight is decoupling prompts from application code. Prompts live in a versioned registry, and applications resolve them at runtime via a sub-5ms API call through a Redis cache.
• Use a git-like versioning model with "staging" and "production" pointers per prompt. Every prompt change goes through an automated eval gate (200 test cases in under 2 minutes) before promotion, so bad prompts never reach users.
• A/B testing with Thompson Sampling (multi-armed bandit) converges to the winning variant in 2-3 days instead of the 14 days required by fixed traffic splits, reducing exposure to underperforming prompts by 80%.
• Prompt-model coupling is the sneaky production killer. A prompt optimized for GPT-4o may score 15-20% lower on Claude 3.5 Sonnet. Bind every prompt version to a specific model identifier.
• Prompt management is a release-engineering problem, not only an authoring-tool problem. Treat prompt changes like code deploys with staging, evaluation, canary rollout, and rollback.

Requirements and assumptions

Functional requirements

• Teams can create, edit, and version prompts through a web UI or API, with each edit producing an immutable version.
• The system resolves a prompt_id to the correct prompt text and parameters (model, temperature, max_tokens) at runtime, supporting per-environment overrides (staging vs production).
• Teams can run A/B tests between prompt versions with configurable traffic splits and automatic metric collection.
• Every prompt change triggers an automated eval suite against a test set before the prompt can be promoted to production.
• Teams can instantly roll back any prompt to a previous version without a code deploy.
• The system tracks per-version metrics (latency, cost, user satisfaction, eval scores) and surfaces them in a dashboard.

Non-functional requirements

• Prompt resolution P99 latency under 5ms (cached) and under 50ms (cold).
• Support 500 active prompts across 20 product teams with 10,000 resolution requests per second at peak.
• Eval suite execution completes 200 test cases in under 2 minutes.
• Rollback takes effect within 5 seconds across all application instances.
• 99.99% availability for the resolution API (this is on the critical path for every LLM call in the company).
• Prompt version history retained for 12 months for audit and compliance.

The hardest engineering problem here: A/B testing prompts is fundamentally different from A/B testing UI changes. Prompt outputs are non-deterministic, evaluation is subjective, and the feedback signal (is this response "good"?) arrives asynchronously or not at all. You need both automated eval metrics and human judgment, and you need a traffic allocation strategy that minimizes exposure to bad variants.

Assumptions

• A prompt version is immutable and includes the model and generation parameters that affect its behavior. Environment pointers select versions; they do not mutate them.
• Resolution is on the hot path for application requests, so the registry, cache, and configuration distribution must remain useful during an authoring or evaluation outage.
• Teams own the meaning of “good” for their use case. The platform supplies versioning, rollout, evaluation, and telemetry rather than one universal quality score.

5-minute approach

Separate the control plane from the resolution data plane. Authors create immutable prompt versions and evaluate them offline; the hot path resolves an environment pointer from a cache, returns the prompt plus parameters, and records the version used.

• Validate variables, output contracts, model compatibility, and required examples before a version can be promoted.
• Use atomic environment pointers, cache invalidation, and canary or A/B allocation for rollout; rollback is a pointer change, not a code deploy.
• Keep evaluation results and live metrics tied to prompt, model, dataset, and policy versions so comparisons are meaningful.
• Treat cache misses, stale configuration, and control-plane outages as explicit failure modes with a last-known-good policy.

The entities and endpoint below define the version, environment, experiment, and resolution contract.

Core entities

Prompt

• prompt_id, name, team_id, description, created_at, updated_at, production_version_id, staging_version_id

PromptVersion

• version_id, prompt_id, version_number, template_text, model_id (e.g., gpt-4o, claude-3.5-sonnet), temperature, max_tokens, system_prompt, few_shot_examples[], created_by, created_at, eval_status (pending/passed/failed), eval_score

EvalRun

• eval_run_id, version_id, test_set_id, total_cases, passed_cases, avg_score, p50_latency_ms, total_cost_usd, started_at, completed_at, gate_result (pass/fail/manual_review)

Experiment

• experiment_id, prompt_id, variants[] (each with version_id, traffic_weight), allocation_strategy (fixed_split/thompson_sampling), start_date, end_date, status (active/completed/stopped), winner_version_id

MetricEvent

• event_id, version_id, experiment_id, request_id, latency_ms, cost_usd, user_rating (1-5, optional), eval_score, timestamp

API design

POST /api/prompts - create a new prompt

Request: {
  "name": "customer-support-classifier",
  "team_id": "team_support",
  "description": "Classifies inbound tickets into categories"
}
Response: {
  "prompt_id": "prm_abc123",
  "name": "customer-support-classifier",
  "created_at": "2026-04-11T10:00:00Z"
}

POST /api/prompts/:prompt_id/versions - create a new version of an existing prompt

Request: {
  "template_text": "Classify the following customer message into one of: {{categories}}.\n\nMessage: {{message}}\n\nCategory:",
  "model_id": "gpt-4o-mini",
  "temperature": 0.0,
  "max_tokens": 50,
  "system_prompt": "You are a precise classification assistant.",
  "few_shot_examples": [
    { "input": "My order hasn't arrived", "output": "shipping" },
    { "input": "I want a refund", "output": "billing" }
  ]
}
Response: {
  "version_id": "ver_def456",
  "version_number": 12,
  "eval_status": "pending"
}

GET /api/resolve/:prompt_id - runtime resolution endpoint (the hot path)

Query params: ?env=production&variables={"categories":"shipping,billing,technical","message":"My order is late"}
Response: {
  "version_id": "ver_def456",
  "resolved_text": "Classify the following customer message into one of: shipping,billing,technical.\n\nMessage: My order is late\n\nCategory:",
  "model_id": "gpt-4o-mini",
  "temperature": 0.0,
  "max_tokens": 50,
  "system_prompt": "You are a precise classification assistant.",
  "experiment_id": "exp_789",
  "variant": "B"
}

This is the most critical endpoint. Every LLM call in the company hits this first. It must be fast (sub-5ms from Redis), reliable (99.99%), and experiment-aware (return the right variant for A/B tests).

POST /api/prompts/:prompt_id/experiments - start an A/B test

Request: {
  "variants": [
    { "version_id": "ver_def456", "traffic_weight": 0.5 },
    { "version_id": "ver_ghi789", "traffic_weight": 0.5 }
  ],
  "allocation_strategy": "thompson_sampling",
  "primary_metric": "user_rating",
  "min_samples": 500
}
Response: {
  "experiment_id": "exp_xyz",
  "status": "active"
}

POST /api/prompts/:prompt_id/rollback - roll back to a specific version

Request: {
  "target_version_id": "ver_abc123",
  "environment": "production"
}
Response: {
  "prompt_id": "prm_abc123",
  "production_version_id": "ver_abc123",
  "rollback_propagated_at": "2026-04-11T10:05:00Z",
  "cache_invalidation_ms": 230
}

45-minute interview approach

Keep the discussion on designing a prompt platform: fast, deterministic resolution plus safe, measurable change management.

• 0–5 min — Clarify scope: ask whether prompts include tools and schemas, which environments exist, who can promote, how experiments are assigned, and how long versions are retained.
• 5–10 min — Requirements and estimates: state prompt count, resolution QPS, hot/cold latency, availability, rollback time, eval size, and experiment traffic.
• 10–18 min — Interfaces and data model: define prompt versions, templates, parameters, environment pointers, policies, eval runs, experiments, assignments, and telemetry.
• 18–28 min — High-level design: draw authoring API/UI, registry, validation/eval workers, pointer store, Redis cache, resolution API, rollout router, and metrics pipeline.
• 28–38 min — Deep dive: explain immutability, model coupling, cache invalidation, A/B allocation, judge calibration, and rollback when the new version is unsafe.
• 38–42 min — Scale and operations: cover hot-key prompts, cache warming, configuration propagation, audit logs, permissions, cost attribution, and stale-cache detection.
• 42–45 min — Trade-offs and close: compare fixed traffic splits with adaptive allocation, central control with team autonomy, and strict gates with release velocity.

High-level design and data flow

The platform has two distinct paths: the authoring path (low-traffic, high-latency-tolerance) where teams create and test prompts, and the resolution path (high-traffic, ultra-low-latency) where applications fetch the active prompt at runtime. These paths share a database but operate at completely different scales.

The authoring path flows through a web UI and API to a Prompt Service that stores versions in Postgres. When a new version is created, the eval pipeline kicks off automatically, running 200 test cases against the new prompt and scoring results. If the eval gate passes, the version becomes eligible for promotion. Teams can then promote to staging, run an A/B test, and eventually promote to production.

The resolution path is the critical path. Every LLM call in your company starts with a prompt resolution. The application sends prompt_id + environment to the Resolution API, which checks Redis first (99%+ cache hit rate). On a cache miss, it reads from Postgres, populates the cache, and returns the resolved prompt. When an A/B test is running, the resolver uses consistent hashing on user_id to assign a variant deterministically.

It is easy to focus on the authoring UI and miss that the resolution API is the hard part. The hot-path question is how every production LLM call gets its prompt in under 5ms.

Here is the prompt deployment pipeline step by step, from authoring to live traffic: