Design an LLM eval pipeline

30-second answer

• Four eval tiers run at different costs and cadences: assertion-based (every commit), RAGAS automated (staging), LLM-as-judge (PR merge), and human eval (weekly).
• The golden dataset is version-controlled alongside the prompt. Changing the prompt without updating the dataset is a common cause of silent regressions.
• LLM-as-judge uses a strong judge model (GPT-4o) separate from the model under test. Never use the same model to judge its own output.
• The CI/CD gate blocks deployment if the primary metric (faithfulness) drops more than 5% from baseline. Secondary metrics trigger warnings, not blocks.
• Regression tracking in a time-series DB with a 7-day rolling average catches slow quality drift that single-run thresholds miss.

Requirements and assumptions

Functional requirements

• The eval pipeline runs 5,000 test cases per pipeline execution, covering real user questions with expected behavior.
• Results integrate with CI/CD: a failing eval run blocks deployment to production.
• Quality metrics (faithfulness, context relevance, answer relevancy) are tracked over time with trend visualization.
• Human evaluators can label a sampled subset of responses through a review UI, and those labels feed back into the pipeline.
• The pipeline runs on every prompt change, not just model changes.

Non-functional requirements

• Full eval run (5,000 cases) must complete in under 30 minutes to fit in a CI pipeline.
• LLM-as-judge must complete within 45 minutes to not block PR merges.
• Cost per full eval run under $50 (5K cases at LLM-as-judge pricing).
• Eval results are immutable once written; each run produces a versioned snapshot for auditing.
• Alert on-call if the 7-day rolling faithfulness average drops below the alert threshold.

Assumptions

• The evaluation target is a versioned application behavior: prompt, model, retrieval configuration, tools, and dataset versions are recorded together.
• No automated metric is treated as ground truth. Human labels calibrate the judge and the acceptance thresholds, especially for subjective quality.
• CI can distinguish blocking checks from warnings. A failed gate is explainable, reproducible, and reversible rather than an opaque score.

5-minute approach

Build the pipeline from cheap, deterministic checks to expensive quality checks. Version the dataset and system under test, run cases in parallel, store immutable results, and gate only on metrics that have a demonstrated relationship to user outcomes.

• Start with a representative golden set and explicit rubrics for faithfulness, relevance, correctness, safety, or latency.
• Run assertions and schema checks on every change, then RAG or retrieval metrics, judge-model scoring, and sampled human review as appropriate.
• Compare candidates with confidence intervals or paired tests against a baseline instead of reacting to one noisy run.
• Feed production failures and human labels back into the dataset, while keeping dataset changes reviewable and tied to a prompt/model version.

The entities and endpoint below define an eval run that can be reproduced months later.

Core entities

EvalCase

• case_id, question, expected_behavior (can be exact string, reference answer, or behavioral description), category (faq/adversarial/edge), created_by, version

EvalRun

• run_id, trigger (commit_hash/pr_id/schedule), prompt_version, model_version, status, started_at, completed_at, cases_run, cases_passed

EvalResult (one per case per run)

• result_id, run_id, case_id, response_text, assertion_pass, faithfulness, context_relevance, answer_relevancy, judge_score, judge_reasoning, latency_ms

HumanLabel

• label_id, result_id, evaluator_id, helpfulness (0-3), accuracy (0-3), conciseness (0-3), override_verdict, created_at

API design

POST /api/evals/trigger — kick off an eval run from CI

Request:  { "commit_hash": "a3f9b12", "prompt_version": "v2.4.1", "run_type": "pr_merge" }
Response: { "run_id": "run_789", "status": "queued", "estimated_duration_s": 900 }

GET /api/evals/runs/{run_id}

Response: {
  "run_id": "run_789", "status": "complete",
  "summary": { "faithfulness": 0.87, "context_relevance": 0.82, "judge_avg_score": 2.4 },
  "gate_decision": "PASS",
  "baseline_delta": { "faithfulness": +0.02 }
}

POST /api/evals/human-labels — submit human review batch

Request:  { "labels": [{ "result_id": "r_abc", "helpfulness": 3, "accuracy": 2, "conciseness": 3 }] }
Response: { "received": 1, "calibration_delta": 0.03 }

GET /api/evals/trends?metric=faithfulness&days=30

Response: { "metric": "faithfulness", "datapoints": [{"date": "2026-04-04", "value": 0.85},...] }

45-minute interview approach

Keep the focus on designing an evaluation system, not on naming one metric or one judge model.

• 0–5 min — Clarify scope: ask what behavior is being evaluated, whether the output has a reference answer, which regressions are release-blocking, and who owns human labeling.
• 5–10 min — Requirements and estimates: state case count, run frequency, time and cost budgets, reproducibility needs, alerting policy, and the expected false-block tolerance.
• 10–18 min — Interfaces and data model: define datasets, cases, prompts, model/config versions, eval runs, per-case results, aggregate metrics, and human labels.
• 18–28 min — High-level design: draw case ingestion, parallel execution, deterministic checks, retrieval metrics, judge service, human review, result store, dashboard, and CI gate.
• 28–38 min — Deep dive: discuss judge bias, variance, golden-set freshness, calibration, statistical thresholds, and how to handle missing or flaky providers.
• 38–42 min — Scale and operations: cover worker concurrency, retries, cost caps, immutable artifacts, trend alerts, canaries, and dataset governance.
• 42–45 min — Trade-offs and close: explain blocking versus warning metrics, speed versus coverage, judge cost versus human confidence, and the rollback path when a gate is wrong.

High-level design and data flow

Every prompt change triggers a CI eval run. The eval runner distributes 5,000 test cases as async jobs across a worker pool, which processes them in parallel and writes results to the scores database. A score aggregator collects individual results, computes summary metrics, and compares to the stored baseline. The gate decides: pass (deploy) or fail (block).

The four eval tiers are stacked by cost and cadence. Assertion-based tests run on every commit in seconds. RAGAS automated metrics run on the staging environment before merge. LLM-as-judge runs on PR merge as the final automated gate. Human eval runs weekly on a sampled subset and recalibrates the judge model's scoring. Each tier catches different failure modes, so skipping any one creates blind spots.

The first dependency to stabilize is the golden dataset. It must be version-controlled with the prompt, reviewed when behavior changes, and augmented regularly with representative production questions. A dataset curated for an old product version will generate misleading scores for a new one.