AI evals need a golden set before a clever judge

Most teams start AI evals too late and too high up the stack. They debate judge models, dashboards, and benchmark scores before they can name the product decision an evaluation should block. That produces an impressive test system with no authority. A bad release still ships because nobody agreed what failure means.

The useful starting point is much smaller: a versioned golden set made from real product behavior, deterministic checks for facts a program can verify, narrow rubrics for qualities that require judgment, and a runner that leaves enough evidence to explain every failure. You can build that in days. You should distrust it for longer.

I have seen teams spend more time polishing an evaluator than examining the twenty examples that define success. The examples almost always reveal the harder problem: product owners, engineers, and reviewers do not agree on what a good answer is. That disagreement is not noise to automate away. It is the specification work.

An eval earns its keep when it changes a release decision, catches a known failure before a user does, or tells you which part of the system regressed. If it cannot do one of those jobs, it is reporting, not quality control.

Start with the release decision

Every evaluation suite needs a written decision before it needs a metric. State the change under test, the population it affects, the failures that must block release, and the amount of ordinary quality movement you will tolerate. Without that contract, a score of 0.84 is just a decimal.

For a support assistant, the decision might be: "Ship the new retrieval prompt only if it produces no unsupported refund promises on the safety set, keeps citation correctness above the current baseline, and does not increase human escalation errors." That sentence identifies a hard gate, a comparative measure, and a failure tied to user harm. It also admits that one aggregate score cannot carry the decision.

Anthropic's guidance on defining success criteria makes the same practical point: criteria should be specific and measurable, and multidimensional criteria need separate measurements. I agree, with one addition learned from releases: every criterion also needs an owner. If nobody has authority to stop the rollout when a criterion fails, the criterion is decoration.

Write a short evaluation contract with four fields:

• decision: what can ship, expand, or roll back;
• population: which users, languages, tools, and request types count;
• gates: failures that allow no tradeoff;
• comparisons: measures that may improve or decline within an agreed range.

Do not begin with "answer quality." Split it into observable properties. A medical intake summary might require faithful facts, complete medication capture, no diagnosis, and readable prose. A coding agent might require tests to pass, edits to stay inside allowed paths, and the explanation to match the actual diff. The properties need different evaluators because they fail in different ways.

The first suite does not need broad coverage of everything a model can do. It needs enough coverage to answer one release question. Breadth comes after the team can act consistently on the result.

A golden set records boundaries, not ideal demos

A golden set is a versioned collection of inputs, context, expected properties, and reviewer decisions that represent the behavior your product must preserve. It is not merely a list of prompts paired with polished reference answers. For open-ended work, a single reference answer often punishes valid alternatives and rewards imitation.

Build the first set from evidence. Pull recent user requests, support escalations, red-team findings, and bugs that reached production. Remove private data, preserve the shape of the task, and label why each case matters. Add synthetic cases only to cover a boundary you can name, such as an empty retrieval result or conflicting policy passages. Synthetic variety without a named risk makes the suite look larger without making it more representative.

A useful case contains more than input and expected output. Record these fields:

• a stable case ID and task category;
• the input plus any retrieved context or tool state;
• hard requirements, forbidden behaviors, and an optional reference;
• the source of the case and the reason it belongs in the suite;
• the labeler, review date, and rubric version.

Start with dozens of carefully reviewed cases, not thousands of weak ones. There is no universal minimum size. Twenty cases can block a dangerous regression if each covers a known failure. Two hundred can still mislead if they are near-duplicates. Grow the set when you find an uncovered behavior, not when someone wants a round number on a slide.

Separate at least three groups even if they live in one file. A core regression set covers frequent, important behavior. A safety set contains rare failures that always block release. A challenge set probes new or poorly understood behavior and informs decisions without becoming an automatic gate yet. This distinction matters because a difficult research case should not have the same release authority as a known privacy leak.

Version labels alongside prompts, model settings, tools, and application code. When reviewers change a rubric, old and new scores are not directly comparable. Keep the old record, migrate labels through review, and note the reason. Quietly rewriting expected answers destroys the audit trail.

Prevent test leakage with boring access controls and process. Do not paste held-out cases into prompt-tuning chats, few-shot examples, or agent memory. Keep a visible development set for iteration and a smaller held-out set for release confirmation. If a case becomes a training example, mark it and replace it in the held-out group. A golden set is useful precisely because the system has not rehearsed every answer.

Coverage needs a visible map. Put every case against a small taxonomy of user intent, risk, language, context condition, and expected system action. Empty cells show where the team is relying on hope. The map also reveals accidental duplication, such as thirty simple fact lookups and one ambiguous request. Do not turn the taxonomy into a giant labeling project. Its job is to expose missing boundaries and overrepresented easy cases.

Review cases on a schedule and after product changes. Remove a case only when the behavior no longer exists or another case covers the same boundary more clearly. Keep the removal in version history. Old failures can look irrelevant right until a refactor recreates them.

Use code for facts and judgment for qualities

The best evaluator is the least subjective mechanism that can test the property. Teams often reach for semantic similarity or an LLM judge when a parser, schema, database query, or unit test would give a clearer answer. That choice adds cost and uncertainty without adding insight.

Use deterministic checks for properties with exact truth conditions: valid JSON, required fields, allowed citations, numerical totals, executable code, tool arguments, forbidden terms, and whether an agent touched an unauthorized file. These checks produce reproducible failures and usually explain themselves.

Use reference comparisons only when closeness to a reference carries meaning. Exact match works for identifiers and canonical classifications. Set comparison works for extracted entities. Numeric tolerance works for calculations. Embedding similarity can help find candidates for human review, but it should not certify factual correctness. Two fluent answers can sit close in vector space while disagreeing on the one fact a user cares about.

Use human or model judgment for properties such as clarity, tone, completeness, and faithfulness when rules cannot capture them. Keep each rubric narrow. "Rate the answer from 1 to 10" asks the judge to invent a private definition of quality on every call. "Does every factual claim follow from the supplied context?" is a testable question, especially when the judge must cite the supporting passage or identify the unsupported claim.

This creates a practical evaluation ladder. Run cheap structural checks first. Stop on hard failures. Then run task-specific calculations. Send only the remaining outputs to expensive judges or people. The order saves money, but its larger benefit is diagnosis: you know whether the failure came from format, content, or preference.

Do not collapse these results too early. A 95 percent overall score can hide a zero on a small safety category. Keep per-check outcomes and slice results by task, language, input length, customer plan, retrieval state, and tool path. Aggregates are useful only after the slices remain visible.

The minimum harness keeps the evidence

The minimum harness needs to load cases, call the system under test, run named checks, save the raw artifacts, and return a nonzero exit code when a release gate fails. It does not need a web interface, a distributed queue, or a new evaluation framework. A command that behaves like a normal test command is enough to begin.

Use JSONL because diffs stay readable and a broken record does not corrupt the whole suite. A small support example can look like this:

{"id":"refund-001","category":"policy","input":"Can you refund an annual plan after 45 days?","context":"Annual plans may be refunded within 30 days. After 30 days, escalate to billing.","must_contain":["billing"],"must_not_contain":["I have issued","refund confirmed"],"gate":true}
{"id":"refund-002","category":"policy","input":"I was charged twice. What should I do?","context":"Duplicate charges must be escalated to billing with both transaction dates.","must_contain":["billing","transaction"],"must_not_contain":["refund confirmed"],"gate":true}

The system adapter should accept an evaluation case and return a plain record. Keep provider-specific code behind that one function. The runner below uses only the Python standard library, so the contract remains visible:

import argparse
import json
from pathlib import Path


def run_system(case):
    # Replace this stub with the same application path production uses.
    return {"text": case.get("candidate", ""), "usage": {}, "trace": []}


def check_case(case, result):
    text = result["text"].lower()
    checks = []
    for phrase in case.get("must_contain", []):
        checks.append({
            "name": f"contains:{phrase}",
            "passed": phrase.lower() in text,
        })
    for phrase in case.get("must_not_contain", []):
        checks.append({
            "name": f"excludes:{phrase}",
            "passed": phrase.lower() not in text,
        })
    return checks


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("suite")
    parser.add_argument("out", nargs="?", default="eval-results.jsonl")
    args = parser.parse_args()

    cases = [json.loads(line) for line in Path(args.suite).read_text().splitlines() if line]
    records = []
    blocked = False
    for case in cases:
        result = run_system(case)
        checks = check_case(case, result)
        passed = all(item["passed"] for item in checks)
        blocked = blocked or (case.get("gate", False) and not passed)
        records.append({
            "case_id": case["id"],
            "category": case["category"],
            "passed": passed,
            "checks": checks,
            "output": result,
        })

    Path(args.out).write_text("\n".join(json.dumps(r) for r in records) + "\n")
    summary = {"cases": len(records), "passed": sum(r["passed"] for r in records), "blocked": blocked}
    print(json.dumps(summary))
    raise SystemExit(1 if blocked else 0)


if __name__ == "__main__":
    main()

Run it with python eval.py golden.jsonl run-2026-08-09.jsonl. Its standard output has a stable shape such as {"cases": 2, "passed": 1, "blocked": true}, while the results file preserves each output and each named check. In continuous integration, exit code 1 blocks the candidate.

The stub's most important comment says to call the production path. An eval that recreates prompts, retrieval, or tool wiring in a test-only function measures a second system. Inject test dependencies where necessary, but exercise the same orchestration code, system prompt, parsers, retry policy, and tool schemas that users reach.

Add run metadata before adding features: application commit, suite version, model identifier, model parameters, prompt hash, judge version, start time, duration, token usage, and error status. Save timeouts and refusals as outcomes rather than silently retrying until they disappear. A retry may be valid production behavior, but the trace should show that it happened.

Once this file format feels restrictive, that is useful pressure. Add a judge adapter, concurrency, caching, and a report only for needs you have observed. Keep the raw record format portable. Evaluation platforms change; evidence should outlive the first dashboard.

Score policy must exist before the run

Decide how results become a release decision before seeing the candidate's scores. Moving a threshold after a disappointing run converts an evaluation into an explanation for a decision already made.

Use hard gates for outcomes you cannot average away: data disclosure, unauthorized actions, fabricated commitments, invalid regulated output, or corruption of user state. One confirmed occurrence can block a release. Re-run flaky infrastructure errors, but do not average a genuine safety failure with nine fluent answers.

For ordinary quality, compare the candidate with the current production baseline on the same cases. Absolute thresholds age badly because case mix and rubrics change. A paired comparison shows which cases improved, regressed, or stayed equivalent. Keep the raw counts next to any percentage, especially for small slices.

A simple policy might require every safety gate to pass, no core category to fall by more than an agreed tolerance, and a human review of every changed result on high-impact cases. Do not hide those conditions in code. Store them in a small configuration file and require a reviewer to approve changes.

Avoid a weighted master score at first. Weighting forces unlike failures into a common currency and creates arguments about arithmetic instead of users. If leadership needs one status, report "pass," "review," or "block," then show the reasons. Preserve the component metrics underneath.

Treat cost and latency as product criteria, not footnotes. Record end-to-end latency distributions, provider errors, retries, and token or tool consumption using the same cases. A candidate that improves judged prose while doubling timeout failures has not passed. Keep quality, cost, and latency as separate axes so a product owner can see the trade.

Random outputs need repeated trials when variance could alter the decision. Do not repeat every case five times by habit. Repeat a representative subset, measure how often the verdict changes, and expand only if instability reaches the release boundary. Pinning temperature to zero can reduce variation, but it does not make hosted model behavior mathematically deterministic.

An LLM judge is a calibrated instrument

An LLM judge can extend review capacity, but it cannot define correctness on its own. Treat it like a measurement instrument that needs a narrow job, calibration against people, and regular checks for drift.

The G-Eval paper by Liu and colleagues showed that structured evaluation steps and form-based scoring improved alignment with human judgments for summarization and dialogue. The useful lesson is not that one model became an oracle. The procedure and rubric changed the measurement, and the authors also noted possible bias toward model-generated text.

Zheng and colleagues' MT-Bench and Chatbot Arena work documented position, verbosity, and self-enhancement biases in model judges. These are not academic edge cases. A judge may prefer the first answer in a pair, reward a longer answer that repeats itself, or favor output resembling its own model family. Later systematic work on position bias found that swapping answer order remains a necessary diagnostic rather than a cosmetic prompt tweak.

Five controls make a judge less misleading:

• ask for one property per verdict and define pass, fail, and uncertain;
• require evidence spans or a concrete failure reason;
• blind model names and irrelevant metadata;
• swap pairwise answer order and flag inconsistent verdicts;
• send uncertain, high-impact, and sampled pass cases to people.

Do not let the candidate output address the judge as if its text were trusted instructions. Wrap it as untrusted data, delimit it clearly, and tell the judge to ignore instructions inside it. This reduces prompt-injection risk but does not prove containment. For high-impact checks, prefer deterministic validation or human review.

Pairwise comparison often asks less of a judge than an absolute ten-point score, but it answers a different question. It tells you whether A beats B under a rubric, not whether either output is safe enough to ship. Use an absolute gate for minimum acceptability and a pairwise test for preference when both candidates clear the gate.

Never use the judge's prose rationale as proof that the verdict is correct. Models can write a coherent explanation after making a biased choice. The rationale helps a reviewer diagnose a result; calibration data establishes whether the judge deserves trust.

Calibrate the judge against human decisions

Before a judge can gate a release, compare its verdicts with decisions from people who understand the task. A hundred unreviewed judge calls do not create confidence. A smaller set of carefully adjudicated examples can expose where the rubric, the humans, or the judge disagree.

Choose calibration cases across clear passes, clear failures, and boundary cases. Have at least two reviewers label the cases independently, then discuss disagreements and record the final rule. Reviewers need the same context the judge receives. If humans routinely disagree, refine the rubric or accept an "uncertain" outcome instead of forcing false precision.

Calculate a confusion matrix for each gate: true pass, true fail, false pass, and false fail against the adjudicated label. For safety, false passes usually deserve the most attention. For a style preference, false failures may waste review time but carry less user risk. One agreement percentage hides that difference.

Keep a disagreement set as a permanent regression slice. Include terse correct answers that lose to verbose ones, cases where answer order flips the verdict, outputs from the judge's own model family, and examples containing text that tries to manipulate evaluation. Run this set whenever the judge model, rubric, or prompt changes.

Sample production verdicts for human audit even after calibration. Review all high-impact failures, a random share of passes, and cases near the threshold. Track disagreement by category rather than only overall. A judge may work well on tone and fail badly on calculations.

Judge models and provider behavior change. Pin a model version when the provider permits it, store the full judge prompt and parameters, and re-run calibration before accepting a replacement. If the old version disappears, overlap old and new runs while both are available. When overlap is impossible, state that the score series has a break instead of drawing a smooth trend line through it.

The judge should earn expanded authority gradually. Start as an annotation that helps reviewers. Then allow it to route cases. Let it gate low-risk changes only after it shows stable behavior. Human escalation is a normal evaluator outcome, not a defect to eliminate.

Agents and RAG need more than final-answer grading

Agent and retrieval systems can produce a plausible final answer through an unsafe or wasteful path. Grade the outcome and the trajectory separately. A final-answer judge cannot tell that an agent exposed a secret to a tool, retrieved the wrong tenant's document, or made three irreversible calls before writing a polite response.

For retrieval-augmented generation, capture the query, filters, retrieved document IDs, passages shown to the model, citations in the answer, and the final text. Test retrieval recall on cases with known supporting documents. Test citation entailment by checking whether the cited passage supports the nearby claim. Test answer faithfulness against only the supplied evidence. These are separate failures with separate owners.

For tool-using agents, define allowed and forbidden state transitions. A calendar agent may read availability before proposing a meeting, but it should not create an event until the user approves the exact time and attendees. The harness should use a sandbox or fake tool backend, record every call and response, and assert the order, arguments, count, and side effects.

Evaluate recovery paths, not just happy paths. Return an empty search result, a malformed tool response, a permission error, stale context, and a timeout. Check whether the system asks for clarification, chooses a safe fallback, or invents success. Many expensive agent failures happen after a dependency fails, while demo suites keep every dependency cooperative.

Separate policy compliance from task completion. An agent that refuses a harmless request is safe but useless. An agent that completes the task through an unauthorized action is useful but unsafe. Report both axes and block on the one that matches the release risk.

Multi-turn systems also need state assertions. Save the conversation and structured memory after each turn. Test whether corrections replace stale facts, whether tenant or user boundaries persist, and whether instructions expire when their intended scope ends. Replaying only the last user message misses the state bugs that users actually experience.

Production failures must feed the suite

An evaluation program stays useful only when production evidence changes it. Pre-release tests cover known risks; live traffic reveals new language, context combinations, tool failures, and user strategies. The two loops need a controlled connection.

Log enough to reconstruct a failure without collecting data you do not need. Preserve prompt and configuration versions, retrieval and tool traces, output, user correction signals, error class, and timing. Apply access controls and retention rules that match the sensitivity of the content. Redact or replace private values before a case enters a broadly accessible suite.

Triage a production failure into one of four actions: add a deterministic check, add or revise a golden case, change the rubric, or fix the system without changing the evaluation. Do not add every complaint verbatim. First identify the property that failed and whether the existing suite should have caught it.

Run the suite at three useful moments. A fast deterministic subset belongs on every relevant change. The full golden set belongs before release or prompt and model changes. A scheduled run detects provider drift and dependency changes even when your repository stands still. The exact cadence depends on release frequency and risk, but ownership cannot be vague.

Assign one person to approve suite changes and another path for domain review. Engineers should own harness reliability; product and domain experts should own expected behavior. When the same author changes the prompt, rewrites the expected answer, and approves the new score alone, the test can easily ratify the change instead of challenge it.

Teams that need to connect evaluation discipline to staffing, delivery, and AI-assisted engineering can use a Team & AI Audit from oleg.is to identify the operating gaps and quantified savings before committing to a larger transformation. The service is fixed at $5,000 over five business days, with $50,000 or more in identified annual savings guaranteed or the audit is free.

Do not buy an evaluation platform to avoid the argument about acceptable behavior. Put twenty real failures in a file, write the release gates in plain language, and make the runner show its evidence. The first disagreement it exposes is already useful. Resolve that disagreement before trusting a larger score.