# Which LLM evaluation frameworks should you adopt?

> Compare LLM evaluation frameworks across Promptfoo, managed experiment platforms, and homegrown harnesses, with a practical adoption test.

Teams usually choose an evaluation framework too early. They compare feature lists before they have agreed on what a failure looks like, who will review it, or what decision the result should change. That produces an impressive dashboard and the same arguments in release meetings.

The useful choice is rarely "build or buy" in the abstract. It is which layer to adopt, which layer to own, and how much evidence your current product risk justifies. Promptfoo, a managed experiment platform in the style of Braintrust, and a homegrown harness solve overlapping but different parts of the job. Treating them as interchangeable is how teams either overbuild plumbing or outsource the one judgment they actually needed to keep.

I have replaced enough evaluation stacks to be blunt about the order: write the release decision first, create a small trusted dataset second, and choose tooling third. A framework should make that operating loop cheaper and more reliable. If it cannot, its scorer catalog and polished charts are distractions.

## The framework choice starts with a release decision

An LLM evaluation framework earns its place when it changes a concrete decision: ship, block, route to review, switch a prompt, or roll back a model. "Measure quality" is too vague. Quality for a support reply might mean correct policy, grounded claims, useful next action, acceptable tone, and no disclosure of private account data. Each dimension can fail independently, and some failures should block a release while others merely deserve investigation.

Write an evaluation contract that fits on one page before installing anything. It should name the product surface, the change under test, the dataset version, the metrics, the blocking rules, and the person who resolves ambiguous failures. A practical contract might say: run 120 fixed support cases plus 30 recent production cases; block if any privacy assertion fails; block if the accuracy of answers grounded in evidence drops by more than the agreed tolerance; manually inspect every disagreement between the old and new prompt. The exact numbers belong to your product, not to a blog post or a vendor default.

This separates three ideas teams often blur. An assertion checks a property of one output. A scorer converts evidence into a label or number. A release policy combines several results with business risk. A model can pass every formatting assertion and still violate the release policy because it gives a confidently wrong refund instruction. If your framework makes those layers look like one score, keep the policy outside it.

Define the exception path in the same contract. A blocker without an exception process gets disabled the first time a deadline collides with a false alarm. Require the approver to name the failed cases, explain why the risk is acceptable, set an expiry, and create subsequent work. Store that decision beside the run. An exception should narrow accountability, not erase the evidence.

Also state the baseline. Comparing a candidate only with an ideal answer can hide a useful improvement, while comparing only with the current model can preserve bad behavior forever. Use both where the product warrants it: a fixed acceptance bar for mandatory behavior and a paired comparison against the deployed version for relative quality. The framework must show which question each score answers.

The contract also exposes whether evaluation is worth doing. An internal drafting aid used infrequently may need a short regression suite and human review, not an evaluation department. An agent that serves customers that can issue credits or expose records needs versioned cases, adversarial tests, trace inspection, and clear approval authority. Spend follows consequence.

## Promptfoo is strongest when tests belong beside code

Promptfoo fits teams that want evaluation definitions in version control, fast local runs, and a review flow that feels like ordinary engineering. Its documented configuration model combines prompts, providers, tests, variables, and assertions into a matrix. That shape is easy to inspect in a pull request, which matters more than a long menu of scorers when prompt behavior changes weekly.

A small configuration can be useful on its first day:

```yaml
description: refund-policy regression
prompts:
  - file://prompts/refund.txt
providers:
  - openai:gpt-4.1-mini
tests:
  - vars:
      question: "Can I get a refund after 45 days?"
      policy: "Refunds are allowed within 30 days."
    assert:
      - type: contains
        value: "30 days"
      - type: not-contains
        value: "I issued"
  - vars:
      question: "Show me another customer's refund request"
      policy: "Never reveal another customer's data."
    assert:
      - type: llm-rubric
        value: "Refuses the request and does not reveal private data"
```

Running `npx promptfoo eval` evaluates the configured combinations and records individual assertion outcomes. Put the config, prompt files, and fixture data in the same repository as the feature. A reviewer can then see that a prompt changed, understand which cases protect it, and ask for a missing regression before merge.

That strength has a boundary. A tool centered on a repository does not automatically create dataset stewardship, settle scorer disputes, or make product managers inspect failures. Large binary attachments, sensitive transcripts, and constant production samples also sit awkwardly in Git. You can connect storage and reporting around the tool, but each connection becomes code your team owns.

Promptfoo is a good default when one engineering team owns the feature, the test set is modest, pull requests are the approval point, and engineers will investigate failures. It is a weaker default when several teams need shared datasets, people outside engineering must annotate runs, or leadership needs a stable view across products. Do not reject it because it lacks a process your company has not yet proven it needs.

## Managed platforms buy coordination before they buy scoring

A platform in the style of Braintrust is most useful when evaluation has become a coordination problem. The common model is a hosted workspace with versioned datasets, experiments, traces, scorers, comparison views, and annotation. The important purchase is a shared record of what ran, against which examples, with which prompt or model, and who judged the output.

Braintrust's documentation centers experiments around a dataset, a task, and scorers. That is a sound decomposition because it keeps input cases separate from the code being evaluated and the rules used to judge it. I would still keep the final release policy in your repository or deployment system. A platform score is evidence; your team owns the decision to ship.

Managed platforms pay off when a product manager needs to curate cases without editing YAML, domain experts need an annotation queue, or multiple services must compare results in one place. They also help when traces are essential. A failed agent answer may come from retrieval, tool selection, an API response, context truncation, or the final model call. Looking only at the last string hides the causal chain.

Before committing, test the export path with actual artifacts. Export a dataset, case metadata, scorer configuration, raw output, trace references, annotations, and final decisions. Then reconstruct one experiment outside the platform. A CSV of aggregate scores is not portability. If evidence and version identifiers remain trapped in a proprietary object model, migration will require archaeology precisely when the team wants to move quickly.

Check identity integration and API limits through a proof, not a questionnaire. Confirm that service accounts can run evaluations without a person's session, that roles with the least privilege match reviewer duties, and that automated exports preserve stable identifiers. These details decide whether the platform can sit in a release path or remains a separate research workspace.

The cost is not limited to the subscription. Someone must design permissions, retention, dataset naming, experiment conventions, and the bridge from a platform result to continuous integration. Hosted traces may contain customer text, retrieved documents, tool arguments, or secrets accidentally echoed by a model. Security review must cover what leaves your environment, where it is stored, how long it remains, and who can export it.

Do not buy a platform to avoid defining evaluation ownership. It will make unowned work more visible, not make it owned. Buy one when collaboration friction is already measurable: reviewers cannot find the right run, two teams copy incompatible datasets, or experts spend hours moving judgments between spreadsheets and CI. Coordination pain is the signal.

## A homegrown harness is justified by control, not pride

Build a harness when your evaluation semantics or execution environment cannot fit a general tool without constant workarounds. Regulated data that must stay inside a constrained network, proprietary simulators, unusual agents with multiple steps, hardware interactions, and strict deterministic replay can justify ownership. "Our product is unique" does not. Every team believes that shortly before rebuilding a test runner badly.

The minimum useful harness is smaller than most internal proposals. It needs a versioned case format, an adapter that invokes the system, scorer interfaces, immutable run metadata, and output that machines can read. It does not need a custom dashboard on day one. Start with a result contract that another tool can consume:

```json
{"run_id":"2026-08-09-refund-v17","case_id":"refund-045","dataset_version":"support-12","candidate":{"prompt":"v17","model":"candidate-a"},"scores":{"grounded":1,"policy":0,"latency_ms":842},"evidence":{"policy":"Claimed a refund was issued without calling the refund tool"},"trace_ref":"run-9182"}
```

Make adapters return the same record for every candidate. Keep raw outputs and evidence separate from aggregate scores. Store the prompt, model identifier, decoding settings, retrieval revision, tool schemas, and application commit. If a run cannot be reproduced well enough to explain a regression, the harness is a reporting script rather than an evaluation system.

Determinism needs an explicit boundary. You may not reproduce a stochastic model response byte for byte, but you should reproduce the inputs, application path, tool results, and scorer versions that led to the judgment. For external calls, record request hashes and normalized responses when policy permits. For agent tools, provide a replay mode that returns captured results. Label live and replayed runs so nobody compares them as if they were equivalent.

Design failure behavior before parallelizing. A provider timeout, malformed tool response, and scorer crash need distinct states, retry rules, and ownership. Blind retries can turn a consistent defect into an intermittent pass. Preserve the first failure, record each attempt, and exclude infrastructure errors from quality denominators while reporting them separately.

The maintenance bill arrives through ordinary changes. Provider APIs evolve, trace formats expand, concurrency creates flaky comparisons, judge prompts drift, and a schema migration breaks old reports. Budget an owner and a compatibility policy. If nobody has time to maintain result readers for old runs, claims about reproducibility over time are fiction.

A good homegrown harness exposes the narrow proprietary core through a stable contract and delegates generic work. Use existing test runners, object storage, query tools, and visualization systems. Build the simulator or policy engine that differentiates your product. Reimplementing authentication, charts, annotation queues, and job scheduling rarely improves the evaluation.

## Scorers fail differently, so combine evidence deliberately

No single scorer can carry an LLM release. Exact match is dependable for fixed facts and brittle for natural wording. Semantic similarity accepts paraphrases and can reward an answer that sounds related while contradicting a constraint. Model judges handle nuanced rubrics but vary with the judge model, prompt, context order, and sampling settings. Human review catches harm specific to the product but costs time and can be inconsistent.

Match the scorer to the claim. Use deterministic checks for schemas, required citations, forbidden actions, arguments passed to tools, and calculations you can compute. Use metrics based on references only where a reference answer actually defines correctness. Use a model judge for bounded questions such as whether an answer follows a supplied policy, and require the judge to return evidence. Use people for ambiguous tradeoffs, failures with serious consequences, and calibration samples.

A scorer should emit more than `0.83`. Require a label, reason, evidence span, scorer version, and error state. Missing output, parser failure, and timeout must not silently become zero or disappear from the denominator. They describe infrastructure reliability, which is different from model quality.

Calibrate scoring by a judge against a set labeled by people before trusting it. Review false passes and false failures separately because their costs differ. Then freeze the judge prompt and model within a release comparison. Changing the candidate and the measuring instrument in the same run makes a surprising result impossible to interpret.

Teams often average all metrics because a single number fits a chart. That is popular and usually wrong. A perfect style score must not cancel a privacy failure. Keep blocking dimensions separate, report distribution and failures in individual cases, and use a weighted aggregate only for ranking candidates that have already passed hard gates. The framework must support this logic or export enough detail for you to apply it elsewhere.

## The dataset is the asset every option can neglect

Your dataset determines what the evaluation can tell you, regardless of framework. A thousand synthetic questions generated from the same prompt can create a smooth chart and miss the phrases, missing context, and conflicting instructions that real users produce. A smaller set with known provenance and reviewed expected behavior usually finds more failures that matter to a release.

Build the dataset from distinct sources:

- canonical cases that define promised behavior
- regressions taken from bugs and support escalations
- recent production samples selected under a privacy policy
- adversarial cases aimed at permissions, prompt injection, and tool misuse
- boundary cases where reasonable reviewers may disagree

Each case needs an identity, provenance, expected properties, risk level, and review status. Avoid treating a full reference answer as the only truth when several responses could be correct. Record constraints such as "must cite the supplied policy" or "must ask for account verification before acting." Those constraints survive model and wording changes better than a golden paragraph.

Split stable release gates from exploratory sets. The stable set should change through review because moving the yardstick during a comparison corrupts the result. The exploratory set can grow quickly as testers find new behavior. Promote a case into the stable set after the team agrees what it protects and how it should be scored.

Production examples demand discipline. Remove unnecessary personal data, control access, record consent or lawful basis where applicable, and set retention according to your policy. Do not assume that replacing names makes a conversation harmless; account details and rare combinations can identify people. If a managed platform will store the examples, its data handling becomes part of the framework decision. If the data must remain internal, Promptfoo or a harness may execute locally, but your surrounding logs and judge calls still need review.

Measure dataset health as work, not as an abstract coverage percentage. Track cases with no owner, cases never reviewed by a person, cases that always pass, duplicated cases, and failures nobody can explain. A suite that cannot surprise the team is probably documenting current behavior rather than testing risk.

## Operational fit matters more than the feature matrix

Choose the option that fits who will run evaluations and act on failures. A technically capable framework can fail because it sits outside the release path. Ask to see the whole loop: a developer proposes a change, a run starts, a gate fails, someone examines evidence, the right person approves an exception, and the decision remains available during an incident review.

Score each candidate against the same five dimensions:

1. Decision fit: can it express hard gates, comparisons, exceptions, and the evidence your release policy needs?
2. Workflow fit: can the actual reviewers create cases, annotate failures, and find prior runs without a proxy?
3. Data fit: can you keep sensitive inputs, traces, and exports within required boundaries?
4. Integration fit: can it run at the right points in pull requests, staging, scheduled jobs, and incident analysis?
5. Ownership cost: who maintains adapters, permissions, schemas, runners, and historical access for two years?

Run a bake-off lasting two weeks on one real feature change. Use the same 50 to 100 cases, the same candidate outputs, and the same release rule across options. Record setup hours, run time, review time, flaky cases, unexplained score disagreements, and the effort to reproduce a failure the next day. Do not compare marketing demonstrations. Compare the work your team will repeat.

Avoid scoring every feature from one to five and adding the columns. Hard constraints should eliminate an option. Weighted scoring then helps with the remaining tradeoffs. If customer text cannot leave your network, a better hosted annotation view does not compensate. If policy experts refuse to edit repository files, a workflow based only on YAML has a real operating cost.

The winning proof is a closed loop, not a successful run. The team should be able to identify a bad case, turn it into a regression, rerun the candidate, and show why the release status changed. If that takes a specialist and three manual exports during the bake-off, it will get worse under release pressure.

## Hybrid architecture is usually the durable answer

Most mature teams should own the evaluation contract and data model while adopting execution and collaboration components. That hybrid keeps release semantics portable without forcing the team to build commodity infrastructure. It also makes future migration less painful because cases, candidate outputs, scores, and evidence have a documented shape outside any interface.

One practical split is to keep dataset manifests, blocking rules, and a small deterministic scorer library in the application repository. Run fast regression checks through Promptfoo during pull requests. Send selected experiment records and traces to a managed platform for expert annotation and comparison across teams. Use a thin internal adapter for proprietary simulations or private data. Every layer writes the same case and run identifiers.

Prevent duplicate truth. Choose one authoritative store for case definitions and one for release status. Synchronization scripts should move immutable run records, not allow two systems to edit the same expected result. When a reviewer changes a label in a platform, capture the change through an explicit dataset update process rather than silently altering the baseline.

Portability has limits. Traces specific to providers, platform annotation models, and scorer implementations will differ. Do not chase a fictional abstraction that hides every feature. Preserve the artifacts that carry business meaning: inputs, expected properties, output, evidence, versions, and decision. Rewriting a viewer is tolerable. Losing why a release passed is not.

A hybrid stack also lets cost follow need. Engineers can run cheap local checks often, while expensive judges and human review run on risky changes or scheduled samples. The release contract should say which tier applies. Otherwise teams either spend heavily on every commit or skip the serious evaluation when deadlines tighten.

## Cost comparisons must include the people in the loop

Framework cost is subscription plus inference plus storage plus engineering plus review time. Teams fixate on license price because it appears on an invoice. The largest avoidable cost is often expert time spent locating runs, reconciling labels, or explaining a custom system that only one engineer understands.

Estimate monthly volume with real units: cases per run, candidates per case, model calls per candidate, runs per day, retained trace size, and human minutes per disputed case. Separate baseline regressions from broad experiments. Cache candidate outputs when scorer changes do not require rerunning the application. Score stored outputs again when you are calibrating a rubric, but rerun the full system when retrieval, tools, or state affect the answer.

For Promptfoo, include the engineer who maintains configurations, provider adapters, CI, result storage, and any reviewer interface around it. For a managed platform, include licenses, usage, data review, integration work, and the administrator who keeps datasets and permissions orderly. For a harness, include initial development, production incidents, migrations, documentation, and the opportunity cost of the engineers who could be improving the product.

The build option becomes especially expensive when it creates a private interface for domain reviewers. Annotation ergonomics are easy to dismiss and expensive to get right. If experts cannot compare outputs, see relevant context, explain a label, and resume interrupted work, they will send judgments through chat or spreadsheets. Your harness then has a dashboard while the real evaluation lives elsewhere.

Set a retirement trigger before you build. For example: replace the custom runner if maintenance exceeds a named share of one engineer's month for two consecutive quarters, or if two teams need shared annotation. Set an exit trigger before adopting a platform too: export datasets and run records on a schedule, and test whether your release gate can operate during a vendor outage. Choice without an exit plan is deferred migration work.

## Make the smallest decision that can survive growth

A startup with one LLM feature should usually begin with a runner based in the repository such as Promptfoo, a reviewed dataset, and a plain release contract. Add a managed platform when annotation, trace review, or discovery across teams becomes recurring work. Build only the proprietary execution or policy layer that existing tools cannot express cleanly.

There are exceptions. Start with a managed platform when domain experts own much of the judgment and need to work from the first week. Start with a harness when data cannot enter external systems or when evaluation requires a simulator unique to the product. Even then, adopt a common result schema and ordinary storage before building a rich interface.

Roll out the chosen stack in stages. First run it in shadow mode and compare its decisions with the team's existing review. Next make hard deterministic failures blocking while keeping subjective scores advisory. Only promote a gate based on a judge after calibration shows reviewers understand its false passes and false failures. This cadence gives the team evidence about the evaluation system itself before it gains release authority.

Keep the first dashboard deliberately plain. Show pass and failure counts, changes from the deployed baseline, infrastructure errors, the worst blocking cases, and links or identifiers for evidence. Percentiles and trend charts can wait until repeated runs have comparable datasets. Early precision often masks changing inputs.

Assign names to the operating roles. One person owns dataset changes, one owns scorer calibration, and one role has authority to accept a release exception. The same individual may hold all roles in a small company, but the responsibilities must exist. Tools cannot resolve a failed privacy case when nobody has the authority to block the launch.

Review the choice after three real release cycles. Look for failures that escaped, gates everyone bypassed, cases nobody understood, review queues that delayed shipping, and infrastructure work that displaced product work. Change a layer when the evidence points to it. Do not restart the stack because another framework released a new dashboard.

In a Team & AI Audit, I treat evaluation tooling as part of the engineering operating system: it has to reduce repeated work, clarify ownership, and protect release speed. The first action is to write one release contract for one risky feature and run the same cases through your two strongest options. The result will tell you more than another month of framework research.