What should an AI delivery pilot prove?

An AI delivery pilot earns more funding only when it proves that the team can ship accepted work at a lower total cost without moving effort into review, defects, or production support. A faster coding demo proves almost nothing. The funding decision needs thresholds that were fixed before the first task entered the pilot, a comparable baseline, and evidence that survived ordinary production conditions.

I have seen pilots declared successful because an agent produced a pull request in minutes. Then senior engineers spent hours correcting it, QA found regressions, and the engineer handling production alerts paid the final bill. The pilot looked fast because its scorecard stopped at code generation. Measure the whole delivery system through acceptance and early production, or do not call the result a business case.

The four gates belong together: cost per accepted feature, review time, defect escape rate, and incident load. Cost tells you whether the economics improved. Review time shows whether AI created a new queue around scarce senior people. Escaped defects reveal whether acceptance was too shallow. Incident load catches the failures that matter most but occur too rarely to average away. A pilot passes only if its planned tradeoffs fit your operating model and no safety gate fails.

Write the funding contract before the first task

The pilot needs a written decision rule that makes failure possible. If the sponsor can reinterpret the result after seeing it, the company bought a demonstration, not evidence. Name the population, baseline window, pilot window, thresholds, exclusions, owners, and the action attached to each outcome.

Use thresholds as gates, not aspirations. A target such as "improve developer productivity" cannot settle a funding argument. A gate such as "reduce median cost per accepted feature by at least 20%, while p90 review time does not rise and no severity 1 incident is attributable to pilot work" can. The exact numbers depend on your margins, release risk, and current performance. What matters is choosing them before the pilot team knows which result will flatter it.

This compact scorecard is enough to expose most evasions:

pilot:
  unit: accepted_feature
  baseline_window_weeks: 6
  pilot_window_weeks: 6
  minimum_accepted_features: 20
  gates:
    cost_per_accepted_feature:
      improvement_percent: 20
    review_time_hours:
      median_max: 6
      p90_max: 24
    defect_escape_rate:
      maximum_percent: 8
      no_worse_than_baseline: true
    incident_load:
      severity_1_max: 0
      responder_hours_per_feature_max: 0.5
  decision:
    pass: fund_next_stage
    mixed: extend_with_named_fix
    fail: stop_or_redesign

Do not copy those sample thresholds blindly. Replace them with values your finance and engineering leads can defend. Keep the shape: one economic gate, two flow and quality gates, one operational veto, and a minimum sample. Record who can classify a feature as accepted and who can attribute a defect or incident to pilot work.

A mixed result needs a narrow rule too. For example, a cost win paired with slow review may justify an extension of two weeks only if the team changes the review design and states the expected effect. "Keep experimenting" is not a decision. It is an unpriced extension.

Freeze a small data dictionary beside the contract. Define when a task enters the pilot, when review starts, what counts as active review, which release marks acceptance, and how long production observation lasts. Name the source for every field, such as the issue tracker, version control history, deployment log, incident system, or finance ledger. Otherwise the final week becomes an argument about whose spreadsheet is authoritative.

Choose a baseline window that represents the work the rollout will face. Avoid a holiday freeze, a migration week, or the month when half the team was new unless the pilot faces the same conditions. If seasonality or a release cycle makes one window suspect, show a second reference window. The point is not to find a baseline that makes AI lose. It is to keep an unusual week from deciding a large investment.

Add stop rules that operate during the test. A severe incident, evidence that confidential data crossed an unauthorized boundary, or repeated failure of mandatory controls may require an immediate pause. A pilot budget does not buy permission to ignore the company's existing safety rules.

Count accepted features, not generated code

The denominator should be an accepted feature: a piece of user or operational value that meets the same completion policy used outside the pilot. Lines of code, agent sessions, pull requests, and story points are activity units. They can rise while delivered value falls.

Define acceptance in observable terms. The change is merged, automated checks pass, the named acceptance criteria pass, required documentation or telemetry exists, and the feature reaches the intended environment. If your normal process requires a security review or customer signoff, the pilot does too. Work that gets abandoned, reverted before acceptance, or split into meaningless fragments stays in the cost numerator and contributes no accepted feature to the denominator.

Feature size still varies, so stratify the sample before comparing it. A simple scheme can classify work as small, medium, or large using the same rubric for baseline and pilot tasks. Better yet, match pilot tasks with recent baseline tasks from the same product area and risk class. Do not let the pilot consume twenty copy changes while the baseline includes payment logic and data migrations.

Teams often use story points because the data already exists. I would not use them as the economic denominator. Points reflect local estimation habits, and a team can unconsciously reprice work once AI enters the process. Use them to match task difficulty if the team estimates consistently, then report accepted features by size band.

The awkward case is a feature that ships but fails its business acceptance test. It is not accepted. The engineering team may have completed its internal checklist, but the pilot is supposed to improve delivery, not checklist completion. State who owns the final acceptance decision, set a time limit for that decision, and freeze the evidence attached to it.

Watch for denominator gaming in planning meetings. One baseline feature may become four pilot features because an agent can produce four pull requests. Preserve the original user outcome as the unit. If splitting work reduces risk and produces independently useful increments, count the increments. If the pieces only make sense together, count one feature.

Keep an acceptance ledger with one row per eligible feature. It should show the original outcome, size band, risk class, entry date, final status, acceptance owner, acceptance evidence, release date, observation maturity, and any linked defect or incident. This ledger lets a skeptical founder trace a polished ratio back to the work that created it. It also makes missing outcomes visible.

Internal platform work needs the same discipline even when no customer clicks a new button. Define the accepted outcome as an observable operational change, such as a supported deployment path, a measured reduction in build time, or removal of a manual recovery step. "Refactored the service" is not enough unless the refactor itself satisfies a stated risk or maintenance outcome.

Cost per accepted feature includes the cleanup

Cost per accepted feature is total labor and service cost attributable to the pilot divided by the number of accepted features. Include planning, prompt and context preparation, implementation, review, test repair, rework, deployment, model or tool fees, security work, and early production support. Excluding senior review or incident response is how a cheap pilot becomes an expensive rollout.

Use loaded labor rates, not salary divided by work hours. Finance should supply one consistent rate for each role that includes the employer costs it normally uses for planning. Precision beyond the company's own budgeting practice adds ceremony without improving the decision. Apply the same rates and cost categories to the baseline.

The calculation can stay plain:

cost_per_accepted_feature =
  (labor_hours_by_role * loaded_rate_by_role
   + AI_tool_cost
   + incremental_infrastructure_cost
   + attributable_rework_and_support_cost)
  / accepted_features

Suppose a hypothetical pilot spends $48,000 across labor, AI tools, extra test infrastructure, rework, and support, then produces 24 accepted features. Its cost is $2,000 per accepted feature. If the matched baseline is $2,600, the observed improvement is about 23%. That clears a 20% cost gate, but it does not clear the pilot until the other gates pass.

Report the median cost within each size band as well as the blended figure. A blended average can improve because the task mix shifted toward smaller work. Show the raw numerator, denominator, and excluded costs beside the ratio so finance can reproduce it. If a cost cannot be attributed cleanly, state the allocation rule before the analysis.

Separate setup cost from recurring cost, then show both views. Initial repository indexing, policy writing, test harness changes, and staff training may be legitimate rollout investments. Hiding them is dishonest, while charging the entire initial setup to every future feature is also misleading. Present pilot economics with setup included, then a recurring estimate that amortizes approved setup across a stated volume. Funding should depend on whether both numbers make sense under a conservative volume.

Do not turn developer time saved into cash savings automatically. A team that saves 300 hours but keeps the same staffing and ships nothing extra has created capacity, not reduced payroll. The business case must say how that capacity will be used: more accepted work, avoided hiring, contractor reduction, or a smaller team after a responsible transition. Finance can value those outcomes differently.

Include the opportunity cost of scarce people. Two hours from a staff engineer may carry the same payroll cost as two ordinary engineering hours in your loaded rate table, yet it can delay architecture, hiring, or a critical customer issue. Keep the formal cost calculation consistent, then flag any role whose pilot demand approaches its available capacity. A rollout that consumes all senior attention cannot grow even when its unit cost looks favorable.

Run a sensitivity check before presenting the savings. Recalculate the result with a lower rollout volume, a modest rise in model fees, and the observed p90 review cost instead of the median. This is not permission to invent a pessimistic forecast. It shows whether the decision survives ordinary variation or depends on every assumption landing perfectly.

Review time exposes displaced work

Review time must capture both elapsed queue time and active reviewer effort. AI can shorten implementation while flooding a senior reviewer with larger pull requests, subtle mistakes, and explanations that look plausible. If you measure only time from first review to approval, you miss the hours a change waited for the one person qualified to inspect it.

Track at least four timestamps: ready for review, first substantive review, approval, and merge. Then record active minutes spent by each reviewer. Elapsed review time equals approval minus ready for review. Queue time equals first substantive review minus ready for review. Active effort comes from time tracking or a brief reviewer entry attached to the change. Perfect telemetry is unnecessary, but consistent definitions are not.

Use median and p90, not the mean alone. The median describes ordinary flow. The p90 exposes the long tail where risky or confusing changes consume senior attention. A pilot that cuts median review from eight hours to four but pushes p90 from twenty hours to three days has not produced a clean win. It made common work faster and difficult work harder.

Segment review results by risk and feature size. Also show reviewer concentration: what share of active review minutes came from the two most experienced reviewers? A pilot can meet its overall review target while becoming dependent on one staff engineer. That dependency will break when the rollout expands beyond a protected team.

Review quality matters more than quick approval. Do not reward reviewers for clicking approve sooner. Pair review time with escaped defects and rework after review. If approvals accelerate while fixes after merge rise, the review gate has been optimized against the company.

The popular recommendation to cap pull request size is useful but incomplete. Small changes reduce reviewer load only when they represent coherent behavior. An agent can produce tiny, interdependent changes that force reviewers to reconstruct the same context repeatedly. Set a coherence rule alongside any size limit: each change must have one testable purpose and enough context to judge it independently.

Count review loops too. A change that receives comments, returns to the author, and comes back with the same class of problem has consumed more than elapsed time. Record the number of substantive review rounds and the active minutes in each round. Repeated loops often reveal weak task context, missing repository rules, or an agent that changes adjacent code without understanding ownership.

Do not use comment count as a quality measure. One reviewer writes ten small comments while another explains the same problem in a paragraph. Classify only what changes the decision: blocking correctness work, required clarification, optional improvement, and approval. The pilot should reduce blocking rework without teaching reviewers to stay silent.

Escaped defects need a stable denominator

The defect escape rate is the share of accepted features that cause at least one attributable defect after crossing the agreed acceptance boundary. Pick that boundary before the pilot. For many product teams it is production release; for regulated or embedded work it may be a formal validation stage.

Count affected features as the primary denominator, not raw defect tickets. One bad feature can create ten duplicate reports, while another causes one severe corruption bug. Report the number of defects and severity mix as supporting data, but calculate the rate consistently:

defect_escape_rate =
  accepted_features_with_one_or_more_escaped_defects
  / accepted_features

Set an observation window long enough for normal use to exercise the change. A feature accepted on the last day of a pilot that lasts six weeks has not had the same chance to fail as one accepted on the first day. Either follow every feature for a fixed period after release or report mature and immature cohorts separately. Do not mark unobserved work as defect free.

Attribution needs a short, documented review. Ask whether the defect came from behavior introduced or changed by the pilot feature, whether existing tests should have caught it, and whether the same classification rule applies to baseline work. Keep the reviewer independent from the person who created the change when severity is high or attribution is disputed.

A raw escape rate can hide severity, so add a veto. One feature causing a major security exposure, irreversible data loss, or a severity 1 incident should not be averaged against nineteen clean features. Define those veto categories using your existing incident and security policies. The pilot should use company language instead of inventing a softer scale.

Do not quietly exclude defects fixed by the pilot team. Fast repair lowers incident duration, but it does not erase the escape. Similarly, a rollback is an operational response, not proof that no defect reached production. Track time to detect and time to restore as diagnostic measures. Keep them out of the main gate unless you wrote them into the funding contract.

Low defect counts create uncertainty. Do not pretend a small sample proves equality. If both baseline and pilot show one escape, say the observed rates are similar and give the counts. Fund a staged expansion with continued measurement when the economic win is clear but the quality sample remains thin.

Measure defect discovery before release separately. Finding more defects before acceptance may be good because tests and reviewers caught them, or bad because initial work quality fell. The cost numerator and review loops already capture much of that burden. A separate count helps diagnose the cause without confusing caught defects with escaped ones.

Reopened defects deserve attention. A quick patch that closes a ticket and fails again inside the observation window remains part of the same escaped feature, but the extra repair hours stay in cost and incident load where applicable. Preserve the event history rather than counting only the final ticket state.

Incident load can veto attractive economics

Incident load is the human and operational burden created by pilot changes after release. Measure responder hours, incident severity, customer impact under your existing definition, overnight pages, rollback work, and subsequent remediation. A model that writes code cheaply can still be a bad investment if it makes production less predictable.

Responder hours per accepted feature gives the load a usable denominator. Count everyone pulled into diagnosis, communication, mitigation, recovery, and the immediate work required to prevent recurrence. Keep broader improvement projects separate unless the incident review assigns them directly.

Severity needs more weight than arithmetic. Define zero tolerance for the categories your company cannot accept, such as a severity 1 outage attributable to pilot work. For lower severities, compare responder hours per accepted feature with baseline and inspect the distribution. Five brief daytime fixes do not equal one overnight incident even when total hours match.

Run the pilot through normal release machinery. A protected sandbox can test whether the tools function, but it cannot prove incident load. At least a representative subset must pass through ordinary deployment, monitoring, rollback, ownership, and responder processes. Do not waive controls to make the pilot look quick, and do not add exceptional supervision that a rollout cannot afford.

Walk through this common failure. An agent produces a configuration change, unit tests pass, and review focuses on syntax. The change reaches production and causes a gradual queue buildup under real traffic. Monitoring catches it hours later, two engineers investigate, one rolls back, and another spends the next morning adding a missing load test. The implementation looked nearly free. The accepted feature cost must absorb all that work, the feature records an escaped defect, and the responder hours hit incident load. One event affects three gates because the gates describe different consequences, not duplicate penalties.

Near misses belong in the evidence even when they do not enter the main incident rate. A deployment caught by a final manual check, an unsafe command blocked by permissions, or a rollback started before customers noticed can reveal a control doing its job. Record the event, the control that stopped it, and whether that control has enough capacity for wider use. Do not relabel a caught problem as an incident merely to make the data dramatic.

Ownership must survive outside the pilot room. Each released feature needs a person who can explain it, respond to an alert, and repair it when the agent is unavailable or its context has changed. If only the pilot champion can operate the result, responder concentration has replaced reviewer concentration.

Compare paired work without protecting the pilot

A credible comparison gives baseline and pilot work similar difficulty, risk, team context, and acceptance rules. Random assignment is ideal when enough suitable tasks arrive, but most startup pilots are too small or operationally constrained for a clean experiment. Matched pairs and transparent controls are usually practical.

Use this sequence:

1. Build an eligible task pool from real roadmap work before choosing which tasks receive AI support.
2. Classify each task by product area, size, risk, required systems, and acceptance owner.
3. Pair similar tasks, then assign one to the pilot and one to the baseline when delivery constraints permit.
4. Apply the same definition of done, review policy, release path, observation window, and cost rules.
5. Publish every eligible task and its final status, including abandoned and reverted work.

Keep the people comparison honest. If the pilot team contains your strongest engineers while the baseline uses new hires, the result measures staffing. The same problem appears when pilot engineers receive private help from the tool vendor or daily attention from an executive sponsor. Record that support as cost and describe whether it can exist at rollout scale.

Training creates a timing problem. Measure too early and you price in basic learning; measure only after months of coaching and you may ignore a costly adoption period. Split the pilot into a short setup phase and a measured phase. Charge setup costs explicitly, require the measured phase to use the intended operating model, and report the learning curve rather than deleting inconvenient weeks.

Contamination also matters. Engineers in the baseline group may use pilot techniques, shared generated code, or improved tests. That can shrink the apparent difference, but forbidding useful learning may harm the company. Record contamination, keep the financial calculation conservative, and focus on whether the operating model improves the whole system.

Choose a minimum sample before launch. Twenty accepted features in the sample scorecard is an illustration, not a universal rule. Work with high volume and low risk may need far more observations to expose quality differences. Rare, risky work may never produce statistical confidence in a short pilot, so use hard safety vetoes, expert review, and a limited next stage instead of claiming certainty.

Do not remove failed or abandoned tasks. They consumed capacity and tell you where the method breaks. Use an intention-to-treat rule: once an eligible task enters the pilot group, its attributable cost stays in the pilot numerator even if the team finishes it manually or cancels it for technical reasons. Business cancellations unrelated to delivery can follow a prewritten exclusion rule applied to both groups.

Fund the operating model, not the demo

The rollout decision should state what scales, for whom, under which controls, and with what stop conditions. A pass does not justify giving every engineer every tool on Monday. It justifies the next funded stage defined in the contract.

Start the decision memo with the four gates and raw counts. Show accepted features, total cost, cost by role, review median and p90, escaped features, severity mix, incidents, and responder hours. Then show task mix, exclusions, setup costs, and confidence limits in plain language. A sponsor should be able to reproduce every ratio from the page.

There are four defensible outcomes. Fund the next stage when all gates pass and the sample covers representative work. Extend narrowly when one diagnosed constraint has a credible fix and the extension has a deadline. Redesign when the tools work but the review, test, or ownership model does not. Stop when economics fail, a safety veto fires, or the team cannot produce trustworthy evidence.

Tie expansion to capacity. If active review time fell but reviewer concentration rose, add teams only after distributing the required expertise. If cost improved on routine work but not on risky changes, restrict the rollout to the proven task classes. If quality evidence is thin, preserve the defect and incident gates through the next cohort.

Price the next stage as a portfolio, not a multiplication exercise. Ten times the pilot team does not produce ten times the result when all teams depend on the same security reviewer, platform engineer, or context maintainer. List shared constraints, their current capacity, the demand observed per accepted feature, and the point at which another hire or process change becomes necessary.

Write stop conditions into the rollout funding as well. The company should pause expansion if p90 review time crosses its ceiling for two reporting periods, a safety veto fires, or accepted feature cost loses its advantage after setup effects settle. Assign the person with authority to pause and the evidence required to resume. Without that authority, a staged rollout becomes an irreversible program.

The decision memo should include what the pilot did not test. It may say nothing about legacy repositories, mobile releases, regulated data, new engineers, or teams in another time zone. Those gaps are not an embarrassment. They define the boundary of the claim and the order of later cohorts.

On oleg.is, I use the five-day Team & AI Audit to build this kind of baseline, identify where AI work will move cost, and put a priced rollout decision in front of the founder. The audit does not replace a paid delivery pilot when production evidence is needed; it keeps the company from spending that pilot on vague productivity theater.

Refuse the seductive number when it stands alone. A claim that coding became three times faster has no funding value until accepted feature cost, review time, escaped defects, and incident load agree with it. Put the thresholds in writing, let ordinary production test them, and accept the result even when it says the rollout is not ready.