Stop thresholds for a multi-agent coding pilot

A multi-agent coding pilot should stop or change shape when its weighted rework rate stays above 35% for two consecutive measurement windows after at least 20 eligible tasks. That is a decision rule, not a universal law. Your own recent human-only baseline decides whether 35% is strict enough, and a security breach, data loss, or serious production incident can stop the pilot immediately.

Raw output volume should never keep a weak pilot alive. Ten agents can produce a week of diffs before lunch. If senior engineers spend the afternoon rejecting, repairing, and explaining those diffs, the pilot has moved labor rather than removed it. Count accepted changes, reviewer minutes, reverted commits, escaped defects, and tool spend together. Those five numbers show whether the system creates deployable work at a lower total cost than the team it is meant to help.

Rework rate needs a denominator you can defend

Use eligible tasks as the denominator and count only work that entered the pilot under a written task contract. An eligible task has a stated outcome, acceptance checks, a repository and environment in scope, and a clear point at which a reviewer can accept or reject it. If the task is vague enough that two engineers would implement different things, the pilot is measuring specification quality.

The simplest rework rate is:

raw_rework_rate = tasks_requiring_material_rework / eligible_tasks

Material rework means a human or agent had to change behavior, architecture, tests, security controls, data handling, or deployment configuration before acceptance. Fixing punctuation in a comment does not count. Replacing an invented API, rewriting a broken migration, adding a missing authorization check, or discarding the change does.

Do not use commits, lines changed, agent messages, or pull requests as the denominator. One task may create twelve agent commits and three pull requests; another may land as one squashed commit. The orchestration topology controls those counts. A task is closer to the unit of intent that the business asked the engineering system to complete.

There is a second distinction that teams routinely blur: pilot rework and production deployment rework are related, but they are not the same measure. DORA defines deployment rework rate as the share of deployments that were unplanned responses to user-facing bugs. That is a production stability measure. A coding pilot also creates pre-merge repair, rejected work, and review effort that never reaches production. Borrow DORA's concern with unplanned corrective work, but do not label your pull request metric as a DORA score.

Classify every eligible task into one final state:

• Accepted without material rework
• Accepted after material rework
• Rejected or abandoned
• Reverted after merge
• Linked to an escaped defect

The states are not mutually exclusive after deployment. A task can be accepted after repair and later reverted. Keep those consequences attached to the original task rather than inventing a cleaner history.

Accepted changes are the unit that survives review

An accepted change meets the task's checks and the repository's normal merge standard without a human silently rewriting the result. Acceptance must mean the same thing for pilot work and ordinary work. Lowering the bar because the demo looks clever corrupts the comparison.

Record three acceptance numbers: first-pass acceptance, eventual acceptance, and accepted change equivalents. First-pass acceptance is the strongest signal that the agents understood the task and constraints. Eventual acceptance shows whether repair produced usable work. Accepted change equivalents let you credit partial delivery without pretending that half a task is complete.

I use 0, 0.5, and 1 as the only allowed values for an accepted change equivalent. A task earns 1 when it delivers the whole contracted outcome, 0.5 when an independently useful part lands and the missing part returns to the backlog, and 0 when reviewers discard it. Do not let teams negotiate values such as 0.8 after seeing the weekly result. That turns a measurement into a defense of the pilot.

The partial-credit rule needs one hard boundary: cleanup is not an independently useful outcome. If the agent writes tests but fails to implement the feature, the tests count only if they expose an existing defect or remain useful for the next implementation. If the agent creates scaffolding that no accepted code uses, credit is zero.

Track rejected work even when it never becomes a pull request. Agent runs that die in a branch, a worktree, or an orchestration queue still consumed tool budget and operator attention. The decision ledger should give each eligible task a stable ID before the first agent starts, so disappearance is visible.

GitHub's pull request documentation describes a pull request as a place to propose, discuss, review, and validate changes before merge. That is useful workflow guidance, but a merged pull request is not proof of independent acceptance. During a hurried pilot, the operator who prompted the agents may also approve their output. Require an accountable reviewer who did not run the task for changes in authentication, billing, data migrations, deployment, or other high-consequence areas.

The ratio that belongs on the weekly scorecard is:

accepted_yield = accepted_change_equivalents / eligible_tasks

If accepted yield is below 65%, the system is already over the proposed 35% rework boundary before heavier consequences enter the calculation. A pilot with 90% eventual acceptance may still be bad when first-pass acceptance is 25%, because reviewers and repair agents are doing most of the real engineering.

Reviewer minutes expose the labor agents displace

Reviewer minutes count from the moment a qualified reviewer starts reconstructing the task until that reviewer records a decision. Include reading the task contract, inspecting the diff, checking relevant code, running targeted tests, writing correction notes, and reviewing the corrected version. Exclude normal queue time and unrelated interruptions.

Do not estimate review effort from the number of comments. A five-minute review can leave eight precise comments. A ninety-minute investigation can end with one sentence: "This migration loses rows during rollback." Use a timer or reconstruct time from a calendar block on the same day. Precision to the nearest five minutes is enough; false precision invites arguments that do not improve the decision.

Compare median reviewer minutes per accepted change with the human-only baseline for the same task class. The median prevents one bizarre task from deciding the pilot. Also report the 75th percentile because a rising tail warns that the pilot is generating changes reviewers cannot bound quickly.

Normalize review labor into a penalty:

review_penalty = max(0, pilot_review_minutes - baseline_review_minutes)
                 / baseline_total_task_minutes

Calculate this per task, then sum the penalties. Baseline total task minutes means implementation plus review for a comparable task completed without the pilot. The formula charges the pilot only for excess review time. Normal review is part of software delivery, regardless of who wrote the code.

Suppose a routine API change takes a human 150 implementation minutes and 30 review minutes. The pilot produces it with 65 reviewer minutes. Its excess review penalty is (65 - 30) / 180, or 19.4% of a task equivalent. The agent may still win if tool cost is low and the change is accepted. It has not saved the full 150 implementation minutes, because 35 extra minutes moved onto a scarcer engineer.

Reviewer seniority matters in the cost model, even if it does not change the rate. Convert minutes to money with a loaded hourly cost when comparing economics. A founder reviewing agent output at midnight has a cost even when payroll records show zero. Use the person's normal loaded rate or a documented internal shadow rate. Free labor makes any automation look efficient.

Watch for review saturation. When the same reviewer handles several agent branches at once, elapsed review time may fall while missed defects rise. Cap concurrent reviews during the pilot, and keep the cap constant across measurement windows. Otherwise the team can improve the review metric by reviewing less carefully.

Separate operator minutes from reviewer minutes even when one person performs both roles. Operator work includes shaping prompts, routing tasks, resolving agent collisions, restarting environments, and assembling a reviewable branch. Review begins only when that branch and its task record are ready for an acceptance decision. This boundary prevents orchestration labor from disappearing into "implementation saved." If the roles overlap during a small pilot, ask the person to switch the timer category when the work changes. The distinction will feel fussy for a week, then it will reveal whether the expensive part is controlling the agents or judging their output.

Reverts and escaped defects need heavier weights

A reverted commit means the team accepted a change and then removed or neutralized it because the change was wrong, unsafe, or operationally unfit. An escaped defect means a pilot-attributed flaw reached a shared test environment, production, or users after the designated acceptance gate. Define the escape boundary before measurement; teams with a serious staging environment may choose production, while teams shipping directly from branches need an earlier boundary.

Not every revert indicts the pilot. Product decisions change, upstream services fail, and releases sometimes roll back as a precaution. Link a revert only when the pilot-created change materially caused the rollback or fix-forward. Require a short reason code and one human owner for attribution.

Use severity weights rather than counting every consequence equally:

• Material pre-merge rework: 1.0 task equivalent
• Rejected or abandoned task: 1.0
• Pilot-caused revert: 2.0
• Escaped defect with limited internal impact: 2.0
• User-facing or security-relevant escaped defect: 4.0

These weights express a management judgment: failure after acceptance costs more than repair during review. You can change them, but freeze them before the first measured task. If your product handles money, health, identity, or irreversible customer data, the user-facing weight should be higher and some categories should become hard stops.

Calculate weighted rework as:

weighted_rework_rate =
  (rework_tasks + rejected_tasks
   + 2 * pilot_reverts
   + 2 * internal_escaped_defects
   + 4 * serious_escaped_defects
   + excess_review_task_equivalents)
  / eligible_tasks

This rate can exceed 100%, and that is intentional. A task that consumed repair time, merged, caused a defect, and required a revert imposed several layers of rework. Capping the score at 100% would erase the difference between a merely rejected change and an operationally expensive failure.

Keep a separate hard-stop list outside the formula. Stop dispatching new pilot tasks after confirmed secret exposure, unauthorized data access, destructive migration behavior, material compliance failure, or a production incident above the severity your company has approved for the experiment. The team can investigate and resume later. A weighted average should never dilute a safety boundary.

DORA's current delivery metrics separate throughput from instability and define change fail rate around deployments requiring immediate intervention, such as a rollback or hotfix. Keep that separation. Agent task throughput can rise while deployment stability falls. Reporting only tasks completed per week hides the exact tradeoff DORA warns teams to measure.

Tool spend belongs in cost per accepted change

Tool spend includes model tokens, agent platform fees, hosted sandboxes, added CI minutes, paid retrieval or search calls, and temporary infrastructure created for the pilot. Attribute shared subscriptions by measured use when possible. If a platform exposes only a monthly price, record the allocation rule in advance and apply it to both successful and failed tasks.

The useful economic measure is fully loaded cost per accepted change equivalent:

pilot_cost_per_accepted_change =
  (tool_spend
   + operator_minutes * operator_rate
   + reviewer_minutes * reviewer_rate
   + repair_minutes * repair_rate
   + incident_cost)
  / accepted_change_equivalents

Compare that with human-only cost for the same task classes. Tool spend alone is a procurement number, not an engineering economics number. A $40 agent run that consumes four hours of principal-engineer review is not a $40 change.

Avoid assigning all tool spend to accepted work. Failed and abandoned runs stay in the numerator. That is why cost per accepted change rises quickly when orchestration fans out too many speculative agents. The extra agents may be cheap individually, but duplicated context loading, conflicting edits, repeated tests, and reconciliation all accumulate.

Set two economic limits. The warning limit should trigger a change to model choice, context size, agent count, or task selection. The stop limit should end the current pilot design. A reasonable starting pair is 1.0 times baseline cost as the warning limit and 1.25 times baseline cost as the stop limit, provided the pilot was meant to reduce cost. A capability-seeking pilot may accept a higher cost if it targets work the team otherwise cannot complete, but that exception must appear in the pilot charter.

Do not claim savings from theoretical headcount changes during a short pilot. Measure capacity returned: implementation hours avoided minus added operator, review, repair, and incident hours. Then ask whether the returned hours were actually usable. Twenty fragmented ten-minute gaps do not equal a focused block an engineer can spend on architecture or customer work.

Report spend by task class. Dependency updates, test generation, small API changes, user interface work, and migrations have different context demands and failure modes. A blended average can hide a profitable narrow use case inside an expensive general pilot. Stopping the global topology while keeping one task class is often the right decision.

Set the stop threshold before the pilot starts

Write a pilot charter with one primary stop rule, economic guardrails, hard safety stops, and a minimum sample. My default for a normal web product is: pause and redesign the pilot when weighted rework exceeds 35% in two consecutive weekly windows, once the pilot has completed at least 20 eligible tasks and each window contains at least 10. Stop the current design immediately if fully loaded cost per accepted change exceeds 1.25 times the matched baseline in two windows or any approved hard-stop event occurs.

Why 35%? Below that line, a pilot can still repay repair with implementation time saved, especially on bounded tasks. Above it, more than one task equivalent in three is being consumed by correction before tool economics enter. The number is deliberately unforgiving because multi-agent systems can make bad work arrive faster than a team notices.

Do not copy 35% without a baseline. If the matched human workflow already has 28% weighted rework, demanding 10% from the pilot may reject useful automation. If the human workflow runs at 8%, accepting 34% because it is below a generic threshold would be reckless. Use this rule:

pilot_stop_threshold =
  min(35%, baseline_weighted_rework_rate + 10 percentage points)

The ten-point allowance buys learning during a time-boxed experiment. It is not the steady-state target. For high-consequence code, remove the allowance and require the pilot to meet the human baseline before expanding scope.

Use a two-window rule to avoid stopping on one rough week, but do not use cumulative averages alone. A strong opening week can conceal three deteriorating weeks because easy backlog tasks ran first. Show the current window, prior window, and cumulative result together.

Small samples require decisions in counts, not decorative percentages. With eight tasks, one revert moves the raw rate by 12.5 points before weighting. Keep the pilot in observation until it reaches the minimum sample unless a hard stop fires. If the team cannot find 20 comparable tasks, the proposed use case may be too rare for a quantitative pilot; use a risk review and case studies instead of pretending the rate is stable.

Match the baseline by task class, repository, and consequence. Do not compare agent-generated dependency bumps in a mature service with human-built payment workflows in a new codebase. Random assignment is best when the backlog permits it. Otherwise tag complexity and risk before assignment, then compare within those tags.

A decision ledger prevents retrospective excuses

Create one row when a task enters the pilot and fill the remaining fields as work progresses. The ledger can live in a spreadsheet, database, or experiment repository, but one owner must reconcile it with pull requests, deployments, incidents, and invoices every week.

This CSV header is enough to run the decision:

task_id,task_class,risk,eligible,baseline_minutes,accepted_equivalent,material_rework,rejected,review_minutes,baseline_review_minutes,revert,escaped_internal,escaped_serious,tool_cost,operator_minutes,repair_minutes,window,notes

Add a reason code to notes for every zero, partial acceptance, revert, or defect. Keep the note factual: "Used removed billing API; reviewer replaced integration" is useful. "Agent got confused" is not. The first describes a failure the team can prevent with repository context or task routing.

Run the weekly review in this order:

1. Freeze the window and reconcile every eligible task ID.
2. Confirm acceptance and consequence attribution with the reviewer or incident owner.
3. Calculate weighted rework, accepted yield, review percentiles, and cost per accepted change.
4. Compare results with the matched baseline and the two prior windows.
5. Record continue, constrain, redesign, pause, or stop, plus the person accountable for the action.

Keep "constrain" separate from "continue." Constrain means the current topology failed for part of the scope, so the next window excludes a task class, reduces agent fan-out, or adds a mandatory gate. It is an intervention, not a green result.

Do not let agent operators adjudicate disputed defects alone. They are invested in the pilot and remember all the clever recoveries that the ledger cannot see. Use the same engineering owner who would classify a human-caused incident. Consistent attribution matters more than protecting either side.

The ledger also needs negative space: eligible tasks assigned to the control group, pilot tasks canceled before execution, and agent runs without accepted output. If those records disappear, the scorecard develops survivorship bias. Only the successful branches remain visible.

At the end of each window, retain the raw records and the exact formula version. A team will change orchestration prompts, models, tools, and gates during a pilot. Mark those changes as interventions. Otherwise an improving graph cannot tell you which change worked.

A worked decision separates repair from cancellation

Consider a four-week pilot with 24 eligible tasks. The agents deliver 18 accepted change equivalents. Six tasks require material rework, including two that reviewers ultimately reject. One accepted task causes a revert. One other task creates an internal escaped defect. Reviewers spend 910 minutes against a matched baseline of 620 minutes, and the baseline total effort for all 24 tasks is 5,800 minutes.

Do not double-count the two rejected tasks as both rework and rejection if your ledger treats rejection as their final rework state. The weighted numerator is six task equivalents for repaired or rejected tasks, two for the revert, two for the internal escape, plus (910 - 620) / 5,800, or 0.05, for excess review. Weighted rework is 10.05 / 24, or 41.9%.

Now split the period into two twelve-task windows. Suppose the first scores 37% and the second 46%. Both exceed the 35% default, the minimum sample has been reached, and the direction is worse. The current design stops even if its tool bill looks cheap. More agent runs will produce more evidence of the same failure.

Suppose instead the first window scores 52% and the second 29% after the team limits work to test generation and small service changes. The two-window stop rule does not fire. The intervention appears useful, but the pilot has not earned general expansion. Run another window with the constrained scope and keep the excluded task classes out.

Economics can reverse either decision. Assume the 18 accepted equivalents cost $7,200 fully loaded, or $400 each, while matched human changes cost $360. The pilot is at 1.11 times baseline: above the warning line but below the 1.25 stop line. Combined with 41.9% weighted rework, that result supports a redesign. If the cost were $500 each, the 1.39 ratio would independently stop the design after the required second window.

This example shows why a single "rework percentage" is insufficient. The six repaired tasks describe pre-merge quality. The revert and escape describe misplaced confidence. Excess review shows hidden labor. Cost per accepted change shows whether the automation paid for any capacity it returned.

Do not rescue the result with future promises. Better repository instructions, stronger tests, a different model, or fewer agents may work, but each is a new intervention with a new measured window. Record the hypothesis, change one or two controls, and make the next design earn its continuation.

Stop the topology before abandoning agents

A failed pilot usually condemns a specific combination of task scope, agent roles, context, gates, and review design. It does not prove that every coding agent is useless. Stop dispatch under the failed design, preserve the ledger, and decide which smaller claim remains worth testing.

Use evidence to choose the change. High rejection with low review time usually means poor task selection or missing context. High eventual acceptance with low first-pass acceptance points to weak validation or excessive decomposition. Low rework with excessive cost points to model choice, context duplication, or too much fan-out. Reverts and escaped defects demand stronger tests, narrower permissions, or removal of the affected task class, not cheaper tokens.

The popular response is to add another reviewer agent. I usually argue against that as the first fix. A reviewer agent can catch mechanical errors, but it shares the same missing repository knowledge and may produce a second layer of confident text for a human to inspect. Add a deterministic check when the failure has a deterministic signature. Add human approval where consequences require judgment.

Resume only with a written delta: which tasks are now eligible, which agent roles changed, what new gate blocks the observed failure, and how the thresholds apply. Do not reset the cumulative history. Show both the new design window and the full experiment so a short recovery does not erase the cost of learning.

For companies that lack reliable baselines or cannot reconcile pilot activity with production outcomes, a Team & AI Audit can establish the task classes, costs, and control points before a broader transformation. The useful deliverable is a decision system your engineering owner can keep running, not a favorable demo.

Multi-agent coding should buy accepted engineering capacity without transferring disproportionate repair and risk to the people who already carry production accountability. When two measured windows say it does not, stop the current design. The agents can wait while the team fixes the experiment.