# Inference cost per merged change needs a business threshold > Set an inference cost per merged change threshold that counts retries, review time, failure risk, and the economic value of shipped work. Paying less per token can make engineering more expensive. A cheap model that needs four attempts, consumes forty minutes of review, and still sends a defect into production costs more than a strong model that finishes cleanly once. The useful routing number is inference cost per merged change, not cost per request, token, or agent session. A threshold should answer a business question: at what expected total cost should this task move to a cheaper model, a stronger model, or a human? That total includes model spend, retry cost, review labor, failure risk, and the value created by a correct change. If the routing rule ignores any one of those, teams end up optimizing an invoice line while losing money elsewhere. The threshold is not one universal dollar amount. A dependency bump, an unclear billing fix, and a production database migration carry different review burdens and failure costs. The practical answer is a small routing table by task class, backed by observed completion data and a formula everyone can inspect. ## Measure a merged change, not an attempt A merged change is the smallest unit that connects AI spend to engineering output. Count all inference used from the first request until the change passes its required checks and a reviewer accepts it. Abandoned attempts belong in the cost of the task class that produced them. Hiding them makes weak routes look efficient. This distinction matters because model vendors sell inference while engineering teams sell completed work. A request can succeed technically by returning code and still fail economically because the code does not compile, misses the acceptance criteria, or takes longer to review than writing it manually. Session success and change success are different events. Define the boundary before collecting data. I use the following rules: - One change maps to one issue or one independently reviewable outcome. - All model calls used to produce, test, repair, or explain that outcome count toward it. - Human review ends when the reviewer approves or rejects the route and takes over. - A change counts as merged only after the normal automated checks pass. - A later production incident is attached to the originating change when the evidence supports that link. Do not divide a large feature into tiny commits merely to improve the metric. Use the unit at which a reviewer can honestly say yes or no. Squashing commits does not change the unit; acceptance does. Keep two denominators. Cost per attempted change exposes demand and abandonment. Cost per merged change exposes delivery efficiency. If ten tasks enter a route and six merge, the inference bill must be divided by six for the delivery metric, not ten. The four failures did not disappear because the dashboard chose a flattering denominator. The same rule applies to human work. A developer may try two approaches before completing a fix. Comparing one AI call with the developer's final coding minutes is nonsense. Compare the full path to an accepted outcome on both sides. ## Put every cost in the same equation A routing threshold becomes useful when every term resolves to money per completed change. The equation can stay simple enough for a spreadsheet: ```text Expected total cost per merged change = (inference cost per attempt + review cost per attempt + retry handling cost) / probability of eventual merge + expected failure cost ``` For a route with several attempts, calculate the numerator from observed totals instead of pretending each attempt is identical: ```text Route cost per merged change = (total inference spend + total review minutes × loaded reviewer rate per minute + total human repair minutes × loaded engineer rate per minute) / merged changes + incident cost attributed to the route / merged changes ``` The loaded rate should include the cost your company actually bears for that person's time, not only salary. Use one internally agreed rate if precision would turn the exercise into an accounting debate. A stable, directionally honest rate beats a supposedly exact rate that nobody updates. Review cost belongs in the formula even when reviewers are salaried and the cash has already left the bank. Review time has an opportunity cost. Thirty minutes spent untangling an agent's patch cannot be spent on architecture, customer defects, or another review. Calling that time free guarantees that routing will push work toward cheap models that create expensive queues. Failure cost needs an expected value, not a dramatic worst case. Multiply the probability of a failure caused by the route by the average cost of that failure class. The cost can include recovery labor, support time, service credits, lost transactions, or a conservative internal proxy when revenue attribution is impossible. ```text Expected failure cost = failure caused by the route rate × average failure impact ``` Keep the estimate coarse at first. Classify failures as local rework, preproduction escape, reversible production incident, and production incident with high impact. Assign each class an approved cost. The point is consistent routing, not a damages model fit for a courtroom. Value sits beside cost rather than inside the cost equation. Calculate expected net value as: ```text Expected net value = probability of successful completion × change value - expected total cost per merged change ``` This prevents a common mistake: rejecting a $40 route for a change worth $20,000 while accepting a $4 route for cleanup worth $3. Cost controls should be strict where value is low and tolerant where delay or failure would destroy much more value. ## Retries reveal the true model price Retry rate is often the term that reverses a routing decision. A model that costs one quarter as much per attempt is not cheaper if it needs five attempts and more reviewer intervention. Record retries as part of the route, including automatic repair loops that users never see. Separate three kinds of retry because they point to different fixes. A generation retry happens when the model fails to produce a usable patch. A validation retry follows a failed test, build, lint, or policy check. A clarification retry happens because the task lacked enough context or acceptance criteria. Switching models may help the first two. Better task preparation often fixes the third more cheaply. Use the probability of success on each attempt to estimate the expected attempt count. If attempts were independent with a constant success probability p, the expected count would be 1/p. Real attempts are not independent. The second prompt often contains a test failure and therefore better information, while repeated failures can show that the route is stuck. Observed attempt counts are better than the geometric shortcut once enough data exists. Set a retry ceiling as well as a cost ceiling. An agent that has failed twice on the same acceptance test usually needs a different route, not a more elaborate prompt. Continuing can inflate cost and anchor the reviewer to the model's mistaken approach. My default policy allows one repair attempt for routine tasks and then escalates. Teams can loosen that limit for cheap, deterministic tasks where tests provide strong feedback. A retry should preserve its cause. Store a short reason code such as build_failed, wrong_scope, missing_context, review_rejected, or tool_error. Without reason codes, a high retry number tells you only that money leaked. With them, you can decide whether to change the model, the context package, the test system, or the task definition. Do not reward a route for avoiding retries by producing giant speculative patches. Patch size, touched components, and coverage of acceptance criteria belong in the comparison group. Otherwise a model can appear to finish on its first attempt while handing risk to the reviewer. ## Review minutes can erase inference savings Review time should start when a human first inspects the proposed change and stop when the person approves, rejects, or takes ownership of the work. Include time spent reconstructing intent, checking suspicious code, rerunning tests manually, and explaining corrections to the agent. Exclude queue time because waiting measures staffing and scheduling, not review effort. Measure active review without turning engineers into timekeepers. A review interface can record focused sessions, or reviewers can choose a coarse bucket such as under 10 minutes, 10 to 30, 30 to 60, and over 60. Coarse data is enough to expose a route that saves $8 of inference and adds half an hour of senior attention. The reviewer rate must match the person the route actually consumes. A patch that requires a staff engineer because it changes authorization logic carries the staff engineer's loaded rate. Using a blended junior rate for every review systematically understates the cost of risky AI work. Review cost also has a nonlinear part. Small amounts fit between other work. Long, unpredictable reviews interrupt planned work and delay other merges. You do not need a complex queuing model on day one, but add a surcharge or escalate the route when the 90th percentile review time exceeds the team's acceptable review window. Track correction minutes separately from reading minutes. Long reading time can mean the diff is hard to understand. Long correction time means the route failed to deliver. Both cost money, but they demand different responses. Better explanations and smaller diffs can reduce reading time; a stronger model or human execution may reduce correction time. ## Failure rates need narrow attribution A production failure should affect a model route only when the change contributed to it. Blaming every incident on whichever model touched the repository creates useless data. Ignoring all incidents because attribution is imperfect is worse. Use the same evidence standard applied in an incident review: timeline, diff, tests, logs, and the actual failure mechanism. Measure at least two failure windows. Failure before merge covers rejected reviews and automated checks that could not be repaired within the retry limit. Failure after merge covers rollbacks, hotfixes, and incidents tied to the change during a defined observation window. Choose the window by system behavior. A web deployment may reveal trouble within days, while a monthly billing path needs longer. Do not compare raw failure rates across unlike tasks. A model routed mostly to documentation and test fixtures will beat a human assigned to payment concurrency bugs. Group by task class and risk tier, then compare routes within those groups. Useful classes include mechanical refactors, test generation, isolated defects, changes across components, security work, data migrations, and ambiguous product work. For rare, severe failures, historical averages can mislead because the sample is small. Apply a risk gate before the economic formula. Some tasks should require human ownership regardless of the current measured model cost: destructive data operations, changes to access control without complete tests, irreversible infrastructure actions, and work where the rollback path is unknown. This is not an argument to ban agents from risky code. Agents can investigate, draft tests, enumerate failure modes, and prepare a patch. The gate decides who owns execution and approval. Routing research assistance to a model differs from delegating the final change. Teams blur those modes and then wonder why their safety policy feels arbitrary. A near miss belongs in review data even if it caused no incident. If a reviewer catches a missing tenant filter, mark the change review_rejected with a reason with high severity. The production loss was zero because the control worked, but the route still demonstrated a dangerous failure mode. ## Change value sets the spending limit The completed change's value determines how much cost and delay the company should tolerate. Estimate value in ranges rather than pretending to know an exact number. A founder, product owner, or engineering lead can classify a change by avoided loss, expected margin, time saved, contract dependency, or strategic learning. Use four value bands with explicit local definitions. A change with low value is discretionary cleanup or convenience work with no current customer impact. A normal change supports planned product or maintenance work. A change with high value protects material revenue, removes a release blocker, or resolves a costly operating problem. An urgent change stops active loss or restores a required service. Value is not the number of lines changed, story points, or the seniority of the requester. A configuration correction on one line can restore checkout. A rewrite that takes a month can produce no measurable return. Ask what becomes possible, what loss stops, and what happens if the change waits one week. Delay cost belongs in the value assessment. If a human can finish tomorrow but the cheapest model route will cycle through review for three days, the route has consumed two days of value. Add a simple delay term when timing matters: ```text Delay cost = value lost per day × expected extra days to merge ``` The change value also caps how much experimentation makes sense. For work with low value, stop when expected delivery cost approaches the value. For work with high value, spend more on the strongest model, deeper review, or direct human execution when that raises the probability of completing safely and on time. Do not let requesters label everything urgent. Require one sentence of evidence for high and urgent bands, such as a blocked contract, current error volume, missed revenue, or a named release dependency. The evidence can be qualitative. It simply makes the trade visible and gives later reviewers something to audit. Some changes have learning value even if their immediate financial return is low. A small prototype can answer whether an integration is feasible. Record that expected decision value explicitly. Otherwise cost routing kills useful experiments because their output is knowledge rather than merged production code. ## A threshold should route the next dollar The routing decision compares the expected total cost and expected net value of the available execution paths. It does not ask whether a model is cheap in isolation. It asks which route should receive the next dollar for this class of change. Consider a hypothetical batch of 40 isolated backend fixes. The figures below are examples, not industry benchmarks. The cheaper model spends $6 per attempt, averages 1.8 attempts for each merge, needs 28 review minutes, and produces a failure cost attributed to the route of $18 per merge. The stronger model spends $18 per attempt, averages 1.15 attempts, needs 12 review minutes, and carries $7 of expected failure cost. The loaded reviewer rate is $2 per minute. ```text Cheaper route = $6 × 1.8 + 28 × $2 + $18 = $84.80 per merge Stronger route = $18 × 1.15 + 12 × $2 + $7 = $51.70 per merge ``` The stronger model wins even though its attempt price is three times higher. If both routes produce the same value and cycle time, paying more for inference saves $33.10 per merged change. A token dashboard would recommend the wrong model. Now change the task class to deterministic formatting fixes. Suppose the cheap route averages 1.05 attempts, takes four review minutes, and has negligible expected failure cost because exact checks cover the output. Its route cost becomes far lower. The correct policy sends formatting work to the cheap model and isolated backend fixes to the stronger one. A threshold can be expressed as the maximum extra inference spend justified by saved human effort and risk: ```text Maximum extra inference spend = review cost saved + repair cost saved + expected failure cost saved + delay cost saved ``` If the stronger model costs $12 more but saves $20 of review and $9 of expected failure loss, the upgrade has $17 of room. Route upward. If it saves only $5, keep the cheaper route unless completion value or a risk gate changes the decision. Compare human execution the same way. Estimate full human build, test, and review cost, then include human success rate and time to merge. Do not treat a developer as perfectly reliable or an agent as unattended. In many ambiguous tasks, a human will win because intent discovery dominates typing. In repetitive, tasks with good tests, the model route may win by a wide margin. The decision table can stay compact: ```yaml routes: mechanical_low_risk: default: cheap_model escalate_after_retries: 1 max_expected_cost_share_of_value: 0.35 isolated_product_code: default: strong_model escalate_after_retries: 1 max_review_minutes: 25 ambiguous_cross_component: default: human model_role: research_and_tests irreversible_or_access_control: default: human_owner model_role: draft_only ``` The cost share is a guardrail, not a law of finance. A company with spare engineering capacity may accept a different share than a startup with limited cash. Set it from your economics, then revisit it with observed outcomes. ## Cheaper models and humans fail in different places Route to a cheaper model when the task has a narrow specification, fast deterministic checks, a small blast radius, and cheap rollback. Examples include formatting, straightforward test expansion, known API migrations with compiler support, and repetitive changes where the repository already contains a correct pattern. Route to a stronger model when reasoning quality reduces retries or review: unfamiliar code, several interacting constraints, subtle bug diagnosis, or a change that needs a coherent plan across files. The stronger route still needs a cost ceiling. Expensive reasoning that keeps circling without producing evidence should escalate after the retry limit. Route to a human when the work depends on unresolved product intent, negotiation across teams, tacit system knowledge, or accountability for an irreversible action. A human is also the right route when validation cannot distinguish a plausible patch from a correct one. Models perform best when the environment can return specific evidence. Human execution does not mean banning model assistance. The owner can use a model to search the repository, draft tests, challenge assumptions, or explain unfamiliar code. Record that inference under the human route. This preserves an honest comparison and avoids the fiction that modern human work uses no AI. Do not route based on model reputation alone. A generally capable model can perform poorly with your monorepo, tools, or prompt format. A smaller model can be excellent on a constrained task with good examples. The threshold needs evidence specific to each route from your system. Do not use one global threshold such as $20 per change. It will overspend on chores and underinvest in important fixes. The minimum viable policy has task class, risk tier, and value band. Model choice follows those inputs. ## Collect enough evidence without building a bureaucracy Start with data already present in the delivery path. Agent logs provide inference spend and attempt identifiers. The code host provides review events and merge status. Continuous integration provides validation failures. Incident records provide outcomes after merge. Join them with a change ID and keep the raw events available for audit. A minimal record looks like this: ```json { "change_id": "chg_1842", "task_class": "isolated_product_code", "risk_tier": "medium", "value_band": "normal", "route": "strong_model", "inference_cost": 21.40, "attempts": 2, "review_minutes": 17, "repair_minutes": 6, "merged": true, "retry_reasons": ["build_failed"], "post_merge_outcome": "clean" } ``` Do not ask engineers to fill twenty fields. Infer what the tools know and ask humans only for task class, value band, active review bucket, and unusual outcomes. If a field never changes a decision, remove it. Review the table after a meaningful batch, not after every surprising change. Small samples swing wildly, especially for incidents. Until a route has enough examples, use a conservative prior: assume review and failure cost will be no better than the established route. New models earn broader routing by producing evidence. Use medians for typical spend and review time, but keep tail percentiles visible. A route that usually costs $10 and occasionally burns $300 can damage budgets and delivery schedules. Set both an threshold for expected cost and a stop loss for each change. Run controlled comparisons where risk permits. Send similar tasks with low risk to two routes, keep acceptance checks constant, and compare merged outcomes. Avoid selecting easy tasks for the favored model. If assignment cannot be random, document the task differences so the conclusion stays modest. Oleg.is uses this kind of operating evidence in Team & AI Audits to identify where model routing, review labor, and team design can actually reduce engineering cost. The useful deliverable is a decision rule tied to your repository and labor rates, not a generic list of model rankings. ## Recalculate when the work changes Routing thresholds decay. Model prices change, model behavior changes, repositories gain better tests, and reviewers learn how to work with agents. Recalculate on a fixed cadence and after a material tool or process change. Keep the old policy and its data so a cheaper invoice does not get mistaken for a better outcome. A budget cap and a routing threshold solve different problems. The budget cap limits aggregate spend over a week or month. The routing threshold decides whether the next change belongs on a particular route. A team can stay under budget while wasting money on weak routes, or exceed a small budget because work with high value arrived. Keep both controls, but never use the remaining monthly allowance as evidence that another attempt is economical. Test thresholds against counterfactuals during each review. Ask what the same change would have cost on the established route and whether the chosen route changed review time, delay, or failure exposure. You will not know the counterfactual exactly, but writing the estimate prevents a false claim that every completed AI task created savings. Completion proves delivery. Savings require a comparison. Assign one owner to the metric. That person should not be rewarded only for cutting inference spend, because the easiest way to do that is to push cost into engineering time. Give the owner a balanced target: lower expected total cost per merged change while holding failure limits and delivery time. The stop rule matters more than another decimal place in the formula. Every route should state when to escalate: a retry count, limit on review minutes, cost stop loss, failed safety check, or missing validation path. Agents are good at continuing. The business needs them to stop when the expected next attempt costs more than it can return. The decision record for a route should fit in a short table: task class, risk tier, value band, expected route costs, chosen owner, and the reason. That is enough for a founder or CTO to challenge the assumptions. If nobody can explain why a $30 model call was rejected while an hour of senior review was accepted, the routing policy is measuring the wrong thing. Set the first thresholds with imperfect data, mark the uncertainty, and make the next twenty changes improve the estimate. The goal is not the cheapest inference bill. It is to buy completed, correct changes at a cost that leaves room for the value they create.