# AI audit for inference costs and delivery savings

> An AI audit for inference costs connects model spend to review, rework, accepted changes, and releases to test whether delivery savings are real.

An AI coding rollout can cut payroll and still make each accepted change more expensive. The only honest test joins model charges to human review, rework, accepted changes, and releases. A five-day audit can establish that baseline if it follows work items through the delivery system instead of comparing an AI invoice with last month's payroll.

I have seen teams celebrate a falling cost per generated line while senior engineers spend afternoons untangling plausible code. I have also seen expensive model usage pay for itself because it removed days of waiting and moved small changes into production. Inference cost alone proves neither case. The unit that matters is the cost of an accepted change, with release frequency and failure signals beside it.

## Five days can answer a bounded cost question

A five-day AI audit can show whether inference costs erase delivery savings for the work your team actually completes during a representative window. It cannot prove an annual return with accounting precision, predict a new model's pricing, or separate every seasonal effect. It can give a founder a defensible decision: expand, restrict, redesign, or stop the current workflow.

The scope must stay narrow. Choose one team, one repository or service group, and two or three recurring change types. Good candidates include routine bug fixes, small product changes, test creation, dependency updates, or support-driven corrections. Do not mix a migration, an incident, and ordinary feature work into one average. Those jobs consume different amounts of context and review.

Five days is enough because most of the required evidence already exists. Model providers record usage. Version control records commits and pull requests. Review systems record comments and approvals. Deployment systems record releases. The missing pieces are usually the links among those records and a consistent definition of accepted work.

The audit should compare at least one prior non-AI period with the observed AI-assisted period. Use the closest comparable period available, normally several working weeks, and preserve its distribution rather than one average. If the team changed size, release policy, or product area, label the comparison as directional. A false precision sign is worse than an honest range.

Write the decision rule before collecting results. For example: continue the workflow only if cost per accepted change falls without a material decline in release frequency or an increase in production rework. This prevents a founder from moving the goalposts after seeing an attractive token bill or an ugly review figure.

## Cost per accepted change is the useful denominator

Cost per accepted change combines machine spend and human effort around work that survives review. Generated tokens, suggestions, commits, and pull requests are activity. They become output only when the team accepts the change and it reaches the workflow state chosen for the audit.

Define an accepted change before day one. For a team with continuous delivery, it may mean a merged pull request deployed successfully to production. For a mobile product, it may mean merged and included in a release candidate. For a regulated system, approval may require tests, security review, and a signed release record. Keep one rule throughout the sample.

Use this calculation for each work item:

```text
machine_cost = model_spend + agent_runtime + allocated_tooling
human_cost = (author_minutes + review_minutes + rework_minutes + release_minutes) / 60 * loaded_hourly_rate
accepted_change_cost = machine_cost + human_cost
```

The loaded hourly rate should include salary, payroll costs, benefits, and any overhead that finance already assigns consistently. Do not spend half the audit inventing a perfect rate. A reasonable company rate applied to both periods is better than different rates chosen to flatter the AI period.

Report the median and the 75th percentile, not only the mean. A few stalled changes can pull the mean upward, while a pile of trivial automated updates can pull it down. The median describes normal work. The upper quartile reveals whether the workflow produces an expensive tail that a team lead must keep rescuing.

Also report cost per accepted change by change type and risk class. A $20 documentation correction and a $900 authentication change should not cancel each other out. If AI helps routine tests but hurts unfamiliar backend work, the answer is a routing policy, not a company-wide verdict.

This distinction exposes a common accounting error. Payroll savings are capacity savings only when the company removes cost, avoids planned hiring, or uses the released time for more accepted work. Treating every saved hour as cash creates fictional savings while payroll stays unchanged.

## Model spend needs work-item attribution

Model spend must be attached to a work item, agent run, team, and time window. A monthly account total cannot tell you whether a merged fix was cheap or whether an abandoned experiment consumed the budget. Attribution can be imperfect, but unassigned spend must remain visible.

Collect billed cost when the provider supplies it. If only usage units are available, preserve input, output, cached input, and any other billed category separately, then apply the price valid for that usage period. Do not recalculate an old baseline with today's price. Provider discounts, prepaid commitments, and routing layers can make list-price estimates diverge from the invoice.

The FinOps Open Cost and Usage Specification, known as FOCUS, defines common billing dimensions and cost measures across technology vendors. Its treatment of billed cost and effective cost is useful when commitments or discounts exist. But FOCUS cannot connect a model charge to an accepted pull request for you. The audit still needs engineering identifiers.

A minimal event record can look like this:

```json
{"timestamp":"2026-07-14T10:42:18Z","team":"payments","work_item":"PAY-1842","pull_request":"4821","agent_run":"run_0192","model":"model-family","input_units":18420,"cached_units":12100,"output_units":2360,"billed_cost_usd":2.84,"status":"completed"}
```

Pass the work item ID into agent runs through environment metadata or a wrapper command. If a tool cannot carry custom metadata, join records by repository, actor, and a tight time window, then mark the confidence of that join. Never silently assign a whole day's spend across merged pull requests. Abandoned runs, retries, prompt experiments, and work without a pull request belong in an `unallocated` bucket.

Shared costs require a written allocation rule. Divide a fixed agent subscription by active users, tracked agent hours, or accepted changes, depending on how the contract behaves. Record the rule next to the result. Hosted runners and search or indexing services count as agent runtime or allocated tooling when the AI workflow causes them.

Keep cash cost and normalized cost side by side. Cash cost answers what the company paid during the window. Normalized cost removes temporary credits or unusual commitments so the team can compare future options. A free credit can make the first week look wonderful and say nothing about steady operation.

## Review time is part of inference cost

Every minute spent validating AI output belongs in the AI workflow's cost, even when it appears on a senior engineer's calendar rather than the provider invoice. Review time often decides whether a cheap model is actually cheap.

Measure active review time, not the elapsed time between review request and approval. A pull request waiting overnight has long lead time but consumes no reviewer labor during the wait. Browser activity, review-session events, or a small timer action can estimate active minutes. If instrumentation is unavailable, ask reviewers to record ranges such as 0 to 10, 11 to 30, 31 to 60, and over 60 minutes immediately after approval.

Separate normal review from AI verification. Normal review asks whether the design and implementation fit the product. AI verification checks claims the author did not personally establish: whether a library call exists, whether an edge case was invented away, whether tests assert behavior rather than repeat implementation, or whether broad edits hide inside generated noise.

Track who reviews. Moving fifteen minutes from a mid-level author to forty minutes from a staff engineer increases cost and creates a scarce-person bottleneck. The pull request may still close faster because the author produced it quickly, yet the organization loses senior capacity elsewhere.

Review comments alone are a weak proxy. One comment can identify a deep design error, while twelve formatting comments take minutes. Capture active minutes, the number of review rounds, and the reason for each requested change. A compact reason set is enough:

- Incorrect behavior or missed requirement
- Security, privacy, or reliability concern
- Maintainability or unnecessary complexity
- Test weakness
- Style or repository convention

Do not ask reviewers to judge whether code "looks AI generated." That opinion changes nothing. Ask what verification work the change required and why. The audit needs cost and failure mechanisms, not a culture argument.

Approval latency still matters beside active minutes. If AI raises the volume of large pull requests, queues grow even when the time per review stays flat. Report median time from ready-for-review to first substantive review and from ready-for-review to approval. Those figures explain why apparent author speed may fail to improve delivery.

## Rework must follow the change after merge

Rework includes corrections before acceptance and unplanned work caused by the accepted change after merge. Counting only the first pass rewards fast generation while another engineer pays for the cleanup.

For pre-merge rework, record minutes spent responding to review, replacing generated code, repairing tests, and reopening the design. For post-merge rework, link hotfixes, rollbacks, incident tasks, and bug fixes to the originating change. Use an explicit relation such as `caused_by`, not a guess based on the same files.

DORA separates delivery throughput from instability and now includes deployment rework rate alongside change fail rate. That separation is right: more releases do not compensate for releases that require intervention. For a five-day audit, DORA metrics provide guardrails, while work-item rework minutes provide the cost detail that the standard delivery measures do not.

One failure pattern appears repeatedly. An agent generates a broad refactor while fixing a narrow defect. The author sees passing tests and opens a polished pull request. A reviewer spends fifty minutes finding a changed retry path, sends it back, and the author regenerates part of the patch. The merged change later causes duplicate requests under timeout, so a different engineer ships a hotfix. If the audit counts the first generation and final merge only, AI appears extremely productive. A complete record assigns the review, regeneration, hotfix, and release labor to the original work item.

Set a rework observation window that extends beyond the five audit days. Fourteen or thirty days is often practical, depending on release cadence. The report can be issued on day five with a provisional flag and then updated when the window closes. Without this tail, Friday's merged changes receive less scrutiny than Monday's.

Do not label all later edits to the same file as rework. Product requirements change, adjacent work lands, and cleanup may have been planned. Require a ticket relation, rollback reference, incident link, or engineer confirmation. Conservative attribution makes the result more credible.

Compare rework as minutes per accepted change and as a share of total human minutes. The share matters because a faster first pass can hide a growing cleanup burden. A workflow that cuts authoring by two hours and adds twenty minutes of review may be sound. One that cuts twenty minutes and adds an hour of senior review is not.

## Release frequency needs quality beside it

Release frequency shows whether lower unit cost turns into delivered capacity. It should sit beside change failure and rework, because a team can inflate deployment count by splitting work or shipping low-risk changes.

Count successful production releases for the audited service, using the team's established definition. If ten pull requests ship in one deployment, that is one release for frequency and ten candidate accepted changes for unit cost. If one pull request deploys independently behind several environment promotions, count the production event, not every promotion.

Use the same service boundary and calendar treatment in both periods. Weekends, release freezes, public holidays, and incident days distort short samples. Note them explicitly. Comparing a normal week with a launch week produces a number, but it does not produce an answer.

Pair frequency with these guardrails:

- Change lead time from commit to successful production deployment
- Change fail rate for deployments requiring remediation
- Deployment rework rate for unplanned corrective deployments
- Failed deployment recovery time

The current DORA guidance groups change lead time and deployment frequency under throughput, and change fail rate and deployment rework rate under instability. I would not compress them into one score. A composite hides the reason a workflow changed and lets arbitrary weights determine the verdict.

Accepted changes and releases answer different questions. Accepted changes measure the economic unit of engineering work. Releases measure flow through the delivery system. When accepted change cost falls but release frequency stays flat, inspect queues, batch size, test duration, and release approval. AI may have improved coding while another constraint owns delivery.

The reverse result also occurs. Release frequency rises because the team ships smaller changes, while cost per accepted change rises slightly. That can still be a good trade if lead time and risk fall, but the founder should choose it knowingly. The audit should show the trade, not declare every higher frequency a win.

## Build one evidence table before calculating savings

One row per work item should contain enough evidence to reproduce every reported metric. Start with raw events, then build the table through explicit joins. Do not begin in a slide deck.

The practical sequence is:

1. Select the team, repositories, change types, comparison window, and acceptance rule.
2. Export provider usage, agent-run records, pull request events, time estimates, and deployment events.
3. Join usage to work items and mark unmatched cost instead of spreading it invisibly.
4. Classify accepted, rejected, abandoned, and still-open work, then attach review and rework minutes.
5. Calculate unit cost distributions and release guardrails, then inspect outliers before making a recommendation.

A useful evidence row includes `work_item_id`, `change_type`, `risk_class`, `ai_assisted`, `machine_cost`, `author_minutes`, `review_minutes`, `rework_minutes`, `accepted_at`, `deployed_at`, `release_id`, and `post_merge_failure`. Add source record IDs so an engineer can trace the row back to usage and delivery systems.

Use a status field rather than deleting incomplete work. `accepted`, `rejected`, `abandoned`, and `open` preserve the cost of attempts. The primary cost-per-accepted-change calculation divides all workflow cost for the chosen cohort by accepted changes. A secondary accepted-only view helps diagnose expensive successes, but it must not erase failed attempts.

Suppose the AI-assisted cohort has $420 in model and runner spend, 96 human hours at a loaded $110 per hour, and 24 accepted changes. Its total cost is $10,980, or $457.50 per accepted change. Suppose the comparable cohort has $80 in tooling, 118 human hours at the same rate, and 22 accepted changes. Its total is $13,060, or about $593.64 per accepted change. The AI workflow saves about $136 per accepted change in that sample, even though inference spend rose by $340.

Now add six hours of post-merge correction linked to the AI cohort. The adjusted total becomes $11,640, or $485 per accepted change. The workflow still wins in the sample, but by less. If two changes remain inside the observation window, report the result as provisional rather than pretending the tail is complete.

Show a range for uncertain time. If review records use time bands, calculate a low and high estimate. A result that wins at both ends is actionable. A result that flips sign deserves more measurement or a narrower routing rule.

## Each audit day has one job

The five-day limit works when every day closes a specific evidence gap and the team avoids rebuilding its analytics stack. The audit is an intensive measurement exercise, not a data warehouse project.

On day one, agree on scope and definitions with the founder, engineering lead, and whoever owns the invoice. Select the cohort, acceptance rule, comparison period, loaded labor rate, observation tail, and prewritten decision thresholds. Pull a small sample of work items and trace them manually from model run to release. This exposes missing identifiers before anyone exports thousands of events.

On day two, collect and normalize machine cost. Reconcile provider usage with the billed or effective cost that finance recognizes, then connect agent runs to repositories and work items. Classify unmatched charges instead of hiding them. By the end of the day, the team should know what share of spend has a reliable work-item link and which tool creates the gaps.

On day three, reconstruct human effort and outcomes. Pull review events, ask engineers for bounded estimates where active time is missing, and classify requested changes. Join accepted work to deployments and identify rejected, abandoned, and open attempts. This is also the day to interview reviewers. A twenty-minute conversation often explains an outlier that event timestamps cannot.

On day four, calculate distributions and challenge the joins. Compare medians, upper quartiles, change types, risk classes, and reviewers. Trace the cheapest and most expensive accepted changes back to raw records. Look at abandoned runs with high cost and merged changes with unusual review or rework. If a result depends on one questionable record, publish it with and without that record.

On day five, decide where the workflow belongs. Present the evidence table, assumptions, confidence levels, unit economics, and delivery guardrails. Turn findings into routing rules, spend limits, instrumentation changes, and named owners. Mark post-merge rework as provisional where the observation tail remains open, and schedule the date when that field will close.

The report should include both findings and data quality. A simple coverage table can show the percentage of model spend attributed to work items, accepted changes linked to releases, pull requests with review-time evidence, and changes with a completed rework window. Low coverage does not automatically invalidate the audit, but it widens the result and may become the first repair.

Do not delay the report because one provider export is messy. Use invoice totals as the team-level machine cost, traced records for work-item analysis, and an explicit unallocated amount. The founder then sees both the economic answer and the instrumentation debt. Pretending that unattributed spend does not exist would bias the result in exactly the direction the audit is meant to test.

The audit also needs access boundaries. Usage exports can contain prompts, repository names, user identifiers, or fragments of sensitive work. Collect the minimum fields needed for cost attribution, restrict raw access, and retain derived records without prompt content when possible. An audit of AI economics should not create a new data exposure.

Finally, preserve a reproducible snapshot. Record export times, pricing inputs, allocation rules, rate assumptions, query versions, and manual corrections. Provider dashboards change as late charges settle. Without a frozen input set, two people can rerun the same analysis a week apart and argue over different totals.

## Savings claims must pass three tests

A delivery savings claim should pass economic, operational, and counterfactual tests. Failing any one means the founder has evidence of activity, not evidence of savings.

The economic test asks whether total cost per accepted change fell. Include model, runtime, tooling, author, review, rework, and release labor. Show cash and normalized machine cost. Keep unallocated model spend in the numerator for the team-level result.

The operational test asks whether delivery remained healthy. Release frequency, lead time, change fail rate, recovery time, and deployment rework should not deteriorate beyond a threshold the team set in advance. A five-day window may contain no failures, so do not call that proof of safety. State the sample size and continue the observation window.

The counterfactual test asks what the company would have paid or delivered without the AI workflow. Use the nearest comparable non-AI cohort and adjust for team size and work type. Avoid story-based counterfactuals such as "this would have taken a week." Repository history and ticket flow are imperfect, but they beat memory.

Translate time into one of three outcomes. Cash savings means an expense actually disappears. Avoided hiring means documented demand can be met without a planned role. Reinvested capacity means the same payroll produces more accepted work or resolves another measured constraint. Keep these labels separate. Only the first changes this month's payroll.

Inference cost erases delivery savings when its increase plus induced human work exceeds the labor or capacity benefit for comparable accepted output. The phrase "induced human work" matters. A model with a lower token price can cost more if it triggers extra runs, larger outputs, more review, and more correction.

Do not annualize a five-day percentage without a range and operating assumptions. Work mix, discounts, model choice, and adoption behavior will change. Apply the observed unit economics to realistic monthly volumes, then show a conservative, base, and high-usage case. That is planning, not a promise.

## The audit should end with a routing decision

The strongest audit result is a policy for which work uses which workflow. A universal "AI works" or "AI fails" conclusion throws away the variation that the evidence table reveals.

Create routing rules from change type, risk, repository familiarity, and observed cost. The data may support AI for test scaffolding, narrow bug fixes, and internal tools while requiring a human-led design for authorization, billing, or unfamiliar infrastructure. It may support a smaller model for classification and a stronger model for implementation. It may also show that one agent configuration should stop entirely.

Set a budget at the work-item level and a stop condition at the run level. If repeated attempts cross the expected human cost, route the item to an engineer. If generated changes exceed a reviewable size, require the author to split or rewrite them. If unallocated spend grows, block runs that lack a work item ID.

Assign ownership for the next thirty days. Engineering operations can maintain usage joins, team leads can sample review classifications, and finance can reconcile cash cost. Review the unit economics weekly at first, then monthly once attribution stabilizes. Do not make developers fill in a large spreadsheet forever; instrument the joins that proved useful and remove fields nobody used.

A Team & AI Audit from oleg.is is a fixed $5,000 engagement completed in five business days, with a guarantee of at least $50,000 a year in identified savings or the audit is free. The useful output is the same one I would demand internally: an evidence table, explicit assumptions, unit economics, delivery guardrails, and a routing decision.

The uncomfortable outcome may be that the current AI setup loses money. That is still a successful audit if it identifies the expensive change types, review bottleneck, or missing attribution that caused the loss. Stop paying for a broad story about productivity. Keep the workflows that reduce the cost of accepted work, and make the rest earn their place with evidence.