An AI coding tool audit can beat another renewal

Buying six AI coding tools is easy. Knowing which ones improve delivery is much harder. A founder facing another annual renewal should buy an audit when one renewal cycle exposes more money than the audit costs and the team cannot connect each tool to accepted production changes. The decision should come from evidence, not developer enthusiasm or a procurement spreadsheet.

The usual comparison is too shallow. Finance sees seat prices, engineering sees useful demos, and nobody counts metered inference, review time, abandoned generated code, or the same capability purchased in several interfaces. A five-day review can settle the renewal question if it follows work from invoice to merged change. It cannot settle it by counting prompts.

I have watched teams cancel a cheap assistant while keeping an expensive agent because the cheap line item looked idle. Three weeks later, the agent bill rose and review queues slowed: the supposedly idle assistant had handled small completions that kept the agent away from routine work. Tool consolidation can save money, but blunt consolidation often moves cost instead of removing it.

Set the decision threshold before collecting data

An audit is worth buying when the value at risk exceeds its price by a comfortable margin and management lacks a defensible renewal decision. Start with annual exposure, not this month's invoice. Renewal exposure includes committed seats, expected usage charges, implementation or integration work, internal administration, and the cost of switching away.

Use this calculation for each tool:

annual exposure = annual seat commitment
                + expected metered charges
                + internal administration hours × loaded hourly cost
                + contractually unavoidable migration cost

Do not add hypothetical productivity gains to exposure. Exposure measures what the company can spend or become committed to spending. Benefits belong on the other side of the decision. Mixing the two lets an optimistic productivity claim hide a real cash obligation.

Suppose six renewals total $72,000 in seats. Metered model use has run at $2,500 per month and is rising, while engineering operations spends roughly eight hours per month on access, billing, and policy work. A $5,000 audit does not need to discover a dramatic failure to pay for itself. It needs to prevent or renegotiate a modest part of the coming commitment, or find delivery waste with the same value.

There are cases where buying an audit makes little sense. If the whole annual stack costs $4,000, every assigned user works in the tools weekly, metered charges have a hard cap, and one owner already reviews outcomes each month, pay attention but do not create ceremony. The same applies when contracts can be canceled monthly with no migration cost. Run the measurement internally and revisit it when spending or team size changes.

Set written decision rules on day one. For example: renew a tool if it owns a distinct job, has sustained active use among intended users, and produces accepted changes at a competitive fully loaded cost. Renegotiate if the tool works but seat count or commitment is wrong. Cancel if another retained tool covers the same job with equal or better results. A rule written after the numbers arrive tends to protect whichever tool has the loudest sponsor.

A paid seat and an active developer are different facts

Seat utilization should answer whether the right people use a paid entitlement often enough for the job it was bought to do. A login, an installed extension, or one prompt during the month does not establish useful adoption. Vendors define activity differently, so copying six dashboard percentages into one table creates false comparability.

Build a seat ledger with one row per person and tool. Record assignment dates, eligible workdays, meaningful active days, the primary job, and the last observed output. A meaningful active day might require an accepted completion, an agent-created branch that reaches review, or a code-review finding that the developer acts on. Define the event before looking at the data and keep it consistent inside each tool.

GitHub's Copilot usage documentation makes a useful distinction between licensed, active, and engaged users. Its metrics also separate suggestions from acceptances and report fields such as lines added. That is better than treating an extension heartbeat as benefit. I would still qualify the vendor metric: accepted editor text is an intermediate event, not proof that the change survived review, tests, and production.

Use two utilization rates rather than one:

seat activation = users with meaningful activity / assigned seats
seat intensity  = meaningful active days / eligible user-days

Activation catches shelfware. Intensity catches a different problem: everyone tries the tool once, so the dashboard looks healthy, but almost nobody makes it part of normal work. Segment both by role and intended job. A reviewer who uses an AI review tool twice a week may be a strong user; a full-time developer assigned an inline assistant but active twice a month probably is not.

Ask about missing use directly. Some seats are idle because the tool is poor. Others are idle because setup fails, repository policy blocks it, latency irritates users, or nobody taught the team where it fits. Canceling the second group without testing the obstacle wastes the setup cost and teaches procurement nothing. A short interview plus evidence from the ledger separates product rejection from deployment failure.

Before renewal, reclaim seats inactive for an agreed period, but keep a small reassignment pool if the contract permits it. Do not force every engineer onto one tool merely to improve the chart. Standardization has operational value, yet a mobile specialist, a database engineer, and a staff reviewer may need different workflows. The ledger should expose justified exceptions rather than erase them.

Duplicate capability costs more than duplicate logos

Six products are not automatically wasteful, and two products can still duplicate nearly everything. Compare the jobs the team buys, not marketing categories. Inline completion, repository chat, multi-file editing, autonomous issue work, pull request review, terminal assistance, test generation, and model access are separate jobs even when a vendor puts them under one subscription.

Create a capability map with tools as columns and jobs as rows. Mark each cell as primary, fallback, incidental, or unused. Require evidence for a primary label: named users, a recurring workflow, and at least a few accepted outcomes. If three tools claim to be primary for multi-file edits, the map has found a decision, not resilience.

The field often blurs capability overlap with workflow redundancy. Capability overlap means two tools can technically do the same thing. Workflow redundancy means the team actually pays, configures, and trains people to perform that job in both. The first can provide useful fallback or bargaining power. The second usually creates cost, fragmented context, and inconsistent controls.

A common recommendation says to choose one vendor suite because consolidation simplifies billing. It is popular because procurement can show an immediate seat reduction. It is wrong when a broad suite handles a specialist job badly enough to increase rework, or when it routes expensive models into tasks a cheaper assistant handled well. Consolidate only after comparing the same job on the same repositories and change classes.

Run a controlled overlap test during the audit. Pick completed, non-sensitive tasks that represent routine work: a small bug fix, a test addition, a dependency update, and a contained refactor. Give comparable starting context to the tools and record hands-on time, elapsed time, inference cost, review findings, and whether the result would be accepted. Do not merge duplicated test work; the point is comparison, not production volume.

The map should also capture non-code differences that affect the renewal. One tool may support central policy while another relies on individual settings. One may fit the team's identity and repository controls while another creates manual access work. These are not vague enterprise features. Convert them into administration hours, blocked workflows, or risk decisions, then place them beside delivery results.

At the end, every retained tool needs a one-sentence charter: who uses it, for which job, under which spending limit, and what evidence will justify the next renewal. If the owner cannot write that sentence, the company is funding optionality without deciding what the option is for.

Inference charges need their own cost model

Seat fees and inference charges behave differently and should never share a single utilization percentage. A seat is paid capacity. Inference is consumption that can vary with model, context size, retries, agent loops, caching, and provider route. A tool with excellent seat adoption can still produce a bad bill if nobody controls expensive runs.

Reconcile costs to financial records first. OpenAI's Usage API documentation explicitly recommends the Costs endpoint or the Costs tab for financial reconciliation because granular usage data may differ from billed costs. That distinction matters across vendors: telemetry helps explain behavior, while invoices establish spend. Use both, but do not ask a prompt count to balance the ledger.

Normalize metered spending into a common table:

date,tool,user_or_team,repository,job,model,cost_usd,change_id,outcome
2026-06-03,Tool A,platform,api,bug_fix,model_x,4.82,chg-184,accepted
2026-06-03,Tool B,platform,api,test_generation,model_y,1.14,chg-185,reworked

Many systems will not provide every field. Add repository and job through project keys, gateway tags, wrapper scripts, or a small developer sample. Never fabricate missing attribution. Put unattributed spend in its own row and treat a large unattributed share as a control problem.

Anthropic's Claude Code gateway documentation describes centralized usage tracking, end-user tracking, budgets, rate limits, and audit logging as reasons to place a gateway in front of model access. A gateway can help, but it is not a complete answer. Subscription traffic may bypass it, IDE telemetry may use another identity, and agents can call external services. Reconcile gateway records with vendor invoices and tool-level activity.

Cost per prompt is rarely useful. A one-line completion and an agent repairing a failing migration are not comparable requests. Cost per token is useful for checking routing and price, but it does not measure work. Group charges by job and change, then inspect the distribution. Averages alone hide a few runaway sessions that account for much of the bill.

Put caps where the team can react to them. Set project budgets, alerts before the hard limit, approved model routes by job, and an owner for exceptions. A hard cap without an escalation path can stop an urgent fix. An alert without an owner becomes email noise. The audit should leave a named person and a response rule, not merely a colorful spending chart.

Rework decides whether accepted output had any value

An accepted suggestion is not an accepted change. The first means a developer inserted generated text. The second means the team reviewed the resulting change, passed its required checks, and kept it. Confusing those events rewards tools that generate large volumes of plausible code and transfers the cleanup to reviewers.

Define an accepted change at the unit your delivery system can trace. For many teams, that is a merged pull request linked to an issue or change request. For small operational fixes, it may be a reviewed commit or an approved infrastructure change. Whatever unit you choose, require a stable identifier that can connect tool activity and cost to the outcome.

Measure rework in observable categories:

• Author correction time after generation but before review
• Reviewer-requested changes caused by correctness or maintainability problems
• Test repair attributable to generated code
• Reverts or follow-up fixes within an agreed observation window
• Security or policy corrections before acceptance

Do not count all review time as AI rework. Humans review human-written code too. Compare against a baseline for similar change classes, or ask reviewers to tag the reason for material revision. A simple label such as ai-correctness, ai-scope, or unrelated produces better evidence than trying to infer intent from comment text later.

Consider a failure I have seen in several forms. An agent completes a dependency upgrade in twelve minutes and opens a convincing pull request. CI passes, but the reviewer spends ninety minutes finding that the agent changed a retry default and removed a compatibility branch outside the requested scope. The author then spends forty minutes restoring behavior and rerunning tests. A dashboard records one completed task and high generated-line acceptance. The company actually bought twelve minutes of generation plus more than two hours of human correction and review.

Track both gross and net time:

gross saved time = baseline hands-on minutes - AI-assisted author minutes
net saved time   = gross saved time - excess review minutes - AI-caused repair minutes

A negative net result does not always require cancellation. The tool may need narrower task selection, better repository instructions, a cheaper model, or a review boundary. But repeated negative results in the job named on the tool's charter remove the case for renewal.

Quality also needs a denominator. Ten AI-related defects sound terrible until you learn the tool touched thousands of changes; one defect sounds harmless until you learn it touched two. Compare defects, reversions, and material review findings per accepted change, segmented by change class. Do not combine documentation edits with authentication changes and call the average quality.

Cost per accepted change makes tools comparable

Cost per accepted change combines cash and human effort at the point where work becomes useful. It is not a universal productivity score, and it should not rank individual developers. It is a decision measure for comparing tools that attempt the same job under similar conditions.

Calculate fully loaded cost this way:

tool cost = allocated seat fees
          + attributed inference charges
          + tool administration cost
          + AI-caused author repair cost
          + excess review cost

cost per accepted change = tool cost / accepted changes

Use loaded hourly rates for internal labor, not salary alone. Keep the rate assumptions visible and apply the same method across tools. If legal, security, or platform staff perform unusual setup work during the measurement window, separate one-time adoption cost from recurring administration so it does not distort steady-state operation.

Allocation needs judgment. Divide seat fees across the jobs a user performed, preferably by meaningful activity or traced changes rather than prompt volume. Assign inference directly through project or change identifiers when possible. If a tool bundles inference into the seat, do not invent a token charge; leave it in the seat fee and compare total cost.

Now take an illustrative month. Tool A costs $3,000 in seats and metered use, adds $1,200 of excess review and repair, and supports 70 accepted changes. Its fully loaded result is $60 per accepted change. Tool B costs only $1,600 on the invoice but adds $2,000 of excess labor and supports 30 accepted changes. Its result is $120. The cheaper invoice is the more expensive production path.

Do not compare Tool A's routine completions with Tool B's complex migrations. Segment changes into a small number of stable classes, such as routine maintenance, tests, feature work, refactoring, and incident work. Compare within a class, then decide whether a tool earns a narrow charter even if its overall average looks worse.

Add a confidence marker to every result. High confidence means costs and outcomes connect through identifiers. Medium confidence means the team sampled work and reconciled totals. Low confidence means interviews or aggregate dashboards carry most of the conclusion. A founder can renew on medium-confidence evidence when the financial risk is modest. A large annual commitment deserves a better trace.

This metric can be gamed, so prohibit its use for performance reviews. Developers will split changes, avoid hard tasks, or hide AI involvement if they think a ratio affects compensation. Report at tool-and-job level with minimum sample sizes that protect individuals. The purpose is purchasing and workflow design.

Five days is enough if every day ends with evidence

A five-day audit can support a renewal decision because it analyzes existing invoices, telemetry, repositories, and interviews. It does not prove long-term productivity in five days. Treat it as a focused financial and workflow investigation, with limited tests to close important gaps.

A practical sequence looks like this:

1. On day one, freeze the tool list, contracts, owners, decision rules, and twelve months of invoices. Build the seat ledger and record renewal dates.
2. On day two, export usage, map capabilities to jobs, and interview a small cross-section of developers and reviewers. Identify missing attribution.
3. On day three, join costs to repositories and accepted changes. Sample rework and run only the overlap tests needed to resolve disputed jobs.
4. On day four, calculate seat activation, intensity, recurring cost, rework, and cost per accepted change by job. Challenge weak assumptions with tool owners.
5. On day five, issue renew, reduce, renegotiate, or cancel decisions, plus owners, savings ranges, control changes, and dates for follow-up.

The working file should be reproducible. At minimum, keep source name, extraction time, covered period, filters, transformation notes, and the person who confirmed each assumption. A spreadsheet is fine for a small team. The artifact matters more than the software used to create it.

Require a decision record for every tool:

tool: Tool A
renewal: reduce from 25 seats to 14
charter: inline completion for active application developers
cash effect: $X annual commitment avoided
workflow effect: move autonomous issue work to Tool C
owner: VP Engineering
review date: 2026-10-01
confidence: medium

Dollar values must come from contracts and current bills; placeholders remain placeholders until verified. Savings should be a range when usage varies. Keep avoided commitment separate from actual cash savings. Reducing a future order is not the same as recovering money already paid.

The audit should also name unresolved questions. Perhaps the repository cannot link agent sessions to pull requests, or an annual contract prevents immediate seat reduction. State the limitation and assign an action. A report that hides uncertainty behind a precise total creates a worse renewal decision than an honest range.

The Team & AI Audit described on oleg.is costs $5,000, runs for five business days, and carries a stated guarantee of at least $50,000 per year in identified savings or it is free. Those terms make the buying threshold explicit, but a founder should still insist on the evidence trail and per-tool decision record described above.

Renewal decisions need an owner and an exit path

The output is not a scorecard. It is a set of commercial and operating decisions with accountable owners. Assign each tool one of four actions: renew, reduce, renegotiate, or cancel. Pair the action with a date, a responsible person, and the workflow changes needed to make it real.

Renew when the tool owns a distinct job, intended users sustain meaningful activity, outcomes meet the team's quality bar, and fully loaded cost compares well within that job. Reduce when the tool works but assigned capacity exceeds use. Renegotiate when usage is healthy but contract structure, included inference, or commitment length places too much risk on the buyer. Cancel when duplicated workflow, poor outcomes, or missing adoption persists after fixable deployment issues have been tested.

Do not cancel on renewal day. Export any records the team needs, remove secrets and integrations, document replacement workflows, reassign affected users, and set a short observation period. Check whether generated code or stored context has retention obligations under company policy. The audit need not become a legal review, but it must flag questions for the right owner.

Negotiation improves when the company can describe use precisely. Ask for a seat band that matches active users, pooled or transferable seats where offered, usage caps, project-level attribution, renewal notice that leaves time to test alternatives, and contract language that separates base commitment from variable inference. Vendors may decline. Evidence still tells the founder which concession matters most.

Keep one compact monthly view after the audit: assigned seats, meaningful users, total recurring cost, unattributed inference, accepted changes by job, rework rate, and cost per accepted change. The owner should explain material movement, not narrate every fluctuation. Review tool charters quarterly and before any headcount or repository expansion.

The most dangerous outcome is a neat consolidation plan that engineers quietly work around with personal subscriptions and untracked API keys. Give teams an approved exception route with a budget, security review, and expiration date. Shadow use often signals a missing job in the official stack. Treat the signal seriously without turning the audit into a hunt for offenders.

A founder should approve the next renewal only when each retained tool has a job, an owner, a spending boundary, and evidence that accepted work costs less or improves enough to justify the difference. Six tools may survive that test. Two may survive. The number is not the target. The target is a stack whose bill and delivery record tell the same story.