Coding agent lock-in needs an exit price

Proprietary coding agents can save real money and still be a bad purchase. The deciding number is not the subscription price, the number of generated lines, or the hours developers say they saved. It is the cost of each accepted feature after review and rework, adjusted for the effort required to leave the vendor.

An engineering audit can produce that number, but only if it treats the agent as part of the delivery system. Measure work that reached production, include human cleanup and workflow maintenance, then run an exit test against a representative repository. If the savings disappear when you price migration and dependency risk, the apparent productivity gain was borrowed from your future team.

Lock-in needs a price, not a label

Coding agent lock-in is the cost and operational disruption of replacing an agent after its assumptions, files, integrations, and habits have entered normal engineering work. Calling a tool "proprietary" does not tell you whether that cost is tolerable. A closed model behind a standard command-line interface may be easy to replace. A tool that exports its prompts may still be hard to leave if every issue, review, permission rule, and deployment depends on its hosted control plane.

I separate lock-in into four balances. The first is stored knowledge: instructions, prompts, memories, examples, and project rules that the agent needs. The second is execution: tools, hooks, permissions, sandboxes, and remote environments. The third is evidence: transcripts, approval records, cost data, and the link between a task and its resulting commits. The fourth is team behavior: how people split tickets, review changes, and recover when the agent fails.

Repository portability covers the first two balances only when the required material lives in version control and another runner can understand it. Workflow portability is broader. It asks whether the team can continue planning, executing, reviewing, and auditing work without the vendor service. A repository that clones cleanly can still carry a thoroughly nonportable operating model.

This distinction changes the audit. Do not assign an emotional score such as low, medium, or high lock-in. Estimate the engineering days and business interruption required to replace each balance. Record what can be converted mechanically, what needs human judgment, what will be lost, and what cannot be tested until a contract ends. An exit price gives a founder something useful to compare with annual savings.

Lock-in is not automatically disqualifying. Teams accept managed databases, cloud queues, and payment providers because the benefit exceeds a known switching cost. The mistake is accepting an unknown switching cost for an agent that touches a growing share of changes. If the vendor dependency saves $180,000 a year and a credible exit costs $25,000, that may be a sensible trade. If nobody can produce the files, people, and elapsed time behind the $25,000 estimate, it is not an estimate.

Accepted feature cost is the productivity measure

The right unit is the cost of an accepted feature or fix that satisfies the team's normal definition of done. "Accepted" means a reviewer approved it, automated checks passed, it reached the intended environment, and it did not immediately return as rework. This avoids rewarding an agent for creating code that a human later rewrites or removes.

Start with comparable work, not all work. Choose completed items from the same repository and work class, such as small API features, production defects, or dependency upgrades. Do not compare agent-assisted maintenance tickets with unaided architecture changes. Tag items before looking at the outcome, or at least have someone other than the agent champion approve the classification.

For each item, calculate:

accepted_feature_cost =
  developer_minutes * loaded_cost_per_minute
  + reviewer_minutes * loaded_cost_per_minute
  + compute_and_license_cost
  + attributable_rework_cost
  + workflow_maintenance_share

Developer minutes include prompting, waiting that blocks other work, inspecting the repository, correcting the agent, resolving conflicts, and preparing the change for review. Reviewer minutes matter because some agents move effort rather than remove it. Compute and license cost should use actual invoices or metered usage. Rework includes follow-up changes caused by defects or missed requirements within a fixed observation window chosen before the audit.

The workflow maintenance share is easy to hide. Count time spent updating instruction files, repairing tool connectors, managing permissions, chasing model regressions, curating context, and teaching the team new vendor behavior. Allocate that cost across accepted items during the measurement period. Do not put initial experimentation into every future feature, but do not pretend recurring care is free.

Use the median cost for the primary comparison because a single incident or unusually large ticket can distort an average. Keep the full distribution beside it. The expensive tail tells you whether the agent fails predictably or occasionally creates a review disaster. Report acceptance rate as a separate measure: submitted agent-assisted items that became accepted items without being abandoned or wholly rewritten.

Some teams have no usable unaided baseline because the agent arrived before time tracking or now touches nearly every ticket. Do not manufacture one from developers' memories. Use recent pre-adoption history if repository and delivery records still support the same work classification, or create a small present-day baseline with the agent disabled on suitable low-risk items. If neither is practical, compare two current workflows and label the conclusion as a relative tool decision, not proof that either beats unaided work. The missing baseline lowers confidence, and the report should say so.

Lines of code, completed suggestions, chat turns, and agent session counts are activity measures. They help diagnose a workflow, but none proves delivered value. A compact deletion can be worth more than a generated module. Developer surveys add context about interruption and confidence, but self-reported hours saved should never enter the savings calculation without repository and delivery evidence.

Build the audit trail before comparing tools

A trustworthy audit connects a work item, agent activity, commits, review, deployment, and rework without sending sensitive prompts into a new analytics system. The minimum event can live as a small record in an internal audit repository or approved data store. It should reference existing systems rather than duplicate source code or full transcripts.

Use a vendor-neutral record like this:

schema: agent-audit/v1
work_item: ENG-1842
repository: billing-api
work_class: defect
agent:
  provider: vendor-a
  mode: repository-agent
session:
  started_at: 2026-06-03T09:14:00Z
  ended_at: 2026-06-03T10:02:00Z
human_minutes:
  developer: 31
  reviewer: 18
result:
  commits: [8f31c2a]
  accepted: true
  deployed_at: 2026-06-04T16:20:00Z
  rework_minutes: 0
portability:
  vendor_files_touched: 2
  hosted_dependencies: [issue-context, approval-log]

The schema makes two useful choices. It identifies the vendor and mode without adopting a vendor's event format, and it records dependencies as data rather than burying them in a comment. Add fields only when they answer an audit question. A sprawling telemetry project will delay the decision and create its own maintenance burden.

Git already gives you a portable link to the resulting object. The Git manual describes notes as annotations attached to objects without changing those objects. An internal notes ref can associate the audit work item with a commit:

git notes --ref=agent-audit add -m 'work_item=ENG-1842 agent=vendor-a' 8f31c2a
git log --show-notes=agent-audit -1 8f31c2a

The output keeps the ordinary commit and adds a notes block:

commit 8f31c2a...
Author: ...

    Prevent duplicate invoice retries

Notes (agent-audit):
    work_item=ENG-1842 agent=vendor-a

Git notes do not solve measurement by themselves. Teams must push the notes ref explicitly, access control still matters, and rewritten commits need a defined notes policy. I use the manual here because it shows a portable mechanism and its boundary. A proprietary dashboard may present richer data, but the audit needs an evidence spine that survives cancellation.

Never store secrets, customer data, raw production logs, or unreviewed transcripts merely to improve attribution. Keep the minimum identifiers needed to reproduce the calculation. If the vendor is the only party that can export usage events, request the export during the audit and verify that timestamps and task identifiers reconcile with your own records.

A fair comparison needs a controlled work sample

Run the comparison as a matched operational sample, not a staged coding contest. Select ordinary backlog items that represent enough cost to matter and low enough risk to ship under normal controls. Include at least two work classes because agents often perform well on repetitive changes and poorly on ambiguous defects.

Before the sample starts, freeze the acceptance rules, observation window, loaded labor rates, and treatment of failed attempts. Also record the available context for each tool. Giving the incumbent six months of curated instructions while handing a replacement a blank repository measures setup maturity, not agent quality. The replacement should receive the portable context that the team claims it owns, but not a hand-built translation that would never happen during a real migration.

A practical sequence has five parts:

1. Sample recently completed work and classify it by repository, risk, and work type.
2. Run new comparable items through the incumbent workflow while collecting human time and vendor costs.
3. Give a replacement agent the repository-owned context and run a smaller matched set under the same review rules.
4. Execute the exit test, including configuration conversion, access changes, evidence export, and one production release.
5. Recheck accepted items after the observation window and assign defect or cleanup work to its originating item.

Do not make developers race. Speed pressure changes review quality and encourages people to rescue the tool silently. Ask them to record intervention while it happens in coarse categories: context preparation, prompting, waiting, correction, test repair, and review response. Ten-second precision is theater; consistent five-minute increments usually support the decision.

DORA defines change lead time as the time from commit to production and pairs throughput with measures of deployment instability such as change fail rate. That is a useful warning against claiming productivity from local completion alone. I would not copy DORA metrics directly into an agent score because a coding agent does not control the whole delivery system. I would use deployment and failure data to check whether cheaper accepted work coincides with slower queues or more unstable releases.

Seasonality and team learning can mislead a short audit. Record releases, incidents, staff absences, and major platform work that affect the window. You do not need a laboratory-grade experiment, but you do need enough context to explain why two apparently similar weeks were not similar.

Repository portability must be demonstrated

A repository is portable when a competent engineer can clone it into a clean environment, discover the operating instructions, run the required checks, and complete a representative agent-assisted change with a different approved tool. A questionnaire cannot prove this. Run the test.

Inventory every file and service the incumbent loads before or during a session. Look for repository instructions, personal instructions, agent profiles, prompt libraries, skills, tool definitions, model settings, hidden ignore rules, and remote memories. Then classify each item as versioned and open, versioned but vendor-specific, external and exportable, or external and trapped.

GitHub's documentation gives a concrete example of why file presence is not enough. It documents repository-wide instructions in .github/copilot-instructions.md, path-specific instruction files, and AGENTS.md for agents, with support varying by environment and feature. Those are inspectable files, which helps. Their paths, precedence, supported frontmatter, and runtime behavior can still bind the workflow to one implementation. An audit should credit the portable prose while pricing conversion of the vendor-specific loading rules.

Run the clean-room check from a new clone and a new user profile. Existing developer machines often contain global instructions, authenticated tools, cached indexes, and environment variables that make the repository look self-contained. Ask the tester to follow only committed documentation and normal secret provisioning. Record every private message needed to get the change through; each message exposes missing repository knowledge.

Use this command to surface common agent-specific material for human review:

find . -type f \
  \( -name 'AGENTS.md' -o -name 'CLAUDE.md' -o -name '*instructions.md' \
  -o -name '*.agent.md' -o -name 'mcp*.json' \) \
  -not -path './.git/*' -print

Its output is only an inventory:

./AGENTS.md
./.github/copilot-instructions.md
./.github/agents/release.agent.md
./tools/mcp.project.json

Do not score fewer files as better. One giant instruction file can hide more coupling than several narrow files. Inspect whether the content describes durable project facts or teaches quirks of a particular model. Project facts, build commands, acceptance rules, and architectural boundaries should remain useful. Repeated advice about how to phrase requests or recover from one vendor's failure mode belongs in migration cost.

The strongest portability test ends with an accepted change, not a successful chat response. The replacement must edit, test, submit, survive review, and reach the same environment. Otherwise the audit has tested syntax compatibility while ignoring delivery.

Vendor workflow maintenance is recurring engineering work

Vendor-specific workflows behave like internal software. They break, need owners, accumulate exceptions, and consume review time. If nobody has a maintenance budget, that work does not disappear. Senior engineers absorb it between visible tasks.

Create an ownership register for instructions, hooks, custom agents, tool connectors, permission policies, and evaluation cases. For each component, record an owner, last meaningful change, trigger for change, test method, and fallback. A file last edited months ago is not necessarily stale, but a critical workflow with no test or owner is a liability.

Count maintenance through work records and version history. Search for changes to agent configuration, incident tickets involving agent behavior, tool-authentication failures, and time spent updating evaluations after model or product changes. Interview data can fill gaps, but ask for a recent example rather than a percentage guess.

Separate project maintenance from vendor tax. Improving a repository's build instructions benefits humans and replacement tools, so charge it to normal engineering enablement. Rewriting prompts after a proprietary model update, repairing a vendor-only connector, or adjusting to changed approval behavior is vendor tax. Some work has both effects; split it rather than forcing a false binary label.

The popular recommendation is to create a large central prompt library before measuring anything. Teams like it because it feels like reusable infrastructure and produces visible artifacts. I argue against it. A central library often collects untested prose, hides which instruction affected an outcome, and increases the surface that must be translated during exit. Keep instructions close to the repository, attach each one to a failure or acceptance rule, and delete instructions that no evaluation can distinguish.

Maintenance also includes cognitive load. If only two people understand why a hook runs, which permissions are safe, or how to recover a stuck remote agent, their availability is part of the dependency. Test the fallback with someone outside that pair. Documentation that has never been followed by a new operator is a claim, not evidence.

The exit drill exposes costs the dashboard cannot

An exit drill replaces enough of the incumbent workflow to ship one representative change without it. It does not require canceling the contract or migrating every repository. It produces an evidence-based estimate of the work, lost capability, and interruption a full exit would cause.

Start the clock when the migration operator receives the repository and approved replacement. Include discovery, export requests, configuration conversion, identity and permission work, replacement tool setup, developer orientation, failed runs, and release completion. Keep procurement lead time separate from engineering time, but include it in elapsed exit time because the business experiences both.

One audit I commonly see in miniature starts well. The team points to committed instruction files and a clean Git history. During the drill, the replacement cannot reproduce issue context because the incumbent fetched private issue fields through a hosted integration. The test agent then fails a deployment check because the approval policy existed only in the vendor console. Reviewers recover, but they cannot trace which agent actions were approved because the export omits the approval log available in the web interface.

That repository was portable in the narrow sense. The workflow was not. The fix is not to ban the incumbent. The team can move issue-context rules into an approved tool interface, version the release policy, export evidence on a schedule, and document a manual release path. The drill turns a vague fear into a small queue of engineering work.

Price full migration by component:

exit_cost =
  discovery_and_export
  + config_conversion
  + integration_replacement
  + retraining_and_slowdown
  + validation_and_release
  + expected_loss_of_nonexportable_history

Use ranges where uncertainty is real. Give each range an owner and assumption, such as "two connectors, one already supports the replacement" or "three repositories share the same policy." Do not hide permanent loss inside labor. Missing transcripts may have no operational value, while missing approval evidence may affect an investigation. State the consequence.

Repeat a small exit drill after major workflow changes or when the vendor moves an important function from files into hosted state. The first drill usually costs the most because it discovers the inventory. Later drills should become cheaper if portability work is genuine.

Savings must clear a risk-adjusted hurdle

Compare recurring savings with the annualized cost of lock-in, not with the seat license. The decision model should be simple enough that a founder can challenge every input and detailed enough that engineering cannot hide review or migration work.

Calculate annual gross savings from the difference in accepted feature cost at the expected annual volume. Subtract recurring workflow maintenance, additional security or compliance work, and contract cost. Then reserve for exit by spreading the estimated migration cost across the decision horizon and weighting it for the plausible chance of a forced or chosen move.

annual_net_value =
  (baseline_accepted_cost - agent_accepted_cost) * annual_accepted_items
  - annual_vendor_cost
  - annual_vendor_workflow_maintenance
  - annualized_risk_adjusted_exit_cost

Do not turn the probability term into fake precision. Show at least three cases: continue normally, replace at renewal, and exit under pressure after a material product or policy change. The pressured case should use more downtime and less vendor help. If the decision changes after a modest assumption moves, approve a shorter commitment or demand portability work before expanding.

Set a hurdle before reviewing the result. A tool that saves slightly more than it costs can still be a poor use of management attention. The hurdle should reflect uncertainty, concentration risk, and alternative investments. A reversible month-to-month workflow can pass with a lower margin than a multiyear commitment tied to hosted state.

Security and legal review belong in the cost model, but they should not become a mysterious veto. Record data classes the agent can access, retention and export terms supplied by the vendor, identity controls, subprocessors if relevant, and the response plan for revoked access. Price the engineering controls and known review work. Escalate contract interpretation to counsel rather than asking engineers to invent legal certainty.

Compare the result with the alternative use of the same money and senior attention. An agent program that returns $80,000 after all adjustments may still lose to removing a slow test suite, fixing unclear ownership, or hiring for a persistent specialist gap. This is why I keep savings and capacity separate in the report. Cash savings require an actual reduction in spend or avoided hiring supported by a staffing plan. Capacity savings mean the same team can accept more or better work, which has value only if the backlog and release system can use it. Calling every saved hour payroll reduction creates a forecast the finance team will rightly reject.

Approval can be conditional. A sensible decision may authorize the agent for two repositories while requiring versioned instructions, a monthly evidence export, and a second exit drill before wider adoption. Conditions need owners and dates. Otherwise they become comforting sentences in a slide deck.

The audit should end with an executable decision

The final output should let leadership approve, limit, renegotiate, or reject the agent without translating a pile of technical observations. Put the decision, annual net-value range, exit-cost range, confidence, and unresolved blockers on the first page. Keep the item-level evidence and calculation behind it for challenge and reruns.

I use four decision states:

• Expand when accepted feature cost improves enough to clear the hurdle and the exit drill succeeds within the agreed tolerance.
• Limit when savings are credible but one workflow dependency or evidence gap needs repair before broader use.
• Renegotiate when economics depend on export access, contract length, usage pricing, or support during migration.
• Replace when accepted work is not cheaper after rework and maintenance, or when the exit exposure exceeds the benefit.

Every recommendation should name the next control. "Improve portability" is not work. "Move the release approval policy from the vendor console into the repository, test it with the replacement agent, and assign the platform lead by September 15" can be tracked. The same standard applies to savings: name the repositories, work classes, expected volume, and review capacity behind the forecast.

A board or investor may ask for a single return-on-investment percentage. Give it only after showing the ranges behind it. The useful management answer is often conditional: the agent pays back under the current volume, stops paying back if review time rises by a stated amount, and becomes unacceptable if the hosted approval record cannot be exported. Conditions make the decision easier to revisit when pricing, models, or team composition change.

A Team & AI Audit is useful here because five business days force the evidence into one decision, and the fixed $5,000 scope comes with a guarantee of at least $50,000 per year in identified savings or the fee is waived. That offer does not make an agent economical; the repository data, time records, matched sample, and exit drill do.

Do not approve proprietary coding agents because lock-in sounds manageable, and do not reject them because lock-in sounds frightening. Put a price on accepted work and a price on leaving. If the vendor will not provide the evidence needed for either price, treat that missing evidence as a dependency you already cannot control.