AI SDLC metrics must measure accepted change

AI coding agents make output cheap. They can open five pull requests while a person is still reading the first one, so a chart that celebrates tickets closed or story points completed can rise while customer value, reliability, and engineering judgment fall. The useful unit is no longer how much code entered the system. It is how much verified change reached users and stayed healthy.

That shift does not require throwing away every software delivery measure. It requires separating flow from acceptance, and acceptance from production success. I have seen teams declare a large productivity gain because merge volume doubled, then spend the next month unwinding generated changes that nobody had understood. Their dashboard was accurate about activity and wrong about delivery.

AI SDLC metrics should tell a founder whether the company is learning and shipping faster without transferring hidden work into review, incidents, security remediation, or next quarter's maintenance. They should also help an engineering leader find the constraint. A board needs a small set of trends. A team needs the detail that explains those trends. Mixing the two produces either a cockpit nobody can read or a scorecard nobody can act on.

Velocity measures a planning convention, not delivery

Velocity was never a universal productivity unit. A story point is a local estimate made by one team under one planning method. Adding points across teams is meaningless, and comparing this quarter's points with last quarter's points becomes even weaker when an agent changes how work gets decomposed.

Agents break the implied relationship between implementation effort and value. A developer may spend ten minutes asking an agent to produce a large migration, then spend two days checking data behavior, rollback safety, and edge cases. Another developer may write a fix of six lines that removes a severe customer failure. Point totals and lines changed reward the first task even when the second has more value.

The same objection applies to commits, pull requests, agent sessions, prompts, tokens, and code acceptance percentages. These are activity or diagnostic measures. They can explain cost and workflow, but they do not prove useful delivery. Put them on an engineering operations page, not in the board pack.

Replace velocity at the executive level with four questions:

• Did a change tied to a customer, revenue, risk, or operating objective reach production?
• How long did accepted work take from commitment to verified use?
• How much of the shipped work caused immediate failure or later rework?
• Did the cost and human attention per accepted change improve?

"Accepted" needs a strict definition. A merged pull request is accepted by version control. A deployed feature is accepted by the delivery system. A useful change has passed its stated production check, such as adoption, task completion, latency, support contact reduction, or a risk control test. The dashboard should name which level it reports. If it silently treats merge as value, agents will make the chart look wonderful.

Do not use this argument to abolish estimates inside a team. Teams can still estimate when estimates help sequence work or expose uncertainty. Just stop presenting the estimate total as company output. It was not comparable before agents, and it becomes easier to game after them.

DORA still measures the delivery system

DORA's software delivery performance measures remain useful because they observe changes crossing a service boundary rather than fingers typing. The current DORA guide groups change lead time and deployment frequency under throughput, then change fail rate, failed deployment recovery time, and deployment rework rate under instability. That division is exactly what an agent-heavy team needs: speed beside consequences.

Keep the definitions close to the source. Change lead time runs from commit to production. Deployment frequency counts production deployments over a period. Change fail rate covers deployments that require immediate intervention. Failed deployment recovery time measures how long recovery takes. Deployment rework rate captures unplanned deployments caused by a production incident. Renaming these casually makes comparisons inside your own company unreliable.

I would add three cuts to the underlying data without creating a separate "AI DORA" score:

• Service or product area, because a mature API and a new mobile client have different risk profiles.
• Change risk class, based on affected components and behavior rather than diff size alone.
• Work mode, such as written by a person, assisted by AI, or executed by an agent, recorded at the change level.

Use those dimensions to diagnose, not to rank people. If database changes executed by an agent show a higher rework rate, adjust the approval and test path for that class. If one engineer has fewer deployments, the data says almost nothing about that person's contribution. They may be removing the architectural constraint that lets everyone else ship.

DORA also warns against turning its metrics into goals. That warning matters because a metric used as a quota loses meaning fast. A target for deployment frequency encourages trivial deploys. A target for lead time encourages teams to split or relabel work. Read the measures as a balanced system: faster flow is good only while instability and rework stay within an agreed range.

These measures still have a blind spot. They can show that a change moved safely, but not whether the change was worth making. Connect each production change to an objective and a verification event. The link can be imperfect at first. An explicit "unknown outcome" is more honest and more useful than pretending every deployment carries equal value.

Accepted change is the replacement for velocity

The replacement for velocity is not one new number. Use a small bundle centered on accepted change, because a single composite score hides which part of the system improved and invites teams to optimize the formula.

Start with accepted change rate: the count of production changes whose promised outcome check passed within a declared observation window. The check must match the work. A checkout improvement might use completion rate and guardrail errors. An internal build change might use queue time and failure rate. A compliance change might use a control test and evidence review. Some maintenance has no direct customer adoption measure, but it can still have an explicit risk or cost condition.

Pair that count with value cycle time. Measure from the moment the company commits to the work, not from the first agent commit. Agent tools can shrink coding time while work waits in product clarification, review, release approval, or customer rollout. Time from commitment to verified use exposes the entire queue. Time from commitment to production is an acceptable interim measure if outcome instrumentation is missing, but label it honestly.

Then add deployment yield: accepted changes divided by production changes. A falling yield means the team is shipping more things that do not meet their stated check. Do not punish experiments that disprove a hypothesis. Mark a properly run experiment as accepted when it produces a decision under its declared test, even if the feature itself loses. The value is the learning, and the record should say so.

Two rework measures stop speed from borrowing against the future. Immediate rework is unplanned corrective work within a fixed window after deployment. Deferred rework is a sampled measure of shipped changes that need substantial cleanup after the window, found through code ownership review, defect clusters, or repeated modification. The exact window differs by product cadence, but changing it every month ruins the trend.

Finally, record human attention per accepted change. Include active review, test design, incident response, and material correction time. Do not treat elapsed queue time as labor. This measure answers the economic question agents are supposed to improve: did the company reduce scarce engineering attention for each useful result? Token spend belongs beside it as cost, but token spend alone has no meaning. A cheap agent run that creates six hours of cleanup is expensive.

A clean metric contract prevents later arguments. For every measure, store the name, business question, formula, event sources, owner, exclusions, observation window, and expected failure modes. If two directors can calculate the same metric differently, the dashboard is not ready.

Do not confuse throughput with occupancy. A team can keep every engineer and every agent busy while work spends longer waiting for decisions. Watch work in progress at the objective level and the age of the oldest committed item. When accepted change rate stalls while active work rises, stop launching more agent runs and clear the decision, review, test, or rollout queue. This often feels slower for a day because fewer tasks appear active. It produces faster delivery because attention stops bouncing among unfinished changes.

Capacity planning should use the same boundary. Forecast how many accepted changes of a given risk and service class the system can absorb, not how many agent tasks it can start. The limiting resource may be a product decision, a staging environment, a security owner, or a customer rollout window. Buying more model capacity will not expand any of those. It will create a larger inventory of generated work that grows stale while it waits.

Review depth needs evidence, not comment counts

Review depth means the strength of the checks applied to the risks in a change. It does not mean minutes spent, number of comments, number of reviewers, or how long the pull request remained open. Those measures are easy to collect and easy to fake. An agent can generate a detailed review comment that proves nothing was tested.

GitHub's documentation describes reviews as approve, comment, or request changes, and its rules can require status checks, code-owner review, and dismissal of stale approvals after new commits. Those controls establish that a gate occurred. They do not establish that the reviewer examined the dangerous behavior. A green approval is an event, not evidence of understanding.

Measure review through risk coverage. Classify each change by the behavior it can affect: authorization, money movement, destructive data operations, public API compatibility, privacy, deployment configuration, or ordinary application logic. Each class has required evidence. An authorization change might require a threat case, negative tests, and review by the owner of that boundary. A copy change should not wait for the same ceremony.

Useful signals of review depth include:

• Required risk evidence present and passing at the final reviewed commit.
• Sensitive components reviewed by an accountable owner independent of the agent run.
• Material review findings resolved through a code or test change, not merely a closed thread.
• Approval refreshed after the final material change.
• Production verification and rollback conditions written before merge for work in the high risk class.

Report coverage as a matrix, not a league table. For example, show that 94 percent of changes to a payment boundary had all required evidence, while 68 percent of ordinary changes had their lighter evidence set. The missing six percent deserves inspection. Averaging both classes into one "review score" conceals the exposure.

Sample quality still matters. Once a week, have a senior engineer inspect a small random set of changes involving an agent and answer concrete questions: Did the change match the issue? Did tests challenge failure behavior? Could the reviewer explain the affected invariant? Did the production check prove what the author claimed? Calibrate reviewers by discussing disagreements. This is closer to audit sampling than performance appraisal.

Never reward comments per pull request. The popular reasoning is that more comments show deeper thought. In practice, the target produces style notes, fragmented conversations, and noise written by agents. Track material findings for diagnosis, but judge the review system by risk coverage and escaped outcomes.

One event model should connect intent to production

Most companies cannot build this dashboard because their issue tracker, repository, deployment system, incident tool, and product analytics disagree about identity. Fix the join before buying another dashboard. Every material change needs a stable change identifier carried from committed work through deployment and outcome verification.

The event model can start small. Emit immutable records with timestamps, actors, service, work mode, risk class, and evidence references. Do not store full prompts or source code in an analytics table. Store identifiers, approved classifications, tool versions when relevant, and references inside the controlled systems that already own sensitive content.

This compact contract is enough to calculate the first useful trends:

{
  "change_id": "chg_01J...",
  "objective_id": "obj_retention_4",
  "service": "billing-api",
  "work_mode": "agent_executed",
  "risk_classes": ["money_movement", "data_migration"],
  "events": [
    {"type": "committed", "at": "2026-07-01T09:12:00Z"},
    {"type": "review_evidence_passed", "at": "2026-07-01T14:05:00Z", "policy": "payments-v3"},
    {"type": "deployed", "at": "2026-07-02T08:30:00Z", "deployment_id": "dep_8841"},
    {"type": "outcome_verified", "at": "2026-07-09T08:30:00Z", "check": "invoice_error_guardrail"}
  ]
}

Four implementation details prevent misleading data. First, make work_mode describe how the final change was produced, not whether somebody opened an AI tool during the week. Second, allow mixed attribution when human and agent contributions are inseparable, but use a documented rule. Third, version the risk policy so historical coverage does not change when requirements change. Fourth, record the commit that received final review so a passing test on old code cannot satisfy the gate.

Derive metrics from events instead of manually maintained issue fields. Manual tags decay under deadline pressure. The repository and deployment pipeline already know when commits, reviews, checks, and releases happened. Product or operational systems know when the declared outcome check passed. The issue tracker should supply intent, commitment, and objective identity, not pretend to be the source for every timestamp.

Start by testing join completeness. Report the percentage of production deployments with a change identifier, risk class, work mode, and objective or explicit operational reason. Until coverage is high and stable, show "unknown" as its own category. Deleting unknown records makes every later chart falsely precise.

Faster pull requests can hide a slower company

Consider a subscription company that enables agents for routine backend work. In the first month, pull requests per engineer rise sharply, median time to first review falls, and merged lines triple. Management announces a productivity gain. The support queue then fills with billing edge cases, two engineers spend most of a week correcting migrations, and the product manager delays the next release to verify old behavior.

The activity dashboard did not contain false data. It contained the wrong boundary. It stopped at merge, counted generated surface area as output, and ignored the people absorbing correction work.

A proper trace tells a different story. The agents reduced the time from first commit to opened pull request. Review queue time also fell because reviewers approved diffs that looked small. Time from commitment to production improved, but time from commitment to verified use grew because rollout checks failed. Deployment yield fell, immediate rework rose, and human attention per accepted change increased. The constraint moved from implementation to specification and verification.

The review records expose why. The migration pull requests carried unit tests for the happy path, yet the risk policy did not require tests for duplicate events or partial rollback. Reviewers approved after checking syntax and test status. Nobody wrote down the invariant that one invoice event must create at most one charge. Agents delivered exactly the narrow behavior they were asked to deliver.

The fix is not a blanket second reviewer. That response is popular because headcount and approval counts are visible. It also slows routine work while leaving the same specification hole in place. Add evidence specific to the risks of money movement, require a human owner to state the invariant, generate adversarial test cases, and verify the final commit. Review capacity then goes where failure costs money.

After the change, pull request volume may fall. That is acceptable if deployment yield recovers, correction work drops, and accepted change cycle time improves. A board trained on activity will call the lower volume a regression. A board shown the full delivery path will see that the company stopped manufacturing work for itself.

The dashboard for the board should fit one screen

A dashboard for the board should answer whether engineering converts money and attention into useful, safe change. It needs trends and exceptions, not repository telemetry. Use a rolling view plus the current period, and annotate major product launches, migrations, or weeks with many incidents that change the mix.

The first row should contain five tiles:

• Outcome: accepted changes by business objective, paired with deployment yield.
• Flow: median time from commitment to verified use, paired with the 85th percentile.
• Stability: change fail rate, paired with recovery time and deployment rework rate.
• Exposure: changes in the high risk class that lack evidence, paired with the oldest unresolved exposure.
• Economics: human attention and tool cost per accepted change, paired with total correction cost.

Median alone hides the stuck work that founders eventually hear about from a customer, so show a tail percentile beside it. Counts alone hide whether the company shipped the right work, so split accepted changes by objective. Ratios alone hide scale, so let every rate drill into numerator, denominator, and absolute counts.

Below the tiles, show a flow line from committed work to production to verified outcome. The gap between each stage identifies the constraint. Add a cohort comparison for changes written by people, assisted by AI, executed by agents, and unknown, but normalize by service and risk class before drawing conclusions. A raw comparison will mostly reveal that teams give agents simpler or different work.

The final panel should list exceptions that require a decision. Examples include a service whose rework rate crossed its agreed range, a risky deployment without evidence on the final commit, an outcome cohort that remained unverified past its window, or an AI cost increase without a gain in accepted changes. Assign an owner and review date. A red chart with no decision attached is decoration.

Keep prompts, tokens, generated lines, acceptance rate of suggestions, agent sessions, and comparisons between models in the engineering detail view. They help explain cost or diagnose a tool change. The board should see them only when they explain a material movement in delivery economics or risk.

Do not collapse the screen into a single traffic light. An executive may want one answer, but engineering performance has legitimate tension. Faster flow with rising exposure requires a choice. A blended score turns that choice into hidden arithmetic.

Roll out the measures without creating a surveillance system

Metrics change behavior as soon as people believe compensation or status depends on them. State in writing that these measures evaluate the delivery system and work classes, not individual developer productivity. Enforce that rule by limiting views, retention, and exports about individuals, not by asking managers to behave well.

A practical rollout takes four phases:

1. Define the metric contracts and sample twenty recent changes by hand. Resolve disagreements about commitment, acceptance, rework, and risk before automating anything.
2. Instrument identity and events, then publish completeness and unknown rates. Do not publish performance comparisons while the joins are unreliable.
3. Run the dashboard privately with engineering and product leaders for several delivery cycles. Compare every surprising chart with actual changes and incidents.
4. Give the board the stable view with five tiles, the definitions, and two or three decisions the data changed. Keep diagnostic measures in the operating review.

Use a baseline that includes several normal delivery cycles and at least one release boundary if your product has them. Avoid a simple comparison across the launch date when the work mix changed. Instead, compare matched service and risk cohorts, show distributions, and annotate process changes. If the company launches an agent only on routine chores, its early numbers cannot predict database migration performance.

The SPACE framework offers a useful check on blind spots. Its authors argue that developer productivity cannot be captured by a single activity measure or one dimension. You do not need to put all five SPACE dimensions on the board. Use periodic team surveys and operational signals to check whether faster delivery coincides with lost focus, poor collaboration, or low confidence. Treat that information as a reason to inspect the system, not to grade a person.

Data access deserves the same care as production telemetry. Agent traces can expose prompts, customer context, security assumptions, and developer behavior. Collect the minimum event metadata needed for the declared decisions. Set retention, access, and deletion rules before the dataset becomes irresistible to every manager with a theory.

Metric ownership belongs with operating leadership

Someone who can change the delivery system must own each metric. Finance can validate cost allocation, data teams can maintain pipelines, and tool vendors can draw charts, but engineering and product leadership must own definitions and responses. Otherwise the dashboard becomes a report about a system nobody in the room feels authorized to change.

Review definitions quarterly and policies whenever the delivery process changes materially. Do not rewrite historical formulas to make the trend smoother. Version the definition, mark the break, and preserve the old calculation long enough to explain it. A board can handle a discontinuity. It cannot make a sound decision from retroactively polished history.

Every metric needs an expected response. If evidence coverage falls for the high risk class, who can pause that change class? If value cycle time rises because outcomes remain unverified, does product operations fix instrumentation or does engineering stop starting work? If human attention per accepted change increases, who examines specification quality, agent routing, and review load? A measure without a response owner will become background color.

This is also where economic claims meet reality. Faster code generation matters only if accepted output rises or attention and cost fall without worse stability. Track fully loaded human time, agent and infrastructure spend, and correction cost under one consistent allocation rule. Do not claim savings by removing coding hours while ignoring senior review and incident hours.

In a Team & AI Audit, I use this evidence to find where agent capacity will remove a constraint and where it will merely flood the next queue. The point is not to install my preferred dashboard. It is to make staffing, controls, and tooling decisions against the same delivery facts.

The first board discussion should not ask, "How much code did AI write?" Ask which verified outcomes reached customers, what attention they consumed, what failed, and which risk checks were actually present. When those answers come from joined events rather than anecdotes, agent speed becomes governable. Until then, a rising velocity chart is only proof that more work entered the pipe.