AI code rework audit using Git history

Tool usage statistics flatter almost every AI coding rollout. Prompt count rises, accepted suggestions rise, and the weekly report says the team is moving faster. Meanwhile, pull requests grow larger, reviewers ask for the same corrections twice, and defects return after somebody declared them fixed.

If you want to know whether an assistant improves engineering output, audit the work that survived contact with review, release, and production. Git history can show what the team rewrote. Pull request events can show how many times reviewers sent work back. Issue history can show whether a defect actually stayed closed. Those are harder measures to game, which is exactly why they are worth using.

This is not an argument against AI assistants. It is an argument against treating activity telemetry as proof of value. An assistant that produces a useful first draft can save serious time. An assistant that produces plausible but poorly bounded changes can move typing effort into review, testing, and repair. The second case often looks productive until you inspect the trail it leaves.

Tool activity measures exposure, not engineering quality

An acceptance rate says that an engineer inserted a suggestion into an editor. It does not say the suggestion passed review unchanged, met the intended behavior, or remained in the codebase long enough to matter.

The same problem appears in prompt counts, generated-line totals, and agent task completions. They all report that a tool participated. None of them reports whether the participation reduced the total work needed to deliver a correct change.

I have seen teams celebrate a sharp rise in generated code while experienced reviewers quietly became the cleanup crew. The reviews did not fail. The metric did. It awarded the first act of writing and ignored every hour spent removing assumptions, restoring omitted edge cases, and explaining why an apparently reasonable change did not belong in that service.

A useful audit starts with a stricter unit of analysis: a change request. A pull request, issue, or linked delivery ticket gives you a boundary for intent, discussion, revisions, tests, and release outcome. The audit then asks four questions.

• How much of the proposed diff did the team materially rewrite before merge?
• How many review rounds did the change require before approval?
• Did the team later revert the delivered behavior or replace it under pressure?
• Did a linked defect reopen after somebody closed it?

These measures still need judgment. That is a feature, not a defect. Engineering output includes judgment, and any dashboard that removes it entirely will reward the wrong behavior.

Do not use the results to rank individual engineers. AI-assisted work crosses authorship boundaries quickly. One person may draft the change, another may direct the assistant, and a third may remove the dangerous parts in review. The right unit is the workflow, repository area, task class, or team. You are testing whether a working method produces expensive churn.

Define rework before you collect a single event

Rework is a later modification that removes, replaces, or corrects work already done for the same intended outcome. It is not every edit after the first commit.

That distinction prevents the audit from turning normal engineering into a failure signal. A pull request often becomes better through review. If the reviewer asks for a missing test and the author adds it, that is completion work. If the reviewer discovers that the central implementation chose the wrong boundary, and the author deletes most of it to build a different approach, that is rework.

Use four separate measures. Do not blend them into one grand score at the start.

1. Rewritten diff share is the portion of meaningful lines introduced in a change request that later disappeared or changed substantially before merge.
2. Reverted change share is the portion of merged changes later reversed, partially rolled back, or superseded because the original behavior was wrong or unsafe.
3. Review-round count is the number of request-changes cycles after the first reviewable version, not the number of comments.
4. Reopened-defect share is the share of resolved defects that return to an active state for the same underlying behavior within your chosen observation window.

The important line is between revision that completes the original plan and revision that replaces the plan. Teams blur these because both show up as added and deleted lines. They are different operationally.

Suppose an assistant drafts a billing endpoint. Review asks for an idempotency test, a timeout, and better names. Those additions may increase the final diff, but the implementation still follows the intended design. Count the work as review effort if it required another round. Do not call the original core diff rewritten.

Now suppose review finds that the endpoint trusts a client-provided price, and the author removes the generated pricing path in favor of a server-side quote lookup. That replacement changes the safety model. Count the removed and superseded implementation as rewritten diff. The number tells you that the first solution created a convincing but invalid path.

Write these definitions in a repository-level audit note before extraction begins. If two people cannot classify five sample pull requests the same way, tune the rules and examples before processing hundreds of them. A metric with vague boundaries produces false certainty at scale.

Git gives you evidence, but not the whole story

Git can establish that patches changed. It cannot reliably tell you why they changed, whether a reviewer blocked them, or whether a customer later saw the defect. Pull request and issue data supply that missing context.

For each audited change request, retain these records:

• base commit at the first reviewable revision
• head commit for every revision pushed before merge
• merge commit or final target-branch snapshot
• review events, including request changes, approval, and comment resolution
• linked issues, incident references, reverts, and later corrective pull requests

The first reviewable revision matters. If you compare only the first commit to the merge commit, you confuse normal local drafting with review-driven rework. If you compare only the last revision to the merged result, you miss the expensive parts that reviewers made the author throw away.

Create an immutable extraction table. Do not let an analyst hand-copy commit IDs into a spreadsheet after the fact. A basic record has enough fields to reproduce every number:

{
  "change_id": "PR-418",
  "repository": "payments-api",
  "base_sha": "a1b2c3d4",
  "first_review_sha": "b2c3d4e5",
  "revision_shas": ["b2c3d4e5", "c3d4e5f6", "d4e5f6a7"],
  "merge_sha": "e5f6a7b8",
  "review_rounds": 2,
  "linked_issue_ids": ["BUG-902"],
  "assistant_exposure": "declared",
  "excluded_paths": ["vendor/", "generated/", "*.lock"]
}

The assistant_exposure field should be optional and coarse. Use it if your team has a reliable declaration or repository convention, such as an AI-assisted label. Do not infer it from a commit message that says "generated" or from the size of a diff. People use tools in uneven ways, and a large change can be entirely hand-written.

Git's own documentation draws a useful boundary here. git diff compares endpoints, while git range-diff compares two commit ranges and tries to pair corresponding patches across revisions. That makes range comparison helpful for an investigator reading a revised series, but it should not be treated as a stable machine interface. Git explicitly describes its range-diff output as human-readable porcelain whose text can change between versions.

Keep raw API responses and Git references alongside the derived table. Six months later, somebody will ask why a pull request received three rounds instead of two. You should be able to show the events, the revisions, and the classification rule, rather than defend a mysterious cell in a dashboard.

Count rewritten diffs with patch lineage, not commit counts

A rewritten-diff measure should answer a practical question: of the meaningful code proposed at a reviewable point, how much did the team later discard or materially alter before merge?

Commit count fails because engineers amend commits, squash work, rebase branches, or force-push a cleaner history. A branch can have ten commits and little rework, or two commits and a complete replacement of its initial design.

Start with revision snapshots. For each pair of consecutive reviewable revisions, calculate a diff from the same base where possible. Then identify removed lines that originated in the earlier revision and additions that replace the same local behavior in the same file or nearby code region. You do not need a perfect semantic code-diff engine to get a useful signal. You need conservative rules and a manual review queue for ambiguous cases.

Use this calculation:

rewritten_diff_share =
  (removed_introduced_lines + materially_replaced_lines)
  / meaningful_lines_in_first_reviewable_diff

Exclude blank lines, pure formatting, generated files, lockfiles, vendored code, and repository-wide mechanical migrations. A team that lets formatting churn into this numerator will learn only that its formatter runs often.

The following shell commands create a review packet for one branch. They do not produce a final metric by themselves. They give an auditor the actual evidence needed to classify a change.

BASE=$(git merge-base origin/main feature/quote-validation)
FIRST_REVIEW=b2c3d4e5
LATEST=d4e5f6a7

# What the first review saw
 git diff --numstat "$BASE" "$FIRST_REVIEW"

# What changed between the first review and the latest revision
 git diff --find-renames --word-diff=porcelain "$FIRST_REVIEW" "$LATEST"

# How the patch series changed after revision
 git range-diff "$BASE".."$FIRST_REVIEW" "$BASE".."$LATEST"

The first command emits tab-separated added-line count, deleted-line count, and path records. The second produces a line-oriented word diff that a script can parse, although Git warns that its word-diff behavior may change with implementation details. The third is for human inspection, especially after a rebase.

For larger audits, build a lineage map. Store a normalized patch signature for every review revision and track whether a later patch has the same signature, a related path and hunk, or no credible connection. Git's patch-id is useful for the first case: it calculates an identifier from a patch while ignoring line numbers, and its stable mode also ignores whitespace and file-diff order. Git describes it as a way to find likely duplicate commits, which is exactly the right level of confidence to assign it. It identifies likely equivalence, not identical intent.

Do not call a moved function rewritten merely because it changed files. Do not call a renamed variable a replaced design. Set thresholds that require meaningful deletion and replacement in the same behavioral area, then sample the high-scoring changes. A metric that misses a few borderline edits is better than one that brands ordinary cleanup as AI failure.

Reverts need a reason code or they will lie to you

A revert is stronger evidence than a revision before merge because the team already accepted the work into a target branch. It still needs classification.

A release manager may revert a safe feature because a customer asked to delay it. A team may revert a migration because the rollout window closed. Neither result says the implementation was poor. Conversely, a partial rollback hidden inside a hotfix may never use Git's Revert commit message and can be the more serious signal.

Track three categories:

• Correctness revert: the code produced incorrect behavior, broke a contract, or created a safety issue.
• Operational revert: the change caused an unacceptable performance, reliability, deployment, or observability problem.
• Business or release revert: the team removed a valid change because priorities, timing, or rollout decisions changed.

Only the first two belong in a quality measure. Keep business reversions visible, but do not put them in the numerator. Otherwise a volatile product roadmap will make the engineering workflow appear worse than it is.

Search for explicit reverts first. Then look for corrective pull requests that reference the original change or touch the same files and issue within a short period. A human reviewer should classify this second group. Automation can suggest candidates through linked issue IDs, commit messages, changed paths, and patch similarity. It cannot safely decide that every nearby fix is a correction of the original change.

Git reflogs are sometimes useful while reconstructing a local investigation because they record previous positions of local branch references and HEAD. They are not a durable source for an organization-wide audit: they are local, can expire, and are often absent from hosted repository exports. Use server-side pull request revisions and protected branch history as your record of fact.

The ratio is simple:

correctness_and_operational_revert_share =
  merged_changes_with_a_qualifying_revert
  / merged_changes_in_the_cohort

Do not wait for enough reversions to make this statistically impressive. Even one correctness rollback in a sensitive area deserves a patch-level review. The point of the measure is not to create a leaderboard. It is to find work types where fast drafting has escaped the controls that should catch it.

Review rounds reveal where the first draft misses the mark

Comment count is a bad proxy for review effort. One careful reviewer can leave twenty useful comments in a single pass. Another can write "please fix" three times without explaining the issue. Count decision cycles instead.

A review round begins when a reviewer requests changes, marks a blocking thread, or records a clearly unresolved concern against a reviewable revision. It ends when the author pushes a later revision that addresses that set of concerns. An approval without requested changes does not create another round.

This produces a measure that founders can understand. If a team used to merge routine changes after one review pass and now needs three passes for the same type of work, the assistant may be generating code faster while moving design work into the review queue.

You need a small policy for awkward cases:

1. Treat several reviewers requesting changes on the same revision as one round.
2. Treat a new blocking concern after a response push as another round.
3. Ignore nonblocking nits that do not delay approval, unless your workflow forces them to block merge.
4. Split the change if a pull request grows into two unrelated features. Otherwise its review count becomes meaningless.
5. Record whether the reviewer found a specification gap, a design error, a test gap, or a simple implementation defect.

That last field changes the conversation. If extra rounds mostly come from missing product decisions, blaming the coding assistant is lazy. If they mostly come from invented APIs, authorization holes, broken error paths, or tests that merely confirm the happy path, the team needs tighter task framing and better guardrails around generated work.

Reviewers should not have to write a novel to make the data usable. Give them a small set of labels in the pull request workflow. Make "AI draft needed design correction" available, but never mandatory. Forced disclosure makes people hide tool use. The audit can measure delivery quality without treating the label as a confession.

A reopened defect measures false closure, not just bugs

A defect reopening is evidence that the team declared a behavior fixed and later learned that it was not fixed, was fixed only in one path, or regressed under conditions the original validation missed.

Count reopened defects only when the same issue returns from a resolved state to an active state, or when a new issue explicitly links to the earlier defect as a recurrence. Do not treat every new ticket near the same component as a reopening. A payments service can have several different bugs in the same week.

Define an observation window that fits your release cadence. For a continuously deployed product, inspect defects reopened within a set number of days after resolution. For enterprise software with scheduled releases, use a window that covers the next meaningful customer deployment and support cycle. Apply the same rule to every cohort you compare.

The audit record should include:

issue_id: BUG-902
original_fix_change: PR-418
resolved_at: 2026-05-07T16:40:00Z
reopened_at: 2026-05-12T09:15:00Z
reopen_reason: missing idempotency behavior on retry
linked_corrective_change: PR-431
classification: same root behavior

Do not reduce this to a number too early. Read the reopened defects. They often expose a repeatable failure pattern that aggregate code metrics hide: the assistant generated an endpoint with no concurrency model, copied validation from a nearby route with a different trust boundary, or wrote tests against mocks that repeated the same wrong assumption.

A reopened defect also distinguishes two failures that teams routinely mix together. A missed defect is a bug that escaped before anyone claimed to have fixed it. A false closure is a bug that passed through a repair cycle and returned. The second says something about your definition of done, test design, review depth, or incident follow-through. If AI-assisted changes show a high false-closure rate, adding another lint rule will rarely solve the underlying problem.

Run the audit as a cohort comparison, not a verdict on one pull request

A single ugly pull request can be a bad day, a rushed deadline, or a hard problem. An audit becomes useful when it compares similar work across a period and looks for concentration.

Build cohorts that answer an operational question. Compare API changes with API changes, not API changes with documentation edits. Compare work handled under an AI-assisted workflow with comparable work from the same team and repository area. If you do not have reliable exposure data, compare before and after a workflow change, but write down what else changed during the period.

For each cohort, report these separate values:

Then read the intersections. High rewritten-diff share with normal review rounds can mean authors catch bad generated code before reviewers see it. That costs time, but it is a different intervention from a review bottleneck. Low rewritten-diff share with high reopened-defect share points toward weak test coverage, shallow acceptance criteria, or a mismatch between staging and production conditions.

The most concerning pattern is high values across all four measures in the same task class. That usually means the first drafts arrive with enough surface polish to pass a quick glance, but not enough grounded understanding of constraints to survive the full delivery cycle.

Do not hide the denominator. A percentage based on three changes is a story prompt, not a performance finding. Show raw counts beside every rate, show exclusions, and keep a list of the sampled pull requests behind each classification. This keeps a founder from making a staffing decision based on a chart that quietly included a repository-wide formatter change.

The audit should produce a changed operating rule

A completed audit is useless if it ends with "use AI more carefully." The finding should change how work enters the engineering system.

If generated changes show high rewritten-diff share in authentication or billing code, require a written invariant and a reviewer-approved approach before generation. If review rounds rise because assistants invent local APIs, make repository architecture notes and interface contracts available in the task context. If defects reopen because tests cover only the happy path, require a failure-mode test plan before the implementation pass.

Change one rule for a defined cohort, then measure the same four outcomes again. Do not change models, prompts, code review policy, and team structure on the same Monday. You will have plenty of activity and no causal answer.

The blunt answer is that many teams do not have an AI productivity problem. They have an evidence problem. They adopted a meter that records keystrokes by proxy, then mistook it for a delivery measure. Git history, review events, and defect recurrence give you a harder but more honest record of where the work went.

If your team cannot produce this evidence cleanly, that is diagnostic too. A Team & AI Audit should leave you with a repeatable extraction method, clear exclusions, and a short list of workflow changes tied to actual rework rather than tool enthusiasm.