Does mutation testing for AI-generated code catch hollow tests?

AI-generated code has changed the failure mode of testing. Teams used to worry that new code arrived without tests. Now a coding agent can produce implementation, mocks, fixtures, and a reassuring green test run in one pull request. The dangerous version of that pull request looks complete: high line coverage, neat test names, and assertions that never prove a user-visible rule.

Mutation testing gives those tests an adversary. You introduce a tiny fault into the implementation and ask whether the suite notices. If a billing total changes, a permission check flips, or a required field becomes optional and every test remains green, the tests were present without protecting behavior.

This is not an argument to run thousands of random mutations against every repository. That wastes CI time and teaches people to ignore the report. Use it where AI can amplify a familiar engineering mistake: it writes tests after reading the implementation, then repeats the implementation's logic in a different file.

A passing generated test can still protect nothing

A test passes because it agrees with the program under test. That agreement may come from a useful specification, or from the test restating the same implementation detail. Code written by an agent makes the second outcome more common because the agent sees the answer before it writes the question.

Consider a generated billing function:

type Invoice = {
  subtotalCents: number;
  taxRate: number;
  discountCents: number;
};

export function totalDueCents(invoice: Invoice): number {
  const discounted = Math.max(0, invoice.subtotalCents - invoice.discountCents);
  return Math.round(discounted * (1 + invoice.taxRate));
}

An agent may produce this test:

it("calculates the invoice total", () => {
  const invoice = { subtotalCents: 10_000, taxRate: 0.08, discountCents: 1_000 };

  const expected = Math.round(
    Math.max(0, invoice.subtotalCents - invoice.discountCents) * (1 + invoice.taxRate)
  );

  expect(totalDueCents(invoice)).toBe(expected);
});

The test exercises every line. It also duplicates the formula. Change the production code so that it taxes the undiscounted subtotal, then make the same change in the test's expected expression, and the suite stays green. The test has become a second implementation, not an independent check.

A behavioral test names an outcome a customer, accountant, or support engineer can recognize:

it("does not charge tax on the discounted portion", () => {
  const invoice = { subtotalCents: 10_000, taxRate: 0.08, discountCents: 1_000 };

  expect(totalDueCents(invoice)).toBe(9_720);
});

This test has a limitation too: a single example can miss boundaries and invalid inputs. But it will fail if the code taxes the wrong base. That is the difference worth protecting.

Line coverage answers, "Did a test execute this statement?" Mutation testing asks, "Would the test fail if this statement did the wrong thing?" Those are separate questions. A suite can claim 100% line coverage and still survive a flipped authorization predicate or a missing validation branch.

Google's mutation testing work describes the technique as injecting bugs and checking whether tests detect them. Its code review approach is worth copying at a smaller scale: surface a specific surviving fault while the author still has the changed code in front of them, rather than sending a broad quality score weeks later.

Mutate decisions that change money, access, and data

Start with business decisions that have a clear wrong outcome. Billing, permissions, validation, tenant isolation, entitlement rules, and state transitions are better targets than controllers that only pass arguments onward.

The useful question is not "What has low coverage?" It is "Which one-line mistake would create a support ticket, a security incident, or a bad financial record?" AI increases delivery volume, so the review system needs to focus on decision points rather than reward a growing count of test files.

For billing logic, inject faults such as:

• charge tax before rather than after a discount
• change > to >= at a pricing threshold
• permit a negative adjustment to increase a refund
• round each line item rather than the final aggregate
• skip a currency or account-status check

For authorization, mutate the predicate itself. A test that merely asserts a 200 response for an administrator says very little. It must demonstrate that a member cannot perform the same action, that an administrator in another tenant cannot cross the boundary, and that an unauthenticated request fails before the business operation executes.

For validation, remove the branch that rejects the input. Do not settle for a schema test that checks whether a library function was called. Send a malformed request through the boundary your application actually exposes and assert the status, error shape, and absence of the side effect.

A good mutation target is small enough that an engineer can explain the intended behavior in one sentence. If nobody can state that sentence, mutation testing will expose a more basic problem: the team has code without an agreed rule.

Billing mutations expose arithmetic tests that only look busy

Billing defects have a special talent for surviving attractive test suites. Agents are good at assembling fixtures with products, invoices, taxes, discounts, credits, and mocked payment providers. They are also good at generating expected values with the same calculation they saw in production.

Use fixed examples for rules that must produce a known amount. Use property tests for relationships that must hold across many inputs. Do both only when the rule earns the extra effort.

Take a subscription upgrade rule. A customer moves from a $100 monthly plan to a $160 plan halfway through a 30-day period. Suppose the contract says the credit uses the old plan's unused time and the charge uses the new plan's remaining time. A concrete test should make the expected net amount explicit. It should not calculate the expected result by calling the same proration helper or by reconstructing its branches.

Then inject mutations that represent actual finance mistakes:

// Intended check
if (creditCents > remainingChargeCents) {
  return { amountDueCents: 0, carryForwardCents: creditCents - remainingChargeCents };
}

// Useful mutant
if (creditCents >= remainingChargeCents) {
  return { amountDueCents: 0, carryForwardCents: creditCents - remainingChargeCents };
}

The equality case matters. If credit equals charge, a system may need to create no payment and no carry-forward credit. A test suite that lacks that case can pass for years because normal examples have unequal amounts.

Another mutation removes the floor on a discount:

const discounted = invoice.subtotalCents - invoice.discountCents;

If tests allow an over-large discount to produce a negative total, you have learned more than "a mutant survived." You have found an unmade product decision. Should the system reject the invoice, clamp it to zero, or issue account credit? The test cannot answer that for you. A product owner or finance owner must.

Do not mutate only arithmetic operators. Billing behavior often hides in gates around arithmetic: whether an invoice is collectible, whether a trial has ended, whether a coupon applies to an annual plan, or whether a charge is idempotent. A test that checks a returned amount but never verifies the ledger entry or payment intent may kill a math mutant while missing a double-charge path.

The practical split is simple. Unit tests should kill mutations in deterministic calculations. Integration tests should kill mutations in the decision boundary that creates a charge, applies an idempotency rule, or records a balance change. End-to-end tests should prove the smallest number of customer journeys needed to verify that the outside world received the intended effect.

Permission tests must prove denial, not just success

Authorization tests often become a list of successful requests. That happens because generating a happy-path test is easy, and because engineers are reluctant to write the fixtures needed for roles, organizations, ownership, and subscriptions. An agent will happily fill that gap with a mocked canAccess function that returns true.

That test proves that a stub can return true.

Put authorization rules in a policy function or a narrow boundary where they can be exercised with real identities and real resource attributes. Then use a mutation that broadens access. If this intended policy is:

export function canEditProject(actor: Actor, project: Project): boolean {
  return actor.organizationId === project.organizationId && actor.role === "admin";
}

the highest-value mutants are not exotic. They are the exact shortcuts people make under deadline pressure:

return actor.organizationId === project.organizationId || actor.role === "admin";

return actor.role === "admin";

return actor.organizationId === project.organizationId && actor.role !== "viewer";

Your suite should kill each one with behavior tests that construct the actor and project deliberately. At a minimum, cover a same-organization admin, a same-organization member, an admin from another organization, and an unauthenticated caller if the boundary accepts one.

Do not overfit the test to the implementation's role strings. A permission test should assert an allowed or denied operation. If an HTTP endpoint changes its policy helper, the test should still tell you whether a cross-tenant edit succeeded. That is why one integration test at the actual route can be worth more than ten unit tests that inspect internal policy calls.

The test must also verify the denied request leaves no trace. If an endpoint returns 403 after it has queued an export, updated a record, or emitted an event, the permission check exists but sits in the wrong place. Mutation testing can help uncover that by moving or removing the guard, but the assertion has to inspect the state change.

This distinction matters: an authorization assertion checks the response, while an authorization protection test checks both the response and the forbidden effect. Teams routinely blur the two and congratulate themselves for a denial that happened after damage.

Validation belongs at the boundary where bad input arrives

Generated validation tests often inspect schemas in isolation. Those tests have a place, but they do not prove that a request path actually applies the schema before it reaches the command handler.

A useful validation mutation removes a required-field check, swaps a minimum boundary, accepts an unknown enum value, or bypasses an ownership constraint embedded in a request. If the suite stays green, find out whether the test never sent the bad input through the boundary or whether the application accepts the input and fails later.

For example, imagine an endpoint that creates a payout:

export type CreatePayoutRequest = {
  accountId: string;
  amountCents: number;
  currency: "USD" | "EUR";
};

The safety rule might include more than type checking. amountCents must be a positive integer, accountId must belong to the caller's organization, and the target account must support the chosen currency. An agent can write a pile of tests for empty strings and omitted JSON fields while leaving the ownership check untested.

Use a boundary-level test with a real request and a controlled repository or test database. The artifact should make a failed test obvious:

it("rejects a payout account owned by another organization", async () => {
  const response = await requestAs(orgAAdmin).post("/payouts").send({
    accountId: orgBAccount.id,
    amountCents: 5_000,
    currency: "USD"
  });

  expect(response.status).toBe(403);
  expect(await payoutRepository.count()).toBe(0);
});

Now remove the organization comparison in the validator or handler. The test must fail. If it does not, do not add a generic expect(response.body).toBeDefined() assertion and move on. Trace the request path until you identify which dependency, fixture, or broad mock hid the rule.

Validation mutation also separates input rejection from error handling. A malformed request that causes a 500 response is not a killed mutant in the business sense. The application noticed something went wrong, but it did not enforce the contract correctly. Assert the intended client error and the absence of a partial write.

A mutation score is a triage signal, not a target

Teams that turn mutation score into a company-wide target will start gaming it. They will exclude awkward files, add brittle assertions that kill superficial mutants, and spend hours arguing with equivalent changes. That result is predictable because one number cannot carry the full meaning of a test suite.

Stryker describes a mutant as killed when at least one test fails with the change active, and survived when all tests pass. It distinguishes no-coverage mutants and timeouts as separate states. Keep those states separate in review. An uncovered authorization branch needs a different response from a timeout caused by an altered loop condition.

Use the report to create a short queue of decisions:

1. Read the surviving diff before reading the test. Decide whether the altered behavior matters.
2. If it matters, add or repair the smallest test that should fail for that behavior.
3. If it does not change observable behavior, mark it as equivalent or exclude that operator with a written reason.
4. If it reveals an unclear requirement, stop treating it as a test task and get a product decision.
5. If the test is slow or flaky, fix the test boundary before increasing the mutation budget.

Equivalent mutants are real. Changing x + 0 to x - 0 does not alter behavior. Replacing a condition in code guarded by an invariant may also be equivalent within the program's reachable states. Do not lie to the tool with a broad ignore rule. Put the reason beside the exclusion, because an invariant that holds today may disappear during a later refactor.

PIT makes a related design choice in its default mutators: it favors operators that are stable and tries to reduce equivalent mutations. It also applies mutations to compiled bytecode, which helps make JVM runs practical in normal builds. That is a useful reminder that operator selection is engineering work, not a checkbox.

A local target can help when it reflects risk. For example, require that changed billing and authorization modules introduce no unreviewed survivors. Do not require an arbitrary repository-wide percentage from a team that has never inspected a mutation report. The first useful baseline comes from reading the survivors, not from drawing a line on a dashboard.

Run mutations where review can still change the code

A full mutation run across a large repository can take too long for every pull request. That does not mean mutation testing belongs in a quarterly quality project. It means scope must match the review loop.

Run fast, targeted mutations on changed business-rule files and their direct tests during pull requests. Run broader jobs on a schedule or before a release when the repository can afford the compute. Keep the report attached to the change that introduced the logic, so the author sees a concrete statement such as "changing > to >= in trial eligibility did not fail a test."

A workable policy for a startup team looks like this:

• Mutate changed code in billing, permissions, validation, and account state transitions on every pull request.
• Fail the pull request only for new survivors in that scoped set, after allowing a documented equivalent-mutant suppression.
• Run a wider mutation job overnight or before a release branch, then assign only the survivors that affect known business rules.
• Keep generated clients, migrations, framework glue, and simple data mapping out of the initial scope.
• Track elapsed time and flaky tests. A report nobody trusts will not improve tests.

Do not let an agent run unrestricted mutation testing and then auto-commit dozens of new tests. That pattern produces volume, not confidence. Give the agent one surviving mutant at a time, the rule it violates, and the test boundary where the behavior belongs.

A prompt with constraints produces better repairs than "improve tests":

The mutation changed `actor.organizationId === project.organizationId`
to `true`, and all tests passed.

Add one integration test for PATCH /projects/:id that uses an administrator
from a different organization. Assert HTTP 403 and assert that the project
name is unchanged. Do not mock the authorization policy. Do not change
production code.

Review the resulting test as if a new engineer wrote it. Confirm it fails against the mutant and passes against the intended code. An agent can produce a plausible test that accidentally uses the wrong fixture, authenticates as the resource owner, or asserts only an error message.

Generated tests need an independent source of truth

The deepest issue is not that an AI coding agent makes arithmetic mistakes. Engineers make those too. The issue is correlation: one system reads an implementation, infers a story from it, and then writes tests that agree with that story. The more code and tests it produces together, the less independent evidence you have.

Break that correlation with inputs the implementation did not author. Use acceptance criteria written before the change, examples from a real bug report, finance rules supplied by the person who owns billing, permission matrices, API contracts, and mutation survivors. Each gives the test author something external to copy from.

For mature areas, ask the agent to propose mutants before it writes tests. Make a human select the mutations that represent costly mistakes. This keeps the workflow tied to your business rather than to whichever generic operator a tool happens to emit.

Meta's engineering team describes a mutation-guided approach that uses focused, relevant mutants rather than indiscriminately generating every possible fault. The principle applies even if you never build an LLM testing system: fewer mutations that resemble your actual risks beat a massive report full of trivia.

For startup founders, this is a management issue as much as a testing technique. If AI lets two engineers ship the output that once required a much larger team, you need evidence that the faster output still protects money, access, and customer data. A green suite is evidence only when it can reject a wrong implementation.

Start with one module that can hurt you: invoice totals, project access, or payout validation. Inject five believable faults by hand if you do not have a tool configured yet. Repair the tests that let them through. Once engineers have seen a green test suite fail that exercise, mutation testing stops looking academic and starts looking like ordinary code review.

If your team is shipping AI-produced changes faster than it can review behavioral risk, a Team & AI Audit can identify where tests, CI, and ownership rules are letting weak changes pass.