AI generated tests developers will actually keep around

Why most generated tests do not survive

Teams keep tests that catch expensive mistakes. They delete tests that cry wolf.

That is why so many generated tests disappear a few weeks after they land. The model copies what the code does today instead of checking what could hurt tomorrow. It reads the current output, selectors, class names, and helper calls, then turns that snapshot into assertions. The result looks busy, but it does not protect much.

A team can get 50 new tests in one pass and still feel less safe. If those tests fail every time someone renames a button, moves a field, or edits page text, people stop trusting the suite. Once trust drops, deletions start.

UI tests suffer most. A small layout update can break a large batch of tests that all depend on the same fragile details. Nothing important changed for the user, but the run turns red anyway. After two or three rounds like that, the suite feels like a tax.

Noise is worse than low coverage. A short suite that fails for real bugs helps developers move fast. A big suite full of weak checks slows review, hides serious failures, and trains people to ignore red builds.

Another problem is misplaced effort. Generated tests often focus on easy paths because that code is easy to read. The riskier behavior usually sits in edge cases: bad inputs, permission checks, retries, pricing rules, timeouts, or state changes after a partial failure. Those are harder for a model to infer, so they get less attention even though they matter more.

Picture a checkout page. The model may write ten tests for field order, labels, and button text. The team keeps none of them if a copy edit breaks half the set. One test that catches a double charge after a retry is worth far more.

More tests do not help when nobody believes them. Teams keep tests that stay calm during normal change and speak up when something important breaks. Everything else is cleanup waiting to happen.

Start with risky paths, not raw coverage

A coverage number can look good while the product still breaks where it hurts most. If a flow can charge the wrong amount, delete data, expose private records, or lock users out, test that first. That is where generated tests earn their place.

Most teams get better results when they aim at parts of the app that already cause pain in production. Look at support tickets, incident notes, postmortems, and bug reports from the last few months. If the same area keeps failing, ask AI to help there before it writes one more test for a tiny helper nobody worries about.

The first targets are usually payment and billing logic, permission checks, imports and exports, state changes in orders or inventory, and anything that broke during recent releases.

This means skipping a lot of low-risk code at the start. A getter that returns one field, a mapper with obvious input and output, or a thin wrapper around a library may raise your coverage report, but it rarely protects the business. Those tests often become noise. They fail during harmless refactors, and nobody feels bad deleting them.

Incident history gives you a better map than coverage ever will. If production logs show that retry logic fails under timeouts, test the timeout path. If a CSV import keeps mangling dates, test messy files and partial failures. If users lose edits during autosave conflicts, test the conflict rules.

The pattern is simple: follow the places where mistakes cost money, trust, or hours of cleanup.

This is also how practical teams use AI in testing. You do not need fifty new tests in safe code. You need a few around the parts that wake people up at night.

Pick stable behaviors the team wants to keep

Teams keep tests when those tests protect decisions the product already made. They delete tests that complain about harmless changes. With generated tests, that difference matters even more, because the model will happily produce a lot of noise if you let it.

Start with business rules, not surface details. A rule like "users on a trial cannot export data" or "orders above a set amount need manual review" is worth testing because the team wants that behavior to stay the same for a long time. A button label, a CSS class, or the exact order of markup usually is not.

Good targets have a long shelf life. Permission rules, price and tax calculations, state changes after an action, API response contracts, and side effects like sending one invoice or creating one record tend to survive refactors.

What to avoid

A lot of generated tests fail for the wrong reason. They lock onto text labels, DOM structure, private function names, or exact timestamps down to the second. Those details change during normal work, so the suite starts to feel annoying instead of useful.

If time matters, check a range or a format. If the UI matters, check that the action works, not that a class name stays identical forever. Small changes should not create extra work for the team.

Contract checks are usually safer than implementation checks. If an API should return a total, a currency, and a payment status, test that contract. Do not test which helper built the response or how many internal calls happened along the way. Developers should be free to rewrite internals without rewriting half the suite.

A simple rule helps: if a developer can change the code heavily and the user should see the same result, the test should still pass. If the test breaks because "Buy" changed to "Purchase," it never protected much in the first place.

A simple process for asking AI to write tests

Asking for a "full test suite" usually gets you filler. You get tests that repeat the code, lock in tiny UI details, or break after a harmless refactor. A better prompt stays small and points at behavior you actually want to protect.

Start with one user flow. Keep it concrete, like "user applies a coupon at checkout" or "admin exports a report." Then add one failure mode that would hurt if it slipped through, such as an expired coupon or a timeout from the billing service.

Next, write the expected behavior in plain language. Skip testing jargon. A simple sentence works: "If the coupon is expired, the app shows an error and keeps the original total." That gives the model a target that matches real product behavior, not just code structure.

A useful prompt usually follows the same shape:

1. Name one flow that matters to users.
2. Name one way that flow can fail.
3. Paste only the code needed to understand that behavior.
4. Ask for two to four focused tests.
5. Review the output and delete weak tests fast.

The context you give matters more than the model you pick. Paste the function, component, or handler under test, plus any small helper it depends on. Include the current test style if you have one. Do not dump the whole file tree. Too much context pushes the model toward broad, noisy guesses.

Ask for a few tests, not coverage theater. One good happy-path test and one or two failure tests beat fifteen shallow checks. If you want stable behavior testing, say so directly. Ask the model to assert on visible outcomes, returned values, saved records, or emitted events. Tell it not to assert on private method calls, exact markup, or other details you expect to change.

Then review with a hard hand. Keep tests that would catch a real bug. Delete tests that only prove the mock framework works, mirror the implementation line by line, or break when you rename a variable.

A short prompt often works better than a clever one. Clear scope, plain expected behavior, and a small slice of code usually produce tests a team will keep.

A realistic example: one checkout flow

A checkout flow is a good place to see what useful generated tests look like. It has money, state changes, and failure paths that can hurt customers fast. That makes it a better target than a static settings page or a screen full of minor UI details.

Take a simple card payment flow. A shopper enters card details, clicks "Pay", and the app creates an order only if the payment provider approves the charge. The tests that earn their keep focus on what must stay true even after refactors.

One test should cover the happy path. After a successful charge, the order should move to a paid state, the cart should clear, and the customer should see a confirmation tied to that order. Those checks matter because they describe the business result, not the screen paint.

A second test should cover a declined card. In that case, the order should stay unpaid or move to a failed state, and the system should avoid creating a second order behind the scenes. If the text on the error banner changes later, that is usually fine. If the order state is wrong, it is not.

A third test should cover duplicate submit. People double-click. Browsers retry. Mobile networks lag. If the customer hits "Pay" twice, the system should still create one order and charge once. This is exactly the kind of risky path teams forget until support tickets pile up.

Notice what these tests do not care about. They do not check the button color, the exact spacing, or a full page snapshot. Snapshot tests for the whole checkout page often break when someone tweaks copy or moves a field by a few pixels. That noise piles up fast.

Tax rules are a different case. If your tax logic rarely changes, keep one focused test for it. For example, a taxable item shipped to one specific region should produce the expected tax total in the stored order. That gives you coverage where mistakes cost real money, without filling the suite with brittle test coverage strategy theater.

Where generated tests usually go wrong

Most bad generated tests try to prove too much, and they prove the wrong thing. A test that checks every field, CSS class, helper call, and timestamp may look thorough, but it breaks the moment the team refactors harmless code.

The first bad pattern is over-asserting. If a refund flow matters because the customer gets money back and the order status changes, the test should focus on that result. It does not need to pin every intermediate value unless that value changes the business rule.

Another common miss is heavy mocking. AI often mocks the database, the queue, the payment client, the clock, and half the app around them. The test turns green, but it only proves that mocked functions return mocked values.

That kind of test rarely catches the bugs teams care about. It avoids the real edges where code, data, and outside services meet.

AI also gets attached to the current shape of the project. It reads helper names, file paths, and internal wrappers, then writes tests that depend on all of them. Rename one helper or move one module, and several tests fail even though behavior stayed the same.

Volume is another trap. When the model finds one risky path, it often creates four more tests that differ by one input string or one flag. You pay the review cost five times, yet coverage barely improves.

Flaky state is the quiet problem. Generated tests often forget cleanup, reuse shared data, or leave background jobs, temp files, and database rows behind. The suite passes on one machine and fails in CI the next morning.

A generated test is usually on the wrong track if it cares more about helper calls than user results, replaces most dependencies with mocks, breaks after a harmless refactor, adds noise without covering new risk, or leaves data behind after it runs.

Good generated tests are a bit boring. They check one behavior that matters, use little mocking, survive refactors, and leave the system clean for the next test.

How to review a generated test before you keep it

A test earns its place when it catches a bug you would care about next month, not when it proves that every label, class name, and pixel stayed the same.

Start with one blunt question: what bug would this test catch? If you cannot answer that in one sentence, the test is probably noise. Good answers sound like real breakages, such as "the user gets charged twice" or "an expired token still opens the account page."

That question also helps you cut weak assertions fast. If an assertion depends on button wording, page layout, or the exact order of harmless UI elements, remove it unless that detail is part of the rule you want to protect. Text and layout change all the time. Business behavior should change much less often.

The name matters more than many teams admit. "renders checkout screen" says almost nothing. "blocks payment when card is expired" tells the next developer what rule the test protects. When a failure appears in CI, that kind of name saves time right away.

Generated tests also love to repeat themselves. They build too much setup, create data no assertion uses, and restate the same check in three slightly different ways. Trim hard. If two helpers do the same job, keep one. If three assertions prove the same outcome, keep the clearest one.

A quick review comes down to five checks:

• The test points to one bug or one business rule.
• The assertions avoid fragile wording and layout details.
• The setup only includes data needed for the scenario.
• The test name explains the rule in plain language.
• The test passes twice in a row on your machine.

That last step catches more problems than people expect. Run the test twice, or a few times if it touches time, randomness, network mocks, or async UI. Flaky behavior often shows up early when generated code relies on timing accidents.

If your team uses generated tests regularly, this review habit keeps the suite lean. You end up with fewer tests, but they fail for reasons that matter.

A short checklist before adding tests to CI

A generated test should earn its place. Once it runs in CI, it stops being an experiment and starts costing the team time every time it fails. If it does not protect something that can hurt users, money, or trust, it probably does not belong there.

Use a quick screen before you keep any AI-written test:

• Tie the test to a real breakage. Ask, "What failure would this catch?" If nobody cares about that failure, delete the test.
• Prefer outcomes that survive refactors. Check totals, saved records, access rules, or visible user results. Avoid private helper calls, exact mock counts, or fragile markup details.
• Read it like a new teammate would. They should understand the setup, action, and expected result in one pass. If they need the original prompt to decode it, rewrite it.
• Make sure it fails for one clear reason. A test that checks three things at once turns every failure into a small investigation.
• Check the upkeep. If the test depends on odd fixtures, prompt tricks, or a pile of mocks that only one person understands, it will rot fast.

This is where many teams get too generous. The AI writes ten tests, seven of them pass, and people keep all seven. That feels productive for a day. A month later, the suite slows reviews and nobody trusts the failures.

A small example makes this concrete. Say the test checks a checkout flow. Keep the test that proves the system rejects a duplicate charge or applies tax correctly. Drop the one that checks the exact order of internal function calls unless that order is the behavior you promise to keep.

Good CI tests read like plain language. Someone should see the file and know what user action it covers, what result matters, and why the test exists. If you cannot explain that in one sentence, the test is still a draft.

The best test to add is often smaller than the first version the AI wrote.

What to do next

Start small. In the next sprint, pick three to five flows where a failure would cost real time or money. That might be login, checkout, invoice generation, permission checks, or an API path that breaks other teams when it changes.

This works better than telling AI to raise coverage everywhere. Generated tests earn trust when they watch risky paths and stable behavior, not every branch the model can find.

Write one short team rule sheet before anyone adds more tests. Keep it short enough to read in a minute. AI can test user-facing behavior, business rules, and bug fixes that already hurt you once. It should stay away from private helper details, timing-sensitive internals, and markup that changes every week. Every generated test needs a human owner. If nobody would fix the test after a refactor, do not merge it.

Then track three numbers for a month: how many generated tests you delete, how many go flaky, and how many catch a real bug before release. Those numbers tell you more than raw coverage ever will.

A simple example makes this concrete. If your team adds ten AI-written tests for a checkout flow and deletes six of them within two weeks, the prompt or the review rule is off. If two stay stable and catch tax or payment edge cases, you are on the right path.

Do the review in the same place you review code. Treat generated tests like production code with lower trust at the start. A test that looks clever but breaks every other Friday is dead weight.

If your team needs help setting the rules, Oleg Sotnikov at oleg.is works with startups and small companies on practical AI-first development and engineering process. That kind of outside review can help when a team wants faster output without turning the test suite into noise.

The next move is boring on purpose: choose a few risky flows, write the rule sheet, and measure what survives. In a month, you will know whether your test coverage strategy is getting sharper or just getting bigger.