How to audit an AI pilot after 30 days with evidence

What you should know after 30 days

After 30 days, skip the mood check and ask one plain question: did the pilot remove real work from a real process?

If people still do the same job and spend extra time checking the AI, the pilot did not help much. A smooth demo can hide that. Daily use does not. Real work has messy inputs, edge cases, missing context, and handoffs between people. A pilot that looks fast in a test can slow a team down once exceptions, reviews, and cleanup start piling up.

That is why how to audit an AI pilot starts with evidence, not opinions. Team feedback still matters, but it should support the numbers, not replace them. Look at what happened in actual work over the month: how often the AI got the task right, how many cases it kicked out, how much time reviewers spent, and whether errors created extra work later.

Use one simple rule: count the whole job, not just the AI step. If an AI draft saves 2 minutes but adds 5 minutes of checking and fixing, the pilot lost time. If it cuts a 20 minute task to 8 minutes with only rare cleanup, that is a different result.

Set the decision rules before people argue about the outcome. Keep them plain. Expand if the pilot saves time in normal use, keeps error rates within your limit, and does not create hidden cleanup. Redesign if one or two weak spots look fixable, such as poor input quality or a review step that is too manual. Stop if the gains only show up in demos, the team spends too much time correcting output, or mistakes create risk you do not want.

This kind of audit is blunt on purpose. It protects you from wishful thinking and gives you a fair basis for the next move.

Run the audit step by step

Start with a tight boundary. If the pilot touched email triage, audit only email triage, not the whole support process. Teams blur the result when they mix pilot work with unrelated manual work.

A clean 30 day review usually follows the same path.

1. Write down the exact workflow the pilot changed in one sentence. For example: "The AI drafts first replies for billing emails." That stops people from pulling in extra tasks like escalation, QA, or follow up reporting.
2. Lock the date range before you look at the results. Use the full first 30 days, then pick a sample that reflects normal work, not only easy cases. If the pilot handled only 40 cases, review all 40. If it handled 2,000, a sample of 50 to 100 cases is often enough to spot patterns.
3. Score every case on the same sheet. Keep it simple so reviewers actually use it. Track whether the output was correct, partly correct, or wrong, whether a person had to edit it, whether an exception happened, and whether any cleanup happened later.
4. Put exceptions, edits, and manual fixes in one log. Do not split this across chat threads, tickets, and spreadsheets. When one case needs a retry, a manual rewrite, and a later fix in another system, log all three against the same case ID.
5. Time the human review from start to finish. Count the whole effort, not only the moment someone clicks approve. Include reading the AI output, checking source data, making edits, and fixing downstream issues if they happen right away.

Keep the scoring sheet boring

Fancy scoring systems waste time. A plain table works better than a clever rubric nobody follows. If two reviewers score the same case differently, rewrite the labels until they agree.

Add a short notes field, but keep it short. One line like "missed refund policy" or "wrong customer tier" is enough to spot patterns later.

When every case goes through the same sheet, arguments get shorter. You can see whether the pilot saves time, shifts work to reviewers, or creates cleanup that cancels out the benefit.

Check accuracy in a way people trust

Accuracy scores only help if two people can look at the same output and give roughly the same judgment. If the team cannot agree on what counts as a good result, the number will start arguments instead of settling them.

Use three labels and define them before scoring starts:

• Correct: the AI output can go through with no meaningful fix.
• Partly correct: the draft is usable, but a person must fix a fact, missing detail, or wording that affects the result.
• Wrong: the output would cause a bad decision, confuse a customer, or force someone to redo most of the work.

Keep the definitions plain. Do not let people invent their own version while scoring. If one reviewer treats small edits as "correct" and another marks the same case as "partly correct," the score sheet needs work.

It also helps to calibrate with a small shared sample first. Have two reviewers score the same 10 to 20 cases, compare the results, and fix the labels before you review the full month. That small step saves a lot of noise later.

Accuracy on its own is never enough, though. A draft can be technically correct and still take too long to verify. That is why the next numbers matter just as much.

Count exceptions before they pile up

An AI pilot can look fine on average and still break the workflow several times a day. Exception counts show that hidden cost fast, especially when the team starts patching problems by hand and stops logging them.

Start with one simple number: how often the system asks for help, gets rejected, or hands work back to a person. Count every interrupt, even small ones. A model that needs help 8 times in 100 tasks feels very different from one that needs help 2 times, even if both have similar average accuracy.

Sort exceptions by cause, not by blame. Notes like "agent fixed it" or "AI failed again" tell you almost nothing. Use buckets that point to the source of the miss:

• missing or messy input data
• unclear prompt or business rule
• model gave the wrong answer with confidence
• integration failed or timed out
• human reviewer could not approve the output

That split matters because each cause leads to a different decision. Bad input may call for a form change. A wrong answer with clean input may mean the task is a poor fit for the pilot.

Some exceptions block the workflow and some only slow it down. Mark blockers clearly. If the AI cannot complete a refund, route a ticket, or draft a usable reply, that is not a minor defect. One blocked case can create three more tasks for the team.

Keep rare edge cases separate from daily misses. One strange customer message in a month is noise. The same billing question failing every Tuesday is a pattern. Daily misses usually point to a rule, data field, or prompt that the pilot sees all the time and still handles poorly.

Watch the trend each week. A flat exception count after prompt edits is useful evidence. A repeating error after four weeks usually means the fix did not work, or the real cause sits somewhere else.

A single total is not enough. Count the volume, sort the cause, flag blockers, and mark repeats. That gives you a clean basis to expand, redesign, or stop.

Measure human review effort

Review time often decides whether an AI pilot saves work or just moves the work to another person. A result can look fast on paper while reviewers still spend hours correcting tone, facts, format, or missing details.

Measure the full review window, not just the moment when someone edits the output. Start when a reviewer opens the item and stop when they approve it, send it back, or replace it with manual work. That gives you the real cost of using the system.

For each reviewed item, log a few plain numbers:

• total minutes spent in review
• whether the reviewer made light edits or a full rewrite
• whether they had to ask follow up questions
• whether the item needed help from a more senior person
• whether the AI draft was approved at all

A simple timer works fine if the volume is low. If the pilot handles more work, pull timestamps from your ticketing system, CMS, or internal workflow tool so the team does not guess later.

Edits matter as much as time. Ten quick grammar fixes are very different from a reviewer rewriting half the draft or checking every claim line by line. Count the number of edits, but also sort them into a few buckets such as minor fixes, moderate changes, and full rewrites. People remember the painful cases, so buckets keep the audit honest.

Then look at the trend across the month. If the average review time stays flat after the first week, the pilot may not be improving, the prompts may be weak, or the task may simply need more structure. If time drops from 8 minutes per item to 3, that is real progress.

Convert that effort into hours per week. If your team reviews 250 items and spends 4 minutes on each, that is more than 16 hours every week. A pilot that saves one operator 20 minutes but adds 16 review hours to the team is not helping much.

Watch for senior staff jumping in. When a lead, manager, or specialist keeps fixing edge cases, count that separately. Their time costs more, and it usually means the AI output still breaks in places that junior reviewers cannot safely approve.

Find downstream cleanup and hidden work

A pilot can look good at the moment the AI finishes its task and still create more work later. That extra work often hides in small fixes, follow up messages, and manual edits that nobody logs as part of the pilot. If you miss that, the results look cheaper and faster than they really are.

Start where the work lands after the main step. Look at notes in the CRM, support tickets, internal comments, spreadsheet edits, and record changes in the system people use every day. You are looking for signs like corrected fields, repeated approvals, reopened tickets, or a second person fixing what the AI produced.

One simple test helps: compare the pilot path with the old manual path from start to finish on the same kind of task. Do not stop when the output gets delivered. Follow both paths until the task is truly done and nobody needs to touch it again.

What to count

Keep the count simple:

• extra minutes spent fixing AI output after handoff
• duplicate work done by another team
• reopened tickets or records changed later
• tool costs tied to the pilot
• rework that delays billing, shipping, or customer replies

Duplicate work matters more than teams expect. A sales team may accept an AI made summary, then operations rewrites it, and finance fixes the record again. The pilot may save 3 minutes up front and waste 12 minutes across the next two teams.

Put all costs in one view. Add tool spend, staff time, and rework together. If the pilot uses a paid model, a review tool, and two extra rounds of checking, combine them into one number per task or per week. That makes the tradeoff easy to see.

If you want a fair read, sample a few completed cases and trace them end to end. Teams that run lean AI operations tend to do this well because they watch the full workflow, not just the first output. That is often the difference between a pilot you can expand and one that quietly shifts work to someone else.

A simple example from a support team

A support team tests an AI tool that drafts replies for incoming tickets. In the first week, the result looks strong. The team reviews 50 easy tickets about password resets, delivery dates, and basic account questions, and 44 drafts need only small edits.

That early score feels better than the daily reality. Once the pilot runs on a normal queue, the mix changes. Now the inbox includes refund disputes, fraud flags, chargeback threats, and angry customers who already contacted support twice. The draft still sounds polite, but it misses policy details, uses the wrong tone, or suggests an action the agent cannot actually take.

The team starts timing review work instead of judging drafts by vibe. On simple tickets, an agent spends about 20 seconds checking the AI reply. On sensitive tickets, review jumps to 2 or 3 minutes because the agent has to read the case history, fix the draft, and make sure the reply does not create legal or billing trouble.

They also check what happens after the reply goes out. Weak drafts create extra cleanup in CRM notes. Agents add missing details by hand, correct the issue category, rewrite internal summaries, and log promises the AI forgot to mention. A draft that looks "almost right" can still push hidden work into the next step.

After 30 days, the picture is clearer:

• Simple ticket accuracy stays high enough to save time.
• Sensitive ticket review effort is too heavy.
• CRM cleanup adds another layer of work on a large share of cases.

That leads to a practical decision. The team can expand the pilot for low risk ticket types, redesign it for sensitive cases with stricter rules and better routing, or stop using it where the review cost cancels out the speed gain. That is the point of the audit. You are not trying to prove the AI is good or bad. You are trying to show where it actually helps and where people still carry the load.

Mistakes that blur the result

A pilot can look better than it is when people remember two great outputs and forget the other 200. Memorable wins stick in the mind. Audit data does not. If one agent says, "The AI nailed that refund case," but the same week three other replies needed rewrites, the pilot is not ready just because one case felt impressive.

Another common miss happens after approval. A reviewer clicks "looks fine," then someone in billing, support, or ops fixes the record later. That cleanup still counts as work. If you ignore it, the pilot seems faster than it is. In practice, many teams move the effort instead of removing it.

Case mix can hide the truth too. If the pilot handled password resets, order status checks, and complex account disputes in one bucket, the final score means very little. Easy cases can lift the average and hide failure on harder work. Split the sample by task type, risk, or difficulty. A 92% success rate on simple requests and a 40% success rate on edge cases tell a much clearer story.

Prompt changes cause another mess. Teams often tweak instructions halfway through the month because the first version did poorly. That is normal, but you need a log. Write down what changed, when it changed, and why. Without that record, you cannot tell whether the system improved or the test conditions moved.

Reviewer time often gets waved away because the staff already exists. That is a bad habit. If a support lead spends 90 minutes a day checking outputs, that cost is real even if payroll did not change. The same goes for senior staff who step in for exceptions. Their time has a price, and it pulls them away from other work.

Most clean audits avoid five traps:

• counting stories instead of totals
• missing cleanup after approval
• mixing simple and messy cases together
• changing prompts without a dated log
• treating review time as free labor

If any of these show up, pause the verdict. Fix the measurement first, then judge the pilot.

Quick checks before you decide

Ask the team to put the whole result on one page. If they need a long deck and a live demo to make the pilot look good, the evidence is probably weak. A solid result should fit in a simple summary with plain numbers and a short note on what changed during the month.

That page should show weekly task volume, accuracy, exception counts, reviewer minutes per item, and cleanup work after the AI output moves into the next step. This is the fastest way to see whether the pilot improved real work or just moved the effort around.

Before you approve more budget or a wider rollout, check a few things:

• Can someone outside the project understand the result from one page?
• Do exception counts stay stable, or better yet drop, from week to week?
• Does reviewer time fall after the first rounds, once people learn the process?
• Does cleanup stay small enough that the time saved still counts as real savings?
• Would the pilot still look good if you removed launch week attention and extra manual checks?

That last point matters more than teams admit. Early pilots often get special care. People watch every output, fix edge cases by hand, and respond faster because leadership is paying attention. That can hide a weak process. If the pilot only works under demo pressure, it is not ready.

Look for a simple pattern. Stable exceptions, lower review time, and limited cleanup usually mean you can expand with care. If accuracy looks fine but reviewers still spend almost the same time and downstream teams keep fixing output, redesign the workflow before you scale it. If the numbers swing hard from week to week, stop and find out why before you add more volume.

This is the short version of how to audit an AI pilot: keep only the numbers that still matter after the excitement wears off. If the team can explain those numbers clearly, the decision usually becomes obvious.

What to do next with the evidence

A 30 day audit should end with a decision, not a vague sense that the pilot "looks promising." If the results are mixed, split the work by task and judge each part on its own.

Expand the parts that work on normal days. That means regular volume, regular staff, and the usual messiness of real work. If a task stays accurate, keeps exceptions low, and does not create extra cleanup, it has earned a wider rollout.

The next move is usually clear:

• Expand tasks that stay stable without extra supervision.
• Redesign tasks that burn review time or create follow up work.
• Stop tasks that only move effort from one team to another.

High review time is a warning sign. So is cleanup that lands on support, ops, finance, or engineering after the AI finishes its part. If people still need to rewrite outputs, fix records, reopen tickets, or explain mistakes to customers, the pilot did not remove work. It just hid it.

Redesign often works better than a full stop. Narrow the task, tighten the input, add simple rules, or keep a human in the few places where mistakes cost the most. Many weak pilots improve once teams stop asking the model to do too much at once.

Some pilots should end. If the tool shifts effort instead of removing it, keeps making the same error, or adds risk around customer data, product behavior, or infrastructure, stop it and document why. That record saves time when the same idea comes back six months later.

If the pilot touches product decisions, internal systems, or production infrastructure, get another set of eyes before you scale it. This is the sort of review Oleg Sotnikov discusses at oleg.is in his Fractional CTO and startup advisory work: checking whether the gains are real, whether the architecture can support wider use, and where the hidden costs still sit.

A careful outside review helps most when the pilot looks good enough to tempt a rollout, but not clear enough to trust without a deeper technical check.