AI review workflow: experts for risk, engineers for rules

Why this goes wrong

Teams often send every AI answer to engineers because engineers already own the model setup, prompts, logs, and integrations. On paper, that looks tidy. One team owns the whole thing, so nobody has to ask who reviews what.

But the biggest AI mistakes usually are not technical. Engineers are good at catching broken tools, bad inputs, missing permissions, weak guardrails, and automations that fail at the wrong moment. They can spot when the model used the wrong source or skipped a validation step. They usually do not own the business rule that decides which refund wording is allowed, what a finance note must include, or when a medical summary goes too far.

That gap causes more trouble than many teams expect. Risk often sits in judgment, wording, and domain rules. A reply can be technically correct and still be wrong in a way that costs money or creates compliance trouble. The model might phrase a policy exception too loosely, soften a warning, or make a recommendation that sounds fine to a generalist but breaks a rule an expert knows immediately.

Finance is a good example. An engineer may confirm that the AI pulled numbers from the right system and formatted the answer correctly. A controller will notice that the wording implies revenue can be recognized this month when it cannot. The risk is not the API call. It is the judgment inside the sentence.

When the wrong people own review, everything slows down. Engineers become a bottleneck for questions they should not answer. Subject experts get pulled in late, after the draft already looks finished. That creates rework, longer approval cycles, and a false sense of safety because someone technical signed off.

The fix is simple. Engineers should own boundaries, access, checks, and failure rules. Subject experts should review outputs where domain risk is high. Mix those jobs together, and the process gets slower and less safe at the same time.

Find the outputs that carry real risk

Most AI mistakes do not start in the model. They start when a team treats every output as equally harmless. A rough internal summary is one thing. A customer email that promises a refund or changes contract terms is something else.

Start by listing every output the AI can draft, recommend, or approve. Think in outputs, not tools. "Chatbot" is too broad. "Refund response," "price quote note," "contract clause rewrite," "incident advice," and "compliance explanation" are specific enough to review.

Use four questions to sort them:

1. Who reads it?
2. Can someone act on it right away?
3. Does it affect money, safety, legal terms, or compliance?
4. What does one wrong answer cost?

Outputs that score high on those questions need human review from the person who knows the subject. Finance should review payment promises and credit decisions. Legal should review contract language. Operations or safety staff should review instructions that could affect equipment, access, or people.

Keep drafts and final answers in separate buckets. An internal first draft for a team member to edit usually carries less risk than a final message sent straight to a customer. Many teams miss this and waste review time on harmless drafts while risky answers get only light checks.

Keep the first pass small. If you try to map every workflow in the company, the work stalls and nobody trusts the process. Pick one or two output types that are both common and risky. A billing reply that can trigger a refund is a solid place to start. So is a contract summary that a sales or procurement team might use in a live negotiation.

This sorting step matters because it tells you where review actually belongs. Once you know which outputs can cause real damage, you can put the right reviewer on them and leave lower-risk drafts under lighter checks.

Put the right reviewer on each output

A billing summary should not wait for an engineer to decide whether a fee disclosure sounds right. The person who already owns that decision should review it. If the output can change money, policy, compliance, or customer promises, send it to the team that already carries that risk.

This is where many teams get stuck. They ask engineers to judge business meaning because the tool is technical. That slows things down and still misses the real issue. Engineers can tell you whether the prompt, model, logging, and access rules work. They should not decide whether a refund email breaks policy or whether a contract summary drops an important term.

In practice, the mapping is usually straightforward. Finance reviews invoices, payment notes, budget summaries, and anything tied to pricing or reporting. Legal reviews contract drafts, policy language, and other compliance-sensitive text. Operations reviews workflow instructions, task routing, and vendor communication. Support reviews customer replies, help content, and escalation messages.

Keep the review scope narrow. Do not ask a support lead to approve the whole system. Ask them to check whether a proposed customer reply is accurate, on policy, and safe to send. Do not ask a finance manager to inspect model settings. Ask whether the output uses the right terms, numbers, and approval logic.

Clear limits make review faster. Each reviewer should know what output they own, what they are allowed to change, and when they must send something back. Legal can approve wording changes, but legal should not change how the model gets data. Engineering can fix retrieval, access control, templates, and audit logs, but engineering should not overrule a policy owner on business judgment.

Teams that do this well keep the split clean. Subject experts approve meaning. Engineers control guardrails, system behavior, and failure paths. It sounds obvious, but it saves a lot of wasted review time and a lot of avoidable mistakes.

Let engineers own the guardrails

Engineers should decide what the model can touch before anyone debates who reviews the output. If the system can read the wrong data, fill in missing facts, or send a final approval on its own, the process is already broken.

In a sound review setup, engineers protect the edges. They choose the allowed data, define stop rules, and make sure every risky action still lands with a person.

What engineers should lock down

Start with data access. The model should read only approved sources, not whatever it can find in a shared drive or old chat history. If a team uses policy docs, a product catalog, and a clean customer record, keep it to those sources and version them.

Then block autonomous approvals. A model can draft a response, suggest a classification, or prepare a recommendation. It should not mark a case as approved, rejected, paid, shipped, or closed without a human step.

Keep a full trail for every decision. Log the prompt, the model output, what a reviewer changed, and the final call. When something goes wrong, that record saves hours. It also shows whether the problem came from weak instructions, bad source data, or a reviewer who had to fix too much.

Add hard stop points when the model does not have enough context. If two records disagree, a required field is blank, or the answer includes uncertainty, the run should pause. Do not let the system smooth over gaps with a confident guess.

A few rules cover most of this:

• Allow only named data sources.
• Block final approvals and external sends.
• Save prompts, outputs, edits, and decisions.
• Stop on missing or conflicting data.
• Assign every failure to one human owner.

That last rule matters a lot. When nobody owns failed cases, they sit in a queue until someone notices. When one person or role owns them, the process stays real instead of theoretical.

This is the work engineers do best. Subject experts decide whether the answer makes sense. Engineers make sure the system cannot wander outside the rules.

Build the review flow step by step

Choose one workflow that can cause real harm if the model gets it wrong. Good starting points include a customer policy answer, a draft compliance note, or a medical intake summary. Keep the goal narrow. You are not trying to use AI everywhere. You are testing one job with a clear result, such as "draft a first response using only approved policy text."

Then write the prompt in plain language and pin down the sources it can use. List the exact documents, tables, or internal notes the model may rely on. If a source is outdated or optional, say that too. It sounds boring, but it saves time later. Reviewers can judge the output against a fixed set of materials instead of guessing what the model pulled from.

Next, keep the review decision simple: approve when the output is correct and safe to use, edit when it is mostly fine but needs changes, and reject when it is wrong, risky, or unsupported. Those three actions are enough early on. More labels usually create noise.

If reviewers edit or reject, ask for one short reason. A few words often do the job: "used the wrong source," "missed an exception," or "tone is too certain." Over time, those notes show patterns faster than long feedback forms.

Keep a record of the original prompt, the draft, the reviewer action, and the final version. Do not make this heavy. A simple table works. What matters is that you can spot the same mistake twice.

Set a weekly review for the logs. Look for repeated edits, common rejection reasons, and places where the model drifts outside the allowed sources. Tighten the prompt, remove weak sources, or add a rule when you see the same failure more than once. If reviewers keep fixing one sentence or one claim, turn that into a rule instead of asking humans to catch it forever.

That is how the process improves: small scope, clear sources, simple decisions, and steady cleanup based on real review data.

A simple example from a finance team

A finance team starts with a narrow job: AI drafts replies to invoice disputes. A customer might say an invoice has the wrong amount, a payment already cleared, or a late fee should not apply. The model writes a first draft, but it does not send anything on its own.

The first reviewer is an accountant. That choice makes sense because the risk sits in the numbers and the policy language, not in the code. If the reply gets the amount wrong or promises a credit the policy does not allow, the business has a real problem.

The accountant usually checks a few plain things: the invoice number, dates, tax, and total; the tone of the reply, especially when the customer is upset; and the wording around refunds, credits, and payment terms.

Engineers still matter, but they do different work. They limit the model to billing records, payment status, and approved reply templates. They block it from pulling random context from email threads or other company files, and they set rules so the draft asks for missing proof instead of guessing.

That boundary does more than reduce mistakes. It also makes review faster because the accountant knows where the draft got its facts. If the reply mentions a late fee, the team can trace it back to the billing record and the approved wording.

After a week or two, patterns start to show. The accountant keeps fixing the same sentence about overdue charges. They also keep changing the date format and softening one line that sounds too blunt. Instead of treating those edits as routine cleanup, the team tracks them and updates the prompt and template.

Now the drafts improve in a way people can measure. Maybe the accountant used to spend six minutes per dispute and now spends two. More importantly, the same errors stop showing up.

Only then does the team automate the low-risk parts. AI can fill in the greeting, summarize invoice facts, and ask for missing payment proof. The accountant still reviews unusual tax cases, disputed credits, or larger amounts. That is what good review looks like: subject experts handle business risk, and engineers hold the system inside clear limits.

Mistakes teams make early

Early teams often make one bad assumption: every AI output carries the same risk. It does not. A draft social post and a payment advice email should not go through the same review path.

When teams ignore that difference, they create two problems at once. Low-risk work gets stuck in review, and high-risk work slips through with too little attention. The process should start by sorting outputs by impact, not by how impressive the model sounds.

Another common mistake is giving subject experts the wrong job. A finance lead, recruiter, or compliance manager should judge whether an answer is correct for the business. They should not spend their day tweaking prompts, tracing API errors, or guessing why the tool changed tone after a model update.

Engineers should own that layer. They can manage prompt versions, model settings, logs, rate limits, and the rules that stop unsafe output before it reaches a person.

Teams also let the model send final answers too early. This usually happens after a few good results, when people start trusting the tool more than the process. That trust is cheap at first and expensive later.

If an output can affect money, contracts, customer promises, or compliance, a person should approve it before it goes out. The model can draft fast. It should not decide alone.

Logs are another thing teams skip and later regret. If reviewers edit an answer but nobody records what changed and why, the team learns nothing. The same errors come back, and people argue from memory instead of evidence.

One more problem shows up quickly: everyone edits prompts. A product manager changes wording on Monday, an engineer adds a rule on Wednesday, and an analyst pastes in a new example on Friday. A week later, nobody knows which change helped or hurt the result.

The warning signs are usually plain. Reviewers spend time fixing style instead of checking risk. Experts report errors but cannot explain the pattern. Engineers cannot trace which prompt version produced an answer. The team trusts the good outputs and ignores the inconsistent ones. Prompt changes keep happening without one clear owner.

This split between business review and system control is a practical one. Oleg Sotnikov often works this way in AI-first operations: subject experts review business risk, and engineers control the tooling, boundaries, and failure paths.

A short checklist before launch

Launch is the wrong time to guess who should approve a risky answer. Before anyone turns on AI review for customers or staff, the team should be able to answer five basic questions.

1. Have you mapped the outputs that can cause harm? Start with anything that can change money, legal terms, customer promises, compliance language, or medical and safety advice.
2. Does each risky output have one named owner? A group mailbox is not enough. One person should own subject expert review for each answer type, even if others can cover for them.
3. Does the system stop and ask for a human at the right moment? Do not rely on people to remember when to review. The product should block sending, publishing, or approving until the reviewer signs off.
4. Do you save edits and the reason behind them? If the reviewer changes a number, removes a claim, or rewrites a sentence, keep that record.
5. Can you roll back quickly? If output quality drops after a model update or prompt change, the team should be able to turn off automation, switch to manual review, or restore the last safe setup in minutes.

This sounds basic because it is. Teams skip one of these steps all the time and then act surprised when nobody knows who approved a bad answer.

A small example makes it real. Say an AI drafts refund replies. If it promises credits outside policy, support costs rise fast. A named operations lead should review those drafts at launch, the system should hold replies with policy exceptions, and the team should log every edit. If the model starts slipping after a change, the team should switch back to human-written replies the same day.

That kind of discipline is boring. It also prevents most avoidable damage.

What to do next

Start smaller than you want to. Pick one output type that can cause real harm if it is wrong, such as a policy answer, a finance summary, or a contract draft. Put only that work into a small review queue, and give it to one reviewer group that knows the subject better than the model does.

This keeps the process simple at the start. You learn faster when one team reviews one kind of output with one set of rules. If you launch with five queues, three risk levels, and a long approval chain, people will stop using it.

For the first few weeks, track a few numbers: edit rate, review time, and blocked mistakes. Edit rate shows how often reviewers change the draft. Review time shows how long a person needs to approve or reject it. Blocked mistakes show where human review pays for itself.

Those numbers tell you what to fix. A high edit rate usually means the prompt, source data, or output format is weak. Long review time often means the model gives too much text, not enough structure, or both.

Do not keep everything in manual review forever. Once a case stays clean for long enough, move it out of the queue and let the system handle it with rules and monitoring. Keep people on the outputs with real domain risk, and let engineers own routing, logging, limits, and fallback rules.

A simple pace works well: start with one queue, measure for two to four weeks, tighten prompts and rules, remove review from the safest cases, and recheck the numbers every month.

If you are setting this up in a small or midsize company, Oleg Sotnikov at oleg.is works as a fractional CTO and advisor on practical AI adoption, engineering guardrails, and lean review workflows. That kind of outside help is most useful when the team needs clear rules and a process it can actually keep running.