Weekly model evaluation for product teams shipping AI

Why teams miss regressions until users complain

A model can look great in a demo and still fail in real use. Demos are tidy. Real work is messy, rushed, and full of edge cases nobody thought to show on a call.

Teams also test with prompts they already know will work. That creates false confidence. If the model handles five familiar examples well, people start to assume it will handle the other five hundred.

Small changes can shift output more than most teams expect. A prompt edit, a new system message, a model version change, or a retrieval tweak can turn a solid answer into something vague, too long, or just wrong. The app still runs, so the problem hides in plain sight.

Users usually catch this first because they see the model under pressure. They ask odd questions, paste broken text, skip context, and still expect a useful answer. When the model starts missing details or inventing facts, support tickets arrive before the team has clear proof.

It gets worse when nobody keeps score. One person says, "It felt better last week." Another says the new version is faster. Someone else remembers a good example from yesterday. Memory is a bad test method, and the loudest opinion often wins.

A simple weekly evaluation fixes that. Instead of arguing from a few stories, the team looks at the same small set of tasks and scores the answers the same way each week. The model still won't be perfect, but regressions become much harder to ignore.

This matters because AI quality drifts quietly. Traditional bugs often break something obvious. Model quality slips in softer ways. The answer gets less precise. It skips one constraint. It sounds confident while missing the point. Customers don't care whether that came from a prompt tweak or a model swap. They only see that the product got worse.

That's why complaints often feel sudden to the team. The drop was already there. Nobody measured it early enough to catch it.

Pick the jobs your model must handle

Start with work people already trust the model to do. Don't begin with the smartest demo prompt or the hardest edge case. Pick the jobs that show up every week and affect real customers, money, or team time.

For most product teams, 5 to 10 jobs is enough. Fewer than that misses too much. More than that turns weekly review into homework nobody finishes.

A "job" is a clear user outcome, not a vague skill. "Answer support questions about refunds" is a job. "Be helpful" is not. "Extract invoice totals from uploaded PDFs" is a job. "Understand documents" is too broad.

A reasonable starting set might include classifying incoming requests, answering product questions with approved facts, drafting replies in the right tone, extracting fields from forms or documents, and flagging risky cases for a human.

Once you have a list, rank each job by impact. Ask two simple questions: how often does this happen, and how bad is it if the model gets it wrong? A wrong label on an internal note may be annoying. A wrong refund answer may cost money and trust.

Put the highest-impact jobs at the top, even if they aren't the most interesting. Teams often spend too much time testing rare edge cases and too little time testing the boring tasks users hit every day.

Then write one success rule for each job. Keep it short enough that anyone on the team can score it quickly. Good rules are concrete: "The reply states the refund policy correctly and asks for the order number if it is missing." Bad rules are fuzzy: "The reply is good" or "The model seems accurate."

If a rule starts a long debate, it's too loose. Tight rules make regression testing useful because different reviewers can look at the same answer and land on almost the same score.

Build a small test set from real work

A useful test set starts with work your team already sees every week. Support chats, bug reports, QA notes, and failed handoffs are better than made-up prompts because they show where the model actually gets confused.

Pull cases from the last few weeks, not from memory. Memory edits out the messy parts. The raw wording, missing context, and odd customer behavior are often the whole point.

Clean each example before you save it. Remove names, email addresses, account numbers, company details, and anything else a customer wouldn't expect your team to copy into an internal test file. If a case only makes sense with private data, rewrite that part with a safe placeholder and keep the structure the same.

You don't need a giant dataset for weekly evaluation. Start with 20 to 50 cases. That's enough to spot obvious drops in quality without creating a maintenance job your team will ignore.

Mix the set on purpose. Include a few easy requests the model should almost never miss, some medium cases with extra context, and some messy ones with unclear wording or conflicting details. A support inbox usually has all three. One customer asks a plain refund question. Another pastes a broken order ID. A third mixes billing, anger, and a vague deadline into one message.

Keep the same few fields for every case: the original input, a short note on the job to do, a brief expected result, and the score your team gave later.

The expected result should stay brief. Don't write a polished answer if your real agents would never respond that way. One or two sentences are often enough: answer the billing question, ask for the missing order number, avoid promising a refund, and keep the tone calm.

If your test set stays close to real work, people will trust it. If scores drop on these cases, you probably have a real problem, not a lab problem.

Write scores your team can apply fast

If a reviewer needs a long debate to grade one answer, the rubric is too hard to use. Weekly evaluation only works when someone can look at an output, score it in under a minute, and move on.

Start with pass or fail for tasks that have a plain right answer. If the model must extract an order number, choose the right support category, or return a valid yes or no, don't add nuance that doesn't help. Either it did the job or it didn't.

Use a 1 to 5 scale when the output can be good, bad, or somewhere in between. Drafting a support reply, summarizing a call, or rewriting a message for a different tone fits this better. A 5 means you would send it as is. A 3 means it's usable but needs edits. A 1 means you would throw it away.

Don't collapse everything into one grade. Score accuracy, safety, and format on separate lines. A response can be factually right and still unsafe. It can also be safe and helpful but ignore the required structure.

A short scoring sheet is usually enough:

• Pass or fail for exact tasks
• Accuracy from 1 to 5
• Safety from 1 to 5
• Format from 1 to 5
• One sentence on why you picked those scores

That last line matters more than teams expect. The sentence keeps people honest, and it gives you something useful to review later. "Wrong refund policy" is better than a silent 2. "Correct answer, but broke the JSON format" tells the team exactly what changed.

Keep the rubric short. If it needs a full page of rules, people will stop using it by week three. Most teams do fine with four fields and a short note.

A support team is a simple example. Say the model drafts 20 replies each week. One answer may earn accuracy 4, safety 5, and format 2, with the note "Good reply, but it ignored the required bullet format." You can spot the problem fast, and you know where to fix it.

If scoring takes longer than running the test set, cut the rubric until it feels easy.

Run a weekly review in 30 minutes

A weekly model evaluation works best when it feels almost mechanical. Use the same test cases, the same rubric, and the same notes format each time. If the setup shifts during the week, the score stops meaning much.

Freeze the exact version of what you're testing before you run it. Record the model name, prompt version, temperature, tools, and any retrieval source that can change the answer. Teams often think the model got worse when the real change came from a prompt edit or a tool setting.

Run the same small test set every week. Keep it stable long enough to spot movement. Even 20 to 40 cases can catch regressions early if those cases reflect real customer jobs.

A 30-minute review is usually enough. Spend about 5 minutes running the cases and recording scores, 5 minutes comparing this week's result with last week's, 10 minutes reading the cases that changed the score, and 10 minutes writing actions and assigning an owner.

Don't spend most of that time staring at the average. Read the failures that moved the result. If three billing replies dropped from "correct" to "partly correct," that tells you more than a small change in the total score. Look for shared causes: a shorter prompt, missing context, a new refusal pattern, or a broken tool call.

Then log one action for each pattern you find. Keep the action concrete. "Review prompt" is too vague. "Add account status to retrieval for billing cases" gives the team something real to test next week.

A simple rhythm helps. On Monday morning, one person runs the set. The team reviews the changed failures, not every answer. By the end of the meeting, each pattern has one owner and one next step. That's enough to make regression testing part of normal product work instead of a rushed fix after users complain.

A realistic example from a support inbox

A SaaS team uses AI to draft refund replies before a human agent sends them. The drafts save time, but only if the wording matches the current refund policy.

On Monday morning, the company changes one rule. Customers on annual plans can no longer get a full refund after the first 30 days. They can get a prorated credit instead. The support lead updates the policy note for the team, but the AI prompt still leans on the old wording.

That afternoon, someone runs the small test set the team keeps for weekly evaluation. It has a dozen real support cases pulled from past tickets, with names and account details removed. One case is simple: a customer on an annual plan asks for a refund on day 45 and sounds upset.

The model writes a polite reply, but it says the customer qualifies for a full refund. That is now wrong. A human agent might miss the mistake because the tone sounds calm and the message reads well.

The score makes the problem obvious. The team checks each answer on a short rubric: does it follow the current policy, does it state the next step clearly, and is the tone respectful and firm?

This reply fails the first check, so the case gets a low score even though the writing looks good. That matters. Nice wording doesn't fix a bad refund term.

The team updates the prompt and the policy snippet the model sees. They tell it to quote the current annual-plan rule and avoid promising a full refund when the account is outside the allowed window. Then they rerun the same case. The new draft offers the prorated credit, explains why, and tells the customer what happens next.

That quick test saves a bigger mess later. Without it, agents could copy the bad draft into live tickets for hours. A tiny test set caught the regression before customers saw it, and the fix took less time than cleaning up even a few wrong refund promises.

You don't need a giant QA process for this. You need a few real cases, clear scoring, and one person who checks them every week and after any policy change.

Mistakes that make scores useless

A score can look neat and still tell you almost nothing. The usual problem isn't math. It's sloppy setup. If your weekly review is built on easy cases, shifting rules, and one blurry average, you'll miss the failure that annoys customers most.

The first trap is testing only clean examples. Real work is messy. Users leave out facts, ask two things at once, paste broken text, or use the wrong terms. If your test set only includes tidy prompts with obvious answers, the model will look better than it is. Keep some ugly cases in the set on purpose.

Rubric drift ruins trust fast. A team starts Monday scoring an answer as "good" if it is correct and polite. By Thursday, one reviewer also expects brevity, another wants citations, and a third starts punishing harmless wording choices. Now the score changed, but the model may not have. Freeze the rubric for the week. Change it later, version it, and rescore if needed.

Teams also confuse cause and effect when they change too much at once. If you swap the model, edit the system prompt, and add retrieval updates in the same release, you can't tell what helped or hurt. Keep one experiment clean. Change one major thing, score it, then move on.

One reviewer scoring every case alone creates another blind spot. People get tired. They also build habits. One person may care too much about tone, while another ignores factual errors if the answer sounds smooth. You don't need a committee for every test, but you do need a second set of eyes on a sample each week.

The last mistake is trusting one average. A model can score 8 out of 10 overall and still fail badly on refund requests, long inputs, or messages written by non-native speakers. Those clusters matter more than the headline number.

Keep five things fixed and visible:

• The exact test set version
• The rubric version
• The model version
• The prompt version
• Scores by segment, not only the overall average

If one of those moves without a note, the score stops being useful. Then you're back to guessing.

Quick checks before you trust the score

A score can look fine and still hide a bad release. Before your team treats the result as real, make sure the test reflects what users actually asked for this week, not what they asked for a month ago.

Weekly evaluation only helps when the sample matches live work. If 60 percent of user requests are support replies and billing questions, but half your test set is creative writing prompts, the score will lie to you.

Before you compare this week to last week, check the task mix, the disputed cases, the required format, safety failures, and scores by task group.

That last check catches a lot of trouble. A model can hold the same overall score while one task group falls apart. Summarization may stay steady while extraction starts missing dates and order numbers. Customers notice the broken part, not the average.

Format checks deserve more attention than teams usually give them. If your product needs JSON, a numbered list is not close enough. If a support draft must include a refund policy sentence, leaving it out is a failure. People often score content quality and forget the output contract.

Unclear cases also need discipline. If one reviewer says an answer is "good enough" and another says it missed the point, your rubric is still too fuzzy. A two-person review on disputed examples keeps the score from drifting with mood.

Safety and policy issues should sit outside the normal score, not be buried inside it. Track them as separate fails. One leaked secret, one unsafe instruction, or one invented policy can outweigh several polished answers.

If you do only one extra step, break the results into small groups and read the misses. That takes five minutes, and it often tells you more than the headline number.

What to do next when scores drop

A lower score should change work that week, not start a long debate. Start with the last risky change and ask a plain question: what changed right before the score moved? In most teams, the cause is recent and boring. A prompt changed, a model changed, retrieval pulled different context, or someone adjusted the rubric.

If the drop is sharp and customer-facing, roll back first. You can study the issue after service is stable again. Teams lose time when they keep a bad change live just to prove they can debug it.

A quick check usually narrows the problem fast. Look at whether the model or prompt changed, whether retrieval or tools changed, whether the rubric changed, or whether the input mix changed because users asked different questions.

Once you spot the likely source, fix one failure pattern at a time. Don't lump five issues into one ticket like "quality dropped." That hides the real problem. Split it into small, visible cases such as wrong tone, missed policy step, fabricated detail, or poor citation to the source text.

Every new bad case should go into the test set. If the model failed on a refund request with mixed language, save that exact case. If it mishandled a short, angry support email, save that too. A small test set gets stronger when it grows from real mistakes instead of imagined ones.

Share the result across product, support, and engineering. Product needs to know what users will feel. Support needs to know what to watch for in live conversations. Engineering needs a tight target for the fix. One short note is enough if it includes the score drop, the suspected cause, the affected task, and the decision: rollback, patch, or monitor.

The whole loop should feel calm and routine. Score, inspect, fix, add cases, repeat.

If your team needs help setting up that habit, Oleg Sotnikov at oleg.is works with startups and smaller companies on AI-first product development, infrastructure, and practical CTO-level guidance. A light outside review can help you build a process that fits real work instead of turning into another heavy QA ritual.