Latency budgets for AI features users will tolerate

Why teams keep arguing about speed

Most teams do not fight about speed in planning. The fight starts after release, when a few users complain and nobody agreed on a target up front.

A polished demo often creates the problem. The team tests the happy path on a clean machine, with warm caches, short prompts, and no queue. It feels fast, so everyone moves on. Real usage looks different. People paste messy inputs, run bigger jobs, retry failed requests, and hit the feature at the same time.

Teams also mix very different AI jobs together. One person is thinking about chat, where a reply should start right away. Another is thinking about report generation, where a user may wait longer if the result saves 30 minutes of work. Both are talking about speed, but they mean different jobs.

Without a latency budget, every opinion sounds reasonable. A product manager says six seconds is fine because the answer is smart. An engineer says six seconds feels broken in chat. Support hears that the tool is slow, but the complaint usually means something more specific: "I could not keep working while it thought."

That is what many teams miss. Users do not judge raw seconds on their own. They judge interruption. If the AI blocks the next action, even a short wait feels long. If it runs in the background and the user can keep moving, a longer wait often feels acceptable.

Waiting eight seconds for an answer in live chat feels clumsy. Waiting 40 seconds for a draft report can feel fine if the user can review another tab, answer email, or keep editing while the job runs.

Release day turns loose guesses into arguments because the product finally shows where delay hurts. The team was not really fighting about engineering. They were talking about different user jobs, and nobody wrote those jobs down as clear AI response time targets. Latency budgets for AI features fix that by turning vague expectations into numbers tied to real user behavior.

What users judge while they wait

People do not judge delay by the clock alone. They judge what the wait does to them.

The first moment after a click matters most. Users want an immediate sign that the product heard them. A button state change, a short status message, or the first words of a reply can calm people fast.

If nothing changes, many users assume something broke. That reaction shows up before they make any careful guess about server time or model size.

Waiting feels bad when people are blocked. If someone cannot move to the next step, cannot edit, and cannot even tell whether work started, a short pause feels long. If they can keep reading, typing, or reviewing partial output, they will often accept a much slower final result.

That is why latency budgets for AI features should start with blockage, not benchmark charts. A tool that streams a rough draft in two seconds can feel faster than one that returns a polished answer in eight, even when the second answer is better.

Visible progress buys patience, but only when it means something. A draft headline, extracted fields, a short summary, or a clear status like "Analyzing your file" gives users a reason to wait. A vague spinner does the opposite. It gives no clue about progress, no hint about quality, and no reason to trust the system.

Useful early output changes the math. If the first response already helps someone decide, edit, or verify something, they forgive more delay for the rest. Think about an AI support assistant: a partial answer with the likely fix earns more patience than a blank loader, even if the full answer takes another five seconds.

Teams often debate speed as if every second has the same cost. Users do not experience it that way. They ask a simpler question: can I do something now, or am I stuck?

Group AI work by job type

Most teams argue about speed because they treat every AI feature like chat. That blurs the real question: what is the user trying to do right now?

Latency budgets work better when you sort tasks by job type, not by model or team preference. A person who is typing needs instant feedback. If the tool suggests the next words, fixes grammar, or fills a short field, even a short pause feels broken. People stay in a typing rhythm, and the product has to keep up.

Live actions

A reply draft sits in a different bucket. The user clicks once, then waits for something bigger than a single suggestion. A few seconds is usually fine because the output replaces real work. If the draft is decent, most people accept the pause.

Summaries can stretch longer than reply drafts. Users often trade a short wait for less reading, and that trade feels fair. If a summary saves five minutes of scanning notes, an extra few seconds rarely causes frustration. The catch is simple: the result needs to be clean and easy to trust.

Deep analysis needs its own flow. Do not force it into the same box as chat. If the system compares documents, reviews a codebase, or finds patterns across many records, users expect a longer wait. They do not expect to stare at a blinking cursor with no clue what is happening.

Delayed actions

Background jobs follow different rules. A nightly classification run, a batch enrichment task, or a long report does not need the same limits as live actions. Users care more about whether the job finishes, whether it fails clearly, and whether they can return to the result later.

A simple test helps: ask what the user pauses to receive. If they pause mid-typing, keep it near instant. If they click and wait for a draft, allow a few seconds. If they hand off a heavy task, make it asynchronous and give clear status updates. That one decision ends a lot of pointless debate after launch.

How to set a budget step by step

Good latency budgets for AI features start with the job, not the model. Technical descriptions usually drag the conversation in the wrong direction. Use plain words instead: "summarize this meeting," "draft a reply," or "find the missing order number."

Then write down what the user is trying to finish. That sounds obvious, but it changes the target. A user who wants a quick suggestion can accept a rough first draft fast. A user who wants a final answer for a customer may wait longer, but only if the result is clearly moving.

1. Name the action in one short sentence. Keep it concrete and tied to one click, tap, or prompt.
2. Define the finish line. Ask, "What can the user do once this returns?" If they still need to edit half of it, the job is not done.
3. Decide what matters more: early feedback or full output. For some tasks, a fast first token or visible partial answer is enough. For others, only the completed result counts.
4. Set three numbers: a target, a warning point, and a stop point. The target is what you expect most of the time. The warning point tells the team the feature feels slow. The stop point is where you stop waiting and switch to a fallback.
5. Put those numbers in the product spec. Include the network conditions you assume, the device that matters most, and who owns the metric after launch.

The fallback matters more than teams expect. If the model runs long, do not leave people staring at a spinner. Return a partial answer, switch to a smaller model, move the task to the background, or let the user choose "notify me when ready."

A small example makes this concrete. If the action is "draft a support reply," you might set two seconds for visible feedback, eight seconds for a usable draft, and 15 seconds as the stop point. After 15 seconds, the product saves the request, keeps working in the background, and tells the user when the draft is ready.

Once those numbers live in the spec, speed stops being a matter of opinion. The team can test against a written budget instead of arguing from gut feel.

Reasonable targets for common jobs

Acceptable AI latency depends on the job, not the model. Users forgive a longer wait when the task is heavy. They do not forgive delay when they expect the screen to keep up with them.

Typing help needs to feel almost instant. If your assistant suggests the next few words, fixes a sentence as someone types, or fills a field inline, aim for about 0.1 to 0.3 seconds. Any slower, and the tool starts to feel sticky.

A short rewrite or reply draft can take a bit longer. Most users will wait around two to five seconds if they expect a complete result, such as a cleaner email, a better headline, or a short response to a customer message. Past that point, many people click again, switch tabs, or assume the request got lost.

Summaries sit in the middle. If someone asks for a page summary, a meeting recap, or a quick extraction of action items, five to 10 seconds usually feels fair. The task sounds heavier, so the wait makes sense.

Large document analysis is different. If the system reads a long contract, checks a policy set, or compares several files, users can accept 10 to 30 seconds. But you need to show movement. Tell them what the system is doing, how far it got, or which file it is on.

Bulk work should almost never block the screen. If a user wants 500 records classified, 200 support replies drafted, or a whole folder summarized, run it in the background and notify them when it is done. People care less about raw speed here. They care that the app does not trap them.

A simple rule works well:

• Under 0.3 seconds feels live.
• Two to five seconds feels like a short task.
• Five to 10 seconds feels like a heavier request.
• Ten to 30 seconds needs visible progress.
• Anything longer should become a background job.

Teams get into trouble when they treat every AI call the same. They are not the same. A product team that sorts jobs this way makes better tradeoffs, spends less on overbuilding, and avoids a lot of post-launch debate.

A simple product example

Picture a support agent opening a messy ticket with 40 messages, file uploads, and notes from two teammates. They do not need every AI task to finish at the same speed. They need the product to react fast, show that work started, and return the first useful result soon enough to keep moving.

The agent clicks "Draft reply." The screen should answer almost at once. A button state change, a short status message, and a loading placeholder should appear in under a second. If nothing happens for even two seconds, many people assume the click failed and press it again.

The first draft has a tighter budget than deeper analysis. In most support tools, a draft reply should land in about two to five seconds. That feels fast enough to stay in the flow and slow enough to allow retrieval, prompt work, and basic guardrails.

Now take the case summary. Summarizing a long thread, checking order history, and pulling earlier notes can take longer. Many agents will tolerate eight to 15 seconds if the product shows progress in plain language, such as "Reading thread" or "Building summary."

A sentiment report is even less urgent. It can run after the draft appears, or in the background while the agent reads. If it arrives 20 or 30 seconds later, that is often fine, especially if the UI marks it as optional analysis instead of blocking the next step.

One ticket can have four different speed targets:

• Click acknowledgment: near instant.
• Reply draft: a few seconds.
• Case summary: longer, with progress.
• Sentiment report: background task.

That split is usually better than forcing every AI action into the same budget. Fast enough depends on the job, not the model alone.

Mistakes that make slow feel worse

A slow AI feature can still feel acceptable. A fast one can feel annoying if the product handles waiting badly.

One common mistake is using one response target for every AI action. That sounds tidy, but it does not match real use. A one-line rewrite, a support reply draft, and a full document analysis do not belong in the same bucket. Users expect each job to move at a different pace.

Another mistake is showing nothing until the full answer is ready. Blank space feels longer than it is. Even a small sign of progress helps: streamed text, a first summary, or a status line that says the system is checking sources.

Teams also fool themselves with averages. An average can look fine while a painful share of users waits far too long. If seven requests return in three seconds and three take 20, the average hides the real problem. Track the slow outliers, not just the middle. Those are the moments users remember.

Mobile makes this worse. A feature that feels smooth on office Wi-Fi can feel broken on a weak 4G connection, on a train, or in a crowded airport. If your product sends large payloads, reloads the screen, or waits on multiple background calls, the delay stacks up fast. Acceptable AI latency depends on the network people actually use, not the network your team used in testing.

Retries are another silent mess. If the app quietly retries the same request again and again, users watch a spinner with no clue what is happening. Set a clear timeout. Tell users when the request failed. Let them retry once on purpose instead of piling extra waiting on top of the first delay.

This is why latency budgets for AI features need product rules, not just model benchmarks. The model matters, but the waiting experience often matters more.

Quick checks before you ship

Most speed fights happen because nobody turns expectations into a visible rule. Before release, write down the target for every AI action in the product. A reply suggestion, image analysis, document summary, and background report do not need the same limit. If the number is not written, people will argue again after launch.

The first second matters more than many teams expect. Users need proof that the product heard them. Change the screen right away. Show a loading state, disable the button, add a placeholder, or open a result panel. Even if the full answer takes longer, that early feedback makes the wait feel shorter.

Longer jobs need a different path. If a task may take several seconds, show progress people can understand, or move the work into the background and notify them when it is done. A user will wait longer for a market report or code review than for a chat reply, but only if the delay feels planned rather than broken.

A short pre-launch review should confirm a few basics: every AI action has a response target in the spec, the interface reacts in under a second after submit, long jobs show progress or background processing, failed generations show plain fallback text with a next step, and logs capture start time, end time, duration, and errors for every request.

Fallback text deserves more care than teams usually give it. "Something went wrong" is lazy and frustrating. Tell users what happened in plain language, keep any input they already entered, and give them one obvious next action, like retrying or switching to a simpler mode.

Logging closes the loop. If you cannot see when a request started, when it ended, and why it failed, you cannot improve performance after release. Timing data is product data, not just a backend detail.

What to do after launch

Launch day gives you guesses. A week of real traffic gives you facts. If you want your latency budget to hold up, check actual wait times every week for each feature, not just one average for the whole product.

A single median number hides the pain. One feature may feel fine on desktop Wi-Fi and awful on a mid-range phone on mobile data. Split your numbers by job type, device, and network so you can see where the delay really lives.

A chatbot reply, a document summary, and a background report do not need the same targets. Review them separately. Users forgive a longer wait when the job is heavy and the result saves real effort. They get annoyed fast when a small action stalls.

What to review each week

Keep the review simple. Check p50 and p95 wait time for each feature, the slowest prompts and context sizes, results by device class and network quality, and the drop-off or retry rate on slow screens.

When one path drifts, trim before you tune infrastructure. Teams often send too much context, repeat hidden instructions, or ask the model to do three jobs in one call. Cutting 30 percent of the prompt can shave off noticeable time without hurting output.

If a task stays slow, move part of the work out of the live flow. Precompute summaries, cache repeated results, or run enrichment after the first answer instead of before it. Users usually prefer a fast first result plus a small follow-up over one long pause.

It also helps to write rollout rules ahead of time. Decide when to switch models, when to cut context, and when to fall back to a simpler response. That stops the same debate from coming back every month.

If your team needs an outside view, Oleg Sotnikov at oleg.is works with startups and small businesses as a fractional CTO and advisor on AI product architecture, infrastructure, and practical automation. This kind of review is often less about chasing one faster model and more about setting better budgets, trimming slow paths, and matching each task to the right user flow.