One engineer on call with AI support: stress test first

Why this setup breaks under stress

A weak on-call plan can look fine for weeks. The trouble starts when several alerts fire at once, the cause is unclear, and one tired engineer has to judge risk before touching production.

One person can do deep investigation or fast triage. They can't do both across four incidents at the same time. When pages keep coming, the work splits into competing tasks: read logs, check dashboards, decide whether to roll back, answer messages, and make a fix. Time spent on one alert leaves another waiting.

AI changes the speed, not the limits. It can summarize logs, suggest commands, and draft a response in seconds. That helps. But it only sees the context you give it. If the alert title is vague, the dashboard has no history, or the notes miss one ugly edge case, the assistant can sound certain while chasing the wrong cause.

Bursts make everything harder. One database slowdown can trigger API latency alarms, queue failures, customer errors, and infrastructure pages within minutes. To one engineer, that feels like four separate fires even when they all started from one root cause.

Small runbook gaps hurt most after hours. During the day, someone can ask a teammate which service to restart first or whether a rollback is safe. At 3 a.m., one missing sentence can add 20 minutes to an outage. An old screenshot, an outdated command, or a note with no rollback steps is enough to turn a manageable problem into a long one.

That is why this model only works when you test the messy version of the night. Quiet hours prove very little. Stress does.

What one engineer and AI should handle

One person on call can cover more than many teams expect, but only when the work is small and boring most nights. The usual mistake is giving that person a mix of routine noise, unclear ownership, and scary edge cases, then hoping AI closes the gap.

Start by splitting alerts into two buckets: routine and urgent. Routine alerts have a known cause, a short playbook, and a safe first action. Disk nearing 80%, a stuck worker, a failed backup retry, or a traffic spike that often settles in five minutes fit here. A true emergency threatens data, payments, login, or customer uptime right now.

AI is useful in the first bucket. It can summarize logs, compare the failure to past incidents, draft a status update, suggest the next command, and turn rough notes into a clean incident record. It should not decide whether to roll back a release, disable billing, fail over a database, or ignore a security signal. Those calls need human judgment because the cost of a wrong guess is too high.

Write down a stop point. If the engineer spends 15 minutes without finding a clear cause, if the fix touches data, or if two services fail at once, wake a second person. Put that rule in plain language. People do better at 3 a.m. when they don't have to debate whether the situation is serious enough.

It also helps to define what "good enough overnight" means for each service. Marketing pages can stay slow for an hour if checkout still works. Internal reports can wait until morning. Password reset emails can queue for 20 minutes. Payment failures need human review now. Anything that risks data loss should move past the solo setup immediately.

This approach works when the engineer and the AI handle repeatable work, the handoff point is documented, and high-risk calls never stay with one person for long. If you can't explain those boundaries on one page, one person should not carry the night alone.

Stress test your alert volume

A solo on-call plan often looks fine on a calm Tuesday and falls apart at 2:13 a.m. when five alerts land at once. AI can summarize logs or draft a response, but it cannot make the queue shorter. If you want this setup to work, measure the load first.

Use real data, not guesses. Pull the last 30 to 90 days of alerts from paging, monitoring, and incident tools. Group them by hour, by service, and by noise level. Daily averages hide the patterns that matter, like one payment service that gets noisy every night at 1 a.m. or a batch job that floods the channel on Mondays.

A simple average can fool you. Ten alerts per day sounds manageable. Ten alerts in 18 minutes is not.

Pick one busy window and replay it like a drill. Open each alert in order and time every step: reading the page, checking dashboards, asking AI for a summary, looking at logs, deciding whether to act, and closing or escalating. Don't skip the tiny steps. They eat most of the time.

A quick scorecard is enough:

• How many alerts arrived within 15, 30, and 60 minutes
• How long one alert took from page to close
• Which alerts needed a human decision
• When the backlog started to grow

That last number matters most. If one person can close six pages an hour and the system throws nine an hour during a noisy period, the setup already fails. AI may save a minute or two on each response, but it will not rescue bad alert design.

Watch for alerts that never change the next action. If the engineer always opens the same dashboard, checks the same metric, and closes the page with no fix, merge that alert with a broader signal or remove it. Repeated noise trains people to ignore the channel.

Teams that run lean systems well, especially those using AI heavily in operations, usually get very picky about paging rules. They should. Fewer alerts with clearer actions beat a clever assistant stuck in a flood of useless pages.

Read your runbooks like a stranger

Most runbooks look acceptable until a tired person opens one at 3:12 a.m. and tries to use it under pressure. That is the test that matters. Pick a common alert, open the runbook, and follow every step exactly as written. Don't fill in missing details from memory.

If step two says "check the queue," the runbook should say where that queue lives, which dashboard to open, what normal looks like, and which command to run if the dashboard fails. If you have to guess, stop and write down the gap. Every hidden assumption becomes delay when one engineer is alone.

A useful rule is simple: if a new hire with production access couldn't complete the first response in ten minutes, the runbook is not ready. Lines like "inspect recent errors" or "restart the worker if needed" waste time. Name the log, the filter, the service, and the safe restart command.

Use a short checklist during review:

• Which exact alert opens this runbook?
• What do I check in the first 60 seconds?
• Which command or dashboard confirms the problem?
• What action is safe right now?
• When do I escalate?

AI needs the same view as the human. If the assistant can't read the notes, logs, recent deploy history, and past incident fixes, it will guess. Guessing often sounds confident, which makes it more dangerous. Give it the same source material you expect the engineer to use, or keep it out of the loop for that alert.

Write the first five minutes in plain language. Start with lines like "Check whether users can still log in" or "See if failures started after the last deploy." A stressed person should not have to decode team shorthand before starting triage.

One small rewrite shows the difference. "Verify recent gateway issues" is vague. "Open payment logs, filter for gateway_timeout in the last 15 minutes, and compare error rate to the usual baseline" gives someone a real first move. That is what a usable runbook looks like at 3 a.m.

Test provider outages on purpose

Most solo on-call plans break at the same point: the helper tools disappear first. Chat goes down, the AI provider times out, code search hangs, or the password manager won't open when you need it most. A setup like this only works if the engineer can keep going alone for a while.

Build a failure map

Write down every outside service between the first alert and the final fix. Keep it simple. For each one, note the backup move beside it. That usually includes the paging tool, team chat, AI provider, code search and repo host, cloud console, docs, and secrets access.

That list gets longer than most teams expect. You may depend on a status page to confirm an outage, a CI runner to ship a rollback, or a phone app for MFA. If any one of those disappears, the incident can stall even when the production issue itself is small.

Then run a practice shift with manual steps only. Turn off the AI assistant. Don't use chat. Find logs the slow way, read the runbook from a local file, and prepare a rollback without code search. If that drill feels clumsy, good. You just found the weak spots before a real customer did.

Keep a local copy of the minimum survival kit on the on-call laptop. That should include runbooks, contact numbers, system diagrams, rollback commands, and break-glass access steps. If your only copy lives in a wiki behind SSO, you do not have a backup.

Set the switch point before the outage

Teams waste time when they keep waiting for a tool to recover. Pick the cutoff in advance. If the AI provider fails three times in ten minutes, stop retrying and work from the runbook. If chat stays down for 15 minutes, switch to phone calls or SMS. If code search is unavailable, use local clones and basic grep.

This is where lean operations matter. Fewer moving parts usually mean fewer ugly surprises at 3 a.m. The test is simple: can one tired engineer still detect, diagnose, and roll back without the smart helpers? If the answer is no, the plan is not ready.

A simple overnight failure story

At 1:40 a.m., a payment alert wakes the on-call engineer. Failed charges jump above the normal range. Two minutes later, checkout latency spikes. Then a third alert lands for a retry queue that is growing too fast. A fourth warning follows when refunds start timing out.

Nothing looks catastrophic on its own. Together, they create the kind of mess that pulls a tired person in four directions at once.

The AI helper does the obvious first step well. It groups the alerts, suggests a few checks, and points to the payment runbook. The engineer follows it and starts with the usual service health checks, database load, and payment gateway logs.

That would have worked last week. It doesn't work tonight.

A recent config change moved part of the payment flow to a different service and changed who owns the rollback. The runbook never got updated. The AI can't guess a step that nobody wrote down, so it keeps recommending checks around the old path. The engineer loses time proving that the documented path is now the wrong path.

The next clue should come from the cloud provider, but the public status page still shows green. One region is failing anyway. Internal metrics show requests hanging only in that region, while the rest of the system looks normal. A green status page gives false comfort when you need a fast answer.

Now the engineer has two problems: find the real owner and find the rollback step that still works with the new config. Messages go to the old payments contact first, then platform, then the person who approved the config change earlier that day. About 40 minutes pass before someone confirms the right feature flag and the safe rollback order.

Traffic shifts, errors drop, and charges recover before most customers notice. The incident still leaves a mark.

The morning review is blunt. The team did not fail because one person was on call, and the AI did not fail because it gave bad advice. The process failed because nobody stress tested the process itself. The team trusted alerts, runbooks, and status pages more than it tested them under pressure. That is where this model usually cracks first.

Mistakes teams make with solo on-call and AI

The one-person overnight model often looks cheap and calm during a normal week. It usually breaks for ordinary reasons, not dramatic ones. Teams count pages, but they don't measure the 20 or 40 minutes each page can eat once someone has to read logs, check dashboards, verify impact, and decide whether the AI answer is safe.

That blind spot matters more than raw alert count. Ten alerts in a night might sound fine. Ten alerts that each need half an hour of real judgment will leave one person tired, slow, and likely to miss the alert that actually hurts customers.

Another common mistake is giving AI too much authority. If the tool replies to every page with confidence, people start trusting the tone instead of the facts. A thin summary, stale context, or a missing dependency can push the engineer toward the wrong restart, the wrong rollback, or a false "all clear."

Runbooks also fail in boring ways. One note lives in the wiki, another sits in a chat pin, a third hides in an old repo, and the newest fix exists only in someone's memory. At 3 a.m., scattered runbooks are almost the same as no runbook.

Provider and network trouble expose the next gap. Teams often assume the AI provider, cloud console, chat tool, and incident channel will all stay up together. Real incidents don't respect that assumption. If DNS fails, the model API slows down, or the main network path drops, the solo engineer needs a plain fallback: local access, cached docs, direct commands, and a human escalation path.

The last mistake is psychological. A quiet week proves almost nothing. Low traffic, lucky timing, or the absence of a real dependency failure can make a weak setup look solid.

A healthier test is simple:

• Time how long alerts take from page to confirmed fix
• Check whether the AI answer cites current, specific evidence
• Keep one runbook source for each service
• Practice one incident with no provider help
• Review near misses, not just outages

If the setup only works when everything else works, it isn't ready for one person to carry overnight.

Quick checks before you go live

If your plan only works on a quiet afternoon, it will fail at 3 a.m. This setup needs a few plain checks before you trust it with a real night.

Most teams skip them because the system looks fine in a demo. Real pressure changes everything. Alerts stack up, runbooks feel thin, and the AI may stop helping right when you need it.

Give one engineer a batch of fake alerts that matches a bad hour, not an average one. Mix repeat pages, noisy warnings, and one or two real failures. The engineer should sort the noise, find the real problem, and take the first repair step without guessing.

Open the runbooks for every page you expect to see often. Read them like a stranger would. Remove stale commands, old screenshots, missing log locations, and steps that only make sense if someone already knows the system.

Turn off access to the AI provider for a short test. The engineer should still have enough notes, shell access, dashboards, and judgment to keep things stable for a while.

Write escalation rules in plain language. Name the backup person, say when the first engineer should call, and set a time limit. "Escalate after 20 minutes if the fix path is still unclear" is much better than "use judgment."

Review the night the next morning, even if nothing fully broke. If an alert caused confusion, update the runbook. If the AI gave a weak answer, note it and add the missing step for next time.

These checks are not busywork. They tell you whether one person can survive a rough hour without burning out or making random changes in production.

A solo setup can work. But it only works when the boring parts are solid: alert load that one person can clear, current runbooks, a backup plan for provider outages, and a real human backup with a clear handoff time.

What to do next

Run two drills before you trust this setup. Do one during office hours, when people can think clearly and fix missing pieces fast. Do another at night or early morning, when the engineer is tired, the AI tool is slower to use, and small gaps suddenly matter.

This model needs proof, not optimism. During each drill, track three things: how many alerts arrive, how long it takes to find the right runbook, and where the handoff between person and AI gets awkward. If one alert fires ten times in an hour, fix that noise first. Another AI tool will not rescue a bad alert.

A short checklist helps:

• Pick one realistic failure, like a database connection spike or a third-party API timeout
• Page the engineer the same way production would
• Make them use the actual runbook, not memory
• Note every step that needs guesswork, missing access, or copy-paste from old chat logs
• Turn the notes into fixes the next business day

Review the setup after every incident, even small ones. A quarterly review sounds tidy, but it is too slow. People forget details after a few days. If the runbook was wrong, change it. If the alert was noisy, tune it. If the AI gave a weak answer, tighten the prompt, add context, or stop using it for that task.

Some teams can already tell when the plan is too thin. Maybe one provider outage would knock out both monitoring and the AI helper. Maybe the on-call engineer still has too many manual steps. That is the point where an outside review can save time.

If you want a second opinion before going live, Oleg Sotnikov at oleg.is helps startups and small businesses review alert flow, runbooks, escalation paths, infrastructure choices, and AI-augmented engineering workflows. A short consultation can expose weak spots before they show up at 3 a.m.