Small SaaS incident runbooks before the first outage

Why runbooks matter before the first outage

Your first incident usually burns more time on confusion than on the bug itself. In a small SaaS team, one person checks logs, another replies in chat, someone else starts a rollback, and nobody owns the next move. Ten messy minutes can turn a minor deploy issue into lost signups, failed renewals, and a pile of support tickets.

Stress makes people forget simple things. Which service changed last? Who can approve a rollback? Where is the last clean backup? Chat threads make it worse. Messages arrive out of order, advice conflicts, and the team starts debating while users keep hitting errors.

A one-page runbook stops that drift early. It gives the team a default path for the first 15 minutes: who leads, what to verify first, what to pause, what to tell users, and when to escalate. That cuts hesitation. It also helps quieter teammates act without waiting for permission while the outage grows.

Most teams do not need fancy tooling or a huge internal wiki. A plain page with short, clear steps usually works better because people can scan it under pressure.

The business cost is simple. If payments fail for 20 minutes, revenue drops. If users get locked out, trust falls fast. If a bad deploy breaks signup, every minute hurts future sales, not just current traffic. Written steps protect money and reputation because they replace panic with a repeatable first response.

When the alert fires, your team should not open a blank chat and ask what to do. The answer should already be written down.

What a good runbook should include

A runbook should remove guesswork when people feel pressure. If someone opens it during an outage, they should know who leads, what triggered the runbook, and what to do first.

Start with names. Put one owner at the top, then one backup who takes over if the owner is asleep, offline, or already busy. Teams lose time when two people both think they are in charge, or when nobody is.

Then define the exact trigger. Name the alert, error, or support pattern that starts the runbook. "Payment API error rate above 5% for 10 minutes" is clear. "Payments look bad" is too vague.

The first actions matter most, so write them in order and keep them short. For most teams, that means confirming the issue, reducing customer impact, and checking the system most likely to have failed. A stressed teammate can follow three plain steps. They will skip a wall of text.

You also need customer message rules. State when the team should post a status update, who approves it, and when support can reply on its own. If approval sits with one founder who is on a flight, the rule is already broken.

Add a simple map of your tools too. Say where logs live, which dashboard shows system health, where backups are stored, and who has access. If that information lives only in one engineer's head, the runbook is not finished.

Leave a small space at the end for timestamps, actions taken, and the final fix. That note turns one rough incident into a better runbook next time.

How to write a runbook people will actually use

The easiest way to write a runbook is to start with something that already went wrong. Pick one real incident, failed deploy, or near miss and rebuild the response from memory, logs, and chat notes. That gives you real actions instead of vague advice.

Write every step as something a person can do right now. Use clicks, commands, or checks. "Open the payment dashboard and confirm whether new charges are failing" is much better than "investigate billing." If a step depends on a decision, say what to check and who decides.

Keep each step short. People read runbooks when they are tired, rushed, or under pressure. Long paragraphs slow them down. One action per line is often enough, especially for deploy failures, user lockouts, or data restore steps.

Test it with someone else

A draft is not ready just because the writer understands it. Ask another person to follow it exactly as written. Watch where they pause. If they ask which dashboard to open, which environment to use, or what a term means, the draft still has gaps.

A quick review usually finds the same problems:

• missing permissions
• unclear command order
• no rollback point
• no owner for approvals
• no way to confirm the fix worked

Store the final version somewhere the whole team can reach fast. A shared internal wiki, repo, or ops folder is fine if everyone can open it during an incident. Put the owner name and last review date at the top. Old runbooks create new problems, so update them after every real incident.

Short template for a deploy failure

A bad release usually shows up in the parts users touch first. Sign-in breaks, checkout fails, or API calls start returning errors. When that happens, stop shipping changes, assign one owner, and follow a simple script.

This runbook pays off almost immediately because deploy problems often have a fast fix: undo the change that caused them.

• Treat it as an incident when a release breaks login, checkout, core pages, or public API requests.
• In the first 10 minutes, stop new deploys, confirm the scope, and name one incident lead.
• Roll back to the last healthy version. If the release changed static assets, config, or sessions, clear the right cache and then watch the error rate.
• Send a customer update that says what broke, who feels it, what the team is doing, and when the next update will come.
• After recovery, write down the bad commit or config change, the check that missed it, and one release guardrail to add.

A small example makes the point. If today's deploy breaks checkout, do not spend 30 minutes debugging production while orders fail. Roll back first. If the error rate drops and successful payments return, you bought time to inspect the change safely.

Keep the aftercare short but specific. "Checkout failed after commit 8f3a. No test covered coupon plus tax flow. Added canary release and checkout smoke test." That is enough for the next person on call. A good runbook helps a tired teammate make the safe move in two minutes. It does not need to read like a perfect report.

Short template for a payment outage

A payment issue can hide longer than a deploy failure. The site may still load, but checkout fails, webhooks stop, or revenue drops fast. This runbook needs clear first moves because repeated charge attempts can make the mess worse.

Use it when the payment provider returns API errors, subscription webhooks stop arriving, or successful payments fall well below the normal rate for the same hour.

1. Assign one owner and check provider status, recent error logs, and your own monitoring. Confirm the scope fast: all payments, one region, one plan, or only renewals.
2. Pause noisy retries and any job that keeps hammering the provider. Save failed payment events to a queue or table so you can replay them later.
3. Test one payment path yourself. Try checkout, renewal, and webhook delivery if possible. Write down what works and what fails.
4. If you have a backup flow, switch to it. That might mean a second provider, a manual invoice path, or temporary "payment pending" handling instead of a hard failure.
5. Post a short customer message in the app or status area. Say checkout may fail, ask users not to retry the same payment again and again, and tell support what to say if people ask about duplicate charges.
6. When the provider recovers, replay queued events, retry failed charges once, and verify account state. Check that users got the right access after payment.

After the outage, clean up the ugly parts. Look for missed renewals, duplicate charges, refund requests, and support tickets from users who got locked out after paying.

A simple rule helps: do not chase every failed payment live. First stop the noise, then preserve the failures, then recover in order.

Short template for a user lockout

User lockouts feel small until support gets ten tickets in five minutes. This runbook needs clear steps because the cause can sit in several places at once: the login form, SSO rules, rate limits, session handling, or an admin change that removed access.

Start by checking scope. One affected user usually points to account state, browser issues, MFA trouble, or a bad password reset. A wave of failures across many users usually points to auth config, SSO, rate limiting, or a recent deploy.

Use a short runbook like this:

• Trigger: repeated login failures, many reset emails, locked admin access, or a sudden rise in auth errors.
• Scope check: confirm whether it affects one account or many. Check auth logs, recent permission edits, SSO status, and any auth changes from the last deploy.
• Safe recovery: verify the user first, then unlock the account, clear broken sessions, resend the reset flow if needed, and review rate limits or MFA rules before you turn anything off.
• Support notes: ask for the account email, device type, browser, exact error message, and last successful login time.
• Aftercare: fix the rule or code path that caused the lockout, such as bad SSO mapping, a session bug, or a mistaken role change.

One rule matters here: never disable security for everyone just to help one person log in. If rate limits block real users, tune them with care and keep a record of what changed.

Close the runbook with two facts: what users saw, and what your team changed. That makes the next lockout much faster to sort out.

Short template for a data restore

Data restores go wrong when teams rush. The first job is to stop the damage, not to recover everything in one move.

Use a simple script that any on-call person can follow at 2 a.m. If a bad migration removed rows, a batch job corrupted records, or someone deleted data by mistake, the same pattern still works.

1. Confirm the trigger and scope. Write down what broke, which tables or objects are affected, and when the bad change started. Name one person to lead the restore so two people do not make different fixes at once.
2. Freeze writes to the affected area. Put the app in maintenance mode for that feature, disable the worker, or block writes at the database layer if needed. Then check the latest backup or snapshot time and note the possible data loss window.
3. Restore to a safe point outside production first. Recover the backup into a separate database or environment, then compare record counts, timestamps, and a few known IDs. Pick a handful of real records from support tickets or internal test accounts and make sure they look right.
4. Tell customers what changed. Keep it plain: what data was affected, what you restored, and what you still need to verify. If some recent changes may be missing, say that directly instead of waiting for users to discover it.
5. Close the incident with aftercare. Record the restore start time, finish time, backup age, and confirmed loss window. If the restore took too long, fix that next. A runbook that restores data in theory but takes three hours to execute is not good enough.

A small example helps. If a migration deleted 1,200 subscription notes at 14:10, stop writes to that table, restore the 14:00 snapshot to a safe place, compare counts, test a few customer records, then restore the missing rows with a reviewed script.

Mistakes that slow teams down

The most common runbook problem is simple: people write a mini article when they need a checklist. During an outage, nobody wants three paragraphs of background. They need clear steps, in order, with plain success and failure paths.

A deploy failure runbook should not explain your release philosophy. It should say who stops the rollout, how to confirm the bad version, how to roll back, and where to post status updates. If a step takes judgment, write down the trigger. "If error rate passes 5% for 10 minutes, roll back" is much better than "assess impact."

Another delay comes from unclear ownership in the first 30 minutes. Teams often assume someone will lead. Then support waits on engineering, engineering waits on ops, and nobody decides. Put one name or role at the top of every runbook for the first response window, even if that person later hands off.

Customer updates also tend to happen too late. Many teams stay quiet until users get angry, then rush out a vague message. A better rule is boring and effective: send a short update early, even if all you can say is that you see the issue, you are working on it, and the next update comes in 15 minutes.

Access problems waste more time than most teams expect. Many runbooks assume everyone can log in everywhere during a crisis. Real outages lock people out of cloud dashboards, payment tools, admin panels, VPNs, or even team chat. Add backup access methods, emergency contacts, and a short note on where recovery codes live.

Backups create false confidence too. A green "backup successful" message does not tell you whether a restore takes 8 minutes or 8 hours. Test restore time on a schedule, write down the result, and update the runbook. If your team has never restored real data under time pressure, the backup plan is still unfinished.

Quick checks before you call it done

A runbook is only ready when someone who did not write it can use it under stress. If a new hire gets stuck, the document still has holes.

Start at the top. Put the owner, the backup person, and the contact channel in the first few lines. During an incident, nobody wants to dig through folders or old chat threads to find who should act.

Then check the steps against your current setup. Old tool names, removed admin roles, and stale permissions cause more trouble than most teams expect. If the runbook says "open the old dashboard" or assumes access nobody has anymore, fix it now, not during an outage.

A short review is enough:

• Ask a new hire to follow it without help.
• Confirm the owner, backup, and contact channel appear at the top.
• Match every step to current tools, names, and access rules.
• Keep the first five minutes on one page.
• Run a drill at least once each quarter.

That one-page rule matters. The first few minutes decide whether your team stays calm or starts guessing. Keep the opening actions simple: confirm the issue, stop the damage, notify the right people, and record what changed.

A drill is the final test. Pick a simple scenario, set a timer, and watch where people hesitate. You will usually find small problems: a missing permission, an unclear step, or a contact method nobody checks anymore.

When the drill ends, update the runbook the same day. A runbook that matched your system three months ago may already be wrong.

A simple incident example

At 4:40 p.m. on Friday, your team ships a small checkout change. Ten minutes later, support sees a pattern: some users can add items to the cart, but the payment step fails for them. Others complete checkout with no issue, which makes the problem easy to miss and hard to explain.

The incident lead does not debate it for half an hour. They freeze new releases, open the deploy failure runbook, and assign clear owners. One person checks logs and error rates. One person starts the rollback. One person watches payments and confirms whether the failure sits in app code or at the provider edge.

A simple response looks like this:

1. Freeze releases so nobody adds more change while the team investigates.
2. Roll back the checkout deploy to the last known good version.
3. Send one short support update: checkout errors affect some users, the team is rolling back now, and the next update comes in 15 minutes.
4. Ask finance to track failed charges and incomplete orders for retry or follow-up after service returns.
5. Confirm recovery with a real test order, then keep monitoring for another 30 to 60 minutes.

That support message matters more than many teams think. Ten custom replies from ten people create noise fast. One clear update keeps customers calm and gives the team room to work.

Finance also needs a seat at the table. If payments fail during the incident, someone should save the list of affected customers, amounts, and timestamps. That turns a messy guess into a clean recovery task on Monday.

Then Monday arrives, and the team should update the runbook while the details still feel fresh. Add the exact alert that fired, the rollback command that worked, and the support note that saved time.

What to do next

If you have nothing written yet, do not try to document every edge case this week. Start with the four runbooks that protect money and access first:

• deploy failure
• payment outage
• user lockout
• data restore

Give each one hour. That time limit keeps the writing simple and honest. Write the owner, the first checks, the stop rule, the rollback or recovery step, and who needs an update.

After that, run a short drill. Fifteen minutes is enough. One person leads, one follows the runbook exactly, and one watches for missing steps, vague wording, or decisions that depend on knowledge stuck in one person's head.

Keep reviewing the runbooks. Update them after every real incident, but also after any change to your deploy process, payment stack, login flow, backup setup, or on-call ownership. A stale runbook can waste more time than no runbook at all.

If you want an outside review, Oleg Sotnikov at oleg.is can look over your runbooks, incident flow, and recovery gaps. That kind of review often finds weak spots quickly, like alerts nobody sees, restore steps nobody tested, or access rules that block the person trying to help.

Put the first draft on the calendar today. Four hours of writing and one short drill can save a day of panic later.