Quarterly disaster recovery drill for small teams

Why small teams skip drills until something breaks

Small teams rarely ignore recovery on purpose. They're busy shipping fixes, answering customers, and trying to keep costs under control. When the product seems stable and backup jobs report "success" every night, a quarterly drill feels optional.

That confidence is often false. A backup file only proves that something got copied somewhere. It does not prove the team can find the latest snapshot, restore it to a clean system, reconnect the app, and confirm that customers can sign in again.

Access is another quiet failure point. The person who set up cloud storage may have changed roles. A token may have expired. Two-factor codes may sit on one phone, and that person may be asleep, on a flight, or simply offline when an outage starts.

Small teams also trust old notes too much. A rollback plan that worked six months ago may point to the wrong branch, the wrong server, or a script nobody has touched since the last infrastructure change. Under stress, even simple steps get harder when people have to stop and guess.

The cost is usually not the drill. It's the delay during a real incident while people ask basic questions: where is the backup, who can log in, which version was safe, and what do we restore first? Ten minutes of guessing can turn a short outage into a long one.

A drill has one job. It should prove that your team can restore service with the tools, credentials, and notes you have today, not the ones you assume still exist. If the session exposes missing access or a broken restore path, that is useful. It is far cheaper than learning the same lesson during production trouble.

What a two-hour drill should prove

This kind of drill should answer one plain question: if something breaks today, can your team recover without guessing or waiting on the one person who knows the steps?

Start with the backup. You do not need to test every backup job in one session. You need to prove that one recent backup exists, that the team can find it quickly, and that it opens correctly in a safe environment. If the file is there but nobody can restore it, the backup is only half real.

The drill should also show that the team can reach every system it needs during an incident. That usually includes the cloud account, source control, deployment tool, backup storage, monitoring, DNS, and the password vault or secrets store. Not everyone needs admin access, but the team does need a clear path into each system if the usual owner is offline.

Rollback matters just as much as restore. Pick one recent, low-risk change and reverse it the same way you would in production. That might mean deploying the previous version, turning off a feature flag, or restoring an older config. If the rollback plan lives only in someone's memory, the drill has already found a real problem.

Keep the scope narrow on purpose. One backup restore test, one access check, and one rollback test are enough for a useful session. Small teams stick with habits they can repeat, and a two-hour drill that happens every quarter is much better than a giant exercise that never happens again.

When the session ends, you want proof, not reassurance. You should know what worked, what took too long, and what needs a written fix before next quarter.

Who should join

Keep the group small. Three to five people is usually enough if they cover the app, the database, and deployment access.

Choose one drill lead before the meeting starts. That person keeps the pace, calls the next step, and stops side trips. Pick a second person to take notes with timestamps, failed checks, and open questions. When one person tries to run the session and document it, details disappear fast.

Invite the people who would actually act during a real outage, not everyone who wants status updates. On most small teams, that means the person who owns the app or backend, the person who manages the database or backups, the person who can deploy and roll back changes, and maybe one extra teammate if too much access sits with one person.

Before the drill starts, put the practical details in one shared document or ticket. Include backup locations, the systems you plan to test, vault access details, and the current rollback notes for recent releases. If part of the plan still lives in one person's head, the drill will expose that anyway.

Check access early. Make sure the team can open the password vault, view backup jobs, reach the server or cloud console, and read the runbook. Losing 15 minutes to login trouble can waste half the session.

Reserve a quiet window and pause non-urgent releases. You do not need a full freeze, but you do need a block of time where nobody pushes a surprise change and muddles the rollback test.

Run the drill step by step

Pick one failure and make it specific. "The production database is gone" is much better than "backup problem." Start a visible two-hour timer as soon as everyone joins. The clock matters because the point is to test what the team can do under normal pressure, not after a full afternoon of debate.

Give each person the written recovery steps and ask them to follow the document exactly as if the system were down. Nobody should rely on memory unless the document tells them to. If someone knows a shortcut, they can use it only after they note that the guide missed a step.

A simple flow keeps the session moving:

1. Name the failure, the affected system, and the recovery target.
2. Walk through the restore or failover steps in the order your notes describe.
3. Pause every time someone needs a missing command, token, secret, hostname, or approval.
4. Write down the gap before moving on.

This is not a skills test. If an engineer has to ask, "Which account owns this backup bucket?" that is a useful result. If a founder is the only person who can approve a DNS change, write that down too. Hidden dependencies are often the real issue.

At about 90 minutes, stop the simulation even if you are mid-recovery. Use the last 30 minutes to fix the worst gaps while people still remember them. Update the runbook, add the missing command, store the secret in the right place, or change permissions so a second person can act.

End with three notes: what worked, what slowed you down, and what must change before the next drill. If the team leaves with cleaner docs and fewer one-person blockers, the session did its job.

Check backups without turning it into a project

A backup that only exists on paper is not much help. One good restore test tells you more than a pile of status emails.

Start with the newest successful backup, not an older file you already trust. Check the timestamp, where it came from, and whether the size looks normal for your app. If yesterday's backup is half the usual size, treat that as a warning now, not a mystery for later.

Restore it into a safe test space. That can be a temporary database, an isolated container, or a throwaway virtual machine. Keep it away from production, and block anything that could send emails, process payments, or call outside services by mistake.

If you use PostgreSQL or a similar database, restore to a new database name instead of replacing an existing one. Small teams make avoidable mistakes when they rush this part.

Then open the restored data and check a few records that matter to daily work. Use something real: a recent customer account, the latest order, a support ticket, or a settings record. You are not proving every row is perfect. You are proving the backup is usable and recent enough to trust.

While you do this, record four details: the backup file name and timestamp, restore start and finish time, restored size or record count, and any manual steps or missing permissions. That short note becomes your baseline next quarter.

It also shows drift early. If restore time jumps from 10 minutes to 35, or one engineer had to remember three undocumented commands, the drill found something worth fixing.

Verify credentials and access

A recovery drill can fall apart in ten minutes if the right people cannot sign in. Do not assume access still works because it worked last quarter. People change roles, MFA devices get replaced, and old emergency accounts often sit untouched until the worst possible moment.

Start with the systems the team would open first during an outage: the password vault, cloud console, code repository, and alerting tool. Have one person log in while another confirms that the account has the right level of access. If login works but the account cannot view logs, restart a service, or reach backups, count that as a failure.

Break-glass accounts need a real test, not a checkbox. Use the stored steps, complete the login, and confirm the account still reaches the same systems your normal admins can reach. If the account depends on one phone number, one hardware token, or approval from one specific person, fix that now. An emergency account that depends on a missing admin is not an emergency account.

Approval paths matter too. Many small teams rely on one founder or senior engineer to approve production changes. That works until that person is asleep or offline. Pick one common action, such as approving a rollback or unlocking deploy access, and confirm that a second person can do it without guessing.

Keep a short record during the drill:

• which accounts logged in successfully
• which accounts had only partial access
• who can approve urgent changes
• which old accounts should be removed

Then clean up the obvious junk. Delete stale accounts, shared logins nobody uses, and access that former staff or contractors still hold. This part is dull, but it removes real risk.

Test rollback on a safe change

Pick a small release from the last quarter, not the biggest one. A tiny UI fix, a config update, or a minor API change works better because the team can focus on the rollback itself instead of debugging a messy deploy.

Use the same records you would use on a normal day. Open the release notes, find the commit, locate the build artifact, and identify the last known good version. If that takes more than a few minutes, the problem is already clear: recovery starts too late because nobody can find the right version quickly.

Then walk through the rollback in the same order you would use in production. Revert the code or redeploy the previous build. Restore the related config values. Check whether the release changed the database schema. Decide whether you need a schema rollback, a forward fix, or no database action at all.

Schema changes need extra care. If the release added a column or index, you may be able to leave it in place and just roll back the app. If the release changed data or removed a field, write down the exact command, migration, or manual step you would use.

Many teams eventually learn that their rollback plan is really just "ask Alex." That is useful to know, but it is also a risk. A real plan still works when Alex is on a plane, asleep, or no longer at the company.

Keep the timer running. Speed matters because stress makes slow steps worse. You want proof that the team can find the last good version, make the change, and verify the service within minutes, not after a long search through old chat threads.

Write down every step that still depends on one person knowing a password, a server name, or the order of commands. That list tells you what to fix next: clearer notes, shared access, or a better release record.

A simple example from a small product team

A three-person SaaS team ran this drill after a bad deploy cut the app off from its main database. The site still opened, but customers could not save anything. They paused normal work, set a 90-minute timer, and treated the problem as if it were live.

One person started a backup restore test in a separate database. That part went well. The data came back clean, and the restore finished within the time they expected. The trouble showed up right after that: the app could not read files from object storage because the access key lived on one engineer's laptop instead of in the shared vault.

At the same time, another teammate rolled the app back to the last stable release. The third person followed the recovery notes and found the weak spot quickly. The notes listed the storage bucket and the database name, but they did not say who owned the storage key, where the current key was stored, or how to rotate it if that laptop was offline.

They wrapped the session in about 85 minutes and left with a short fix list:

• move the storage key into the shared vault
• add plain recovery notes for database, storage, and deploy access
• save the rollback command and the checks to run after rollback

That is what a good drill should do. It should find the hidden single point of failure before a real outage does.

The same team ran the drill again the next quarter. The backup still restored, the rollback took only a few minutes, and nobody had to dig through old chat threads for credentials. They finished faster because the process was clearer and the access path was shared.

Mistakes that waste the drill

A two-hour session only works if the scope stays tight. The easiest way to ruin it is to turn a short check into a full incident exercise with extra scenarios, long status updates, and random side work. Save that for another day. This session has a smaller goal: can the team restore data, reach the right accounts, and roll back one risky change?

Another common mistake is trying to test every system in one sitting. Small teams do this out of anxiety, then rush through everything and prove nothing. One database restore, one access path, and one rollback test are enough. Next quarter, rotate to different systems.

Old docs are another trap. A runbook that looked fine six months ago can fail on the first real step. People change roles, tools move, secrets rotate, and commands drift. If nobody has opened the instructions in months, do not trust them. Have one person follow the document exactly while another watches for broken steps, missing context, or access requests that no longer make sense.

A few habits keep the session useful:

• put a time limit on each part of the drill
• test a narrow slice and save proof that it worked
• fix the document during the session when you spot a problem
• assign one owner to each follow-up task
• set due dates before the meeting ends

The last two matter more than many teams admit. Plenty of drills end with a quick "we should update that" and no one owns the work. Two weeks later, the same gap is still there. If the backup test failed, name the person who will fix it. If only one engineer could reach the rollback tool, assign the access change on the spot.

A drill without owners and dates is just a discussion. Discussions do not restore production.

Before you close the session

Before anyone leaves, spend ten minutes on a hard pass-fail review. If a step felt vague, rushed, or depended on one person remembering a trick, mark it as a gap.

Use a short checklist and answer each item with yes or no:

• Restore data and open it in the app, a database client, or the file viewer your team actually uses. Seeing a backup file in storage is not enough.
• Test every login the recovery path depends on. That includes cloud access, backup storage, deployment tools, secrets managers, DNS, and any emergency account you keep for bad days.
• Repeat one rollback path on a safe change. Pick something small, such as reverting the last deployment or switching to the previous container image, and make sure another teammate can do it too.
• Assign every fix before the meeting ends. Put one name and one date on each task. "We should clean this up later" usually means it will sit there until the next outage.

Pay attention to timing too. If restoring a small dataset took 35 minutes when the team expected 10, write that down. Slow recovery is still a problem, even if the restore eventually works.

Watch for hidden single points of failure. Maybe one password worked, but only because it lived in one engineer's browser. Maybe the rollback succeeded, but only after someone edited a script by hand. Those details matter more than a clean demo.

A useful drill ends with a short record: what passed, what failed, who owns each fix, and when the team will verify the fix. That gives the next session a clear starting point instead of a fuzzy memory.

What to do after the drill

Right after the session, capture what actually happened, not what the team meant to do. Memory gets fuzzy fast. If the restore took 18 minutes, write 18 minutes. If someone needed a code from a former admin, write that too.

Update the runbook the same day if you can. Small edits matter most: the exact backup location, the current owner of each account, the rollback command that worked, and the step that confused people. A runbook that matches reality beats a polished document nobody trusts.

Then make three decisions before everyone goes back to regular work:

• pick the top two gaps to fix first
• assign one owner for each fix
• put the next drill on the calendar now

That order matters. Teams often leave with ten notes and fix none of them. If one broken access path and one unclear rollback step slowed the drill, start there. Minor cleanup can wait.

Scheduling the next session now is part of the work, not admin overhead. These drills only help if they stay on the calendar before launches, vacations, and customer issues take over.

Keep the follow-up short. A 15-minute review is enough to confirm who changed what and when you will recheck it. If a fix touches backups or credentials, ask the owner to prove it with a quick test, not a verbal update.

If your team keeps finding the same weak spots, an outside review can help. Oleg Sotnikov at oleg.is works as a fractional CTO and startup advisor, and a fresh review of your recovery notes, access model, and rollback path can catch thin spots before they turn into downtime.