Capacity planning for background jobs on a lean team

Why nightly jobs spill into the workday

Nightly jobs usually miss their window because small delays stack up. A batch starts 40 minutes late because an upstream import runs long. A few jobs take twice as long because the database is busy. Then retries return after timeouts or rate limits. Each problem looks minor on its own. Together, they push the queue into the morning.

That is why capacity planning for background jobs cannot start with a rough guess about worker count. You have to protect a fixed overnight window. If your safe window is 10 p.m. to 6 a.m., that is eight hours. A late start cuts it to seven. Slow processing cuts it again. Retries add extra work on top of the original load.

Most teams notice the spillover before they understand the cause. Support hears that dashboards are stale, exports are missing, or emails arrived late. Internal staff signs in to a queue that is still draining, so morning tasks now compete with unfinished night work. Customers feel it as lag, old data, or random delays. The team feels it as a rushed start to the day.

A one-off spike is different from a pattern. If a partner sends a huge file once a month and that single event runs long, you have an exception to handle. If jobs finish at 8:30 a.m. three times a week, you have a capacity problem even if nothing fully fails.

The pattern usually comes from the same few issues: jobs arrive in a burst instead of spreading out, one slow job blocks many short ones, retries add more work than anyone counted, and workers sit idle in some periods but choke in others.

Morning pain usually starts the night before. Protect the overnight window first, then size the system so it can finish on a bad night, not just an average one.

What to measure before you guess

You cannot size workers from a daily total. A queue that gets 120,000 jobs per day may look safe on paper, but the real problem often sits inside a two-hour spike after midnight. Measure arrivals by hour, and by smaller slices if traffic is bursty. A daily average hides the moment when the queue starts to pile up.

Run time needs the same treatment. Measure the average, but do not stop there. You also need the slow-case run time, such as the 95th or 99th percentile, because a small group of slow jobs can keep workers busy long after the fast jobs finish. If most jobs take 8 seconds but some take 90, those slow jobs decide whether the queue clears before people log in.

Retries change the math more than most teams expect. A 5% retry rate sounds small until failures cluster in the same time window and return a few minutes later. Track both numbers: how many jobs retry, and how long the system waits before retrying them. Without that delay, you cannot tell whether retries spread the load or create a second spike right before business hours.

A short measurement pass is usually enough. Count new jobs per hour, not only per day. Record average run time and slow-case run time. Track retry rate and retry delay. Then note worker concurrency, database throughput, and any outside API caps.

That last part matters because workers are not the only limit. You can double worker count and still finish later if the database starts locking, disk I/O saturates, or an external API only allows a fixed request rate. Teams often blame the queue when the real bottleneck is somewhere else.

If you want one simple rule, measure the busiest hour, not the calmest one. Those are the numbers you can trust when you size workers and decide whether you need more machines, fewer retries, or a different job schedule.

Sort jobs by type and deadline

Teams get into trouble when they treat every background job as if it has the same urgency. It does not. A password reset email, a nightly sales report, and a cleanup task can all sit in the same queue, but they should not carry the same deadline.

Start by splitting jobs into plain groups people can understand. In most systems, that means imports, emails, reports, cleanup work, and webhooks. Keep the first pass simple. If a job fits two groups, put it where the delay hurts most.

Then mark each group by deadline. Ask a blunt question: what must finish before staff log in? That usually includes overnight imports that feed dashboards, reports finance reads first thing, and any sync that keeps customer data current. If those jobs slip into the morning, people notice fast.

Some work can wait. Cleanup jobs, archive tasks, large exports, and non-urgent reports often fit better in quieter hours. Moving them later does not fix a slow system, but it stops low-priority work from blocking jobs people actually need.

One more split matters. Group jobs by the resource they hit. If five job types all hammer the same database table or the same third-party API, they compete even if the queue looks balanced on paper. This is where capacity planning for background jobs often goes wrong. Teams count jobs, but they do not count contention.

A simple worksheet helps: job type, latest finish time, whether it can wait, the database table or API it touches most, and what breaks if it runs late.

A quick example makes the point. Say your app sends invoices, builds reports, processes webhooks, and deletes old logs overnight. Webhooks and invoice emails may need to finish before 9 a.m. Reports might need to land by 8 a.m. Log cleanup can run at noon if needed. If reports and cleanup both hit the same read replica, separate them before you even think about adding workers.

This step feels basic, but it saves a lot of bad math later.

Forecast queue growth hour by hour

A nightly queue rarely breaks all at once. It grows in one hour, shrinks in another, and then hangs around longer than anyone expected. The clean way to see it is simple: track each hour, then subtract completions from arrivals.

Use this formula for every hour:

end backlog = start backlog + arrivals - completions

If the result drops below zero, set it to zero. A queue cannot be less than empty.

Do this with the busiest real night you can find, not the average night. Average numbers make the system look calmer than it is. If one import run, partner sync, or billing cycle creates a much heavier night, that is the night you should plan around.

The forecast only needs four numbers for each hour: jobs that arrive, jobs workers finish, backlog left at the end of the hour, and whether business hours have started.

Two moments matter most. First, when backlog starts. That is the first hour when work arrives faster than workers can clear it. Second, when backlog clears. The gap between those points shows how long the queue really lives.

A small example helps. Say 2,000 jobs arrive at 11 p.m., 4,000 at 12 a.m., 3,000 at 1 a.m., and 1,000 at 2 a.m. If workers finish 2,500 jobs per hour, the backlog ends each hour at 0, then 1,500, then 2,000, then 500. The queue still exists after 2 a.m. It only clears in the next hour.

That is the part teams often miss. They look at total nightly volume and assume the work fits before morning. Hourly math shows whether the queue crosses into business hours. Once it does, customers, support, and on-call engineers all feel it.

You do not need fancy software for this. A plain spreadsheet is enough. For worker count planning, the shape of the night matters more than one daily total.

Add retry load to the math

Retries can quietly eat the capacity you thought you had. A queue with 100,000 planned jobs is not really a 100,000-job queue if failed work keeps coming back all night.

Start with the number of jobs that fail on the first attempt. Then count how many of those fail again, and how many come back for a third try. Teams often stop at first retries, and that hides a real part of the load.

A simple estimate looks like this:

• planned jobs: 100,000
• first-attempt failures: 4,000
• second-attempt failures: 1,600
• third-attempt failures: 400

That queue is not handling 100,000 attempts. It is handling 106,000. If your workers were already close to full overnight, that extra 6,000 attempts can push work into the morning.

Use real retry rates from logs, not hopes from a quiet week. Split the numbers by job type when you can. Email sends, report generation, imports, and API syncs usually fail for different reasons and return at different rates.

One bad dependency can wreck the whole forecast. A slow payment API, a locked database table, or a rate-limited vendor can make thousands of jobs fail in a short burst. When that happens, retries do not arrive as a trickle. They pile up together and compete with fresh work.

Delay policy changes the shape of that pile. Fixed delays are easy to reason about, but they often create waves. A five-minute retry delay means a failure spike at 1:00 returns as another spike at 1:05.

Exponential backoff spreads load better and gives a broken dependency time to recover. But it can also push work past a job's deadline, so check the trade-off. A report that must finish before 8 a.m. cannot wait through long retry gaps.

When you size workers, count expected attempts, not just original jobs. That small shift makes the overnight plan much more honest.

How to size workers step by step

Start with the deadline, not the current worker count. If the queue must be empty by 8 a.m. and jobs begin at 12 a.m., you have eight hours to finish. Do not treat all eight hours as usable time. Leave room for slow queries, brief outages, and the usual overnight mess.

A good rule is to plan around 70% to 80% worker use. With an eight-hour window and a 75% safety level, one worker gives you six hours of reliable work, or 21,600 seconds.

Then estimate how much work arrives overnight in job seconds. Multiply the number of jobs by the average run time for each job type. If one group runs much longer than the rest, calculate that group on its own. A simple average can hide a problem.

The sizing flow is straightforward:

1. Set the finish time and count the available hours.
2. Reduce that window to safe worker time.
3. Calculate total overnight job seconds.
4. Add extra load from retries and spikes.
5. Divide by safe worker seconds and round up.

Say 18,000 jobs arrive overnight and the average run time is 2.4 seconds. That gives you 43,200 job seconds. Now add retry load. If 6% of jobs retry once and 1% retry twice, your real load is about 8% higher. That brings the total to 46,656 seconds.

Add headroom after that, not before. A 10% to 20% buffer is usually enough for slow jobs, uneven arrival patterns, and small forecast errors. With a 15% buffer, the load becomes 53,654 seconds. Divide that by 21,600 safe seconds per worker, and you get 2.48. Round up to 3 workers.

Do one more check before you trust the number. Run that worker count against database limits, connection pools, and outside APIs. More workers do not help when the database is already pegged or an API starts rejecting requests. In that case, fix the slow part first, then recalculate.

A simple example with real numbers

A team runs one vendor import at 11:00 p.m. and one report batch at 2:00 a.m. On a normal night, 12 workers are enough.

• The import queue gets 300,000 jobs at 11:00 p.m.
• Each worker usually finishes 4 import jobs per second.
• At 2:00 a.m., the report queue gets 60,000 jobs.
• Each report worker can finish 2 report jobs per second.

With one shared pool of 12 workers, the import finishes around 12:44 a.m. The report batch then finishes about 42 minutes after it starts, well before morning.

Now change one thing. The vendor API slows down, so each import worker drops from 4 jobs per second to 1. That cuts import capacity from 172,800 jobs per hour to 43,200.

Retries make the backlog worse. If 25% of import jobs fail once and 10% need a second retry, the queue no longer holds 300,000 attempts. It holds 405,000.

By 2:00 a.m., the workers clear only 129,600 import attempts. That leaves 275,400 import attempts still waiting. When the 60,000 report jobs arrive, the total backlog grows to 335,400 jobs.

With one large first-in, first-out pool, the reports sit behind the import backlog. The import queue clears about 6.4 hours later, around 8:22 a.m. Only then do reports begin, so the last report lands around 9:04 a.m. That is how nightly batch jobs spill into the workday.

Split queues change the result. Give 9 workers to imports and 3 workers to reports. The import now runs slower and ends around 11:30 a.m., which is ugly. But the report queue gets protected capacity. Those 3 workers clear 60,000 report jobs in about 2 hours and 47 minutes, so reports still finish around 4:47 a.m.

If the import must also finish by 6:00 a.m., recalculate the worker count from the slow case, not the normal one. You have seven hours from 11:00 p.m. to 6:00 a.m. to process 405,000 import attempts. That means you need about 57,900 jobs per hour. At 1 job per second per worker, you need 17 import workers, not 12.

This is why capacity planning for background jobs should start with slowdown and retry load estimation. One big worker pool looks efficient on paper. Two queues often protect the jobs that have a real deadline.

Mistakes that create false confidence

False confidence usually starts with clean averages. A dashboard says the queue clears in six hours, but that average hides the hour when job arrivals spike and workers fall behind. If you plan from daily totals instead of the busiest hour, nightly batch jobs will keep leaking into the morning.

Teams also overcount workers. Ten workers do not mean ten jobs run at once if the database, an external API, or the job runner allows only two or three of that job type at a time. On paper, capacity looks fine. In production, workers sit idle or wait on the same shared bottleneck.

Rare slow jobs cause more damage than most teams expect. If 95% of jobs finish in 10 seconds, that feels safe. But the 5% that take four or five minutes can pin workers long enough to delay everything behind them, especially when they land near the end of the night run.

Retries create another false sense of safety. A higher retry count can make failure graphs look better because more jobs eventually succeed. It can also double or triple the load. If 1,000 jobs hit a flaky dependency and even 8% retry several times, you did not process 1,000 jobs anymore. You processed a larger queue and burned worker time on the same broken step.

Adding workers to a hot database often makes the system worse. More workers send more reads, more writes, and more lock pressure into the same weak spot. Queue time may drop for a few minutes, then climb again as queries slow down and retries pile up. Capacity planning for background jobs has to include database headroom, not just worker count.

A quick sanity check helps. Compare capacity against the busiest hour, not the average day. Check how many jobs of each type can run at once. Measure how many jobs take much longer than the median. Count retry attempts as real load. Watch database wait time when workers ramp up.

One small example makes the risk obvious. Say a team sees 12,000 jobs per night and 20 workers. That sounds healthy. But if the peak hour receives 4,000 jobs, one slow job class ties up six workers, retries add 15% extra attempts, and the database starts queuing writes after midnight, the schedule was never safe. The worker count looked fine because the model left out the parts that actually set the finish time.

If the database queue climbs when workers wake up, stop there. The next worker will not save the morning run.

A quick check before you add workers

Adding workers feels like the obvious fix, but it often hides a traffic problem instead of solving it. A short audit can show whether jobs are stuck on one shared resource, one retry pattern, or one bad schedule.

Check where time actually goes

Start with the slowest jobs. If many of them read from or write to the same table, extra workers may only create a bigger line. The same thing happens when several job types call the same external API. More workers do not help much if they all end up waiting on the same bottleneck.

It helps to split job time into a few simple buckets: time doing actual work, time waiting on database locks, time waiting on API limits, time sleeping before retry, and time stuck behind older jobs.

That breakdown changes the conversation fast. If workers spend 40% of the night waiting on rate limits, buying more capacity is a weak fix.

Look at retries next. Teams often give every failed job the same retry delay, so failures come back in one sharp wave. At 2:00 a.m. the queue looks normal, then at 2:15 a.m. hundreds of retries wake up together and crowd out fresh work. Spread those retries out before you add a single worker.

Separate urgent work from bulk work

A queue can look busy for the wrong reason. If urgent jobs sit behind bulk imports, report generation, or cleanup tasks, users feel the delay even when total throughput is fine. Reserve some workers for urgent jobs, or move bulk work into its own queue with a looser deadline.

A small schedule change can fix more than most teams expect. Try moving one heavy batch 20 or 30 minutes later, after another job family finishes. Or start retries on a different minute than your largest nightly import. One test night can tell you whether spillover comes from low capacity or bad timing.

A simple example: if invoice emails miss their window because a product sync starts at the same minute and hammers one API, adding four workers may change nothing. Moving the sync back half an hour might clear the whole morning backlog.

If workers mostly wait, bunch up, or fight over the same resource, fix that first. Add workers after you know they will spend their time working, not standing in line.

Next steps for a lean team

If jobs still bleed into business hours, stop guessing and write down the numbers that decide the outcome. For each job group, record arrival rate, run time, retry rate, and the latest time it must finish. Do not mix everything into one average. Imports, reports, emails, and media work put very different pressure on the queue.

A plain spreadsheet is enough at first. Split arrivals by hour, then compare that workload with the worker time you actually have overnight. If one group has a hard morning deadline, treat it separately. That keeps a low-priority batch from hiding a real risk.

A simple weekly routine works well:

• update arrival counts for each job group
• check median and slow-case run time, not just the average
• recalculate retry load after any code or infrastructure change
• compare total overnight work with the hours your workers can provide
• mark any group that cannot finish before its deadline

Set an alert on the morning backlog, not just on failures. If the queue is still draining at 6 a.m. or 7 a.m., that is already a production problem even when jobs eventually succeed. A second alert on retry spikes helps too, because retry storms often show up before the queue misses its deadline.

For a lean team, the goal is simple: know when backlog starts, know why it grows, and know which jobs must never wait behind bulk work. Once you have those numbers, worker count planning gets much easier.

If your team is small and you are sorting through queue behavior, database limits, and retry storms at the same time, an outside review can save a lot of trial and error. Oleg Sotnikov at oleg.is works with startups and smaller companies on architecture, infrastructure, and AI-first engineering operations, and this kind of bottleneck analysis is often where a short advisory pass pays for itself.