What makes an engineering efficiency audit baseline fair?

Most engineering efficiency audits fail before anyone opens a dashboard. They compare an ordinary quarter with a launch sprint, a holiday period with a fully staffed month, or a team that lost two engineers with the same team on paper. The resulting percentage looks precise, the slides look serious, and the conclusion is wrong.

A fair baseline is not an average of whatever happened recently. It is a documented picture of normal operating capacity: how much useful work a specific team delivers, how long that work waits, what interrupts it, and what level of reliability it maintains when the calendar is not on fire. For a startup, that is the only baseline worth using to decide whether process changes, AI tooling, contractors, or another hire will pay off.

The goal is not to create a score for developers. It is to find where work spends time and money. That distinction changes the data you collect, the weeks you exclude, and the claims you are allowed to make afterward.

A baseline must describe normal work, not recent history

Recent history includes normal work, exceptional work, and organizational noise. An audit needs all three, but it must not blend them into one number.

Take a team that closed 80 tickets in the previous ten weeks. That number tells you almost nothing. Perhaps the team shipped a major customer migration in two weeks, spent one week recovering from a database incident, lost an engineer for a month, and took a company holiday. The remaining weeks might show a steady delivery rate of six completed customer-facing items per week. Or they might show a pile of small support requests that displaced roadmap work. The same total can describe radically different operating conditions.

Use three categories from the beginning:

• Baseline-eligible work is work completed during weeks that represent ordinary staffing, ordinary demand, and ordinary release practice.
• Exceptional work includes launches, severity-one incidents, migrations, deadline pushes, acquisition work, audits, and other concentrated events that change how the team operates.
• Capacity modifiers include public holidays, planned leave, vacancies, onboarding, prolonged sickness, part-time schedules, and temporary reassignment to sales support or recruiting.

Do not throw away exceptional work. It answers useful questions. A launch week can show whether deployment controls survive pressure. An incident can expose missing ownership and weak observability. A hiring gap can show how much delivery depends on one person. Those facts belong in the audit narrative and in separate measures. They do not belong in the denominator for normal weekly capacity.

This is where founders often make a costly mistake. They ask, "What did the team do last quarter?" because it sounds objective. The more useful question is, "What can this team repeatedly deliver when it is staffed as planned and not handling an abnormal event?" The first question produces a retrospective. The second gives you something you can use for a budget and operating plan.

Eight to twelve weeks is long enough if the weeks are comparable

For most startups, an eight to twelve week window is the practical range for a baseline. Shorter windows overreact to one release, one blocked integration, or one absent engineer. Longer windows often mix different product priorities, staffing models, and delivery rules until the comparison becomes historical trivia.

Use eight weeks when the team deploys regularly, work items are usually small enough to finish within a few weeks, and staffing has remained stable. Use twelve weeks when releases happen less frequently, the team works in larger batches, or you need enough normal weeks after exclusions. A team that deploys once per month does not have enough signal in a four-week sample, however attractive the spreadsheet may look.

The window begins with completed weeks, not a convenient calendar boundary. If you start an audit on a Thursday, do not include a partial week as if it were representative. Pull the prior eight to twelve complete weeks, then classify each one.

A good baseline should leave you with at least six eligible weeks. If you start with twelve weeks and exclude six because of a launch, an outage, two holidays, and a staffing transition, you have learned something important: the business has not operated normally enough to support a confident productivity claim. Do not solve that by quietly including the abnormal weeks. State that the evidence is insufficient, extend the observation period, and keep tracking.

The 2024 DORA report separates software delivery performance into throughput and instability measures, rather than treating speed as the whole story. Its measures include change lead time, deployment frequency, failed deployment recovery time, change fail rate, and deployment rework rate. That framing is useful in an audit because a faster team that creates more urgent repair work has not become more efficient.

DORA measures work at the system and team level. Keep it there. A baseline based on individual commits or pull requests creates a different game: people split work into tiny changes, avoid review and mentoring, and choose visible tasks over unpleasant maintenance. You will get prettier activity charts and worse engineering behavior.

Classify weeks before calculating a single average

Make eligibility a visible decision, not a judgment you reconstruct after an inconvenient result appears. I use a weekly register because it forces every exclusion to have a reason that another person can challenge.

A simple register can live in a spreadsheet, a database table, or a lightweight file beside the audit notes. The format matters less than consistent definitions.

week: 2026-W18
eligible_for_baseline: false
exceptions:
  - product_launch
  - extended_release_freeze
capacity:
  planned_engineer_days: 25
  available_engineer_days: 24
  open_role_days: 0
  onboarding_engineer_days: 0
work_mix:
  planned_product: 0.58
  defects: 0.12
  support_and_escalations: 0.10
  platform_and_maintenance: 0.20
notes: "Launch support continued through Thursday. Do not use for normal throughput."

This artifact prevents a quiet failure that turns up constantly in audits: somebody calls a week "normal" because people attended work, even though they were running a launch bridge, handling customer escalation calls, or unable to deploy because of a release freeze. Attendance is not available delivery capacity.

Set the exclusion rules before inspecting the outcome measures. For example:

1. Exclude a week from normal throughput if a defined launch required a release freeze, extended support coverage, or work outside the team’s normal planning cycle.
2. Exclude a week if a severity-one or severity-two incident consumed a material block of the team’s planned capacity. Record the incident in a separate reliability view.
3. Mark a week capacity-adjusted, rather than excluded, if the team had fewer planned workdays because of scheduled holidays or leave but delivery practice otherwise remained normal.
4. Mark a staffing transition if an open role, departure, or onboarding period changed the team’s effective capacity. Decide in advance whether the transition makes the week ineligible or simply requires normalization.
5. Do not exclude a week merely because it contains a difficult feature, a defect, or a poor result. If hard work is routine, the baseline must include it.

The last rule matters. Teams can game almost any measurement system if bad outcomes make data disappear. A fair baseline excludes changes in operating conditions, not evidence that the team has a delivery problem.

Keep the register small enough to maintain. If each week needs forty labels, nobody will update it and the labels will become fiction. Five to seven clear fields are usually sufficient.

Launches deserve their own comparison, not a hidden adjustment

A major launch can make a team look unusually productive or unusually slow. Both readings can be false.

During a launch, engineers often defer cleanup, accept more interruptions, batch changes that would normally ship separately, and spend time in coordination meetings that do not show up in repository activity. The team may close a large amount of work because scope was prepared for weeks beforehand. Or it may close very little because every person is watching production, supporting customers, and repairing small failures. Neither outcome describes ordinary feature flow.

Create a separate launch view with four questions:

• How much planned scope reached production by the committed date?
• How much unplanned repair or support work followed the release?
• Did release controls change, such as a code freeze, approval gate, or manual deployment?
• What work did the team defer to make room for the launch?

That last question prevents a familiar executive illusion. A successful launch can appear to prove that the team increased capacity, when it actually borrowed capacity from reliability work, tech debt, customer requests, and the next sprint. The team may have executed well. It still may not have created additional sustainable output.

Treat launch weeks as a stress test. Compare them with other launches if the product has a repeatable release cadence. Do not compare them with normal weeks, and do not use their ticket total as the target for a new AI-assisted workflow.

There is one exception: if every week is effectively a launch week because the company runs continuous delivery and product work flows without a separate release ceremony, then that is normal work. The label should follow the operating model, not the emotional intensity around a release.

Holidays and absences change capacity before they change output

A four-day week does not become a normal five-day week because the Jira board has no holiday field. The same applies when half the team takes planned leave, a senior engineer spends three days interviewing candidates, or an engineer joins and needs close support.

Calculate capacity in engineer-days, then state what it means. If five full-time engineers were planned to work five days, planned capacity is 25 engineer-days. If one person took two days of leave and another spent one day at a customer workshop, available capacity is 22 engineer-days. That does not mean the team had 88% of its normal output. Coordination costs and work ownership may magnify the effect. It does mean you should not compare the raw weekly count with a 25-day week as though staffing were unchanged.

Use capacity adjustment for predictable, modest changes. Exclude a week when the change breaks normal operating behavior. A single public holiday with a team that still plans and deploys normally can be adjusted. A year-end week when half the company is away, vendors are unavailable, and production changes are frozen should be excluded from flow comparisons.

Do not normalize every metric by engineer-days. Deployment frequency can depend on release policy and product risk, not only available people. Incident recovery can depend on who was on call and whether a specialist was away. Normalization is most useful for measures tied to planned delivery, such as completed work items of a defined size, accepted scope, or time spent in the delivery system.

The sharp distinction is between capacity and output. Capacity is the effort the organization had available to apply. Output is what reached a customer or production environment. A team can have full capacity and low output because work waits in review, testing, approval, or product decisions. A team can have reduced capacity and high output because it finished work prepared earlier. When an audit confuses these two, it starts blaming engineers for queues elsewhere in the company.

Hiring gaps must be measured as lost capability, not empty seats

An open headcount is not a harmless blank square on an org chart. It changes what the existing team can take on, how much review it can perform, and how much operational knowledge remains available during an incident.

Record the dates when the role became vacant, when the replacement started, and when that person began owning work independently. Do not pretend a new hire contributes at the same level on day one. The team pays an onboarding cost before it receives an onboarding return, especially when the codebase has weak documentation or one person holds most of the deployment knowledge.

For an audit baseline, classify hiring gaps in one of two ways:

• If the team worked with the same reduced staffing for the entire baseline period, that reduced staffing is the current normal. Use it, but label the baseline with the actual staffed capacity.
• If a departure or hiring delay happened during the window, split the observation period. Do not average pre-departure and post-departure output into one supposed normal state.

A founder may object that the team should have "covered" the missing role. Sometimes it can, for a short period. But coverage has a cost. Senior people absorb reviews and production duties. Product work waits. Specialists avoid vacation because nobody else knows the system. The audit should show this cost in the work mix and waiting times rather than assume the remaining people suddenly became less capable.

This is also where raw velocity becomes especially misleading. If the team closes roughly the same number of tickets after losing an engineer, investigate before calling it a victory. Perhaps the backlog was re-scoped into smaller tickets. Perhaps maintenance work was deferred. Perhaps the team is working longer hours. Sustainable capacity is what matters, not a single unchanged chart.

Measure flow, reliability, and work mix together

A useful audit uses a small set of measures that explain one another. One metric alone always invites the wrong story.

Start with flow. For a selected work type, measure elapsed time from active development to production, plus the time spent waiting in named states such as review, QA, product approval, security review, and release queue. Do not collapse those waits into a single "cycle time" if you can avoid it. The whole point of the audit is to find the constraint.

Then add delivery reliability. DORA defines change lead time as the time from code commit to successful production deployment and distinguishes failures caused by changes from external outages when measuring recovery. That distinction is more than terminology. A cloud provider outage may hurt customers, but it does not tell you whether the team’s release process produces defective changes.

Finally, measure work mix. Use broad categories that represent economic tradeoffs:

• planned product work
• defects and production repair
• customer escalations and support
• platform, security, and maintenance work
• internal requests, meetings, and unclassified work

Do not make engineers classify every fifteen-minute task. Ask for enough fidelity to identify whether the company funds its roadmap or keeps spending its week on repair and interruption. If unclassified work exceeds a small, agreed threshold for several weeks, inspect it. Unclassified work often contains hidden operational burden, sales engineering, manual data fixes, or product decisions that arrived too late.

The SPACE framework makes a related point: developer productivity has several dimensions, including satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. It warns against treating a single activity measure as productivity.

That does not mean an audit needs a giant employee survey. It means you should include a short, structured conversation with the people doing the work. Ask where work waits, which interruptions recur, which systems require manual handling, and what they stopped doing to keep delivery moving. If their answers contradict the dashboards, inspect the data pipeline before you dismiss the people.

Medians expose the wait that averages hide

Use medians and percentile bands for elapsed time. Averages are easy to calculate and easy to misuse.

Imagine eight eligible work items reached production in 2, 3, 3, 4, 4, 5, 6, and 45 days. The average is 9 days. A founder who reads that as "the team delivers in nine days" will be misled. The median is 4 days, which describes the typical item. The 45-day item deserves investigation because it shows a separate problem: perhaps a dependency, approval queue, environment issue, or unclear product decision.

Report three numbers for time-based measures:

1. The median, which describes a typical work item.
2. The middle 50% range, which shows whether results are stable or spread out.
3. The slowest few items with their waiting reasons, which exposes constraints that customers feel.

For count measures such as deployments or accepted work, show weekly values rather than only a total. A total of 40 deployments over ten weeks can mean four steady deployments each week. It can also mean zero for nine weeks and 40 during a launch. Those are not the same operating system.

Avoid a target that says every item must move quickly. Some work should take longer because it changes payments, identity, data retention, or system architecture. The audit should separate small, standard changes from large or high-risk changes. Otherwise, an organization responds by slicing a risky change into smaller tickets without reducing the actual risk or wait.

A worked baseline shows why exclusions change the answer

Consider a six-person product team with ten completed weeks of data. The leadership team wants to know whether AI-assisted development can reduce the need for two planned hires. Raw results show 96 completed product items and a median active-development-to-production time of 11 days.

That sounds like a usable starting point. It is not.

The weekly register shows the following:

The raw average blends all ten weeks. The launch inflates completed items because work prepared earlier crossed the finish line together. The incident increases elapsed time because feature work paused. The departure changes capacity. If you compare a post-AI period with this raw figure, any outcome can look like improvement or decline depending on what happened to fall inside the window.

A fair audit produces two baselines instead:

• Full-team normal baseline: weeks 1, 2, 3, 6, and 7, with week 3 capacity-adjusted. This describes delivery when the team had six people and ordinary operating conditions.
• Reduced-team normal baseline: week 10, followed by several more weeks of measurement. This describes the current staffing state, but it is not yet strong enough to support a large claim.

The audit then examines why the full-team baseline had an 11-day median. Suppose the data shows two days in active development, four days waiting for product acceptance, three days in review because only two senior engineers approve changes, and two days waiting for a manual QA environment. AI may reduce some active-development time. It will not fix the product decision queue or a shared QA environment by itself.

That finding protects the company from a bad purchase decision. Buying more coding capacity when the work mostly waits outside coding raises cost while leaving the main delay untouched.

Compare after changes with the same rules, or do not compare at all

Once you have a baseline, freeze the definitions. Use the same work types, start and end events, week classifications, capacity fields, and reliability measures after you introduce a change.

This applies to AI tools, a new staff engineer, a contractor, a rewritten CI pipeline, or a product management process. If you change the metric definition at the same time as the operating model, you lose the ability to say what changed.

A reasonable before-and-after review should answer five questions:

• Did the mix of work change materially?
• Did staffed capacity change?
• Did release policy, approval policy, or deployment frequency change?
• Did reliability worsen, improve, or remain stable?
• Where did elapsed time actually fall?

The last question keeps the discussion honest. If median time improved from 11 days to 7 days, identify whether active coding fell, review queues shrank, test failures declined, or product decisions arrived earlier. If nobody can explain the mechanism, wait for more data before declaring a gain.

Do not demand mathematical certainty from a small startup dataset. You rarely have enough observations for that. Demand an auditable explanation instead. A fair comparison says, "With the same work classification and similar staffing, normal feature delivery moved faster, and the reduction came from fewer review iterations and automated test setup. Rework did not increase." That is much stronger than, "AI made the team 40% more productive."

A Team & AI Audit should begin with this baseline work, because any savings estimate built on mixed weeks is theater. The point is to find the constraint before you spend money moving it somewhere else.

The first action is to label the last twelve weeks

Do not begin with a developer survey or a tool demo. Pull the last eight to twelve completed weeks, place one row per week in a register, and label launches, incidents, holidays, absences, vacancies, and changes in release practice.

Then calculate normal delivery only from the weeks that earned that label. Keep the excluded weeks visible beside it. If the normal baseline is weak because too few weeks remain, that result is useful: the organization has been operating under exceptions, and it needs a longer observation period before it claims an efficiency problem or an efficiency win.

Founders usually want a single number because it seems easy to manage. Build the explanation first. The number will then have a chance of meaning something.