Engineering capacity after AI adoption: what to measure

Why ticket counts hide delivery problems

A ticket count looks simple: compare this week's total with last week's. But teams can raise that number without shipping anything more useful. One engineer may log a single ticket for a complete billing change, while another splits the same work into separate tickets for research, code, tests, review, release, and follow-up.

AI widens this gap. A coding assistant can produce more pull requests and smaller tasks, yet customers may see the same release pace. Delivery can even slow down when reviewers spend more time checking generated code or correcting unclear changes.

Ticket totals also ignore waiting. A feature might take two hours to build, then sit in a review queue for four days. The backlog looks active, and the completed-ticket chart may look healthy at month end. The customer still waits, while engineers lose time switching to other work.

Consider a startup that adds AI assistance and reports a 40% jump in completed tickets. That sounds good until the founder checks releases. The team shipped the same number of customer requests, production defects rose, and senior engineers now spend part of each afternoon reviewing larger batches of generated changes. The ticket count captured activity, not delivery.

Customer outcomes offer a clearer test. Did a requested change reach production sooner, work reliably, and cost less to deliver? A completed outcome could be a self-service export that customers can use, a checkout bug that no longer blocks payment, or a report that replaces a manual weekly task.

Tickets still help organize work. They show workload, reveal recurring requests, and clarify ownership. They are a poor headline metric for engineering capacity after AI adoption because they reward task splitting and visible motion.

Use ticket counts as a supporting signal alongside measures that follow work to a real result:

• Time from commitment to production
• Time changes wait for review
• Defects customers find after release
• Service availability and incident duration
• Engineering time and payroll cost for each shipped outcome

This changes the conversation. Instead of asking engineers to close more tickets, leaders can ask why a customer request took 12 days, why it spent three days waiting for review, and whether AI reduced that delay. A fuller board does not prove that a team delivers more.

Five measures that show real delivery

Ticket totals can rise after an AI rollout while customers wait longer for fixes and engineers spend more time reviewing generated code. Engineering capacity shows up in the flow of work, the quality of releases, and the cost of results.

Measure the path to a working release

Cycle time tracks work from the moment an engineer starts it until customers can use it. Use the median, not only the average. A few stalled tasks can make an average look harmless, while the median shows the normal experience.

When possible, split cycle time into active coding, review waiting, testing, and release. If AI cuts coding from four days to one but review waits grow from one day to five, the team has not gained much delivery speed.

Review queue age measures how long a pull request waits before someone gives it useful attention. It often reveals the first bottleneck after a team adopts coding assistants. AI can produce more code than reviewers can safely assess.

Track the age of open pull requests and set a simple expectation, such as a first review within one business day. A growing queue suggests that the team should reduce work in progress, share review duty, or make pull requests smaller.

Measure whether releases help customers

Escaped defects are problems customers find after a release. Count customer reports, urgent support cases, rollbacks, and production fixes tied to a release. Do not treat every typo like a service outage. Use severity levels so the team can see whether AI changes the type of defect as well as the total.

Uptime records whether the product remains available after release. Pair it with incident duration. A team that ships twice as often but causes regular 30-minute outages has traded one problem for another. For a SaaS product, track the percentage of time the customer-facing service works and note incidents caused by recent changes.

Cost per shipped outcome connects spending to work that created a customer result. An outcome might be a paid feature, a removed support issue, or a fixed checkout failure. Add labor costs for the people involved, AI tool fees, cloud costs caused by the change, and contractor spend. Then divide by completed outcomes in the same period.

This measure requires judgment. Ten tiny interface edits do not equal a feature that lets customers complete a purchase. Define what counts as an outcome before comparing months.

A small startup might find that AI helped it release 12 outcomes instead of eight each month. That sounds promising. If review waits doubled, escaped defects rose, and payroll plus tool costs climbed, the gain is weaker than it first appears.

Build a baseline in five steps

A baseline gives you something solid to compare against after the team starts using AI coding tools. Without one, a busy month can look like progress even when releases take longer or defects rise. Choose one product area that ships often enough to produce useful data, such as the customer dashboard or checkout flow.

1. Set the comparison window

Pull four to eight weeks of data from before the AI change. Use a longer period if releases vary widely, but avoid mixing in a major rewrite, holiday rush, or staffing change. Those events skew the result.

Record the start and end dates. Product and engineering leads should agree that the period reflects normal work. If the team released almost nothing during those weeks, choose a different product area rather than forcing a weak comparison.

2. Define each measure once

Teams often argue about numbers because they count different moments. Put a short definition beside every measure and keep it unchanged during the comparison.

For example, cycle time might run from the first commit on a work item to its production release. An escaped defect might mean a customer-reported production bug that needs an engineering fix. Review queue age might start when an engineer requests review and end when a reviewer responds.

Use the same source for each number every week. Pull uptime from monitoring records, release dates from deployment records, and cost from payroll or contractor spend. Do not combine estimates with system data unless you label the estimate clearly.

3. Record the middle and the spread

Averages can hide painful outliers. If nine changes ship in two days but one waits 30 days, the average suggests a smoother process than the team experienced.

For each measure, record the median and the usual range. A weekly note might say: median cycle time was 4.5 days, and most work finished between two and nine days. Keep the count of shipped outcomes beside the cost figure so cost per shipped outcome has a clear denominator.

4. Keep work categories visible

Separate planned feature work, defects, operational work, and urgent customer requests when possible. A team that spends half a month fixing an outage should not compare that period directly with a quiet feature month.

You do not need perfect labels. A small set of consistent categories gives product leaders context when delivery speed changes.

5. Review the baseline every week

Engineering and product leads should spend 20 minutes reviewing the same small table. Ask what changed in the work, not which person caused a number to move. If review queue age rises while cycle time stays flat, reviewers may be carrying too much work.

Keep this routine after AI adoption. The baseline becomes a practical test: did the team ship useful work faster, keep uptime steady, and reduce cost per shipped outcome?

Read the measures together

One number can flatter an AI rollout. A team may close more tickets because AI drafts code quickly, while customers still wait for releases or report more bugs. Capacity appears in the full path from request to reliable use.

Start by separating work types. A small bug fix, a customer request, and a new billing feature move through different paths. Track cycle time for each group, from the moment a developer starts work until the change runs in production. Otherwise, a pile of quick fixes can make feature delivery look faster than it is.

Compare coding time with waiting time. If AI cuts implementation from four days to one but pull requests sit in review for three days, total cycle time barely moves. The same applies to release: code may pass review quickly, then wait for a manual test slot or a weekly deployment window. The customer experiences the whole delay.

Read escaped defects beside cycle time. Include bugs that reach production, support issues caused by a change, and urgent fixes after release. A modest increase in defects may be acceptable while a team learns a new process, but frequent hotfixes erase the benefit of faster coding. Severity matters too. Ten cosmetic issues differ from one outage that blocks checkout.

Uptime adds another useful signal. Frequent changes can improve delivery when the team keeps releases small and easy to reverse. But a drop in uptime after larger AI-assisted changes suggests that review, testing, or system knowledge has not kept pace. Check whether incidents cluster around certain services, work types, or generated code patterns.

Cost per shipped outcome ties these measures to the budget. Include salaries, contractor invoices, and AI tool costs. Then divide the total by an outcome the business can recognize, such as a released customer feature, a completed integration, or a resolved reliability issue. Do not divide by tickets. Their size and business value vary too much.

For example, a six-person startup spends $72,000 a month on engineering, contractors, and AI subscriptions. Before AI, it ships four customer-facing features per month. After AI, it ships six, but production incidents rise from one to five and two engineers spend much of Friday fixing them. The apparent cost falls from $18,000 to $12,000 per feature, yet the team should investigate quality before calling the change a success.

Use the measures as a group:

• Shorter cycle time with stable defects and uptime usually indicates real progress.
• Faster coding with an older review queue points to a review bottleneck.
• More releases with more escaped defects points to weak tests or rushed review.
• Lower cost per outcome matters only when customers can use the outcome reliably.

A simple example from a startup team

A three-person startup used an AI coding assistant during a billing update. The work included a new invoice status, changed payment reminder rules, and clearer amounts in the customer portal. Each developer began closing more small tickets than before. On the weekly board, output looked twice as high.

That first number gave the founder the wrong impression. Extra tickets created a longer review queue. Pull requests waited about two days longer because each developer opened more of them, while nobody owned reviews clearly. Code sat ready but unmerged, so customers did not receive the update any sooner.

The team then rushed the release near the end of the week. Several customers reported invoice totals that did not match what they expected. The team paused planned work, inspected payment records, corrected the logic, and answered support messages. Ticket counts still looked good. The billing result did not.

The team stopped treating each small ticket as a unit of output. It grouped related work into one shipped billing outcome: customers could receive accurate invoices and reminders under the new rules.

It tracked five numbers for that outcome:

• Cycle time from the first code change to production release
• Review queue age from opening a pull request to its first review
• Customer-reported invoice errors after release
• Billing service uptime during the change
• Total engineering and support cost for the completed update

The bottleneck became plain. AI helped developers write and revise code faster, but review work limited delivery. The team assigned one rotating reviewer each day and required reviewers to handle open pull requests before starting new tickets. It also tested a few real invoice cases before each release.

During the next billing change, cycle time fell because code moved through review sooner. Customer defect reports stayed flat, and the service remained available. The team did not need to claim that AI doubled capacity. It could show that a billing outcome reached customers faster without adding rework or support cost.

Mistakes that distort the numbers

AI can make a team look busier before it makes the team faster. The easiest mistake is to count generated lines of code, pull requests, or tickets closed. Those numbers describe activity, not whether a customer received a working improvement.

A developer may use an AI assistant to produce 2,000 lines for a feature that later needs two weeks of repair. Another developer may delete 500 lines, remove a recurring failure, and prevent support requests. The second change delivers more, even though the activity count looks smaller.

Compare like with like

A launch week and a maintenance week should not sit in the same comparison without context. Launch work may include new features, urgent fixes, and unusual review hours. Maintenance work often contains smaller, safer changes. Tag work by type, then compare similar periods over several weeks.

Do not rely on average software delivery cycle time. One item stuck for 30 days can make a healthy week look slow, while several tiny fixes can hide a delayed customer request. Use median cycle time and inspect the 75th or 90th percentile. That shows what a typical item takes and how often work gets seriously stuck.

Keep the measurement simple:

• Separate feature work, defects, technical maintenance, and urgent incidents.
• Record median cycle time and the oldest unfinished items.
• Compare periods with similar work mix and team size.
• Record unusual events, such as a launch, staff absence, or major outage.

Count impact, not defect volume

Treating every defect as equal creates bad incentives. A typo in an internal screen and a payment failure should not carry the same weight. Record whether a defect affected customers, blocked revenue, caused data loss, or required an emergency response. Escaped defects matter because customers found them after release, but severity matters more.

AI often speeds up the first draft of code. It can also move the bottleneck into review, testing, security checks, or release approval. If pull requests wait three days for review, faster coding only lengthens the queue. Track review queue age and the time from approved change to production alongside coding time.

Cost per shipped outcome can also mislead when the outcome is vague. "Built an analytics page" says little. "Released an analytics page that lets account owners export monthly usage data" gives finance and product teams something concrete to assess. Include people time, AI tool costs, cloud spend caused by the change, and follow-up repair work.

A review of engineering capacity after AI adoption should ask whether customers get reliable outcomes sooner at a lower total cost. If review queues grow, defects reach customers, or releases slow down, ticket volume will not explain the problem. The delivery path will.

A short weekly delivery check

Run this check at the same time each week, using a fixed 30-minute slot. Bring one view of work that reached production, customer reports, incidents, and current pull requests. This helps the team spot blocked flow before a monthly report hides it behind averages.

Start with completed outcomes. Ask whether a customer received a useful change sooner than last month. A released ticket does not always count: a billing fix that removes failed payments counts, while five internal refactoring tickets may not.

Use a short checklist:

• Compare median cycle time for work completed this week with the prior four weeks.
• Find the oldest open code review, name its owner, and agree on a decision date.
• Read customer-reported defects tied to recent releases and note repeat causes.
• Check whether deployment days match uptime incidents, error spikes, or rollbacks.
• Divide recent engineering cost by completed customer outcomes, then compare the trend.

Cycle time may fall after the team starts using AI coding tools. That helps only if review delays, escaped defects, and outages stay flat or improve. If developers produce twice as many pull requests but reviewers wait four days to respond, the bottleneck moved. The team did not gain usable delivery speed.

Keep the discussion specific. Instead of saying "reviews are slow," say that the oldest review has waited nine business days because only one engineer can approve database changes. The owner can then pair with another engineer, document approval rules, or reduce the size of future changes. A vague complaint rarely changes next week's numbers.

Treat defects with the same care. If customers report three issues after a Friday release, record the release, affected outcome, and repair time. Do not blame AI by default. Check whether the team skipped tests, merged a large change late, or lacked a clear acceptance check. AI engineering metrics should describe the whole delivery path, not only code generation.

End the meeting with one action, an owner, and a due date. For example: "Maya will clear the two reviews older than five days by Thursday." Revisit that action the following week. Small, repeated corrections keep delivery cost tied to results customers can see.

Next steps for your team

Start small enough that people can trust the numbers. Choose one product area and the team that owns it, then measure its work for a full delivery cycle. A checkout flow, reporting module, or customer onboarding feature makes a better first scope than every project in the company.

Write down exactly how you calculate cycle time, escaped defects, review queue age, uptime, and cost per shipped outcome. Decide whether cycle time starts when a developer begins work or when the team commits to it. Decide which production issues count as escaped defects. Small definition changes can make a healthy trend look worse or hide a real problem.

Keep those definitions unchanged for at least one quarter. AI adoption changes how people draft code, test changes, review pull requests, and handle incidents. A week of data mostly reflects deadlines, holidays, and one unusually difficult release. Twelve weeks gives the team enough time to spot a repeatable pattern.

Talk through the numbers with the people doing the work. Developers may see that AI produces more pull requests but leaves reviewers with larger, harder-to-check changes. A release engineer may find that deployment time stays flat while uptime drops after releases. Those details explain the metric better than a dashboard alone.

Do not turn this review into a scorecard for individual developers. Teams will protect themselves by splitting work into tiny tickets or avoiding risky fixes. Find blocked work and remove its cause, whether that is unclear product decisions, an overloaded reviewer, weak tests, or an AI workflow that creates too much cleanup.

After a quarter, compare the baseline with current results. Capacity has improved when useful work moves faster without higher customer defects, review delays, downtime, or cost. More tickets alone do not prove anything.

If the data exposes problems but the team cannot agree on the cause, Oleg Sotnikov's Team & AI Audit reviews engineering costs, delivery workflows, and AI use in five business days. It gives founders a clear savings plan before they change team structure or add more AI tools.