AI coding productivity needs both DORA and METR

AI coding productivity cannot be judged by choosing DORA or METR as the winner. DORA tells an SMB whether its delivery system is moving useful changes to production without creating more damage. METR tests a narrower causal question: did access to specific AI tools change the time experienced developers needed for defined tasks in repositories they already knew? A buyer needs both views because faster task completion can disappear in review, testing, release, or repair.

For a mature codebase, I would make accepted task time the local outcome and DORA metrics the system guardrails. I would also measure rework, reviewer effort, and escaped defects. That combination catches the two expensive errors I see in AI trials: rejecting a useful tool because a broad delivery metric barely moved, and buying a tool because developers felt fast while everyone downstream absorbed the cost.

DORA measures the delivery system, not the typist

DORA measures how an application or service delivers changes. Its current model groups five measures into throughput and instability: change lead time, deployment frequency, failed deployment recovery time, change fail rate, and deployment rework rate. These are outcomes of a whole delivery system. They include product decisions, ticket readiness, architecture, review capacity, test speed, release controls, and operations, not merely typing or code generation.

That scope is why DORA evidence matters to an owner. Payroll only turns into business value when a safe change reaches a user. If AI cuts implementation time but increases review queues, larger changes, emergency fixes, or recovery work, the company has not gained useful capacity. DORA exposes that transfer of effort even when an editor reports thousands of accepted suggestions.

The scope also limits what DORA can prove. The annual research uses broad survey data and statistical models to study relationships among practices, AI adoption, developer experience, delivery, and organizational outcomes. The public 2025 survey questions show that many inputs are reported by respondents about their primary application or service. This design can describe patterns across many environments, but it does not randomly assign AI to one team and withhold it from another. An association between adoption and throughput is not a causal estimate for your twelve year old billing service.

Release year matters. The 2024 Accelerate State of DevOps analysis reported that higher AI adoption was associated with lower delivery throughput and stability, including estimated reductions of 1.5% in throughput and 7.2% in stability for a 25% increase in adoption. The 2025 State of AI-assisted Software Development report found higher adoption associated with greater throughput and greater instability, and framed AI as an amplifier of the surrounding system. Quoting one number without its report year, model, and outcome produces a confident but useless purchasing argument.

DORA itself warns against turning a metric into a target or comparing unlike applications. Follow that advice. Measure one service against its own history. A payment API that deploys under regulatory controls should not compete with a marketing site on deployment frequency. The useful question is whether the same service delivers better after the intervention, while stability holds or improves.

METR isolates task time inside familiar repositories

METR measures whether AI access changes elapsed task completion time under controlled conditions. In its early 2025 randomized trial, 16 experienced open source developers completed 246 real tasks in mature projects they knew well, with about five years of repository experience on average. Researchers defined tasks before random assignment, allowed early 2025 AI tools for one condition, prohibited them for the other, and used screen recordings plus repository evidence to inspect the work.

That design answers a question DORA cannot: for this participant and this task pool, did tool access cause work to finish sooner? The answer surprised nearly everyone involved. METR estimated that AI access made completion 19% slower, with its reported interval running from 2% to 39% slower. Before the work, developers expected a 24% speedup. After doing it, they still believed they had been 20% faster. The disagreement between measured time and felt speed is the finding an SMB should remember.

The result is strong evidence for its setting, not a permanent tax on AI coding. Participants were unusually experienced with their repositories. Their tasks averaged about two hours. They used tools and models available from February through June 2025, mainly Cursor Pro with Claude 3.5 or 3.7 Sonnet. A coding agent handling a bounded migration in a private service today is a different treatment. A junior developer learning an unfamiliar module is a different population.

METR made this limitation more explicit in a February 2026 update. Its later experiment suffered selection bias because developers who valued AI often did not want to submit work that might be assigned to the no AI condition. Parallel agents also made task time harder to record. METR said the later data gave an unreliable signal, while participant reports suggested current tools probably helped more than the early 2025 tools. That is good scientific behavior: the researchers weakened their own claim when the design stopped supporting it.

Do not turn the early trial into a slogan that AI slows good engineers. Use it as a warning about local measurement. Experienced developers can misjudge their own speed, mature repositories impose context costs, and a clean looking benchmark can describe the wrong task mix.

The studies answer different management questions

DORA and METR sit at different levels of the same value chain. Treating them as rival scoreboards blurs causality, scope, and timing.

1. DORA observes application or service delivery. METR observes a developer completing a predefined task. Your trial should observe an accepted change within one service.
2. DORA's primary outcomes are throughput and instability. METR's outcome is task completion time. Your trial needs active engineering time plus delivery guardrails.
3. DORA shows how practices and outcomes move together across organizations. METR estimates the causal time difference for its studied sample. Your trial can support a scoped claim about one workflow, team, and period.
4. DORA cannot isolate typing time or prove a local causal effect. METR cannot measure the whole production system or current tools forever. Your trial will have a small sample and behavior changes caused by observation.
5. Use DORA to detect displaced cost and operational damage. Use METR to design a fair comparison and distrust perception alone. Use the local result to decide where to expand, change, or stop.

A task timer is a microscope. Delivery metrics are the vital signs of the organism. The microscope can show a local change that the vital signs are too coarse to reveal. The vital signs can show that the local improvement harmed the larger system. Neither substitutes for the other.

This distinction resolves an apparent contradiction. AI can reduce active implementation time while deployment frequency stays flat because release approval is the constraint. It can also increase deployment frequency while change fail rate rises because the team ships larger batches. Or it can slow a repository expert on subtle maintenance work while helping another engineer write tests in an unfamiliar module. One average cannot represent all four effects.

For an SMB, the decision is not whether DORA or METR is more authoritative. Decide which link in the value chain each measure can inspect. Use task evidence to estimate direct labor capacity. Use delivery evidence to verify that the capacity reaches production intact.

A mature codebase changes the expected payoff

A mature codebase rewards context more than syntax. The hard part usually sits in unwritten invariants, old migration paths, surprising call sites, production data, and the reason an ugly branch still exists. An experienced maintainer may type slowly and decide correctly. An AI tool may produce a plausible patch quickly and force that maintainer to prove which assumptions are wrong.

This is where benchmark enthusiasm often fails. A green test suite only proves what the suite expresses. It does not prove that a retry remains idempotent, an old client can parse the response, a permission boundary still holds, or an operations runbook still matches reality. On an established product, discovery and verification can dominate generation.

Task type therefore belongs in the experiment, not in a footnote. Classify work before assignment so a pile of easy tests does not hide poor performance on risky maintenance. A practical taxonomy has four buckets:

• Local work with a clear pattern, such as adding one validator beside similar validators.
• Repository discovery, where the engineer must locate behavior and explain dependencies before editing.
• Cross module changes involving contracts, migrations, permissions, or backward compatibility.
• Verification work, including tests, review, incident repair, and release preparation.

Expect different answers. AI may do well on repetitive local work and test scaffolding, break even on discovery, and lose time on a small change governed by an obscure invariant. That is not inconsistency. It is the result needed for routing work intelligently.

Repository readiness also changes the treatment. Clear build commands, fast tests, searchable decisions, typed interfaces, and small modules give an agent usable feedback. A flaky integration suite and tribal knowledge make every generated patch expensive to check. DORA's 2025 amplifier claim fits this mechanism: AI magnifies the system it enters. The practical response is to record readiness conditions, not to excuse every failure as bad tooling.

One team I would distrust immediately is the team that changes its repository, agent instructions, model, and review policy halfway through a trial, then reports a single average. Tool learning is real, but uncontrolled changes destroy the comparison. Record every treatment version. Analyze stable periods separately.

Lines of code and ticket counts will mislead you

Lines of code, suggestions accepted, prompts sent, commits, pull requests, and tickets closed are activity measures. They are cheap to collect and easy to improve without creating more value. That popularity makes them dangerous in a payroll decision.

AI often changes the shape of work. It can generate more verbose code, split work into extra commits, add low value tests, or turn one ticket into several. A team can raise every activity count while spending more reviewer time and shipping the same customer outcome. METR defined tasks before randomization partly to avoid this exact ambiguity: output volume can change while the amount of intended work does not.

Story points are worse as a cross condition outcome. Teams estimate them, renegotiate them, and learn the local scale. If developers know leadership wants a productivity gain, the scale moves. Keep points only as a rough stratification field, never as the numerator in an AI return calculation.

The unit worth paying for is an accepted change that meets its original conditions and stays healthy after release. Count a task as complete when the agreed checks pass, a reviewer accepts it, and required operational or documentation work is present. Reopened work returns to the same task record. A follow up fix caused by the change is rework, not fresh output.

Do not use surveillance as measurement. Screen recording made sense in a consented research study with paid participants and a defined protocol. An employer usually needs event timestamps, task classifications, tool condition, review effort, and outcomes. Collect the minimum evidence needed, publish the rules before the trial, and never rank individual developers. People who expect a leaderboard will optimize the ledger instead of the software.

Perceived productivity still belongs in the data, but as a separate measure. Ask a short weekly question about focus, frustration, and whether the tool helped. The gap between felt and observed time may reveal learning cost or hidden review work. It cannot replace the clock.

A permissions change shows why this matters. Suppose an agent finds the authorization middleware, adds a role check, writes unit tests, and opens a clean pull request in forty minutes. The standard workflow took another engineer ninety minutes on a similar ticket. A timer declares a large win. During review, the maintainer remembers that older tenants can inherit roles through a migration table the unit tests never load. The AI patch needs a second implementation, a data fixture, and another review round. If those minutes land on a new ticket or in the reviewer's calendar, the trial still credits the first task with a forty minute finish.

The error did not begin with bad arithmetic. It began with a weak definition of completion. The accepted outcome must include the inherited role path, the required regression fixture, and reviewer approval. The record must attach the rejected attempt and second pass to the same task. With that definition, the trial can still find a gain, but it measures the work the company actually needed.

Now consider the reverse case. An engineer asks an agent to trace every caller of an old pricing function and produce a change plan before touching code. Implementation time falls only ten minutes, yet review drops by an hour because the plan exposes two hidden consumers early. A code generation metric misses the benefit, while accepted task time captures it. This is why prompts and tokens are poor outcomes. The workflow includes discovery, planning, implementation, and proof. Measure the boundary around all of them.

Run a controlled crossover without crippling the team

A six week crossover can produce a useful SMB decision if the team has a baseline, enough repeated task types, and honest completion rules. You do not need a university lab. You do need to decide the analysis before seeing the result.

1. Spend two weeks checking telemetry and defining task classes, completion, rework, active time, and exclusions. Use existing history to calculate the DORA baseline for one service.
2. Choose one AI workflow and freeze its model, repository instructions, permissions, and review policy for the comparison period. Train participants during paid work before measurement starts.
3. Preclassify eligible tasks by risk, type, and rough size. Randomly assign AI allowed or standard workflow within each class, while exempting incidents, sensitive work, and anything where withholding the tool creates unacceptable risk.
4. Run two two-week measurement blocks. Rotate participating engineers across both conditions so differences in repository knowledge do not become tool effects.
5. Analyze accepted tasks, abandoned tasks, rework, reviewer minutes, and delivery outcomes together. Keep the task level result separate from the service level result.

Randomization must happen after a task is written and before implementation begins. If developers choose the condition, they will naturally send boilerplate to AI and subtle bugs to themselves. The resulting comparison measures task selection, not tool impact. METR's later recruitment problem shows the other side: once people strongly prefer AI, forcing a no AI condition can change which tasks and developers enter the study. Record refusals and excluded tasks because absence is data.

Pairing reduces noise. If two similar migration tasks exist, place one in each condition. If the same engineer handles both, alternate order to limit learning effects. Do not run the AI version second every time; knowledge gained on the first task would make the second look better.

The primary task measure should be active engineering minutes per accepted task, not wall clock time alone. Track waiting for CI, review, or another team separately because AI may move a bottleneck rather than remove it. Report medians and the full range by task class. A grand average can be dominated by one incident or a batch of tiny edits.

Set decision thresholds before the pilot. For example, expand only if median active time falls by at least 15% in two eligible task classes, reviewer time does not rise by more than 10%, and no DORA stability measure materially worsens. Those numbers are operating choices, not universal research findings. Pick values that cover license cost, training, and the cost of organizational change.

Capture task evidence that survives argument

A small event record is enough to reconstruct most disputes. Store one record per task in version controlled trial data or an internal analytics table. Keep prompts and generated code out unless a separate security review approves them.

{"task_id":"BILL-1842","service":"billing-api","class":"cross_module","condition":"ai_allowed","engineer":"anon-07","started_at":"2026-08-03T09:12:00Z","active_minutes":138,"ci_wait_minutes":24,"review_minutes":41,"completed_at":"2026-08-04T14:20:00Z","accepted":true,"rework_14d_minutes":35,"escaped_defect":false,"treatment":"agent-v1"}

This shape prevents three familiar arguments. The condition and treatment fields prove what was tested. Separate active, CI, and review time shows where effort moved. The fourteen day rework field keeps a rushed patch from looking cheap merely because repair happened in another sprint.

Write the task card before assignment. It should state the intended behavior, relevant constraints, required checks, and what counts as done. Do not prescribe the implementation, because that would remove part of the work the tool is supposed to help with. A reviewer who does not know the condition should judge acceptance where practical. Full blinding is hard when generated style is obvious, but hiding the label still removes some expectation bias.

Use a weekly rollup that a founder can read:

class          condition       accepted  median_active  median_review  rework_14d
local_pattern  standard        11        74m            18m            9%
local_pattern  ai_allowed      12        48m            21m            8%
cross_module   standard         7       166m            39m           14%
cross_module   ai_allowed       6       181m            57m           27%

The example does not declare AI good or bad. It says to route local pattern work toward the tested workflow and keep cross module work out until instructions, context, or verification improve. That is more useful than a companywide claim of 23% productivity.

Review abandoned and censored tasks. If a developer stops a bad AI attempt and finishes manually, keep the AI time in the task. If the pilot excludes the hardest work, show the excluded share. Otherwise a narrow success turns into an exaggerated capacity forecast.

Keep DORA as the production guardrail

Task gains count only while the service remains safe and deliverable. Pull DORA measures from deployment and incident systems for the same application before, during, and after the trial. Use consistent definitions across all periods.

Change lead time should begin at commit and end at production deployment. Deployment frequency counts successful production deployments for the service. Failed deployment recovery time covers the interval needed to recover from a deployment that requires immediate intervention. Change fail rate is the share of deployments needing that intervention. Deployment rework rate is the share of unplanned deployments made to address a production incident. If your current dashboard uses older names or definitions, document the mapping instead of silently relabeling the chart.

Read the five measures as a set. Lower task time with longer change lead time suggests a downstream queue. More deployments with a higher change fail rate suggests the team bought speed by spending reliability. Stable delivery with lower task time may be a real local gain even when deployment frequency cannot rise because product approval caps releases.

Use rolling weekly values for operations, but make the purchase decision over a window long enough to observe rework and incidents. A mature low volume service may not produce enough deployments for a stable percentage in six weeks. In that case, do not manufacture precision. Report counts, continue the guardrail after the task trial ends, and make only a limited rollout decision.

Keep batch size beside the DORA set. It is not one of the five current performance metrics, but DORA's AI analysis points to larger changes as a plausible path from faster generation to instability. Track changed files or reviewable change size as a diagnostic, never as a performance target. When change size rises, tighten task boundaries before blaming the model.

A service comparison also needs business demand context. Deployment frequency may fall because the product team paused changes, and lead time may rise because a compliance review started. Annotate these events. Telemetry does not remove the need for a weekly conversation with the people doing the work.

Decide with task economics and delivery health

The purchase decision needs two dimensions: local task economics and system delivery health. Approve broader use only where both are acceptable. This avoids demanding that every tool move every metric, while preventing a local speedup from hiding operational cost.

1. Expand the task classes that run faster with equal or lower review and rework while DORA guardrails stay stable or improve.
2. If tasks run faster but review or stability worsens, fix batch size, tests, instructions, or permissions, then repeat the trial.
3. If task time does not improve and DORA stays stable, stop that task class unless learning or quality has a separate measured value.
4. If tasks run slower but quality improves or operational load falls, price the avoided cost before deciding.
5. If tasks run slower and delivery is worse or unchanged, stop the workflow and keep the evidence.

Convert accepted time into money only after the scorecard passes. Use loaded engineering cost, not salary alone. Subtract license cost, training time, extra review, rework, and the maintenance cost of agent instructions and integrations. Do not convert every saved hour into payroll savings unless demand can absorb the capacity or headcount can actually change.

A practical formula is:

monthly_net_value = accepted_hours_saved * loaded_hourly_cost
                  - tool_cost
                  - training_and_admin_cost
                  - added_review_and_rework_cost

Run low, expected, and high cases because a small trial has wide uncertainty. Keep the result by task class. A 25% gain on work that consumes 10% of the month is a 2.5% gross capacity gain before costs, not a 25% team transformation.

Founders often ask for a single verdict because they want a budget answer. Give them one, but attach scope: expand, revise, or stop for named workflows in one service. A Team & AI Audit from oleg.is can set up this baseline and identify savings before a broader transformation, but the evidence should remain inspectable by the team that will live with the decision.

DORA provides the production boundary. METR provides the discipline to test time instead of trusting enthusiasm. Your own crossover supplies the only evidence that prices the current tool, current team, and current codebase. If the company cannot state the task classes, completion rule, and stability guardrails before the trial, it is not ready to quote a productivity percentage.