The AI Adoption Metrics Trap: Why Activity and Pull Request Counts Don't Prove AI ROI
TL;DR: Pull requests merged measures code output, while activity metrics such as seat activation or acceptance rate measure tool usage. Neither shows whether the AI spend bought more delivery. The defensible AI return on investment (ROI) answer is a linear regression of each delivery outcome against token spend, read off the slope, with each outcome's trend over time for the same team as the supporting check. minware's AI impact reports run this regression across teams and time periods, reporting the slope with its R-squared value. A clean pre-rollout baseline isn't required, so connect token spend data to delivery data as soon as you can.
AI vendor dashboards report seat activation, acceptance rate, and pull request activity, while boards ask what the AI spend did for delivery. Those metrics answer a different question. Usage metrics confirm the tools are running, yet many teams still present merged pull request counts as the return on that spend. This guide covers what adoption dashboards measure, which question proves AI ROI, and how to build a case an engineering leader can defend in an executive review.
What AI adoption dashboards actually measure
The metrics on a typical AI vendor dashboard fall into a few groups, each covered below.
Pull requests merged measure code output
The trap worth naming in AI ROI reporting is pull request merge counts presented as the return on AI spend. One 2026 enterprise study followed a company committed to doubling merged pull requests per engineer. Merge counts do climb with adoption. A study of Microsoft's rollout of command-line agents found adopters merged roughly 24% more pull requests, while noting that a merged PR is a different thing from the value it delivers.
A rising merge count proves code output rose, but it says nothing about whether that code moved a roadmap item. Pull requests merged, commits, and lines of code are throughput metrics. Delivery shows up in the two value delivery metrics, roadmap value delivery and on-time delivery rate.
Seat activation and acceptance rate measure usage
Seat activation counts the licensed seats that have started using the tool. Acceptance rate, sometimes reported as completion rate, is the share of AI suggestions a developer accepts. Both are activity metrics that confirm the tool is running.
Acceptance rate is a legitimate usage signal for suggest-and-accept tools like GitHub Copilot. Claude Code also reports a suggestion accept rate across its Edit, Write, and NotebookEdit tools, so the metric exists for coding agents too. The problem comes when either number is presented as ROI evidence, because accepting a suggestion is a different event from delivering value. An engineering leader who brings seat activation to a board review is answering a question the board did not ask.
Vendor dashboards stop at the tool's own data
Vendor dashboards report what the tool itself can see. GitHub groups its Copilot metrics into adoption, engagement, acceptance rate, lines of code, and pull request lifecycle metrics. Its impact dashboard also groups users into adoption cohorts and connects that adoption to pull request output. Anthropic's Claude Code analytics page suggests measuring ROI by tracking PRs per user as adoption increases. Each of these stops at throughput. Roadmap value delivery and on-time delivery rate depend on ticket data from your project management system, such as Jira, Linear, Azure Boards, or GitHub Issues. That ticket data sits outside AI tool usage reports. The table sorts each metric by what it can answer.
| Metric | Category | What it measures | What it answers |
|---|---|---|---|
| Seat activation | Activity | Licensed seats in use | Whether the tool is running |
| Acceptance rate | Activity | Share of suggestions accepted | Whether developers use the suggestions |
| Pull requests merged | Throughput | Code output volume | Whether code output increased |
| Roadmap value delivery | Value | Expected value of completed work, by default story points on non-bug tickets with a parent epic | Whether roadmap work shipped |
| On-time delivery rate | Value | Estimated duration divided by actual duration per roadmap item, capped at 100% | Whether delivery held to its estimates |
| Token spend | Input | Dollar cost of AI tool usage | What the AI tools cost |
Which question proves AI ROI
Two parts make up the answer: the question to ask, then the outcomes that answer it.
Spend level replaces usage as the question
The fix is to change the question from whether people use the tool to whether higher spend buys more delivery. Token spend level is the durable variable, because a yes-or-no usage flag no longer separates teams as adoption spreads. Spend level answers the CFO's actual question: is higher spend buying more delivery, or just a bigger bill?
Token spend here means the dollar cost of AI tool usage, as reported by vendor APIs or OpenTelemetry integrations. Answering the question takes token spend data connected to delivery outcome data, plus a method for reading the relationship between them. That method is linear regression, a statistical model that fits a best-fit line between an outcome metric and an input variable.
Three outcomes carry the analysis
minware's Lean AI Framework regresses three outcomes separately against token spend: roadmap value delivery, on-time delivery rate, and overhead cost. Roadmap value delivery is the expected value of completed work items. When no explicit value field is configured, it defaults to story points completed on tickets that are not bugs and have a parent epic. On-time delivery rate divides each roadmap item's estimated duration by its actual duration, capped at 100%. Overhead cost covers personnel cost and token spend on work outside roadmap value delivery. Non-AI work needs no separate cohort, because a regression against spend level already includes it as the zero-spend endpoint.
How to build the defensible case
Building the case takes three steps, in this order.
Configure data collection early
The renewal conversation is usually when the board asks for proof, so the regression slope and same-team trends need to exist before it. Connect telemetry as soon as you can. Most teams have already rolled out AI coding tools, so the question that matters is whether usage data is being captured from this point forward. A clean pre-rollout baseline isn't required, because the regression compares spend levels across teams and time periods. OpenTelemetry has no historical backfill, which makes it the integration to configure first. The Claude Code Analytics API can still backfill earlier usage for Claude Console organizations at a coarser per-user, per-model, per-day grain.
Connect token spend data
Most AI coding tools now expose usage data well beyond seat counts through vendor enterprise APIs or OpenTelemetry integrations. GitHub Copilot and Cursor expose it through their APIs, Claude Code through both routes, and Codex through OpenTelemetry. Pull token spend at per-user granularity, so it can be rolled up by team and linked to commits in your version control system, such as GitHub, GitLab, Bitbucket, or Azure DevOps. Those commits connect the spend to the tickets that carry delivery outcomes. Where a tool reports token counts without a cost figure, as Codex's OpenTelemetry data does, calculate spend from those counts at the vendor's published per-token rates.
AI tool APIs and OpenTelemetry generally report what token cost would be on a consumption-based plan with no discount. Claude Code, for example, computes the dollar figure locally from token counts at list price. On flat-rate plans such as Claude Pro, reported spend can differ from the amount actually paid. Where significant discounts apply, multiply reported costs by the total bill divided by total reported spend to reflect the average discount.
Run one regression per outcome
Each regression fits a single metric pair, so correlating spend against three outcomes means running three regressions. The data points come from teams and time periods, so the model needs spend that varies across them. Normalize each data point to per-person-day by dividing token spend and the outcome metric by active contributor work days. That normalization keeps team-size differences from distorting the slope. Run the regressions on the same grouping every reporting cycle, so the slopes stay comparable from one review to the next.
Why cohort comparison fails for AI ROI
Two common alternatives to regression run into problems, covered in turn below.
Selection bias skews AI versus non-AI groups
Comparing AI-assisted work to non-AI work is losing meaning as adoption spreads. METR reported in 2026 that wider AI adoption made task-level productivity harder to measure, with 30% to 50% of developers in its study holding back tasks they didn't want to do without AI. The non-AI work that remains skews toward code where AI helps least. One 2026 enterprise study found throughput gains barely present in legacy repositories. These findings point to two confounders: task-level selection and covariates such as code base age or code quality.
Bucketing spend recreates the cohort problem
Splitting token spend into low, moderate, and high groups looks like a compromise that keeps some of regression's precision. It is still cohort analysis, because bucketing a continuous variable discards the variation the regression relies on. It also adds arbitrary group boundaries that shift the result depending on where the lines fall. Keep spend continuous and fit a linear regression of each delivery outcome against it. Work with zero spend then sits at the low end of the same line, with no binary AI versus non-AI split. The regression still carries task-level and person-level selection as confounders, so name them whenever a slope is presented.
What the regression tells you
Reading the output comes down to two things: the ROI figure itself and the metrics that explain it.
The slope is the incremental ROI figure
The regression slope is the change in the outcome metric per additional dollar of token spend, for example story points completed per dollar. That marginal figure is the ROI answer. Swapping the axes expresses the same relationship as dollars per outcome, the cost-per-outcome figure stakeholders often ask for. Dividing total outcomes by total spend produces the wrong version of the same figure, because it credits AI with all delivery, including work humans would have shipped anyway.
Regressions also report an R-squared value, the share of the outcome's variation the spend explains, on a 0-to-1 scale. A low R-squared means spend isn't strongly correlated with that outcome, without identifying what else drives delivery.
Guardrails explain why the trend moves
Efficiency and quality metrics have two specific jobs alongside the regression. As guardrails, cycle time should hold steady or fall as AI spend rises, which confirms the spend isn't creating a bottleneck further down the pipeline. Bug rate, bugs created divided by pull requests merged, plays the same guardrail role for quality. As diagnostics, ticket cycle time and bug rate are where to look when delivery outcomes don't move with spend. Proving that the investment worked stays with the value delivery regression, which is why both metrics follow roadmap value delivery and on-time delivery rate in any report.
What same-team trends add
Each outcome metric's trend over time for the same team is the supporting check alongside the regression. Comparing a team to itself as its AI maturity increases controls for differences between teams, such as code base age, domain, and process. A same-team trend showing how roadmap value delivery moved as the team's token spend grew is more persuasive than any AI versus non-AI cohort comparison.
It also gives the regression a second angle. If the slope across teams is positive and individual teams trend the same way over time, the case holds up from both directions. If the two disagree, the guardrail metrics are the first place to look for the reason.
What to say in your next executive review
Three parts make the presentation hold up under questioning.
Lead with the business outcomes
Open with roadmap value delivery and on-time delivery rate, with overhead cost on the cost side. Efficiency and quality metrics follow as supporting signals. DORA metrics often get presented as the numbers boards care about. DORA names five software delivery metrics: deployment frequency, lead time for changes, change failure rate, failed deployment recovery time, and deployment rework rate. They form an efficiency and quality set. None of the five is a value delivery metric, so DORA can't answer on its own whether delivery improved. The DORA metrics belong in the review as context for the value delivery trend.
Show the slope, R-squared value, and confounders
Show the regression slope, the R-squared value, and the same-team trend for each outcome. State the method in one line: a linear regression of each delivery outcome against token spend, grouped by team and time period, normalized to per-person-day. Then name the confounders: task-level selection, person-level selection, and covariates such as code base age. Stating them up front tells a skeptical board member the analysis has already considered the obvious objections. The answer is defensible because every part of the calculation is visible and every limitation is named before anyone has to ask.
Acknowledge what the analysis cannot prove
A regression shows that spend and delivery move together, while causation needs more. A correlation needs a plausible causal story before it drives a decision. One candidate explanation is that AI tools shorten some categories of work, which frees capacity for more roadmap delivery. The regression and the same-team trend together make that story more or less plausible. Neither can prove it, because the data is observational, so the analysis can only control for bias. Saying so directly is part of what makes the rest of the analysis credible.
How minware handles this
Here is what the reports produce and what getting them running takes.
What the AI impact reports calculate
minware's pre-built AI impact reports regress roadmap value delivery, on-time delivery rate, and overhead cost against token spend, reporting a slope with its R-squared value for each. The regressions group by team and time period, with each data point normalized to per-person-day. Every metric's calculation is visible and editable through minQL, minware's formula language, so the number can be defended in an executive review.
The patented hypercube data model handles the data preparation a custom build would need. It recovers relationships between entities that have no structured link, such as agent sessions and the tickets they belong to. It also resolves developer identities across systems with different usernames or emails, and it normalizes vendor data formats into one model. Because relationships are recovered automatically, analysis can start before source data is cleaned up.
What setup and customization involve
Many analyses, including the regression of story points completed against token spend, work as soon as data is connected. Org-specific questions need a new or adjusted metric or report, which customer success typically sets up within a single call and about 24 hours. Historical backfill at first connection takes hours depending on repository size, a one-time setup step.
"The platform comes with a great set of engineering reports out of the box, so you can start getting value immediately. At the same time, those reports are highly customizable, and minQL makes it possible to build your own metrics and dashboards tailored to your team's workflow rather than being limited to predefined reports." - Verified user on G2
Pricing is publicly listed at $25/contributor/month for Professional with no seat minimum, and the 14-day trial is fully self-serve with no credit card required. You can connect version control, project management, and an AI coding tool, then explore the pre-built AI impact reports with your own data before talking to our team.
How to weigh build versus buy
Building your own data pipeline means writing your own extract, transform, load (ETL) stack. A custom build can work for a small team. Once several teams or stakeholders rely on the same numbers, it needs consistent metric definitions across all of them. It also works only until a vendor API changes or a stakeholder asks how a number was calculated. From then on, someone on your team owns that maintenance and that explanation. The cost most teams underestimate is the explanation, because someone has to answer how each metric is calculated every time a stakeholder asks.
| Consideration | Custom pipeline | minware |
|---|---|---|
| Initial setup | High, typically multiple weeks of engineering time to build the pipeline | Connect data sources, historical backfill takes hours |
| Vendor API changes | Your team owns compatibility | minware maintains compatibility as APIs evolve |
| Entity resolution | Your team builds relationship linking and identity matching | Hypercube data model recovers relationships and resolves identities |
| Data normalization | Your team maps each vendor's format | Hypercube data model normalizes across vendors |
| Org-specific questions | Your team writes new transformations | Customer success configures a new or adjusted metric or report, typically within about 24 hours |
| Metric definition questions | Whoever built the pipeline, indefinitely | Visible minQL formulas, customer success answers without an engineering escalation |
| Ongoing maintenance | Engineering capacity diverted from product work | Included in subscription |
What to bring to your next budget cycle
Seat activation, acceptance rate, and pull requests merged measure usage or code output, while the budget question is whether the spend bought more delivery. The defensible answer is three separate linear regressions against token spend, one each for roadmap value delivery, on-time delivery rate, and overhead cost, each read off its slope, with same-team trends as the supporting check. That gives the next executive review figures backed by a visible calculation, a stated method, and named confounders.
Start a 14-day free trial at minware.com with no credit card required, then connect your version control, project management, and AI coding tool data to see token spend regressed against delivery outcomes.
FAQs
What if we don't have a clean pre-rollout baseline?
You don't need one. minware regresses delivery outcomes against token spend level across teams and time periods, with non-AI work included as the zero-spend endpoint. Connect telemetry as soon as you can and capture usage from that point forward.
How do we track outcomes without building custom pipelines?
minware connects version control, project management, CI/CD, and AI coding tools, then normalizes their data into a single layer. Its pre-built AI impact reports regress delivery outcomes against token spend without custom ETL work.
Is higher token spend always better?
No. The regression slope shows whether higher spend is buying more delivery or just a bigger bill. A flat or negative slope means additional spend isn't moving that delivery outcome, and the guardrail metrics are where to look for the reason.
What if our data is too messy for this analysis?
minware works with data as it exists today. Its hygiene metrics, such as PRs Traceable to Ticket and Tickets Completed with Estimate on the Best Practices report, surface exactly where the process is breaking down. Each gap becomes a specific item someone can work through.
How long does it take to see a meaningful correlation?
Many analyses work as soon as data is connected, because historical version control and ticket data backfills at initial setup. Token spend history depends on the source. The Claude Code Analytics API can backfill earlier usage for Claude Console organizations, while OpenTelemetry data starts from the day it's configured. The regression grows more reliable as more teams and months of spend data accumulate.
Key terms glossary
Seat activation: The number of licensed seats that have started using an AI tool. An activity metric that shows whether the tool is in use.
Acceptance rate: The share of AI suggestions a developer accepts, also reported as completion rate. An activity metric reported by suggest-and-accept tools such as GitHub Copilot. Claude Code also reports it across its Edit, Write, and NotebookEdit tools.
Pull requests merged: The total number of pull requests merged, to or from any branch. A throughput metric that measures code output volume.
Token spend: The dollar cost of AI tool usage, as reported by AI tool APIs or OpenTelemetry. On flat-rate, tiered, or discounted plans, reported spend may differ from the amount actually invoiced.
Linear regression: A statistical model charting the best-fit line between an outcome metric and an input variable such as token spend. Each regression fits one metric pair.
R-squared (R²): In a regression chart, the share of the outcome metric's variation explained by the input variable, on a 0-to-1 scale.
Roadmap value delivery: The expected value of completed work items. If no explicit value field is configured, it defaults to story points completed on tickets that are not bugs and have a parent epic.
On-time delivery rate: The estimated roadmap item duration divided by the actual roadmap item duration, capped at 100% per roadmap item.
Overhead cost: Personnel cost and AI token spend that goes toward work outside roadmap value delivery.
Same-team trend: Each outcome metric's movement over time for the same team as AI maturity increases. The supporting check that controls for differences between teams.
minQL: minware's patented formula language, used to define every metric and calculation with full visibility into the underlying logic.
OpenTelemetry: An open protocol and data format for telemetry. Claude Code uses it to report token usage and cost, while Codex reports per-turn token counts with no cost figure.
DORA: DevOps Research and Assessment, the research program behind five software delivery metrics: deployment frequency, lead time for changes, change failure rate, failed deployment recovery time, and deployment rework rate.
Extract, transform, load (ETL): The sequence of steps a custom data pipeline runs to pull data from source systems, normalize it, and store it for reporting.