On-Time Delivery Rate and AI ROI: Connecting Token Spend to Commitments

All Posts
Share this post
Share this post

TL;DR: Engineering leaders whose boards judge AI by commitments met should regress on-time delivery rate against token spend. On-time delivery rate divides each roadmap item's estimated duration by its actual duration, capped at 100%. Roadmap value delivery is the companion value delivery metric, and the same regression applies to it. The regression slope is the ROI readout, and R-squared shows how much variation the spend explains. Use each team's trend over time as the supporting check, since AI-vs-non-AI cohorts suffer from selection bias. PR lead time, ticket cycle time, and bug rate explain why the trend moves. minware runs this regression from ticketing, version control, and AI tool data.

Your board does not ask how many tokens the team burned last quarter. They ask whether the team hit its commitments. Most AI ROI reporting cannot answer that question because it stops at adoption metrics, code output volume, or efficiency signals like PR lead time and ticket cycle time that do not connect to value delivery outcomes like on-time delivery rate.

On-time delivery rate is the commitment-level outcome that closes that gap. It is a value delivery metric, one of the outcomes to regress against token spend alongside roadmap value delivery. It answers the question boards ask most directly: did we deliver what we promised, when we promised it. This article explains how to source on-time delivery rate from ticketing data, read its regression slope against token spend, and use supporting metrics to explain why the trend is moving.

Framing AI ROI around commitments

Boards judge AI investment by commitments met, which makes on-time delivery rate the metric that answers their question. Sprint completion gives an early read on delivery risk.

Identifying what boards demand to see

Boards typically ask a specific question: did the team deliver what it promised, when it promised it. That question maps to one metric: on-time delivery rate, calculated as each roadmap item's estimated duration divided by its actual duration, capped at 100% per roadmap item.

On-time delivery rate is a value delivery metric, alongside roadmap value delivery as one of the two value delivery outcomes to regress against token spend, separate from the efficiency, quality, and throughput metrics that help explain why delivery moved.

When the board asks whether AI investment is working, the answer that holds up is a regression slope: the change in on-time delivery rate per unit change in token spend, with R-squared showing how much of the outcome's variation the spend explains.

Recognizing why adoption metrics mislead leaders

Adoption metrics like seats activated and completions accepted typically indicate whether the tool is being used. Throughput metrics like pull requests merged reflect code output volume. Neither answers whether delivery improved. Self-reports don't close the gap either. In METR's early-2025 study, experienced open-source developers took 19% longer to finish tasks with AI tools while believing AI had sped them up by 20%.

Pull request merge counts are a common stand-in for AI ROI, but a team can merge more pull requests without shipping more value because merge counts measure output volume.

DORA's 2025 report found that AI adoption raises delivery throughput while it continues to hurt delivery stability. More output doesn't guarantee that commitments land.

Measuring AI impact on sprint success

Sprint completion is a predictability metric that measures story points completed against the sprint's original commitment, capped at 100%. It can serve as an early signal of on-time delivery risk. A decline in sprint completion while token spend rises is an early warning that on-time delivery rate may follow. Teams that do not run sprints can approximate this with a weekly measure such as story point velocity.

For on-time delivery rate specifically, the measurement relies on three points in a roadmap item's history: when it moved to in progress, its due date at that moment, and when it was marked done. Jira, Linear, and Azure Boards expose these fields. Teams on GitHub Issues often track dates through milestones, which carry due dates.

Measuring on-time delivery rate

On-time delivery rate comes from ticketing data, measured against the original due date with success criteria set before measurement. Tracking roadmap items shows more than tracking sprints.

Mapping ticketing data to outcomes

On-time delivery rate is derived from ticketing data. A roadmap item's estimated duration runs from when it's marked in progress until the original due date set at that time. Its actual duration runs from in progress until it's marked done. By default, minware treats each epic as a roadmap item, and teams can configure higher-level tickets instead.

The definition of done varies across organizations. Some teams count a ticket as done once it reaches a "Done" status. Others exclude tickets still in user acceptance testing. minware's metric definitions are transparent and customizable, so teams can align the definition of done with how they actually work.

The data typically comes from the project management systems the team already uses: Jira, Linear, Azure Boards, or GitHub Issues. minware connects to each and normalizes the data into a canonical model, so a ticket in Jira and a ticket in Linear produce the same metric definition.

Setting clear criteria for success

Define the inputs before you measure: which ticket level counts as a roadmap item, and which status counts as done.

The definition of done is configurable, and teams can, for example, exclude certain ticket statuses like user acceptance testing from efficiency metrics for specific teams. Customization happens through report configuration, with deeper changes handled by customer success in a single call and about 24 hours, while many analyses work as soon as data is connected.

Keep the criteria stable across reporting periods. Changing the definition mid-year makes the trend uninterpretable.

Managing scope creep in delivery data

Scope creep shows up in ticketing data as tickets added mid-sprint, due dates moved after commitment, and estimates revised upward. Each affects the actual duration measured against the original estimate.

minware measures on-time delivery rate against the original due date set when a roadmap item went in progress. An epic whose due date moved from March 15 to March 30, then completed on March 28, is measured against March 15. minware's ticketing history captures field changes, so the original due date remains visible even after the team moves it.

Comparing roadmap items with sprint tracking

On-time delivery rate shows which specific roadmap commitments slipped and by how much. Sprint completion aggregates story points and hides individual failures. A team can complete 85% of its sprint points and still miss every commitment that mattered, if the 15% that slipped was the work the board was tracking.

A manager looking at a flagged on-time delivery rate can see exactly which tickets missed, which pull requests were involved, and which agent sessions contributed.

Regressing on-time delivery rate against token spend

The slope of on-time delivery rate regressed against token spend is the ROI readout, with token spend sourced as dollar cost from vendor APIs and OpenTelemetry integrations. Delivery speed and spend scaling help explain the result.

Mapping token spend to delivery metrics

Token spend means dollar cost, not a count of tokens. AI tool APIs and OpenTelemetry generally report what token cost would be on a consumption-based plan with no discount, which may differ from the amount actually paid on flat-rate plans such as Claude Pro.

The data sources are vendor enterprise APIs and OpenTelemetry integrations. Cursor's Admin API returns token and cost data per request. GitHub Copilot reports usage through its metrics API. Claude Code reports estimated cost through OpenTelemetry as well as its Analytics API. Codex supports OpenTelemetry export. The granularity varies, but the category-level availability is broad enough that token spend is measurable for most AI coding tools in use today.

Quantifying AI ROI via regression

The methodology used here is a linear regression of on-time delivery rate against token spend, rather than a comparison of groups. The slope is the delta in on-time delivery rate per unit change in spend. The R-squared value says how much of the outcome's variation the spend explains, on a 0-to-1 scale.

Name the confounders before the board asks: task-level selection (engineers reach for AI on work where it is likely to help), person-level selection (early adopters are often already the strongest performers), and covariates like code base age and quality.

Measuring impact on delivery speed

Delivery speed is a supporting signal that explains whether AI is creating bottlenecks downstream. PR lead time and ticket cycle time show whether the review and execution stages are absorbing extra code volume.

Pair them with on-time delivery rate as guardrails. Cycle time should hold steady or fall as AI spend rises, confirming the spend is not creating a bottleneck further down the pipeline. When delivery outcomes do not move or fall as token spend rises, look at ticket cycle time and bug rate to find out why.

Explaining why token spend fails to scale

Token spend can rise without on-time delivery rate improving, because more code output creates review bottlenecks and scope creep. The regression slope captures this directly: a flat or negative slope means higher spend is not buying more on-time delivery.

This is the answer to the CFO's question: is higher spend buying more delivery, or just a bigger bill.

Using internal benchmarks to verify AI ROI

Each team's trend over time is the supporting check for the regression, because AI-vs-non-AI cohort comparison suffers from selection bias. Sprint outcomes can be read alongside token spend in the same report.

Recognizing how selection bias skews AI data

AI-vs-non-AI cohort comparison is losing meaning as adoption spreads, and the non-AI groups that remain are prone to selection bias. An engineer who skips AI is often working where it is unlikely to help, such as a legacy repository with poor test coverage, so comparing that engineer's output to an AI-heavy engineer's output does not isolate the AI effect.

The supporting check to use instead is each outcome's trend over time for the same team, which controls for differences between teams because the team is compared to itself.

Validating AI ROI via team baselines

Track on-time delivery rate for the same team over time as AI maturity increases. If the trend rises alongside token spend, the correlation supports the regression slope. If it stays flat or declines, treat the slope with caution.

The regression doesn't need a pre-AI baseline. What matters is configuring data collection, especially OpenTelemetry, as early as possible so usage is captured from that point forward.

Linking AI costs to sprint outcomes

Sprint completion and on-time delivery rate both come from ticketing data, so they can be read alongside token spend in the same report. Sprint completion reflects the team's ability to estimate and commit. On-time delivery rate reflects the team's ability to deliver against commitments. AI that improves estimation without improving delivery can show up as stable sprint completion with flat on-time delivery rate. AI that improves both may show up as both metrics rising together.

Reading the supporting metrics

PR lead time, ticket cycle time, and bug rate explain why on-time delivery rate moves. When they contradict the trend, investigate before reporting the ROI number.

Reading PR lead time signals for AI ROI

PR lead time runs from a pull request's first commit to deployment. AI usage tends to show up first in PR review time, the time from when a pull request is opened to when it receives its first review from another person, because more code arriving for review lengthens that wait. Google's engineering practices set one business day as the maximum time to respond to a code review, a useful reference point for that wait.

A rising PR lead time alongside rising token spend is the signature of the throughput trap: more code arriving, same review capacity, longer waits, and eventually missed commitments. The on-time delivery rate regression captures the outcome. PR lead time explains the mechanism. This matches DORA's 2025 finding that AI adoption is associated with increasing instability.

Connecting cycle time to sprint success

Ticket cycle time runs from when a ticket moves to an in-progress status to when it is marked done. It's an efficiency metric measuring internal delivery speed. On-time delivery rate measures whether commitments are met, so the two are best read side by side. A rising ticket cycle time alongside flat on-time delivery rate can suggest the team is taking on more work without delivering faster. A falling ticket cycle time alongside rising on-time delivery rate may indicate that AI is accelerating execution.

Using bug rates to validate quality

Bug rate is bugs created divided by pull requests merged into a main branch. A rising bug rate alongside rising token spend can suggest AI-generated code is creating rework that puts on-time delivery at risk. Rework is the hidden cost of code output volume.

Handling supporting metrics that contradict the trend

If on-time delivery rate improves but bug rate rises, the improvement may be unsustainable. The team is hitting commitments by shipping code that creates downstream rework, which will eventually slow delivery.

Contradictions between the outcome metric and its supporting signals should prompt investigation before the ROI number is reported.

Metric Category Role in AI ROI
On-time delivery rate Value delivery Outcome metric, regressed against token spend
Roadmap value delivery Value delivery Outcome metric, regressed against token spend
Sprint completion Predictability Leading indicator of on-time project delivery
PR lead time Efficiency Supporting signal, explains delivery speed
Ticket cycle time Efficiency Supporting signal, explains delivery speed
Bug rate Quality Supporting signal, explains rework and sustainability
Pull requests merged Throughput Misleading as ROI evidence, denominator for bug rate

Presenting the results to the board

A defensible board presentation leads with the on-time delivery rate regression, names the confounders, and closes with a recommended action. A build-vs-buy comparison and a bottleneck checklist support that presentation.

Correlating AI costs with team output

In this approach, the regression groups data by team and time period and normalizes each data point to per-person-day by dividing token spend and on-time delivery rate by active contributor work days. Team-size differences do not distort the slope because each point reflects output per person, not raw totals.

Addressing board-level inquiries

Prepare for the executive review with a specific structure. A defensible approach is to lead with on-time delivery rate regressed against token spend. Read the slope as the change in on-time delivery rate per additional dollar of token spend per person-day. The R-squared value shows how much of the variation the spend explains.

Show the same-team trend over time as the supporting check. Name the confounders before the board asks: task-level selection, person-level selection, and covariates like code base age.

Close with the specific action: which teams, tools, or workflows to scale, stop, or investigate. A regression slope without a recommended action leaves the board without a decision to make.

Calculating build-vs-buy for commitment-level reporting

Factor Internal build minware
Initial setup Multi-week pipeline project Configuration is required for org-specific questions, and many analyses work as soon as data is connected.
API maintenance Your team owns it as APIs change minware maintains compatibility
Metric definitions Custom code, tribal knowledge Visible, editable minQL formulas
Data normalization Custom mapping per vendor Canonical model across all sources
Entity resolution Manual identity matching Automatic relationship recovery and identity resolution
Ongoing support Whoever built the pipeline, indefinitely Customer success, single call and ~24 hours

A recurring cost that's easy to underestimate is the governance layer: consistent metric definitions and calculations across teams. With a vendor, a data lineage question typically becomes a support conversation. With an internal build, it is whoever built the pipeline, indefinitely.

minware's Professional plan is publicly listed at $25/contributor/month with no seat minimum, and the 14-day trial is self-serve with no credit card required.

Assessing AI-induced bottlenecks with a checklist

Use this checklist to identify whether AI-generated code volume is creating downstream bottlenecks that threaten on-time delivery rate.

  1. PR review time rising: The wait for a first review from another person is lengthening, potentially as more code arrives for review.

  2. Bug rate rising: The rate of bugs relative to merged pull requests is increasing, which can suggest rework is absorbing delivery capacity.

  3. Ticket cycle time rising: Tickets are taking longer from in-progress to done, even as code output increases.

  4. On-time delivery rate flat or declining: The team is not hitting more commitments despite higher code output.

  5. Token spend rising: AI tool costs are increasing quarter over quarter.

If three or more of these are true, the regression slope of on-time delivery rate against token spend may well be flat or negative. The spend is not buying more delivery. The supporting metrics tell you where the bottleneck is.

"It gives us clear visibility into code quality, defect rates, and overall development health. I like the project-specific metrics that allow us to track individual initiatives, along with the leadership dashboards." - George V. on G2

Answering the board's commitment question

On-time delivery rate answers the board's question about AI investment. Regress it against token spend, read the slope, and show the same-team trend over time as the supporting check. Use PR lead time, ticket cycle time, and bug rate to explain why the trend is moving. Name the confounders before the board asks. That is the defensible answer.

Start a 14-day free trial, no credit card required, and connect your ticketing, version control, and AI coding tool data to see on-time delivery rate regressed against token spend with your own numbers.

FAQs

Does AI coding improve on-time delivery rate?

It depends on the team and the regression slope. Some teams may see on-time delivery rate rise with token spend, while others may see it stay flat even as code output rises. In this framework, the slope of on-time delivery rate against token spend is the answer for your org, and the R-squared value shows how much of the variation the spend explains.

What if our on-time delivery rate has not changed since AI rollout?

A flat on-time delivery rate alongside rising token spend means the spend is not buying more on-time delivery. Check PR lead time and bug rate to see whether review bottlenecks or rework are absorbing the extra code volume.

How long does it take to see on-time delivery rate improve?

On-time delivery rate moves slowly because it reflects commitments made weeks or months earlier. Track the same-team trend over time rather than a single before-and-after snapshot, and expect the signal to emerge over several reporting cycles. Sprint completion operates on a shorter cycle and can provide an earlier read.

Can we measure sprint completion AI ROI without historical baselines?

Yes. The regression methodology does not need a clean pre-AI baseline. What matters is configuring data collection, especially OpenTelemetry, as soon as possible so you start capturing usage from that point forward. The Claude Code Analytics API can still backfill usage for Console organizations at a coarser per-user, per-model, per-day grain.

What if our ticketing data is incomplete or messy?

minware works with data as it exists today, and hygiene metrics such as PRs Traceable to Ticket surface exactly where the process is breaking down, giving you a specific list of items to clean up.

Key terms glossary

Bug rate: A quality metric measuring the number of bugs created divided by the number of code changes, by default the number of pull requests merged into a main branch.

Linear regression: A statistical model charting the best-fit line between an outcome metric and an input variable. It is the primary AI ROI methodology, fitting a line through token spend and a delivery outcome metric across teams and time periods.

On-time delivery rate: The estimated roadmap item duration (time from when a roadmap item is marked in progress until the original due date set at that time) divided by the actual roadmap item duration (time from in progress until it is marked done), capped at 100% per roadmap item.

PR lead time: The time from a pull request's first commit to when it was deployed. Deployment defaults to when the code merges into a main branch.

Same-team trend: A team's outcome metric tracked over time as AI maturity increases. It controls for differences between teams and replaces AI-vs-non-AI cohort comparison, which selection bias undermines as adoption spreads.

Sprint completion: The total story points completed each sprint divided by the story points in the original commitment, capped at 100% per sprint. Late additions to the sprint are not penalized.

Ticket cycle time: The time from when a ticket was first marked in progress to when it was completed.

Token spend: The dollar cost of AI tool usage, as reported by AI tool APIs or OpenTelemetry. On flat-rate, tiered, or discounted plans, reported spend may differ from the amount actually invoiced.