How to Detect AI Slop in Your Codebase

TL;DR: AI slop shows up in delivery data before it shows up in production. Leading indicators such as ticket size, ticket WIP, PR complexity, pre-merge churn rate, and PRs with no ticket link can surface it within weeks. Bug rate, bug lead time, and change failure rate confirm it over the following months. A linear regression of roadmap value delivery against token spend shows whether rising spend is buying delivery, and a flat slope while spend climbs is the pattern to investigate. You can run the audit from spreadsheet exports, or use minware, which connects version control, project management, and AI tool data, to run the regression in pre-built reports.

AI slop accumulates in oversized tickets, large PRs, and commits that never link to a ticket, until delivery slows with no clear cause. By the time quality metrics spike, you may have shipped months of syntactically valid but architecturally incoherent code.

The detection methodology that works at leadership level is statistical. You can't review every AI-generated PR, but you can read the delivery data: watch leading indicators across your development tools, confirm with lagging indicators, and use token spend regression to separate AI churn from normal development churn.

This guide walks through the specific metrics to track, how to calculate them from your existing tools, and how to build a defensible detection framework.

Spotting AI slop in your repository

Merriam-Webster defines slop as digital content of low quality produced usually in quantity by means of artificial intelligence. minware's Lean AI Framework defines AI slop as work or output that consumes tokens or human time without contributing to delivered value, which is why delivery data can surface it. In a codebase, its most visible form is syntactically valid code that lacks architectural intent. It passes linting and often passes tests, but doesn't fit your team's established design patterns.

Some technical debt is a deliberate tradeoff, such as taking a shortcut to make a release. Inadvertent debt is common too, and Martin Fowler argues it is inevitable even for teams that design well. In code, AI slop emerges when changes merge without sufficient review of their fit within the larger system.

Tracing the delivery impact

The signature of AI slop is volume without coherence: code output increases while roadmap value delivery stays flat or declines. Roadmap value delivery is the expected value of completed work, and it defaults to story points completed on non-bug tickets with a parent epic when no explicit value field is configured. Teams ship more pull requests, but that value doesn't move. PR complexity and pre-merge churn rise along with the output, so many of the earliest signals show up in your version control system, such as GitHub, GitLab, Bitbucket, or Azure DevOps. Ticket size and ticket WIP show the planning side of the same problem.

PR review time stretches because reviewers face larger, less coherent changesets. Bug rate climbs as architectural inconsistencies introduce subtle defects, and bug lead time stretches because nobody fully understands the code behind them. That makes bug rate, bug lead time, and change failure rate the natural confirmation signals.

Choosing metrics that surface quality gaps

Five metrics surface AI slop before it compounds:

• Ticket size: The story point estimate on each ticket, flagging any ticket worth a week or more of one person's time.

• Ticket WIP: The time each person spends with a ticket in progress per day, with a target of 1.5 or below.

• PR complexity: Added code lines per pull request, starting with a 500-line threshold.

• Pre-merge churn rate: Lines added that never make the final merge, investigating PRs above 50%.

• PRs Traceable to Ticket: The share of merged PRs linked to a valid ticket.

Each detection method catches a different set of patterns:

Detection method	What it tends to catch	What it tends to miss	Effort required
Manual code review	Obvious style violations and logic errors, particularly in small PRs	Architectural drift across many PRs and systemic patterns	High, difficult to scale beyond small teams
Static analysis tools	Syntax errors, common security patterns, and complexity metrics	Issues requiring broader context, including architectural intent	Medium, requires tool configuration
AI likelihood scoring	Code resembling generative patterns, such as repetitive structures	Context-appropriate AI use and output quality, with false positives a known risk	Medium
Delivery data regression	Code volume rising without matching delivery outcomes, or spend without delivery correlation	Individual PR quality and specific code issues	Low with automation, high with manual analysis

Tracking the leading signals

Leading indicators appear in version control and your project management system (such as Jira, Linear, Azure Boards, or GitHub Issues). Signals from these two sources can catch AI slop before it impacts customers and disrupts future work.

Spotting oversized tickets and rising WIP

Slop often starts before any code is written, when a ticket is too large or too vague for an agent to work on well. Large tickets leave room for runaway token use, and they hide missing requirements until rework arrives.

Ticket size uses each ticket's story point estimate by default. minware recommends breaking down any ticket worth a week or more of one person's time, which is half or more of a two-week sprint. Aim for an average of two days of capacity or less.

Ticket WIP shows the cost on the human side. It measures the time each day that a person has a ticket in progress, divided by a day. minware recommends a target of 1.5 or below. Higher values point to someone overloaded, waiting on a slow automated process such as an agent, or pulled away by bugs and urgent work. Read the two together, since oversized tickets can make WIP look better than it is.

Spotting rising PR complexity

AI lets developers produce far more code in the same amount of time, and DORA's research on generative AI finds it "possible, even likely" that changelists are growing in size as a result. DORA has consistently shown that larger changes are slower and more prone to creating instability. An agent's changeset can also include error handling for edge cases that don't exist in your environment.

PR complexity measures the number of added code lines in a pull request by default. minware recommends starting with a 500-line threshold, then having a senior engineer check each flagged pull request. The question for each one is whether it could and should have been split up. The threshold catches verbose agent output along with every other cause of oversized pull requests, so adjust it up or down depending on how many flagged PRs turn out to have a real issue.

Pair PR complexity with PR review time, which runs from when a pull request opens to its first review from another person. A sustained rise in both may suggest that AI-generated volume is outpacing your review capacity.

Spotting high pre-merge churn

Pre-merge churn rate is lines added that never make it to the final merge, divided by total lines including both churned and merged lines. minware's rule of thumb is to investigate individual PRs above 50% and keep the average below 30%. A rising average points to code that needed substantial revision after initial generation, often a sign of open-ended prompts or thin up-front planning.

Very low churn needs a check too. For a developer or team with less than 5% pre-merge churn over a longer period, look at overall commit frequency. That confirms people are committing regularly and pushing every commit. It can also surface lax code review.

Identifying untracked code changes

The clearest leading indicator of governance breakdown is code bypassing your planning process entirely: branches, commits, and PRs with no ticket reference.

Start with commit messages, and look for messages describing what changed without explaining why.

Then calculate PRs Traceable to Ticket, the metric minware tracks for this: pull requests merged into a main branch with a link to a valid ticket, divided by all pull requests merged into a main branch. A sustained drop below your team's historical baseline means more work is happening outside planned tickets.

This matters because AI makes it easy to generate code for problems that weren't worth solving. A refactor an agent produces on request can look plausible enough to merge without anyone asking whether it was on the roadmap.

minware's Best Practices report tracks PRs Traceable to Ticket as a hygiene metric because it surfaces this pattern.

Confirming with lagging indicators

Lagging indicators help confirm whether leading indicator signals were real. These metrics move weeks to months after code merges, when the maintenance cost of AI slop becomes visible in quality, efficiency, and churn metrics.

Detecting rising bug rates and change failures

Bug rate is the broader of the two quality signals. It divides all bugs created by pull requests merged, so it catches quality problems that never trigger a production incident. A bug rate that rises with token spend could indicate slipping quality. A regression of bug rate against token spend shows whether that's happening, though bug rate can also rise when AI is used to find more bugs.

Change failure rate is one of five DORA metrics. DORA defines it as the ratio of deployments that require immediate intervention following a deployment. DORA moved from four metrics to five in 2024, adding deployment rework rate.

By default, minware calculates change failure rate as highest-priority bugs divided by pull requests merged to a main branch. Teams that want DORA's deployment-based version can reconfigure the denominator to production deployments from Git tags or CI/CD pipeline runs. A rising change failure rate signals a more severe problem needing immediate attention.

Detecting longer bug lead times

Bug rate counts how many bugs are created. Bug lead time shows how hard they are to fix. It runs from when a bug is reported to when it's fixed, by default from ticket creation to ticket resolution for bug tickets.

Bug lead time matters for AI slop because the code behind the bugs is often code nobody on the team fully understands. When it breaks, the fix takes longer. A rising bug lead time alongside a rising bug rate means defects are both more frequent and harder to resolve.

Detecting AI slop via post-merge churn

Post-merge churn rate catches code that passed review but didn't hold up once merged. It divides the merged lines removed within 14 days by the total lines merged, weighted by line count so larger PRs count for more. minware's rule of thumb is to investigate developer-week data points above 15%.

Elevated post-merge churn can indicate code that looked correct but failed in production use. Sharply elevated rates may signal AI producing plausible implementations that don't match actual requirements.

Read it alongside pre-merge churn rate from the leading indicators. High pre-merge churn means the code needed rework before merge, while high post-merge churn means the rework arrived after merge.

Tracking slower ticket cycle times

Ticket cycle time runs from when a ticket is first marked in progress to when it's marked done. When AI slop accumulates, cycle time stretches even as code output increases. The extra code creates review burden and integration complexity that slows overall flow.

Track cycle time at the 50th and 90th percentiles. If the 90th percentile stretches sharply (for example, from roughly eight days to fifteen) while the median moves only slightly, outliers are likely driving the increase. Check whether those outliers are tickets where AI-generated code needed substantial rework.

The leading and lagging indicators work together to surface AI slop at different stages:

Indicator type	Metric	What it can signal	Data source
Leading	Ticket size	Oversized, underspecified work	Project management
Leading	Ticket WIP	Context switching and overloaded people	Project management
Leading	PR complexity (added lines)	Verbose or oversized changes, heavy review load	Version control
Leading	Pre-merge churn rate	Code requiring substantial revision	Version control
Leading	PRs Traceable to Ticket	Work bypassing planning	Version control plus project management
Lagging	Bug rate	More bugs filed per pull request merged	Project management plus version control
Lagging	Bug lead time	Bugs taking longer to fix	Project management
Lagging	Change failure rate	Severe quality problems	Project management plus version control by default
Lagging	Post-merge churn rate	Code requiring quick fixes	Version control
Lagging	Ticket cycle time (50th and 90th percentile)	Slower flow from start to done	Project management

Separating slop from normal churn with regression

Regression analysis helps separate AI slop from normal development churn. You correlate token spend against delivery outcomes and read the slope. This is the same methodology used for proving AI return on investment (ROI), run in reverse as a detection framework.

Correlate token spend with delivery metrics

Token spend is the dollar cost of AI tool usage, a separate data point from token consumption. OpenTelemetry is an open-source observability framework that AI coding tools such as Claude Code or Codex use to export usage telemetry. Most AI tool APIs and OpenTelemetry integrations report what token cost would be on a consumption-based plan, which may differ from what you actually pay on a flat-rate plan.

The regression setup: plot roadmap value delivery per person-day on the Y-axis against token spend per person-day on the X-axis. Each data point represents one team for one week or sprint, and a reliable read needs enough data points across teams and time periods to show a consistent trend. A positive slope suggests spend is buying delivery, while a flat or negative slope means delivery isn't rising with spend, which is the pattern to investigate. A low R-squared value means token spend explains little of the variation in delivery, without saying what does.

Name the standing confounders whenever you draw a conclusion: engineers reach for AI on tasks where it's likely to help, and early adopters are often already the strongest performers.

The regression covers non-AI work too, since it carries zero token spend and sits at the zero-spend endpoint without a separate non-AI comparison group. The methodology is documented in minware's Lean AI Framework.

Track token spend as a continuous variable

AI-vs-non-AI cohort comparison doesn't hold up as a check. Non-AI groups are disappearing as adoption spreads, and the ones that remain are prone to selection bias. The more durable approach treats token spend as a continuous variable across teams and time periods. Compare team-week or team-sprint data points where spend varies naturally.

minware's Lean AI Framework regresses three outcomes separately against token spend: roadmap value delivery, on-time delivery rate, and overhead cost. Each metric gets its own regression.

The supporting check is each team's trend over time. Track roadmap value delivery for each team month over month, since a team whose delivery stays flat while token spend climbs is showing the pattern AI slop leaves. Once confounders such as harder work or more interruptions are ruled out, ticket cycle time and bug rate help explain why. If cycle time stretched and bug rate climbed during the same period, the team may be generating code faster than it can validate it.

Auditing your delivery data in six steps

A systematic audit takes six steps. Run it manually with exports from your existing tools, or automate it with a platform connecting the data sources.

Step 1: Export your current delivery data

Export data from version control, project management, and AI coding tools, such as Claude Code, Cursor, GitHub Copilot, or Codex, for the last six months. You need commit timestamps with sizes, PR open and merge times, ticket status transitions with story point estimates, plus token spend by user. Confirm the export includes full ticket status history, and that commit authorship maps to identifiable contributors.

Step 2: Sync your engineering data

Match identities across systems by building a lookup table mapping email addresses or usernames to a canonical identity. Link commits and PRs to tickets using branch naming conventions, PR descriptions, or explicit ticket IDs. Where structured links don't exist, use time-based correlation: match commits to tickets the same person had in progress at that time. Then normalize each tool's export into one shared format, so the same concept, such as a contributor or a day of token spend, means the same thing across every source.

Step 3: Scan for leading indicators

Calculate the leading indicators: average ticket size, ticket WIP per person, PR complexity, pre-merge churn rate, plus PRs Traceable to Ticket. Flag teams showing two or more warning signs at once: tickets worth a week or more of one person's time, ticket WIP above 1.5, pull requests above 500 added lines, average pre-merge churn above 30%, or a drop in PRs Traceable to Ticket below the team's baseline.

Step 4: Confirm with lagging indicators

Calculate the lagging indicators: bug rate, bug lead time, change failure rate, post-merge churn rate, plus ticket cycle time by percentile. Chart each one by team as a trend across the six months of data. Treat a sustained rise in bug rate or change failure rate as confirmation that the leading signals translated into real quality problems.

Step 5: Measure AI ROI against delivery data

Run the regression analysis: roadmap value delivery per person-day against token spend per person-day, grouped by team and week. A slope near zero or negative is the signal to investigate. Rule out confounders first, such as teams taking on more complex work or absorbing more disruptions, then check bug rate and ticket cycle time to see whether slop explains the flat line.

Step 6: Report AI findings to leadership

Package your findings as a trend report: leading indicators that surfaced early, lagging indicators that confirmed damage, and the regression showing whether spend correlates with delivery. Link every flagged metric to the actual pull requests or tickets behind it. Leadership needs to see that the problem is concrete and addressable.

The build vs. buy decision determines whether you run this methodology manually or automate it:

Criterion	Build custom pipeline	Use minware
Setup time	Typically multiple weeks of engineering time for initial extract, transform, load (ETL) work across version control, project management, CI/CD, and AI tools	Hours for the initial historical backfill, depending on repository size and data volume
Entity resolution and normalization	Manual lookup tables for identities, custom time-based logic to link commits, tickets, and agent sessions, plus custom code to normalize each tool's export format	Automatic relationship recovery, identity resolution, and data normalization in the patented hypercube data model, which links every entity from every data source
Ongoing maintenance	Continuous, someone owns API changes and stakeholder questions indefinitely	Vendor handles API changes and answers calculation questions
Metric transparency	Depends on how well your team documents the custom code	Visible formulas in the UI, editable via configuration or customer success
Customization	Full control over calculations	Pre-built metrics with breakdowns by any dimension through the hypercube data model, with most changes handled by customer success, typically in a single call and about 24 hours
Cost	Engineer time allocated to data pipeline maintenance, at a median software developer wage of $135,980 a year, as of May 2025	$25 per contributor per month for Professional, $45 per contributor per month for Enterprise (50-contributor minimum)

Running detection and acting on the results

The detection checklist condenses the methodology into weekly, monthly, and quarterly checks. The two sections after it cover what to do when a check fires.

Working through the weekly, monthly, and quarterly checks

Use this checklist to audit your codebase for AI slop on a rolling basis. Apart from the framework's rules of thumb for ticket size, ticket WIP, PR complexity, and churn, each threshold is relative to the team's own baseline.

Weekly checks (leading indicators):

• Review ticket size, flag any ticket worth a week or more of one person's time

• Track ticket WIP per person, flag values above the 1.5 target

• List pull requests above 500 added lines, and have a senior engineer check whether each should have been split

• Calculate pre-merge churn rate, flag an average above 30% or individual PRs above 50%

• Measure PRs Traceable to Ticket, flag a sustained drop below the team's historical baseline

Monthly checks (lagging indicators):

• Track bug rate and change failure rate trends by team, flag a sustained rise

• Track bug lead time, flag a sustained rise alongside bug rate

• Calculate post-merge churn rate, flag developer-week data points above 15%

• Review ticket cycle time at the 50th and 90th percentiles, flag a 90th percentile pulling away from the median

Quarterly checks (regression analysis):

• Run a regression of roadmap value delivery per person-day against token spend per person-day

• Track on-time delivery rate trend by team

• Analyze bug rate and change failure rate trends across all teams

Adding process gates and guardrails

Once you've detected AI slop, remediation focuses on process gates that catch low-quality code before it merges. Enforce ticket linkage requirements for all AI-generated commits. Break down tickets worth a week or more of one person's time before work starts. Have a senior engineer review pull requests above 500 added lines, and split those that bundle changes that could be reviewed independently.

Every detection metric also needs a guardrail that catches gaming: a second metric that shows whether gains in the primary metric are real. Pair roadmap value delivery with bug rate to expose corner-cutting on validation. Pair PR complexity with PR review time, since splitting a change into dependent pull requests that can't be reviewed on their own only hides its complexity. Splitting pull requests also lowers bug rate and change failure rate when pull requests merged is the denominator, so consider a denominator such as story points completed.

The manager holds final judgment on metric interpretation. Metrics inform that judgment without replacing it, which removes most of the incentive to game a number.

Recalibrating AI coding workflows

Recalibrate when the regression slope stays flat or negative across multiple consecutive quarters despite process adjustments. Check diagnostic metrics to find the bottleneck. Rising PR review time points to review capacity, and a climbing bug rate points to validation gaps. Rising ticket cycle time with stable PR lead time means the delay sits outside the pull request, such as waiting before coding starts or in stages after merge like deployment or user acceptance testing.

Then find where the slop concentrates. Break down metrics by repository to see which repositories combine high PR complexity, low ticket linkage, and rising post-merge churn rate. Code-based metrics such as commits and PR lead time break down cleanly by repository, while ticket-based metrics such as roadmap value delivery break down more cleanly by project. Legacy repositories are a good place to look first, since an agent has no record of the design decisions behind older code.

Adjust prompt templates and tool configuration for the repositories with the worst patterns. Legacy codebases may need more explicit architectural constraints in prompts, while greenfield projects can usually take more agent autonomy.

Catching AI slop before it compounds

Delivery data catches AI slop earlier than production incidents do. Leading signals move first, quality metrics confirm the pattern over the following months, and the token spend regression tells you whether the extra AI spend is turning into delivery. Engineering leaders who run this audit can walk into the next exec review with the specific teams, repositories, and pull requests behind the trend.

The six-step audit and the checklist above run on spreadsheet exports alone. If you would rather automate the data connection and regression analysis, start a 14-day free trial at minware.com with no credit card required, then connect your first data source to run this detection methodology against your own engineering data.

FAQs

What is the difference between AI slop and technical debt?

Some technical debt is a deliberate tradeoff, taken on to make a release with the cost known up front. AI slop is broader: any tokens or human time that don't turn into delivered value. The slop that reaches the codebase is closer to inadvertent debt: code that merged without adequate review of its architectural fit, so nobody chose the cost. Both require remediation, but AI slop is harder to track because it doesn't come with a documented decision.

Can you detect AI slop without AI tool telemetry?

Yes. Version control and project management data show leading indicators like ticket size, PR complexity, pre-merge churn rate, plus ticket linkage, regardless of whether you have AI usage metrics. Commit and pull request messages can also show an AI tool as author or co-author, with no AI integration needed. The token spend regression requires AI tool usage data from vendor APIs or OpenTelemetry, but you can detect delivery problems from outcome metrics alone.

How long does it take for AI slop to show up in delivery metrics?

Leading indicators can show it within weeks of the code being written, before it reaches customers. Lagging indicators like bug rate, bug lead time, and change failure rate take weeks to months to move measurably. The regression analysis becomes more reliable as it accumulates data across more teams and time periods, with roughly 20 to 30 data points needed for a reliable read.

How often does AI-generated code introduce security vulnerabilities?

Veracode's 2026 GenAI Code Security Report found that in roughly 44% of tasks, AI code generation introduced a risky security vulnerability in testing. Results vary by model, with the best performer passing 68% of security tasks while more than half of tested models sat at 50% to 53%.

How do I separate AI slop from normal development churn?

Normal churn shows stable ticket linkage, consistent PR complexity, and steady cycle times. AI slop shows rising PR complexity, dropping ticket linkage, and flat delivery despite rising code volume. The regression of delivery outcomes against token spend is the strongest separator, and minware runs it in pre-built AI impact reports. Normal churn shows no consistent relationship with spend, while AI slop shows up as flat delivery alongside rising spend.

Key terms glossary

Bug lead time: The time from when a bug was reported to when it was fixed. By default, calculated as the time from ticket creation to ticket resolution for all bug tickets.

Bug rate: Bugs created divided by pull requests merged, by default. A quality metric capturing the effect of quality problems on delivery.

Change failure rate: By default in minware, highest-priority bugs divided by pull requests merged to a main branch. Configurable to DORA's deployment-based definition, the share of deployments requiring immediate intervention such as a rollback. One of five DORA metrics.

Deployment rework rate: The ratio of deployments that are unplanned but happen as a result of an incident in production. DORA's fifth metric, added in 2024.

On-time delivery rate: Estimated roadmap item duration divided by actual duration, capped at 100% per item. Estimated duration runs from when the item is marked in progress to its original due date, and actual duration runs from in progress to done.

Overhead cost: Personnel cost and AI token spend going to work not counted in roadmap value delivery, such as bugs or tickets with no parent epic.

Post-merge churn rate: Merged lines removed within 14 days, divided by total lines merged, weighted by line count so larger PRs count for more.

Pre-merge churn rate: Lines added across a pull request's commits that never make it to the final merge, divided by total lines including both churned and merged lines. Investigate PRs above 50% and keep the average below 30%.

PR complexity: The risk and review effort associated with a pull request, measured by default as the number of added code lines. minware recommends a 500-line threshold as a starting point for review.

PR lead time: The time from a pull request's first commit to deployment. Deployment defaults to when the code merges into a main branch, and can be configured to use Git tags or CI/CD deployment pipeline runs.

PR review time: The time from when a pull request opens to when it receives its first review from another person.

PRs Traceable to Ticket: Pull requests merged into a main branch with a link to a valid ticket, divided by all pull requests merged into a main branch.

R-squared value: The share of the outcome metric's variation explained by the input metric, on a 0-to-1 scale.

Regression slope: The change in the outcome metric per unit change in the input metric in a linear regression. For AI ROI analysis, this represents the change in delivery outcome per unit change in token spend.

Roadmap value delivery: The expected value of completed work items. Configurable to any field where a team sets explicit value estimates, it defaults to story points completed on tickets that are not bugs and have a parent epic when no value field is set.

Story points completed: The total of the story points field for all completed tickets in a time period.

Ticket cycle time: The time from when a ticket is first marked in progress to when it's marked done.

Ticket size: A ticket's story point estimate, by default. Tickets worth a week or more of one person's time should almost always be broken down.

Ticket WIP: The total time during a day that a ticket is in progress and assigned to a person, divided by a day. minware recommends a target of 1.5 or below.

Token spend: The dollar cost of AI coding tool usage, typically reported as what the tokens would cost on a consumption-based pricing plan. May differ from the amount actually paid on flat-rate plans.