Measuring The Value Of AI Coding Without Ignoring The Costs

🔍 Read the full analysis: Measuring The Value Of AI Coding Without Ignoring The Costs on ThorstenMeyerAI.com

Before you orderOffer from Amazon

Get the latest gadgets delivered free with Prime

• Fast, free delivery on millions of items
• Prime Video, Amazon Music and more included
• Member-only deals all year

Start your free Prime trial →Free trial for eligible customers · Cancel anytime

As an affiliate, we earn on qualifying purchases.Hide

Engineering teams are reporting AI coding bills from about $150 to $3,000 per developer per month, but recent workplace and vendor data show no consistent link between spending and delivered value. A report published October 23 by ThorstenMeyerAI.com, drawing on interviews at 27 workplaces and other studies, says companies preparing 2027 budgets need to track accepted work, rework and failures—not just tool usage or pull-request counts.

The report summarizes an October 20 analysis by Gergely Orosz of The Pragmatic Engineer, based on conversations with engineering leaders and senior engineers at 27 workplaces, plus aggregate data from AI code-review vendors CodeRabbit and Gitar. The source says the cited material comes from the freely readable portion of Orosz’s article. Reported monthly budgets range from a $150-per-developer cap at one large company to about $3,000 per developer at a two-person European agency. Company size alone did not predict spending.

Output measures differ as sharply as costs. CodeRabbit reported that developers on its platform created 113% more pull requests per week than a year earlier. In a study of tens of thousands of engineers during an early-2026 rollout, Microsoft found adopters merged about 24% more pull requests over four months than they otherwise would have. Those figures measure activity, not necessarily customer impact. The report says companies in Orosz’s sample ranged from flat pull-request frequency to increases of five to ten times, with some producing larger PRs rather than more of them.

Controlled research gives a less settled picture. METR’s early-2025 study found experienced open-source developers took 19% longer with AI. Its February follow-up estimated an 18% speedup for returning developers and a 4% speedup for new developers, but both estimates had confidence intervals that included zero. METR described the data as unreliable, in part because many participants held back tasks they preferred to do without AI. These findings do not establish a single productivity effect across teams.

Budgets Need Outcome Measures

The findings matter as companies decide whether to renew subscriptions, expand access or absorb usage-based bills. The source reports that some discounted subscriptions are due to expire, with companies anticipating bills could rise five- to tenfold if they do not change plans or usage. It also says API-plan spending at some companies peaked in June and July and has roughly halved since, a change Orosz attributed to Opus 5.5 and a move to open models. Those are reported developments, not a universal cost trend.

More activity can create costs elsewhere. Larger pull requests may take longer to review, while extra generated code can increase maintenance and security work. The source cites Veracode’s latest test, in which 44% of AI coding tasks produced a known vulnerability. That result is a test finding; it does not establish the vulnerability rate in every company’s production code. Still, it highlights why teams should track review capacity and security alongside speed.

The report recommends measures tied to work delivered: cost per accepted change, rework, change failure rate, and whether teams close more user stories than arrive. These metrics can help leaders judge whether an AI tool improves delivery rather than simply increasing token consumption or PR volume. They also make comparisons between teams more meaningful when work complexity differs.

From Usage Caps to Value

The cost question is becoming more immediate as billing arrangements change. The source says GitHub moved every Copilot plan to usage-based billing on June 1, charging tokens at each model’s API rate and removing the cheaper fallback model. GitHub explained the change by saying, “Agentic usage is becoming the default” and brings higher computing and inference demands. That policy makes consumption a more direct budget concern for organizations using Copilot.

Some teams describe specific gains that aggregate output measures may miss. At a SaaS company with about 600 engineers, a staff engineer said customer-issue triage that could take a day sometimes takes minutes of agent work plus up to an hour of human checking. Code archaeology and prototypes were also reported to take less time. Ix Technologies, a small agency using a uniform C# and Blazor stack, said it is clearing a long-standing backlog. These are company accounts, not controlled comparisons, and should not be treated as typical results.

Results from METR also illustrate why estimates need careful qualification. Its survey of 349 technical workers found a median self-reported change of 1.4 to 2 times in the value of work, but METR said participants in its 2025 research had overestimated AI’s effect by 40 percentage points. Self-reports, vendor platform data and controlled studies measure different things, so their numbers are not directly interchangeable.

Productivity Gains Remain Uneven

There is no common measure across the cited evidence for what counts as productivity or delivered value. Pull-request counts, merged changes, survey answers and task completion capture different outcomes. The reported increases do not show whether code required more review, led to more defects, or improved customer results. Nor do the examples establish that similar gains would carry over to teams maintaining complex legacy systems.

The source does not provide a standardized comparison of total costs across companies, including model bills, subscriptions, human review, rework and security remediation. It also does not resolve how much of reported output growth is caused by AI rather than changes in staffing, workflow or the kinds of tasks being measured. The 44% Veracode result applies to its test tasks; the material does not give enough detail to infer an equivalent rate for all AI-assisted software development.

Teams Set 2027 Spending Plans

Companies are preparing 2027 budgets while weighing renewals, usage caps and the end of some discounts. The source does not identify a single upcoming industry benchmark or timetable for resolving the ROI question. For now, it points engineering leaders toward tracking costs against accepted changes, rework, failures and story completion, then comparing results across similar types of work.

As billing shifts toward usage and agent-based workflows demand more computing, spending reports may become easier to obtain while value remains harder to establish. The next useful evidence will need to show not only how many changes AI-assisted teams produce, but how many are accepted, maintained and delivered without added failure or review burdens.

Key Questions

How much are companies spending on AI coding tools?

The report describes budgets ranging from about $150 to $3,000 per developer per month among cited workplaces. Those examples are not a representative industry-wide estimate.

Do the reported pull-request increases prove AI coding is more productive?

No. Pull-request counts measure activity, and the Microsoft study authors cautioned that a merged PR is not the same as delivered value. The cited studies also use different methods and report varying results.

Where does the report say AI coding has helped most?

Company accounts in the report point to triage, code archaeology, prototyping and backlog cleanup, particularly on simple, uniform technology stacks. These are reported examples, not guarantees for other teams.

What should engineering teams measure?

The report recommends tracking cost per accepted change, rework, change failure rate and whether completed user stories outpace incoming work, rather than relying on usage or PR volume alone.

Is the reported 44% vulnerability figure an industry-wide rate?

No. The source attributes the figure to Veracode’s latest test of AI coding tasks. It does not establish that 44% of AI-assisted production code across companies contains a known vulnerability.

Source: ThorstenMeyerAI.com