To judge AI developer productivity, compare how work moves through a team with how much faster people think AI has made them. Neither feeling nor headline proves a gain. In the 2025 survey from DORA, a software delivery research programme, more than 80% believed AI had increased their productivity. That tells you what respondents thought, rather than how quickly their teams delivered. This companion to episode 2 explains why estimates can miss in either direction, what the research can establish, and how to check one of your teams against its own guess.
What one careful study found
In early 2025, the research group METR ran a randomised trial with 16 experienced developers working on open-source projects they knew well: mature code bases where they had, on average, five years of experience. Across 246 tasks, each task was randomly assigned to allow AI tools or not.
Before they started, the developers expected AI to cut their time by 24%. Afterwards, they believed it had cut it by 20%. Measured, their tasks took 19% longer.
The gap between belief and clock is the point: a small version of what’s often called the AI productivity paradox, where enthusiasm runs ahead of measured gains. These weren’t careless people. They had done the work themselves, and they still misjudged it. But a small trial with experienced maintainers and particular tools cannot tell you what every team will experience.
Why AI can feel faster than it is
Think of taking the back roads to dodge a traffic jam. You never stop, so it feels quicker. You’re always moving, always busy. Then you look at the clock.
METR recorded the developers’ screens and labelled a sample of the recordings. With AI allowed, less of their time went on writing code and searching for information, and more on writing prompts, waiting for answers and reviewing what came back. Those activities can feel like progress. They also take time. The recordings show how time was spent; they do not establish which activity caused the slowdown.
The 2026 update: probably faster, hard to prove
That study used early-2025 tools. In February 2026, METR reported that developers were probably faster with the tools available then, but that its new experiment gave “an unreliable signal”. The reason is telling: 30% to 50% of developers said they were choosing not to submit some tasks, because they didn’t want to do them without AI. The experiment was missing some of the work people thought would benefit most.
So does AI increase productivity at all? Other controlled studies have found gains, of very different sizes:
- A trial at Google, on one standard enterprise coding task with Google’s own tools, estimated about 21% faster. The uncertainty was wide: its best-fit estimate could not rule out no gain.
- A study run partly by researchers from the tool’s maker, on a set JavaScript task, found developers using GitHub Copilot finished 55.8% faster. Its participants estimated a 35% gain: they underestimated, the opposite of the METR group.
So no single AI developer productivity study settles it. Self-estimates miss in both directions, and results depend on the task, the people and the tools. The METR and Copilot versions cited here are arXiv preprints, shared before formal peer review; Google’s trial was peer reviewed and presented at ICSE 2025. Neither a narrow coding task nor a personal estimate measures the whole delivery system.
How most organisations judge AI developer productivity
LeadDev describes its 2026 sample as 600 engineering leaders, although respondents also included engineers without management roles. The most common way respondents assessed AI’s impact on productivity was employee feedback about the tools: 63% use it. Only 22% use change failure and pull request reversion rates: measures of changes that need correcting or undoing.
People’s experience matters, but pressure can shape what they report. It’s a view I hold strongly: “AI makes you go faster” raises the pressure on teams to deliver. If leaders expect a gain, make it safe to report where the tools add work as well as where they save it.
How to measure developer productivity without asking for opinions
Start with three measures of the work:
- Cycle time: how long work takes from start to finish. The Kanban Guide, by Daniel Vacanti and John Coleman, defines it as the time between a work item starting and finishing. Keep those boundaries consistent. Use the median, the middle completion time, so one stuck item doesn’t dominate the result.
- Work item age: how long each piece of unfinished work has been in progress, counted up to today. It shows the queue that completed-work figures cannot reveal.
- Change failure rate: how often a change needs fixing once it’s live. DORA counts the share of deployments needing immediate intervention, such as reversing the change or applying a quick fix.
Notice that none of them measure typing. Microsoft’s New Future of Work Report 2025 cites studies finding that developers spend only between 15% and 25% of their time writing code. In my view, the rest is where delivery is won or lost: review, decisions, testing and rework. Faster coding doesn’t automatically mean faster delivery, the subject of episode 1 and what code review is for when agents write the first draft.
A worked example: guess first, then measure
Imagine a team that has used AI coding tools for six months (an illustrative example, not a real team). You ask how much faster AI has made them. They say about a third. Write that down before looking at the data.
Compare their cycle time for the three months before adoption with the last three months. Suppose the median moved from eight days to seven. In this example, work finishes sooner, but that alone does not show AI caused the difference. Check whether the work became smaller, staffing changed or another improvement happened at the same time.
Their work item age also shows items open for more than three weeks, most waiting for review. Their change failure rate has crept up. That suggests looking at review capacity and batch size: how much work arrives for checking at once. It does not establish that AI caused either problem.
Use the same exercise across your teams, comparing each with its own baseline. A department average could hide a growing queue in one team. If a team is only starting with AI, take its baseline now.
How you’ll know it’s working
Look for conversations that start with a measure and a time period, alongside what changed in the work. A shorter cycle time is useful only if quality holds and the work still serves users. Teams should be able to identify waits, test a small change and check the result without having to defend an AI success story.
For a wider view of where AI is straining delivery across your teams, the free AI Delivery Health Check gives you a score, the area to work on first and one thing to try this week. What the Health Check covers explains how it works.
