Research note
Read the update before repeating the productivity number
Use METR’s early-2025 study and its 2026 follow-up to understand populations, selection effects and changing evidence.
THE SHORT ANSWER
A study estimates an effect in a setting. A later update can change what that estimate means.
Why it matters
A memorable percentage can outlive the conditions that produced it. Before using a productivity result to buy a tool or change a workflow, identify who was studied, which work they did and whether newer evidence changes the interpretation.
Read the evidence
METR’s July 2025 report studied 16 experienced open-source developers completing 246 repository tasks. Allowing early-2025 AI tools increased completion time by 19% in that setting. Its February 2026 update says newer speedups appear plausible, but participation, task selection and concurrent-agent timing make the follow-up estimates unreliable. Neither result is a universal verdict on today’s coding tools.
How to use this resource
Read both reports as one evidence chain. For your own future evaluation, define the completed change rather than counting code output.
- Record the population, task type, tool versions and comparison condition.
- Read exclusions, confidence intervals and follow-up notes before the headline.
- Include review, debugging, waiting and rework in the task boundary.
- Separate elapsed time, active human effort and final quality.
- Keep unsuccessful tasks in view instead of reporting only finished demonstrations.
Limits and evidence
These are historical research findings and methodological lessons. CooScope did not participate in or reproduce either study. Experienced maintainers of mature repositories are not interchangeable with beginners, small website owners or every kind of software work.
Put it to work.
- Read the original and its follow-up.
- Keep the population attached to the number.
- Define complete work, including review.
- Do not turn a snapshot into a permanent ranking.
Go to the source.
Original material, with its own scope and publication date. This note is an explanation, not a CooScope performance test.
- METR — Early-2025 developer productivity study (10 July 2025)Link / source review: 2026-10-04
- METR — Developer productivity experiment update (24 February 2026)Link / source review: 2026-10-04