NBER study of 100,000 developers finds AI tools boost commits by up to 180% but shipped software by 30% to 50%

Big paper on AI coding agents using Github & other data

The auto-complete tools (Copilot) led to 2.2x more code, local agents like original Claude Code led to 7.4x, & current remote coding agents 17.3x(!)

But human bottlenecks in coding means actual releases "only" went up 30%

Big paper on AI coding agents using Github data

The early auto-complete tools (Copilot) led to 2.2x as much code, local agents like original Claude Code led to 7.4x, & current remote coding agents 17.3x(!)

But human bottlenecks in coding means actual releases "only" went up 30%

Here's the paper, which confirms that big changes are happening very quickly. Some really interesting approaches to teasing out what is happening in the software development cycle as well: https://papers.ssrn.com/sol3/papers.cfm?abstract_id=6843118

Link to paper: https://www.nber.org/papers/w35275

@emollick Here is a great example of an agents beating the state of the art quantum algorithms.

It’s because it’s fully software verifiable.

@twst12612648 @florianederer @_vonarchimboldi Tyler Cowen said something like this: humans are the biggest bottlenecks, more intelligence doesn't mean faster growth. intelligence is just one bottleneck. world already added billions of additional human minds through the economic rise of China, ASEAN, even India.

@florianederer @arpitrage Which technological revolutions involved things that were already done on a large scale getting massively cheaper and faster to do, vs things that were not economically feasible to scale becoming possible, vs things that weren’t even technically possible before…

@emollick 17x the code. 1.3x the releases. The bottleneck sent a bill for its services.

@florianederer @albrgr I assume the “projects / releases” stat doesn’t include internal facing apps, which have much lower bar for deployment than customer facing apps

There’s also a distributional shift here.

People with no technical skills are now making a run at app development.

People with professional development skills are going after much more ambitious projects.

What’s being *attempted* is going up relative to the skill of the development. This is going to manifest in software gains that take off more slowly than if everyone were simply doing the same shit they were before.

@Jtyles What a terribly AI answer. You should prompt better. Blocked.

@florianederer It has long been known in software industry that measuring productivity is a pain. You get what you measure. So if you want me to produce more lines of code, I will do that for you.

@florianederer Cc @sebkrier @alexolegimas @soumitrashukla9

@florianederer so the bottleneck shifts to validation, not coding

It took Amazon about seven and a half years (90 months) to become profitable. Founded in July 1994, the company recorded its first profitable quarter in Q4 2001 (earning $5 million) and its first profitable full year in 2003 ($35 million net income

Commercial AI exploded into the mainstream for everyday users with the release of Large Language Models (LLMs) like OpenAI's ChatGPT in late 2022.

Give it five years.

@florianederer the commit count is the tell. AI multiplied the cheap step, writing code, and left the expensive one untouched: proving it works before you ship. that verification gap is exactly where releases stop tracking commits. https://assrt.ai/r/exfn7cdc

@emollick the 30% number is the honest one. bottleneck moved - not writing code anymore, it's reviewing it and shipping it. 17x output still hits the same pipeline.

@florianederer Does this look like a productivity revolution? New apps surge but downloads and ratings are flat. Marginal software is invisible to end users. Consumer surplus is not materialising.

@Noahpinion commit velocity and shipped value decoupling is the vanity metric problem with a new coat of paint

@Noahpinion Probably shows that for now, the bottleneck is still having ideas that are worth realising. AI let‘s you work on more stuff, but the stuff that didn‘t make the cut before AI has a low chance of making a big splash.