Study suggests Goldman Sachs was right about AI: it's not 'replacing' coders
Many in finance think coding is going out of fashion due to AI. UBS' chief economist Paul Donovan thinks the skill is a 'stranded asset', while Barclays' global head of FIG banking suggests one AI powered coder could do the work of eight workers not imbued with the benefits of AI. Goldman Sachs didn't buy the hype, however, and a recently published paper affirms that the reality may be much more marginal.
'The Effects of Generative AI on High Skilled Work', a paper by researchers from MIT, Princeton, Microsoft and more, looks at generative AI in the workplace, rather than in a controlled environment. The workplaces it scrutinises include Microsoft, tech consultancy Accenture and a "fortune 100 electronics manufacturing firm", examples of which are Hewlett Packard and NVIDIA.
Under the study, 4,867 software engineers (including a control group) were mandated to implement Github CoPilot into their workflow. There was a 26% increase in pull requests by engineers using the technology - a signifier that more tasks had been completed. The implication is that four AI-powered engineers could do the same number of tasks as five non-AI engineers...
In Goldman's generative AI report, 'Too much spend, too little benefit,' the bank's equity research head Jim Covello said AI is good at making code more efficient, but "estimates of even these efficiency improvements have declined."
At Microsoft, benefits to the process of adding code were even more marginal. Commits, which represent a successful change to a developer's codebase, were up 18%. Builds, which check whether new code was successful or not, were up 23%. The implication is that developers write around 20% more code when using AI. Build success rate, used to measure the quality of code, was down a marginal 1.3%.
Accenture's result was even more peculiar. It had the smallest number of developers at 316, and had a massive 92% increase in the number of builds when AI was deployed. However, it also saw its build success rate decline 17.4%.
The study also found that AI was more effective at raising task completion for junior engineers than for senior ones. This is likely because junior developers are often left to handle the more mundane tasks that AI would be well-equipped to handle.
It found a similar distinction between short-tenured and long-tenured employees, which aligns with what we've seen in the industry. At Goldman Sachs, for example, AI is used to both write code and explain it, in cases where new engineers are carrying on the work of ones that have left.
Have a confidential story, tip, or comment you’d like to share? Contact: Telegram: @AlexMcMurray, WhatsApp: (+1 269 237 3950). Click here to fill in our anonymous form, or email editortips@efinancialcareers.com. Signal also available.
Bear with us if you leave a comment at the bottom of this article: all our comments are moderated by human beings. Sometimes these humans might be asleep, or away from their desks, so it may take a while for your comment to appear. Eventually it will – unless it’s offensive or libelous (in which case it won’t.)
