Copilot Research Asks Who Will Clean Up The Mess
Written by Sue Gee   
Wednesday, 31 January 2024

A whitepaper from GitClear sets out to investigate the quality and maintainability of AI-assisted code compared to what would have been written by a human. It highlights a worrying increase in "code churn" due to reliance on Copilot. 

We've previously reported on devs' enthusiasm for ChatGPT, Copilot and other forms of AI assistance and have reported on claims for improved productivity. Now some research has been published that raises doubts. It comes from GitClear and the findings are reported in a whitepaper:

gitclear titlepage

Backing up the reference in the subtitle to the negative impact Copilot is having on code quality, the report shared a tweet from Adam Tornhill, the author of Your Code as a Crime Scene:

churn

making the comment:

GitHub claims that code is written "55% faster" with Copilot. But what about code that shouldn't be written in the first place? The problem here is that code spends 10x more time being read than being written, according to Robert Martin, author of Clean Code: A Handbook of Agile Software Craftsmanship. Writing bad code faster implies considerable pains for the subsequent code readers.

To investigate the influence of AI-assistance GitClear used its Diff Delta tool to analyze 153 million changed lines of code, authored between January 2020 and December 2023 and projected the trends for 2024:

churntab

GitClear noted that since the introduction of Copilot in 2022 the percentage of 'added code' and 'copy/pasted code' had increased in proportion to 'updated,' 'deleted,' and 'moved 'code commenting:

In this regard, AI-generated code resembles an itinerant contributor, prone to violate the DRY-ness [don't repeat yourself] of the repos visited."

What GitClear finds most concerning is the increase in "churn" which it defines as the percentage of lines of code that are reverted or updated less than two weeks after being authored, showing that it is projected to double in 2024 compared to its 2021, pre-AI baseline:

churnchart

The paper concludes:

There's no question that, as AI has surged in popularity, we have entered an era where code lines are being added faster than ever before. The better question for 2024: who's on the hook to clean up the mess afterward?" 

Hopefully the answer to this is developers aided by Copilot.
AI-powered coding tools are expected to learn from their own mistakes and as they see more and more lines of code they will increasingly distinguish between the good, bad and the ugly. Copilot will be able not only to identify the substandard code that it helped to generate, but also to refactor it.

After all, we are looking to AI to take on the spade work and that includes mucking out the stables as well as providing fresh fodder. 

copilot chat sq

 

More Information

Coding on Copilot

Related Articles

Copilot Chat Improves Confidence and Enjoyment

GitHub Copilot Provides Productivity Boost

The Impact of Copilot Chat

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

 

Banner


Love Learning With Udacity
09/02/2024

To celebrate Valentine's Day, Udacity has a one-week promotion for new subscribers with 40% of the first month's payment. This seems like a good incentive for trying it out. Here we look at four cours [ ... ]



WasmCon 2023 Sessions Now Online
01/03/2024

The recorded session of the premier conference for technical developers and users interested in exploring the potential of WebAssembly are now online.


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Wednesday, 31 January 2024 )