Developers Like Code Assistants Even When They Are Incorrect
Written by Sue Gee   
Wednesday, 05 June 2024

Over half of ChatGPT answers to programming questions contain misinformation, yet the majority of developers are still keen to use AI tools and report both personal satisfaction and increased productivity from using them.

sosurvey

Findings about the adoption of AI tools and developers perceptions of them come from a recent survey by Stack Overflow which asked 1,700 developers across its community about their use of code assistants and the overall effect these tools have on their productivity.

The survey reveals that 76% of participants are already using or plan to use AI code assistants. The roles that use these tools the most amongst professional developers are academic researchers (87%), AI developers (76%), frontend developers (75%) and data scientists (67%).

Asked which code assistants developers use ChatGPT clearly  dominates with a massive 84% of respondents nominating it. GitHub Copilot is in second place with 49% using it.  Visual Studio IntelliCode came in third with 11%, followed by Codeium (7%) and Google Gemini, formerly Duet (4%). Eight products are used between 3% and 1% of survey participants and a further 4 by fewer than 1%.

sosurveytools

Notice that the top two bars of this chart have been truncated and that the scale applies only from Visual Studio Intellicode.

In terms of ease of use, the top scores were awarded to Codeium (84%), GitHub Copilot (76%) and ChatGPT (61%) and the same three in order came out with the highest satisfaction scores (86%, 72%, 65% respectively) and highest increase in productivity (x2.5, x2.3, x2.2 respectively).

sosurveytools2

In the Stack Overflow survey 38% of developers report code assistants provide inaccurate information half of the time or more. This corresponds well with a finding by researchers from Purdue University presented last month at the Computer-Human Interaction conference that 52 percent of programming answers generated by ChatGPT contain misinformation.

The study addressed the question Is Stack Overflow Obsolete? by investigating how well ChatGPT performs in answering programming questions compared to Stack Overflow. Specifically the researchers took 517 programming questions asked on Stack Overflow and examined the correctness, conciseness and consistency of ChatGPT answers. They found that 52 percent of ChatGPT answers contain misinformation, 77 percent of the answers are more verbose than human answers, and 78 percent of the answers suffer from different degrees of inconsistency to human answers.

Despite these shortcomings, by conducting a small user study with 12 participants, they found that ChatGPT answers were preferred 35% of the time due to their comprehensiveness and their use of well-articulated language style and they overlooked the misinformation in the ChatGPT answers 39% of the time. which led the Purdue researchers to state:

This implies the need to counter misinformation in ChatGPT answers to programming questions and raise awareness of the risks associated with seemingly correct answers.

As we reported recently, see OpenAI Enriched By Stack Overflow, Stack Overflow is playing its part in this endeavour by partnering with OpenAI and giving it access to its entire repository of programming questions and answers. Another way to improve ChatGPT answers is to raise the quality of prompts ensuring that they contain clear information and also indicate the risk of a wrong or incomplete answer. 

chatgptsq

How can it be that accuracy is so unimportant to us that we ever prefer the answers of an AI agent to a human?

More Information

Stack Overflow code assistant survey results

Is Stack Overflow Obsolete? An Empirical Study of the Characteristics of ChatGPT Answers to Stack Overflow Questions

Related Articles

OpenAI Enriched By Stack Overflow

AI Helps Generate Buggy Code

Stack Overflow Traffic Slumps As Devs Turn to ChatGPT

Copilot Chat Improves Confidence and Enjoyment

GitHub Copilot Provides Productivity Boost

The Impact of Copilot Chat

Tell A Chatbot "Take a deep breath ..." For Better Answers

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

 

Banner


Eclipse IoT Developer Survey 2024
04/12/2024

The Eclipse Foundation’s IoT Working Group has released the results of its 2024 IoT Developer Survey. Industrial automation and automotive are now the leading industry sectors and connectivity is th [ ... ]



The Feds Want Us To Move On From C/C++
13/11/2024

The clamour for safe programming languages seems to be growing and becoming official. We have known for a while that C and C++ are dangerous languages so why has it become such an issue now and is it  [ ... ]


More News

espbook

 

Comments




or email your comment to: comments@i-programmer.info

 

Last Updated ( Thursday, 25 July 2024 )