IE 11 Not Supported

For optimal browsing, we recommend Chrome, Firefox or Safari browsers.

How often does ChatGPT answer programming questions incorrectly?

Answer: 52 percent of the time.

In the background, the ChatGPT logo and text that says "Welcome to ChatGPT" in white font on a black background. In the foreground a hand holds up a smartphone with the ChatGPT app open on the screen.
If you’re turning to ChatGPT to help you with computer programming, you may want to be extra careful to double-check its answers. A new study has found that 52 percent of the popular chatbot’s answers to computer programming questions contain inaccurate information.

The study was conducted by Purdue University and presented at the Computer-Human Interaction Conference in Hawaii this month. The researchers looked at 517 programming questions on Stack Overflow before feeding them to ChatGPT.

In addition to over half of the bot’s answers containing incorrect information, they found that 77 percent of the answers were verbose. What’s more, the programmers who participated in the study did not always catch the bot’s inaccuracies, overlooking “the misinformation in the ChatGPT answers 39 percent of the time.” However, ChatGPT is still a favorite among programmers, with the study participants preferring “ChatGPT answers 35 percent of the time due to their comprehensiveness and well-articulated language style.”