ChatGPT was introduced in November 2022 and has been making headlines ever since for its sheer intelligence. The AI chatbot has aced different tests that individuals have put it through, including acing various competitive examinations, assisting students with their homework, acting as a tutor in some situations, writing music and poetry, and much more. However, one problem has been consistent over this time: the accuracy of its responses. According to a new study, the chatbot’s accuracy is once again being scrutinized, since it failed to accurately answer 52% of software engineering questions.
According to an IANS report, a study conducted by Purdue University in the United States called the chatbot’s accuracy into question. Researchers examined ChatGPT’s responses to 517 Stack Overflow (SO) inquiries and discovered that 52% of these responses were incorrect and 77% were “verbose.” The team also discovered that the incorrect answers were mostly caused by the AI chatbot’s failure to grasp the rationale underlying the questions. According to the researchers, even though ChatGPT understood the question, it couldn’t solve it, which contributed to a higher frequency of conceptual errors. The team also questioned the AI tool’s poor thinking.
“In many cases, we saw ChatGPT provide a solution, code, or formula without foresight or consideration for the outcome,” the research team told IANS. “While prompt engineering and human-in-the-loop fine-tuning can help ChatGPT understand a problem to some extent, they are still insufficient when it comes to injecting reasoning into LLM.” As a result, it is critical to understand the causes of conceptual errors as well as to correct errors caused by reasoning limitations,” they stated.
In related news, a recent article in Analytics India Magazine noted a decline in ChatGPT users. In June, the chatbot’s user base decreased. The usage of ChatGPT further decreased the next month. The popular AI chatbot had 1.7 billion users in June, but that number dropped to 1.5 billion in July, a 12% fall. The article continued by stating that OpenAI is currently not profitable and that Microsoft’s investment of USD 10 billion may be keeping the business afloat at this time. This is due to the company’s growing losses and the fact that its funding comes from investor pockets.