ChatGPT fails to understand software engineering concepts, answers 52 per cent of questions incorrectly.

A recent study has highlighted that ChatGPT, the AI chatbot, exhibited an error rate of 52 percent when responding to software engineering questions, failing to grasp the underlying concepts.

Thank you for reading this post, don't forget to subscribe!

Launched in November 2022, ChatGPT has been gaining attention for its remarkable capabilities, excelling in competitive exams, aiding students with assignments, composing music and poetry, and more. While its versatility has been celebrated, concerns about the accuracy of its responses have persisted. According to a recent study, ChatGPT’s precision is once again scrutinized, as it was found to answer 52 percent of software engineering questions inaccurately.

Researchers at Purdue University conducted an analysis, examining 517 questions from Stack Overflow. Their findings revealed that 52 percent of the chatbot’s responses were incorrect, and 77 percent were deemed excessively verbose. The study identified that the incorrect answers primarily resulted from ChatGPT’s inability to grasp the underlying concepts of the questions.

Furthermore, even when ChatGPT understood a question, it struggled to arrive at a viable solution, leading to a higher incidence of conceptual errors. The research team criticized the AI’s limited reasoning ability, noting instances where it provided solutions, code, or formulas without a comprehensive understanding of the outcomes.

The researchers noted that while prompt engineering and human-in-the-loop fine-tuning could aid in helping ChatGPT understand problems to some extent, these methods still fell short in instilling reasoning capabilities into the language model. They emphasized the importance of addressing both the factors contributing to conceptual errors and rectifying issues stemming from the model’s limited reasoning capacity.

In a related matter, an article by Analytics India Magazine revealed a decline in ChatGPT’s user base. In June, the chatbot experienced a decrease in users, and the trend continued in July with a 12 percent drop, reducing the active users to 1.5 billion from the initial 1.7 billion. The report pointed out that OpenAI, the company behind ChatGPT, has yet to achieve profitability and is relying on Microsoft’s investment of USD 10 billion to sustain its operations. The mounting losses raise concerns about the company’s financial viability in the long run, especially considering that it costs approximately USD 700,000 daily to operate ChatGPT. Consequently, if OpenAI does not achieve profitability in the near future, the risk of bankruptcy becomes a looming possibility.

Leave a Comment

Your email address will not be published. Required fields are marked *

Amazon India Today's Deals

Scroll to Top