Recent research has shown that ChatGPT can give better responses to health-related questions than human physicians.
The study, which was published in the JAMA Internal Medicine, was conducted by licensed healthcare professionals who compared ChatGPT-generated responses to health-related questions to those given by physicians to a certain level of accuracy and empathy.
Surprisingly, ChatGPT responses turned out superior to those provided by actual physicians in 79% of the cases.
In recent years, we have seen remarkable advancements in technology, with AI at the center stage of automation in nearly every sector of the economy due to its potential to revolutionize how we do things.
The most notable advancements have been witnessed in the field of Generative AI, where large language models (LLM) have been used for chatbots and AI co-pilots to supplement human support teams.
With the development and public availability of natural language processing, AI systems that can communicate in natural human language have opened a new era of generative AI, such as ChatGPT, which can give responses based on questions prompted.
Although generative AI was first met with skepticism, with the first models not being as capable as the current versions, AI is rapidly becoming part and parcel of our day-to-day lives. It cuts across every sector of the economy— from manufacturing automation to customer support chatbots.
Now, researchers have found that ChatGPT can produce more acceptable responses to health-related questions than human physicians.
This underscores the rapid development of large language models (LLM), from robotic-sounding responses to empathetic and more human-sounding responses that can surpass the quality of some medical professionals.
The Study
Given that the volume of messages sent by patients to healthcare professionals has exponentially increased over recent years, contributing to healthcare employee burnout, Study author John W. Ayers and his team wanted to find out whether AI systems like ChatGPT can help respond to patients’ queries adequately.
According to the researchers, the study aimed to find out the effectiveness of using ChatGPT for responding to healthcare-based questions, as some physicians were already integrating ChatGPT-backed chatbots on their platforms to respond to questions, despite having no prior research about such uses in the healthcare sector.
Since ChatGPT was not developed specifically for the healthcare industry, the researchers wanted to assess the quality of the response ChatGTP generated based on typical questions patients ask healthcare professionals.
The researchers picked a sample of 195 questions and answers from conversations between physicians and patients from Reddit’s r/AskDocs forum, a sub-Reddit group where patients post questions and physicians respond to them.
The questions from Reddit were prompted in ChatGPT, and responses were evaluated by a team of health professionals drawn from pediatrics, internal medicine, geriatrics, infectious disease, oncology, and preventive medicine based on the quality of information and empathy without knowing the sources of the responses.
Unexpected Outcome
In the study, the researchers found that 27% of the physicians’ answers were below the accepted quality, with only 3% of ChatGPT responses regarded as of low quality.
The evaluators also rated ChatGPT-generated answers as better in 79% of the cases. On average, ChatGPT results were rated as more empathetic compared to those from physicians.
However, it’s important to note that the responses studied were from volunteer physicians on a free platform.
Although the study provides insights into the quality of healthcare-related responses given by ChatGPT, the authors concluded that studying the inclusion of AI assistants in patient messaging workflow is key to unlocking AI’s potential to improve outcomes for both physicians and patients.
