29th April 2023

ChatGPT outperforms human doctors in medical advice

When measured on quality and empathy, OpenAI's ChatGPT is shown to outperform human doctors in responding to patient queries.

Can artificial intelligence provide responses to patient queries that are of comparable quality and empathy to those written by physicians? That is the question a group of scientists set out to find the answer to, in new research published yesterday. The result of their study – which appears in The Journal of the American Medical Association (JAMA) – is a very clear "yes".

The team used ChatGPT, the revolutionary chatbot developed by OpenAI, which has taken the world by storm since its release in November 2022. Led by Professor John Ayers from the University of California San Diego (UCSD), they drew 195 random questions from patients in a social media forum, and then compared human physician and ChatGPT responses, blinded to the origin of each.

The panel preferred chatbot answers over physician answers, rating them significantly higher for both quality and empathy. The AI's responses had "good" or "very good" quality 79% of the time, compared with 22% of doctors' responses, while 45% of the AI's responses appeared "empathetic" or "very empathetic" compared with just 5% of doctors' replies.

"The opportunities for improving healthcare with AI are massive," said Professor Ayers. "AI-augmented care is the future of medicine."

"These results suggest that tools like ChatGPT can efficiently draft high-quality, personalised medical advice for review by clinicians, and we are beginning that process at UCSD Health," said paper co-author Dr Christopher Longhurst.

"ChatGPT messages responded with nuanced and accurate information that often addressed more aspects of the patient's questions than physician responses," said Jessica Kelley, a nurse practitioner with San Diego firm Human Longevity and study co-author.

"I never imagined saying this," added Aaron Goodman, MD, Associate Clinical Professor at UCSD School of Medicine and study co-author, "but ChatGPT is a prescription I'd like to give to my inbox. This tool will transform the way I support my patients."

Comments »

If you enjoyed this article, please consider sharing it: