Artificial Intelligence (AI) models frequently provide incorrect advice to users regarding medical questions, raising concerns about their widespread deployment in the public sphere, according to a Feb. 9 peer-reviewed study published in the journal Nature Medicine.
Lead medical practitioner on the study Dr. Rebecca Payne said in a statement that people should be aware that asking LLMs about their symptoms can be dangerous, as these models may provide incorrect diagnoses.
AI “just isn’t ready” to take the role of a physician, Payne said.
In the study, researchers recruited nearly 1,300 individuals aged 18 or older from the UK. These individuals were presented with a medical scenario and tasked with identifying potential health conditions and recommending a course of action.
Participants were split into four groups. Three groups were provided with an AI large language model (LLM)—GPT-4o, Llama 3, and Command R+—for assisting them in completing the task. The fourth was a control group that was asked to use any methods they typically would use at home to complete the task.
Researchers also fed the scenario and questions directly into the AI models to assess their performance without interacting with participants.
“Tested alone, LLMs complete the scenarios accurately, correctly identifying conditions in 94.9 percent of cases and disposition in 56.3 percent on average,” the study said. Disposition refers to the recommended course of action.
“However, participants using the same LLMs identified relevant conditions in fewer than 34.5 percent of cases and disposition in fewer than 44.2 percent, both no better than the control group.”
Participants in the control group had 1.76 times the odds of identifying a relevant condition as those in the LLM-based groups.
AI was found to have generated several misleading and incorrect pieces of information. In two situations, LLMs initially provided correct responses but later produced incorrect answers when participants provided additional details.
In one case, two users were given opposite advice despite sending similar messages describing symptoms of a subarachnoid hemorrhage.
“In our work, we found that none of the tested language models were ready for deployment in direct patient care. Despite strong performance from the LLMs alone, both on existing benchmarks and on our scenarios, medical expertise was insufficient for effective patient care,” the researchers wrote.
“We recommend that developers, as well as policymakers and regulators, consider human user testing as a foundation for better evaluating interactive capabilities before any future deployments.”
The way users interacted with AI was also deemed “a challenge” to LLM deployment for providing medical advice, researchers wrote. Researchers found that users overall failed to provide the AI with sufficient information to reach a correct recommendation.
In many cases, participants provided partial information. Some users reported their symptoms only after being prompted by the LLM.
In a Feb. 9 statement, the University of Oxford, whose researchers were part of the study, said that current standard evaluation methods for testing LLMs fail to account for the complexity involved when these models interact with people.
Dr. Adam Mahdi, senior author of the study, said the disconnect between benchmark test scores of the LLMs and their performance in the real world should be a “wake-up call” for AI developers and regulators.
“Our recent work on construct validity in benchmarks shows that many evaluations fail to measure what they claim to measure, and this study demonstrates exactly why that matters,” Mahdi said.
“We cannot rely on standardised tests alone to determine if these systems are safe for public use. Just as we require clinical trials for new medications, AI systems need rigorous testing with diverse, real users to understand their true capabilities in high-stakes settings like healthcare.”
Health AI Implementation
The study comes as OpenAI released an update to its popular ChatGPT chatbot last month, launching ChatGPT Health, a dedicated bot to tackle health and wellness queries from users.“Health is a dedicated space in ChatGPT where you can ask health and wellness questions and choose to connect your health data (like medical records and wellness apps) so responses can be grounded in that context. It is designed to support, not replace, medical care,” OpenAI said.
“You can optionally connect sources like Medical Records and Apple Health, and your Health chats, memories, and files stay separate from the rest of ChatGPT.”
Dr. Rebecca Andrews, chair of the board of regents for the American College of Physicians (ACP), told The Epoch Times that ACP “firmly believes” AI technologies should complement the logic and decision-making of physicians rather than supplant it.
“This is so important because AI cannot perform a clinical exam, which is one of the most essential components of medical care,” Andrews said.
In an Oct. 9 testimony before the Senate Committee on Health, Education, Labor, and Pensions, Dr. Russ B. Altman, a senior fellow at the Stanford Institute for Human-Centered AI, highlighted the benefits of the new technology.
According to Altman, AI can augment clinical diagnosis and treatment, enhance patients’ understanding and control of their care, and accelerate drug discovery.
However, “while I am optimistic about these applications, we will only fully realize their benefits if healthcare systems build interdisciplinary teams that thoroughly evaluate tools for clinical effectiveness and safety,” he said.
