Can Chatbots Counter Medical Misinformation? Unproven Treatments, Patient Literacy, and the Challenges of Evaluating LLMs by Dr. Zubin Master

Join us Monday, November 2 at 4:00 PM in Alumni Hall 286! Dr. Zubin Master, Associate Professor Social Sciences and Health Policy Wake Forest Institute of Regenerative Medicine Wake Forest University School of Medicine will share his insight on topic entitled Can Chatbots Counter Medical Misinformation? Unproven Treatments, Patient Literacy, and the Challenges of Evaluating LLMs.

Abstract

Older adults and caregivers increasingly confront persuasive online claims about unproven
therapies for which evidence is limited, uncertain, or absent, even as these products and
services are marketed directly to consumers. Examples include unproven stem cell
interventions, natural products and supplements, peptides, and repurposed drugs promoted for
conditions such as cancer. Medical misinformation also surrounds well-established interventions
with strong evidence of safety and efficacy, including vaccines and statins. Such misinformation
can contribute to misunderstandings about disease and treatment, drug-supplement risks,
delayed or forgone evidence-based care, and strained conversations with clinicians. Its effects
on patient attitudes and behavior also depend on broader factors including prior beliefs,
experiences with health care, and trust in medicine and science. At the same time, consumer
artificial intelligence chatbots are increasingly being used for health advice. These systems
could potentially help patients ask better questions, understand uncertain evidence, evaluate
unsubstantiated claims, and prepare for more productive clinical encounters. Prior to
implementing chatbots to patients for health advice, a rigorous evaluation of LLM accuracy,
comprehensiveness, and appropriately cautious advice is necessary. Yet the rapidly expanding
literature evaluating chatbot performance for medical advice has important limitations: studies
vary substantially in the models, prompts, clinical domains, reference standards, evaluation
criteria, and human judgments making LLM performance difficult to interpret or compare. In this
presentation, the speaker will cover how medical misinformation reaches patients, who may be particularly vulnerable, and strategies used to address it. Dr. Master will then use two case studies—including a comparative study from our group—to illustrate methodological challenges in evaluating LLM-generated medical advice. Finally, he will consider whether consumer chatbots could improve
patient science literacy and help counter misinformation, and the research needed to
understand how patients use these systems and what safeguards physicians would want before
recommending them.

All are welcome!