Written by Babatunde Carew
Spoon Feed
Introducing even a small amount (0.001%) of misinformation into medical LLM training data significantly increases potentially harmful outputs.
AI – artificially informed?
Since the launch of ChatGPT and the resulting AI boom, large language models (LLMs) have been increasingly utilized across many industries. While LLMs offer significant potential in healthcare, concerns have been raised regarding their reliability, potential for bias, and ability to perform complex tasks when used in clinical settings.
This study evaluates the vulnerability of medically trained LLMs to data poisoning. By injecting medical misinformation into an LLM training dataset, the researchers demonstrated that replacing just 0.001% of data with misinformation increased the likelihood of inaccurate responses by ~5% (P = 0.03836). Compromised LLMs performed comparably to unaltered models on standard medical benchmarks, indicating that current evaluation methods fail to detect data poisoning. These findings highlight the risk of LLMs inadvertently amplifying misinformation and underscore the need for rigorous validation mechanisms. A significant limitation of this study is the use of only a single training dataset. (Ironically, a small portion of this text was AI-generated and physician edited.)
How does this change my practice?
This study supports my current approach to AI tools in practice. Current medically centered AI tools like DoximityGPT and OpenEvidence are great supplements to traditional medical databases such as UpToDate. However, I am critical of the answers they provide, given the known propensity of AI tools to hallucinate or provide misinformation. Future studies comparing the efficacy of novel validation tools to detect and curtail potentially harmful AI outputs are needed, especially as predatory journals and the volume of “peer-reviewed” literature increase.
Source
Medical large language models are vulnerable to data-poisoning attacks. Nat Med. 2025 Jan 8. doi: 10.1038/s41591-024-03445-1. Epub ahead of print. PMID: 39779928
