Written by Doug Wallace
Spoon Feed
Verified physician answers to patient questions on a public social media forum were compared to Chatbot answers to the same questions by rating the quality and empathy of responses. Guess who won?
ChatGPT vs Physicians: FIGHT!
Artificial Intelligence (AI) utilization is on the rise in health care settings, prompting plenty of discussion (ChatGPT Articles, ChatGPT JF). One question is: how will AI change provider workflows?
In the context of increasing electronic patient messaging with pressure for prompt responses by clinicians, this cross-sectional study plugged 195 public, de-identified patient questions from a social media forum (Reddit’s r/AskDocs) into ChatGPT. Chatbot answers were then compared to verified physician responses to the same questions. Answers from all sources were anonymized, randomized, and blinded before being presented to “licensed health care professionals” (Re – the authors of the study).
Responses were evaluated in terms of “quality” and “empathy” on a Likert scale. Key study results include that physician responses were on average shorter (52 vs 211 words) and rated as being lower quality and less empathetic. Check out the article for some interesting examples. The authors mention that perhaps chatbots could be used in the future to draft responses that physicians could then edit.
How will this change my practice?
Given this study’s inherent limitations and questionable generalizability, this will not change my practice. Having the authors evaluate the responses, even in a blinded fashion, introduces clear bias. Furthermore, Reddit Q/A may not be reflective of typical electronic patient messaging or physician responses (which the authors acknowledge).
Perhaps a more realistic comparison would be actual patients’ evaluation of physician vs AI responses to actual patient questions. Nonetheless, this is an intriguing study, and I applaud the authors for exploring this timely topic. More content in this space is undoubtedly on the horizon.
Source
Comparing Physician and Artificial Intelligence Chatbot Responses to Patient Questions Posted to a Public Social Media Forum. JAMA Intern Med. 2023 Jun 1;183(6):589-596. doi: 10.1001/jamainternmed.2023.1838.
One Comment