Back to Search Start Over

The role of large language models in self-care: a study and benchmark on medicines and supplement guidance accuracy.

Authors :
De Busser B
Roth L
De Loof H
Source :
International journal of clinical pharmacy [Int J Clin Pharm] 2024 Dec 07. Date of Electronic Publication: 2024 Dec 07.
Publication Year :
2024
Publisher :
Ahead of Print

Abstract

Background: The recent surge in the capabilities of artificial intelligence systems, particularly large language models, is also impacting the medical and pharmaceutical field in a major way. Beyond specialized uses in diagnostics and data discovery, these tools have now become accessible to the general public.<br />Aim: The study aimed to critically analyse the current performance of large language models in answering patient's self-care questions regarding medications and supplements.<br />Method: Answers from six major language models were analysed for correctness, language-independence, context-sensitivity, and reproducibility using a newly developed reference set of questions and a scoring matrix.<br />Results: The investigated large language models are capable of answering a clear majority of self-care questions accurately, providing relevant health information. However, substantial variability in the responses, including potentially unsafe advice, was observed, influenced by language, question structure, user context and time. GPT 4.0 scored highest on average, while GPT 3.5, Gemini, and Gemini Advanced had varied scores. Responses were context and language sensitive. In terms of consistency over time, Perplexity had the worst performance.<br />Conclusion: Given the high-quality output of large language models, their potential in self-care applications is undeniable. The newly created benchmark can facilitate further validation and guide the establishment of strict safeguards to combat the sizable risk of misinformation in order to reach a more favourable risk/benefit ratio when this cutting-edge technology is used by patients.<br />Competing Interests: Conflicts of interest: The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.<br /> (© 2024. The Author(s).)

Details

Language :
English
ISSN :
2210-7711
Database :
MEDLINE
Journal :
International journal of clinical pharmacy
Publication Type :
Academic Journal
Accession number :
39644377
Full Text :
https://doi.org/10.1007/s11096-024-01839-2