1. Evaluating ChatGPT text-mining of clinical records for obesity monitoring
- Author
-
Fins, Ivo S., Davies, Heather, Farrell, Sean, Torres, Jose R., Pinchbeck, Gina, Radford, Alan D., and Noble, Peter-John
- Subjects
Computer Science - Information Retrieval ,Computer Science - Computation and Language - Abstract
Background: Veterinary clinical narratives remain a largely untapped resource for addressing complex diseases. Here we compare the ability of a large language model (ChatGPT) and a previously developed regular expression (RegexT) to identify overweight body condition scores (BCS) in veterinary narratives. Methods: BCS values were extracted from 4,415 anonymised clinical narratives using either RegexT or by appending the narrative to a prompt sent to ChatGPT coercing the model to return the BCS information. Data were manually reviewed for comparison. Results: The precision of RegexT was higher (100%, 95% CI 94.81-100%) than the ChatGPT (89.3%; 95% CI82.75-93.64%). However, the recall of ChatGPT (100%. 95% CI 96.18-100%) was considerably higher than that of RegexT (72.6%, 95% CI 63.92-79.94%). Limitations: Subtle prompt engineering is needed to improve ChatGPT output. Conclusions: Large language models create diverse opportunities and, whilst complex, present an intuitive interface to information but require careful implementation to avoid unpredictable errors., Comment: Supplementary Material: The data that support the findings of this study are available in the ancillary files of this submission. 5 pages, 2 figures (textboxes)
- Published
- 2023