1. Evaluating ChatGPT's competency in radiation oncology: A comprehensive assessment across clinical scenarios.
- Author
-
Ramadan S, Mutsaers A, Chen PC, Bauman G, Velker V, Ahmad B, Arifin AJ, Nguyen TK, Palma D, and Goodman CD
- Abstract
Purpose: Artificial intelligence (AI) and machine learning present an opportunity to enhance clinical decision-making in radiation oncology. This study aims to evaluate the competency of ChatGPT, an AI language model, in interpreting clinical scenarios and assessing its oncology knowledge., Methods and Materials: A series of clinical cases were designed covering 12 disease sites. Questions were grouped into domains: epidemiology, staging and workup, clinical management, treatment planning, cancer biology, physics, and surveillance. Royal College-certified radiation oncologists (ROs) reviewed cases and provided solutions. ROs scored responses on 3 criteria: conciseness (focused answers), completeness (addressing all aspects of the question), and correctness (answer aligns with expert opinion) using a standardized rubric. Scores ranged from 0 to 5 for each criterion for a total possible score of 15., Results: Across 12 cases, 182 questions were answered with a total AI score of 2317/2730 (84 %). Scores by criteria were: completeness (79 %, range: 70-99 %), conciseness (92 %, range: 83-99 %), and correctness (81 %, range: 72-92 %). AI performed best in the domains of epidemiology (93 %) and cancer biology (93 %) and reasonably in staging and workup (89 %), physics (86 %) and surveillance (82 %). Weaker domains included treatment planning (78 %) and clinical management (81 %). Statistical differences were driven by variations in the completeness (p < 0.01) and correctness (p = 0.04) criteria, whereas conciseness scored universally high (p = 0.91). These trends were consistent across disease sites., Conclusions: ChatGPT showed potential as a tool in radiation oncology, demonstrating a high degree of accuracy in several oncologic domains. However, this study highlights limitations with incorrect and incomplete answers in complex cases., Competing Interests: Declaration of competing interest The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: [DAP reports a Clinician-Scientist Grant from the Ontario Institute for Cancer Research, royalties from Uptodate.com, and a consultant role with equity from Need Inc. TN declares a speaker fee with the Radiosurgery society and a consultant role with equity from Need Inc. CG declares consulting fees and stock options from Need inc. GSB declares an advisory board position with advanced accelerator applications, a director position with the Centre for Translation Cancer Research, and a research grant from Siemens. PCC declares he is an employee of Need Inc. AM declares honoraria from Sanofi. The other authors declare no competing interests.]., (Copyright © 2024. Published by Elsevier B.V.)
- Published
- 2024
- Full Text
- View/download PDF