Search

Chat Generative Pre-Trained Transformer (ChatGPT), an artificial intelligence program that uses natural language processing to generate conversational-style responses to questions or inputs, is increasingly being used by both patients and health care professionals. This study aims to evaluate the accuracy and comprehensiveness of ChatGPT in radiation oncology–related domains, including answering common patient questions, summarizing landmark clinical research studies, and providing literature reviews with specific references supporting current standard-of-care clinical practice in radiation oncology. We assessed the performance of ChatGPT version 3.5 (ChatGPT3.5) in 3 areas. We evaluated ChatGPT3.5's ability to answer 28 templated patient-centered questions applied across 9 cancer types. We then tested ChatGPT3.5's ability to summarize specific portions of 10 landmark studies in radiation oncology. Next, we used ChatGPT3.5 to identify scientific studies supporting current standard-of-care practice in clinical radiation oncology for 5 different cancer types. Each response was graded independently by 2 reviewers, with discordant grades resolved by a third reviewer. ChatGPT3.5 frequently generated inaccurate or incomplete responses. Only 39.7% of responses to patient-centered questions were considered correct and comprehensive. When summarizing landmark studies in radiation oncology, 35.0% of ChatGPT3.5′s responses were accurate and comprehensive, improving to 43.3% when provided the full text of the study. ChatGPT3.5′s ability to present a list of studies related to standard-of-care clinical practices was also unsatisfactory, with 50.6% of the provided studies fabricated. ChatGPT should not be considered a reliable radiation oncology resource for patients or providers at this time, as it frequently generates inaccurate or incomplete responses. However, natural language programming-based artificial intelligence programs are rapidly evolving, and future versions of ChatGPT or similar programs may demonstrate improved performance in this domain. [ABSTRACT FROM AUTHOR]

Background: Noninferiority (NI) and equivalence trials evaluate whether an experimental therapy's effect on the primary endpoint (PEP) is contained within an acceptable margin compared to standard-of-care. The reliability and impact of this conclusion, however, is largely dependent on the justification for this design, the choice of margin, and the analysis population used., Methods: A meta-epidemiological study was performed of phase 3 randomized NI and equivalence oncologic trials registered at ClinicalTrials.gov. Data was extracted from each trial's registration page and primary manuscript., Results: We identified 65 NI and 10 equivalence trials that collectively enrolled 61,632 patients. Sixty-one trials (81%) demonstrated NI or equivalence. Sixty-five trials (87%) were justified in the use of an NI or equivalence design either because of an inherent advantage (53 trials), a significant quality-of-life improvement (6 trials), or a significant toxicity improvement (6 trials) of the interventional treatment relative to the control arm. Sixty-nine trials (92.0%) reported a prespecified NI or equivalence margin, of which only 23 (33.3%) provided justification for this margin based on prior literature. For trials with time-to-event PEPs, the median NI margin was a hazard ratio of 1.22 (range, 1.08-1.52). Investigators reported a per-protocol (PP) analysis for the PEP in only 28 trials (37%)., Conclusions: Although most published NI and equivalence trials have clear justification for their design, few provide rationale for the chosen margin or report a PP analysis. These findings underscore the need for rigorous standards in trial design and reporting., (© The Author(s) 2024. Published by Oxford University Press. All rights reserved. For permissions, please email: journals.permissions@oup.com.)