Search

Background: Recent advancements in artificial intelligence, such as GPT-3.5 Turbo (OpenAI) and GPT-4, have demonstrated significant potential by achieving good scores on text-only United States Medical Licensing Examination (USMLE) exams and effectively answering questions from physicians. However, the ability of these models to interpret medical images remains underexplored., Objective: This study aimed to comprehensively evaluate the performance, interpretability, and limitations of GPT-3.5 Turbo, GPT-4, and its successor, GPT-4 Vision (GPT-4V), specifically focusing on GPT-4V's newly introduced image-understanding feature. By assessing the models on medical licensing examination questions that require image interpretation, we sought to highlight the strengths and weaknesses of GPT-4V in handling complex multimodal clinical information, thereby exposing hidden flaws and providing insights into its readiness for integration into clinical settings., Methods: This cross-sectional study tested GPT-4V, GPT-4, and ChatGPT-3.5 Turbo on a total of 227 multiple-choice questions with images from USMLE Step 1 (n=19), Step 2 clinical knowledge (n=14), Step 3 (n=18), the Diagnostic Radiology Qualifying Core Exam (DRQCE) (n=26), and AMBOSS question banks (n=150). AMBOSS provided expert-written hints and question difficulty levels. GPT-4V's accuracy was compared with 2 state-of-the-art large language models, GPT-3.5 Turbo and GPT-4. The quality of the explanations was evaluated by choosing human preference between an explanation by GPT-4V (without hint), an explanation by an expert, or a tie, using 3 qualitative metrics: comprehensive explanation, question information, and image interpretation. To better understand GPT-4V's explanation ability, we modified a patient case report to resemble a typical "curbside consultation" between physicians., Results: For questions with images, GPT-4V achieved an accuracy of 84.2%, 85.7%, 88.9%, and 73.1% in Step 1, Step 2 clinical knowledge, Step 3 of USMLE, and DRQCE, respectively. It outperformed GPT-3.5 Turbo (42.1%, 50%, 50%, 19.2%) and GPT-4 (63.2%, 64.3%, 66.7%, 26.9%). When GPT-4V answered correctly, its explanations were nearly as good as those provided by domain experts from AMBOSS. However, incorrect answers often had poor explanation quality: 18.2% (10/55) contained inaccurate text, 45.5% (25/55) had inference errors, and 76.3% (42/55) demonstrated image misunderstandings. With human expert assistance, GPT-4V reduced errors by an average of 40% (22/55). GPT-4V accuracy improved with hints, maintaining stable performance across difficulty levels, while medical student performance declined as difficulty increased. In a simulated curbside consultation scenario, GPT-4V required multiple specific prompts to interpret complex case data accurately., Conclusions: GPT-4V achieved high accuracy on multiple-choice questions with images, highlighting its potential in medical assessments. However, significant shortcomings were observed in the quality of explanations when questions were answered incorrectly, particularly in the interpretation of images, which could not be efficiently resolved through expert interaction. These findings reveal hidden flaws in the image interpretation capabilities of GPT-4V, underscoring the need for more comprehensive evaluations beyond multiple-choice questions before integrating GPT-4V into clinical settings., (©Zhichao Yang, Zonghai Yao, Mahbuba Tasmin, Parth Vashisht, Won Seok Jang, Feiyun Ouyang, Beining Wang, David McManus, Dan Berlowitz, Hong Yu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 07.02.2025.)

Shortly after the first case of SARS-CoV-2 was diagnosed a public health emergency (PHE) was declared and a multi-agency response was initiated within the US federal government to create and propagate testing capacity. As part of this response, an unprecedented program designated Rapid Acceleration of Diagnostics (RADx) Tech was established by the National Institutes of Health (NIH) to facilitate the development of point-of-care tests for the COVID-19. The RADx Tech Clinical Studies Core (CSC), located at the University of Massachusetts Chan Medical School (UMass Chan), with partnering academic, private, and non-governmental organizations around the country, was tasked with developing clinical studies to support this work. This manuscript details development of a biorepository specifically focused on the collection and storage of samples designed for diagnostic platform development. It highlights the unified collection and annotation process that enabled gathering a diverse set of samples. This diversity encompasses the geography and backgrounds of the participants as well as sample characteristics such as variant type and RT-PCR cycle threshold (CT) value of the corresponding reference sample on a uniform clinical reference platform., Competing Interests: Declarations Ethics approval and consent to participate This study was approved by the UMass Chan Institutional Review Board reference number [H00022475]. Informed consent for participation was obtained from all participants. In the case of pediatric participants, informed consent was obtained from the parent or legal guardian as well as assent from the child. Clinical trial number: not applicable. Consent for publication Not applicable. Competing interests The views expressed in this article are those of the authors and do not necessarily represent the views of the National Institute of Biomedical Imaging and Bioengineering; the National Heart, Lung, and Blood Institute; the National Institutes of Health (NIH); or the U.S. Department of Health and Human Services., (© 2024. The Author(s).)

Background: It is important to document the performance of rapid antigen tests (Ag-RDTs) in detecting SARS-CoV-2 variants., Objective: To compare the performance of Ag-RDTs in detecting the Delta (B.1.617.2) and Omicron (B.1.1.529) variants of SARS-CoV-2., Design: Secondary analysis of a prospective cohort study that enrolled participants between 18 October 2021 and 24 January 2022. Participants did Ag-RDTs and collected samples for reverse transcriptase polymerase chain reaction (RT-PCR) testing every 48 hours for 15 days., Setting: The parent study enrolled participants throughout the mainland United States through a digital platform. All participants self-collected anterior nasal swabs for rapid antigen testing and RT-PCR testing. All Ag-RDTs were completed at home, whereas nasal swabs for RT-PCR were shipped to a central laboratory., Participants: Of 7349 participants enrolled in the parent study, 5779 asymptomatic persons who tested negative for SARS-CoV-2 on day 1 of the study were eligible for this substudy., Measurements: Sensitivity of Ag-RDTs on the same day as the first positive (index) RT-PCR result and 48 hours after the first positive RT-PCR result., Results: A total of 207 participants were positive on RT-PCR (58 Delta, 149 Omicron). Differences in sensitivity between variants were not statistically significant (same day: Delta, 15.5% [95% CI, 6.2% to 24.8%] vs. Omicron, 22.1% [CI, 15.5% to 28.8%]; at 48 hours: Delta, 44.8% [CI, 32.0% to 57.6%] vs. Omicron, 49.7% [CI, 41.6% to 57.6%]). Among 109 participants who had RT-PCR-positive results for 48 hours, rapid antigen sensitivity did not differ significantly between Delta- and Omicron-infected participants (48-hour sensitivity: Delta, 81.5% [CI, 66.8% to 96.1%] vs. Omicron, 78.0% [CI, 69.1% to 87.0%]). Only 7.2% of the 69 participants with RT-PCR-positive results for shorter than 48 hours tested positive by Ag-RDT within 1 week; those with Delta infections remained consistently negative on Ag-RDTs., Limitation: A testing frequency of 48 hours does not allow a finer temporal resolution of the analysis of test performance, and the results of Ag-RDTs are based on self-report., Conclusion: The performance of Ag-RDTs in persons infected with the SARS-CoV-2 Omicron variant is not inferior to that in persons with Delta infections. Serial testing improved the sensitivity of Ag-RDTs for both variants. The performance of rapid antigen testing varies on the basis of duration of RT-PCR positivity., Primary Funding Source: National Heart, Lung, and Blood Institute of the National Institutes of Health.

Importance: Widespread distribution of rapid antigen tests is integral to the US strategy to address COVID-19; however, it is estimated that few rapid antigen test results are reported to local departments of health., Objective: To characterize how often individuals in 6 communities throughout the United States used a digital assistant to log rapid antigen test results and report them to their local departments of health., Design, Setting, and Participants: This prospective cohort study is based on anonymously collected data from the beneficiaries of the Say Yes! Covid Test program, which distributed more than 3 000 000 rapid antigen tests at no cost to residents of 6 communities (Louisville, Kentucky; Indianapolis, Indiana; Fulton County, Georgia; O'ahu, Hawaii; Ann Arbor and Ypsilanti, Michigan; and Chattanooga, Tennessee) between April and October 2021. A descriptive evaluation of beneficiary use of a digital assistant for logging and reporting their rapid antigen test results was performed., Interventions: Widespread community distribution of rapid antigen tests., Main Outcomes and Measures: Number and proportion of tests logged and reported to the local department of health through the digital assistant., Results: A total of 313 000 test kits were distributed, including 178 785 test kits that were ordered using the digital assistant. Among all distributed kits, 14 398 households (4.6%) used the digital assistant, but beneficiaries reported three-quarters of their rapid antigen test results to their state public health departments (30 965 tests reported of 41 465 total test results [75.0%]). The reporting behavior varied by community and was significantly higher among communities that were incentivized for reporting test results vs those that were not incentivized or partially incentivized (90.5% [95% CI, 89.9%-91.2%] vs 70.5%; [95% CI, 70.0%-71.0%]). In all communities, positive tests were less frequently reported than negative tests (60.4% [95% CI, 58.1%-62.8%] vs 75.5% [95% CI, 75.1%-76.0%])., Conclusions and Relevance: These results suggest that application-based reporting with incentives may be associated with increased reporting of rapid tests for COVID-19. However, increasing the adoption of the digital assistant may be a critical first step.

Purpose: To validate an algorithm based upon International Classification of Diseases, 9(th) revision, Clinical Modification (ICD-9-CM) codes for acute myocardial infarction (AMI) documented within the Mini-Sentinel Distributed Database (MSDD)., Methods: Using an ICD-9-CM-based algorithm (hospitalized patients with 410.x0 or 410.x1 in primary position), we identified a random sample of potential cases of AMI in 2009 from four Data Partners participating in the Mini-Sentinel Program. Cardiologist reviewers used information abstracted from hospital records to assess the likelihood of an AMI diagnosis based on criteria from the Joint European Society of Cardiology and American College of Cardiology Global Task Force. Positive predictive values (PPVs) of the ICD-9-based algorithm were calculated., Results: Of the 153 potential cases of AMI identified, hospital records for 143 (93%) were retrieved and abstracted. Overall, the PPV was 86.0% (95% confidence interval; 79.2%, 91.2%). PPVs ranged from 76.3% to 94.3% across the four Data Partners., Conclusions: The overall PPV of potential AMI cases, as identified using an ICD-9-CM-based algorithm, may be acceptable for safety surveillance; however, PPVs do vary across Data Partners. This validation effort provides a contemporary estimate of the reliability of this algorithm for use in future surveillance efforts conducted using the Food and Drug Administration's MSDD., (Copyright © 2012 John Wiley & Sons, Ltd.)