Search

Background: Recent advances in Artificial Intelligence (AI) are changing the medical world, and AI will likely replace many of the actions performed by medical professionals. The overall clinical ability of the AI has been evaluated by its ability to answer a text-based national medical examination. This study uniquely assesses the performance of Open AI's ChatGPT against all Japanese National Medical Licensing Examination (NMLE), including images, illustrations, and pictures., Methods: We obtained the questions of the past six years of the NMLE (112th to 117th) from the Japanese Ministry of Health, Labour and Welfare website. We converted them to JavaScript Object Notation (JSON) format. We created an application programming interface (API) to output correct answers using GPT-4 for questions without images and GPT4-V(ision) or GPT4 console for questions with images., Results: The percentage of image questions was 723/2400 (30.1%) over the past six years. In all years, GPT-4/4V exceeded the minimum score the examinee should score. In total, over the six years, the percentage of correct answers for basic medical knowledge questions was 665/905 (73.5%); for clinical knowledge questions, 1143/1531 (74.7%); and for image questions 497/723 (68.7%), respectively., Conclusions: Regarding medical knowledge, GPT-4/4V met the minimum criteria regardless of whether the questions included images, illustrations, and pictures. Our study sheds light on the potential utility of AI in medical education.

Background: Recent advancements in artificial intelligence, such as GPT-3.5 Turbo (OpenAI) and GPT-4, have demonstrated significant potential by achieving good scores on text-only United States Medical Licensing Examination (USMLE) exams and effectively answering questions from physicians. However, the ability of these models to interpret medical images remains underexplored., Objective: This study aimed to comprehensively evaluate the performance, interpretability, and limitations of GPT-3.5 Turbo, GPT-4, and its successor, GPT-4 Vision (GPT-4V), specifically focusing on GPT-4V's newly introduced image-understanding feature. By assessing the models on medical licensing examination questions that require image interpretation, we sought to highlight the strengths and weaknesses of GPT-4V in handling complex multimodal clinical information, thereby exposing hidden flaws and providing insights into its readiness for integration into clinical settings., Methods: This cross-sectional study tested GPT-4V, GPT-4, and ChatGPT-3.5 Turbo on a total of 227 multiple-choice questions with images from USMLE Step 1 (n=19), Step 2 clinical knowledge (n=14), Step 3 (n=18), the Diagnostic Radiology Qualifying Core Exam (DRQCE) (n=26), and AMBOSS question banks (n=150). AMBOSS provided expert-written hints and question difficulty levels. GPT-4V's accuracy was compared with 2 state-of-the-art large language models, GPT-3.5 Turbo and GPT-4. The quality of the explanations was evaluated by choosing human preference between an explanation by GPT-4V (without hint), an explanation by an expert, or a tie, using 3 qualitative metrics: comprehensive explanation, question information, and image interpretation. To better understand GPT-4V's explanation ability, we modified a patient case report to resemble a typical "curbside consultation" between physicians., Results: For questions with images, GPT-4V achieved an accuracy of 84.2%, 85.7%, 88.9%, and 73.1% in Step 1, Step 2 clinical knowledge, Step 3 of USMLE, and DRQCE, respectively. It outperformed GPT-3.5 Turbo (42.1%, 50%, 50%, 19.2%) and GPT-4 (63.2%, 64.3%, 66.7%, 26.9%). When GPT-4V answered correctly, its explanations were nearly as good as those provided by domain experts from AMBOSS. However, incorrect answers often had poor explanation quality: 18.2% (10/55) contained inaccurate text, 45.5% (25/55) had inference errors, and 76.3% (42/55) demonstrated image misunderstandings. With human expert assistance, GPT-4V reduced errors by an average of 40% (22/55). GPT-4V accuracy improved with hints, maintaining stable performance across difficulty levels, while medical student performance declined as difficulty increased. In a simulated curbside consultation scenario, GPT-4V required multiple specific prompts to interpret complex case data accurately., Conclusions: GPT-4V achieved high accuracy on multiple-choice questions with images, highlighting its potential in medical assessments. However, significant shortcomings were observed in the quality of explanations when questions were answered incorrectly, particularly in the interpretation of images, which could not be efficiently resolved through expert interaction. These findings reveal hidden flaws in the image interpretation capabilities of GPT-4V, underscoring the need for more comprehensive evaluations beyond multiple-choice questions before integrating GPT-4V into clinical settings., (©Zhichao Yang, Zonghai Yao, Mahbuba Tasmin, Parth Vashisht, Won Seok Jang, Feiyun Ouyang, Beining Wang, David McManus, Dan Berlowitz, Hong Yu. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 07.02.2025.)

Radiation therapy (RT) is a critical component of multidisciplinary cancer care, but has inconsistent curricular exposure. We characterize the radiation oncology (RO) content on the standardized undergraduate medical examinations by comparing its context and prevalence with other domains in oncology. National Board of Medical Examiners (NBME) self-assessments and sample questions for the United States Medical Licensing Exam (USMLE) Steps 1-3 and NBME clinical science shelf examinations were accessed (n = 3878). Questions were inductively analyzed for content pertaining to oncology and treatment modalities of RT, systemic therapy (ST), and surgical intervention (SI). Questions were coded using USMLE Physician Tasks/Competencies and thematic analysis. Descriptive statistics and analyses using the Kruskal-Wallis test are reported. A total of 337 questions (8.6%) within the USMLE and shelf exams included oncology content, with 101 questions (2.6%) referencing at least one cancer treatment modality (n = 35 RT, 45 ST, 57 SI). Treatment questions were more common on USMLE Step 2 CK (n = 35/101, 32%) compared to Step 1 (n = 23/101, 23%) and Step 3 (n = 8/101, 8%) (p < 0.001). RT was significantly less likely to be the correct answer (2/35, 6%) compared to ST (4/45, 9%) and SI (18/57, 32%) (p = 0.003). Therapeutic oncology questions are uncommon on the examination material, with an under-representation of radiation-related content, and contextual bias favoring surgical approaches. We advocate for greater RO involvement in the content creation of such examinations to help trainees better understand multidisciplinary cancer care., Competing Interests: Declarations. Ethical Approval: Not applicable. Conflict of Interest: M.T.M, L.C.L, B.K.N, P.S.T, Mi.D.M, K.T.S report no disclosures. T.C.W reports receiving ACRO Luther Brady Grant funding and American Radium Society Executive Board membership, outside of the submitted work. J.J.C reports receiving funding from the UCSF Helen Diller Family Comprehensive Cancer Center Diversity, Equity, Inclusion and Accessibility Award and membership on the Radiation Oncology Education Collaborative Study Group Executive Board, outside of the submitted work. Ma.D.M reports receiving research funding from the Radiation Oncology Institute and New Jersey Health Foundation and funding from Bristol Myers Squibb Foundation for Diversity in Clinical Trials Career Development Program, outside of the submitted work., (© 2024. The Author(s).)

The United States Medical Licensing Examination (USMLE) is a critical step in assessing the competence of future physicians, yet the process of creating exam questions and study materials is both time-consuming and costly. While Large Language Models (LLMs), such as OpenAI's GPT-4, have demonstrated proficiency in answering medical exam questions, their potential in generating such questions remains underexplored. This study presents QUEST-AI, a novel system that utilizes LLMs to (1) generate USMLE-style questions, (2) identify and flag incorrect questions, and (3) correct errors in the flagged questions. We evaluated this system's output by constructing a test set of 50 LLM-generated questions mixed with 50 human-generated questions and conducting a two-part assessment with three physicians and two medical students. The assessors attempted to distinguish between LLM and human-generated questions and evaluated the validity of the LLM-generated content. A majority of exam questions generated by QUEST-AI were deemed valid by a panel of three clinicians, with strong correlations between performance on LLM-generated and human-generated questions. This pioneering application of LLMs in medical education could significantly increase the ease and efficiency of developing USMLE-style medical exam content, offering a cost-effective and accessible alternative for exam preparation.

Objective: To evaluate the accuracy and parsing ability of GPT 4.0 for Japanese medical practitioner qualification examinations in a multidimensional way to investigate its response accuracy and comprehensiveness to medical knowledge., Methods: We evaluated the performance of the GPT 4.0 on Japanese Medical Licensing Examination (JMLE) questions (2021-2023). Questions are categorized by difficulty and type, with distinctions between general and clinical parts, as well as between single-choice (MCQ1) and multiple-choice (MCQ2) questions. Difficulty levels were determined on the basis of correct rates provided by the JMLE Preparatory School. The accuracy and quality of the GPT 4.0 responses were analyzed via an improved Global Qualily Scale (GQS) scores, considering both the chosen options and the accompanying analysis. Descriptive statistics and Pearson Chi-square tests were used to examine performance across exam years, question difficulty, type, and choice. GPT 4.0 ability was evaluated via the GQS, with comparisons made via the Mann-Whitney U or Kruskal-Wallis test., Results: The correct response rate and parsing ability of the GPT4.0 to the JMLE questions reached the qualification level (80.4%). In terms of the accuracy of the GPT4.0 response to the JMLE, we found significant differences in accuracy across both difficulty levels and option types. According to the GQS scores for the GPT 4.0 responses to all the JMLE questions, the performance of the questionnaire varied according to year and choice type., Conclusion: GTP4.0 performs well in providing basic support in medical education and medical research, but it also needs to input a large amount of medical-related data to train its model and improve the accuracy of its medical knowledge output. Further integration of ChatGPT with the medical field could open new opportunities for medicine., Competing Interests: Conflict of Interest Statement. The authors declare that there is no conflict of interest with any financial organization or corporation or individual that can inappropriately influence this work., (© 2024. Huazhong University of Science and Technology.)

Policy Points The reinstitution of pre-COVID-19 pandemic licensure regulations has impeded interstate telehealth. This has disproportionately impacted patients who live near a state border; geographically mobile patients, such as college students; and patients with rare diseases who may need care from a specialist outside their state. Several promising and feasible reforms are available, at both state and federal levels, to facilitate interstate telehealth. For example, states can offer exemptions to licensure requirements for certain types of telehealth such as follow-up care or create licensure registries that impose little reduced paperwork and fees on physicians. On the federal level, congressional interventions that mimic the Department of Veterans Affairs Maintaining Internal Systems and Strengthening Integrated Outside Networks (VA MISSION) Act of 2018 can waive provider licensing and geographic restrictions to telehealth within certain federal programs such as Medicare. Any discussion of medical licensure reform, however, must also consider the current political climate, one in which states are taking divergent stances on sensitive topics such as reproductive care, gender-affirming care, and substance use treatments., (© 2024 Milbank Memorial Fund.)

Background: This study aimed to evaluate the performance of GPT-3.5, GPT-4, GPT-4o and Google Bard on the United States Medical Licensing Examination (USMLE), the Professional and Linguistic Assessments Board (PLAB), the Hong Kong Medical Licensing Examination (HKMLE) and the National Medical Licensing Examination (NMLE)., Methods: This study was conducted in June 2023. Four LLMs (Large Language Models) (GPT-3.5, GPT-4, GPT-4o and Google Bard) were applied to four medical standardized tests (USMLE, PLAB, HKMLE and NMLE). All questions are multiple-choice questions and were sourced from the question banks of these examinations., Results: In USMLE step 1, step 2CK and Step 3, there are accuracy rates of 91.5%, 94.2% and 92.7% provided from GPT-4o, 93.2%, 95.0% and 92.0% provided from GPT-4, 65.6%, 71.6% and 68.5% provided from GPT-3.5, and 64.3%, 55.6%, 58.1% from Google Bard, respectively. In PLAB, HKMLE and NMLE, GPT-4o scored 93.3%, 91.7% and 84.9%, GPT-4 scored 86.7%, 89.6% and 69.8%, GPT-3.5 scored 80.0%, 68.1% and 60.4%, and Google Bard scored 54.2%, 71.7% and 61.3%. There was significant difference in the accuracy rates of four LLMs in the four medical licensing examinations., Conclusion: GPT-4o performed better in the medical licensing examinations than other three LLMs. The performance of the four models in the NMLE examination needs further improvement., Clinical Trial Number: Not applicable., Competing Interests: Declarations. Ethics approval and consent to participate: Not applicable. Informed consent: Informed consent were not required. Institutional review board: This study does not involve human or animal subjects and therefore did not require approval from the Institutional Ethical Board. Competing interests: The authors declare no competing interests., (© 2024. The Author(s).)

An international medical graduate (IMG) is a doctor who has received their basic medical qualification from a medical school located in a different country from that in which they practice or intend to practice. IMGs are known to face difficulties in their working lives, including differential attainment in assessment. The objective of this review is to map key concepts and types of evidence in academic and gray literature relating to international medical graduates' experiences of clinical competency assessment and to identify knowledge gaps on this topic by systematically searching, selecting, and synthesizing existing knowledge. All studies will relate to IMGs. The concept of interest will be IMGs' experiences of assessment. The context will be postgraduate, licensing or credentialing medical assessments of clinical competence. This review will be conducted in accordance with the Joanna Briggs Institute (JBI) methodology for scoping reviews. Seven electronic databases will be searched for literature published between 2009 and 2024: the Australian Education Index, British Education Index, ERIC, PubMed, PsycINFO, Scopus, and SocINDEX. Gray literature will be searched using Google, Google Scholar, and published reports from postgraduate training bodies and medical licensing organizations. Documents will be independently screened, selected, and extracted by two researchers using a piloted data-extraction tool. Data will be analyzed and presented in tables and in a narrative format. Trial resgistration: Scoping review registration: Open Science Framework: https://osf.io/8gdm7., Competing Interests: The authors have declared that no competing interests exist., (Copyright: © 2024 Hynes et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Background: Recent studies, including those by the National Board of Medical Examiners, have highlighted the remarkable capabilities of recent large language models (LLMs) such as ChatGPT in passing the United States Medical Licensing Examination (USMLE). However, there is a gap in detailed analysis of LLM performance in specific medical content areas, thus limiting an assessment of their potential utility in medical education., Objective: This study aimed to assess and compare the accuracy of successive ChatGPT versions (GPT-3.5, GPT-4, and GPT-4 Omni) in USMLE disciplines, clinical clerkships, and the clinical skills of diagnostics and management., Methods: This study used 750 clinical vignette-based multiple-choice questions to characterize the performance of successive ChatGPT versions (ChatGPT 3.5 [GPT-3.5], ChatGPT 4 [GPT-4], and ChatGPT 4 Omni [GPT-4o]) across USMLE disciplines, clinical clerkships, and in clinical skills (diagnostics and management). Accuracy was assessed using a standardized protocol, with statistical analyses conducted to compare the models' performances., Results: GPT-4o achieved the highest accuracy across 750 multiple-choice questions at 90.4%, outperforming GPT-4 and GPT-3.5, which scored 81.1% and 60.0%, respectively. GPT-4o's highest performances were in social sciences (95.5%), behavioral and neuroscience (94.2%), and pharmacology (93.2%). In clinical skills, GPT-4o's diagnostic accuracy was 92.7% and management accuracy was 88.8%, significantly higher than its predecessors. Notably, both GPT-4o and GPT-4 significantly outperformed the medical student average accuracy of 59.3% (95% CI 58.3-60.3)., Conclusions: GPT-4o's performance in USMLE disciplines, clinical clerkships, and clinical skills indicates substantial improvements over its predecessors, suggesting significant potential for the use of this technology as an educational aid for medical students. These findings underscore the need for careful consideration when integrating LLMs into medical education, emphasizing the importance of structured curricula to guide their appropriate use and the need for ongoing critical analyses to ensure their reliability and effectiveness., (© Brenton T Bicknell, Danner Butler, Sydney Whalen, James Ricks, Cory J Dixon, Abigail B Clark, Olivia Spaedy, Adam Skelton, Neel Edupuganti, Lance Dzubinski, Hudson Tate, Garrett Dyess, Brenessa Lindeman, Lisa Soleymani Lehmann. Originally published in JMIR Medical Education (https://mededu.jmir.org).)

Importance: False medical information disseminated dangerously during the COVID-19 pandemic, with certain physicians playing a surprisingly prominent role. Medical boards engendered widespread criticism for not imposing forceful sanctions, but considerable uncertainty remains about how the professional licensure system regulates physician-spread misinformation., Objective: To compare the level of professional discipline of physicians for spreading medical misinformation relative to discipline for other offenses., Design, Setting, and Participants: This cross-sectional study analyzed and coded publicly reported medical board disciplinary actions in the 5 most populous US states. The analysis included data from January 1, 2020, through May 30, 2023, for California, Florida, New York, and Pennsylvania and from January 1, 2020, through March 30, 2022, for Texas., Main Outcomes and Measures: Medical board disciplinary proceedings that resulted in some form of sanction were analyzed. Codes were assigned for the different types of offenses relied on by medical boards for imposing physician discipline., Results: Among 3128 medical board disciplinary proceedings in the 5 most populous states, spreading misinformation to the community was the least common reason for medical board discipline of physicians (6 [0.1%] of all identified offenses). Two reasons tied for third least common: patient-directed misinformation (21 [0.3%]) and inappropriate advertising or patient solicitation (21 [0.3%]). The frequency of misinformation conduct was exponentially lower than more common reasons for discipline, such as physician negligence (1911 [28.7%]), problematic record-keeping (990 [14.9%]), and inappropriate prescribing (901 [13.5%]). Patient-directed misinformation provided a basis for discipline 3 times as often as spreading misinformation to the community. The frequency of disciplinary actions for any reasons related to COVID-19 care, even if not about misinformation, was also quite low (10 [0.2%]). Sanctions in misinformation actions tended to be relatively light., Conclusions and Relevance: The frequency of discipline for physician-spread misinformation observed in this cross-sectional study was quite low despite increased salience and medical board warnings since the start of the COVID-19 pandemic about the dangers of physicians spreading falsehoods. These findings suggest that there is a serious disconnect between regulatory guidance and enforcement and that medical boards relied on spreading misinformation to patients as a reason for discipline 3 times more frequently than disseminating falsehoods to the public. These results shed light on important policy concerns about professional licensure, including why, under current patient-centered frameworks, this form of regulation may be particularly ill-suited to address medical misinformation.

Aims: The aim of the present study was to evaluate the performance of various artificial intelligence (AI)-powered chatbots (commercially available in Switzerland up to June 2023) in solving a theoretical cardiology board exam and to compare their accuracy with that of human cardiology fellows., Methods: For the study, a set of 88 multiple-choice cardiology exam questions was used. The participating cardiology fellows and selected chatbots were presented with these questions. The evaluation metrics included Top-1 and Top-2 accuracy, assessing the ability of chatbots and fellows to select the correct answer., Results: Among the cardiology fellows, all 36 participants successfully passed the exam with a median accuracy of 98% (IQR 91-99%, range from 78% to 100%). However, the performance of the chatbots varied. Only one chatbot, Jasper quality, achieved the minimum pass rate of 73% correct answers. Most chatbots demonstrated a median Top-1 accuracy of 47% (IQR 44-53%, range from 42% to 73%), while Top-2 accuracy provided a modest improvement, resulting in a median accuracy of 67% (IQR 65-72%, range from 61% to 82%). Even with this advantage, only two chatbots, Jasper quality and ChatGPT plus 4.0, would have passed the exam. Similar results were observed when picture-based questions were excluded from the dataset., Conclusions: Overall, the study suggests that most current language-based chatbots have limitations in accurately solving theoretical medical board exams. In general, currently widely available chatbots fell short of achieving a passing score in a theoretical cardiology board exam. Nevertheless, a few showed promising results. Further improvements in artificial intelligence language models may lead to better performance in medical knowledge applications in the future.

Objective: To investigate interview and match outcomes of medical students who received pass/fail USMLE reporting vs medical students with numeric scoring during the same period., Design: Retrospective analysis of a cross-sectional survey-based study., Setting: United States 2023 residency match., Participants: Medical student applicants in the 2023 residency match cycle who responded to the Texas Seeking Transparency in Application to Residency (STAR) survey., Results: Among 6756 applicants for the 2023 match, 496 (7.3%) took USMLE Step 1 with pass/fail reporting. Pass/fail reporting was associated with lower USMLE Step 2-CK scores (245.9 vs 250.7), fewer honored clerkships (2.4 vs 3.1), and lower Alpha Omega Alpha membership (12.5% vs 25.2%) (all p < 0.001). Applicants with numeric USMLE Step 1 scores received more interview offers after adjusting for academic performance (beta coefficient 1.04 (95% CI 0.28-1.79); p = 0.007). Numeric USMLE Step 1 scoring was associated with more interview offers in nonsurgical specialties (beta coefficient 1.64 [95% CI 0.74-2.53]; p < 0.001), but not in general surgery (beta coefficient 3.01 [95% CI -0.82 to 6.84]; p = 0.123) or surgical subspecialties (beta coefficient 1.92 [95% CI -0.78 to 4.62]; p = 0.163). Numeric USMLE Step 1 scoring was not associated with match outcome., Conclusions: Applicants with numeric USMLE Step 1 scoring had stronger academic profiles than those with pass/fail scoring; however, adjusted analyses found only weak associations with interview or match outcomes. Further research is warranted to assess longitudinal outcomes., (Copyright © 2024 Association of Program Directors in Surgery. Published by Elsevier Inc. All rights reserved.)

Objectives: This study assessed the relationship between academic performance and National Licensing Examination (NLE) scores of Ethiopian anaesthetists and proposed academic performance thresholds that predict failing the NLE., Design: A retrospective cross-sectional study., Setting: Academic performance measures were collected from eight universities and matched to total and subdomain NLE scores for anaesthetists. Four universities were first generation (oldest), three were second generation and one was third generation (newest)., Participants: 524 anaesthetists., Measures: Selected course grades, yearly (Y1-Y4) and cumulative grade point averages (GPAs) and in-school exit exam status were correlated with NLE scores. Linear regression was used to identify predictors of NLE score. Receiver operating characteristic curves were plotted to estimate GPA thresholds that predict failing the NLE., Results: Total and general surgery anaesthesia subdomain NLE scores showed moderate to strong linear relationships with most academic performance indicators. However, correlations between the rest of the subdomain NLE scores and composite course scores were either small or negligible (r<0.5). For every 1-point increase in a 4-point cumulative GPA and composite senior year practicum course score, the first-attempt NLE score increased by 12.61 (95% CI 10.03 to 15.19; p<0.0001) and 3.98 (95% CI 1.77 to 6.19; p=0.001) percentage points, respectively. Those entering university directly from high school and those attending a third-generation university would, on average, score 2.79 (95% CI 0.23 to 5.35; p=0.033) and 4.34 (95% CI 1.42 to 7.26; p=0.004) higher NLE scores compared with nurse entrants and first-generation university graduates, respectively. On a 4-point grading system, the thresholds for failing the NLE were below 2.68 for year-1 GPA, 2.62 for year-2 GPA, 2.72 for year-3 GPA, 3.04 for year-4 GPA and 2.87 for cumulative GPA., Conclusions: The Ethiopian anaesthetist's NLE score modestly correlated with most academic performance indicators. Based on pass/fail thresholds, the NLE could assist training programmes in improving NLE pass rates., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2024. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: The ability of an expert's item difficulty ratings to predict test-taker actual performance is an important aspect of licensure examinations. Expert judgment is used as a primary source of information for users to make prior decisions to determine the pass rate of test takers. The nature of raters involved in predicting item difficulty is central to set credible standards. Therefore, this study aimed to assess and compare raters' prediction and actual Multiple-Choice Questions' difficulty of the undergraduate medicine licensure examination (UGMLE) in Ethiopia., Method: 815 examinees' responses to 200 Multiple-Choice Questions (MCQs) were used in this study. The study also included experts' item difficulty ratings of seven physicians who participated in the standard settings of UGMLE. Then, analysis was conducted to understand experts' rating variation in predicting the actual difficulty levels of examinees. Descriptive statistics was used to profile the mean rater's and actual difficulty value for MCQs, and ANOVA was used to compare the mean differences between raters' prediction of item difficulty. Additionally, regression analysis was used to understand the interrater variations in item difficulty predictions compared to the actual difficulty. The proportion of variance of actual difficulty explained from rater prediction was computed using regression analysis., Results: In this study, the mean difference between raters' prediction and examinees' actual performance was inconsistent across the exam domains. The study revealed a statistically significant strong positive correlation between the actual and predicted item difficulty in exam domains eight and eleven. However, a non-statistically significant very weak positive correlation was reported in exam domains seven and twelve. The multiple comparison analysis showed significant differences in mean item difficulty ratings between raters. In the regression analysis, experts' item difficulty ratings of the UGMLE had 33% power in predicting the actual difficulty level. The regression model also showed a moderate positive correlation (R = 0.57) that was statistically significant at F (6, 193) = 15.58, P = 0.001., Conclusion: This study demonstrated the complex process for assessing the difficulty level of MCQs in the UGMLE and emphasized the benefits of using experts' ratings in advance. To ensure the exams maintain the necessary reliable and valid scores, raters' accuracy on the UGMLE must be improved. To achieve this, techniques that align with the evolving assessment methodologies must be developed., (© 2024. The Author(s).)

Purpose: The United States Medical Licensing Examination (USMLE) is an examination series required for allopathic physician licensure in the United States (US). USMLE content is created and maintained by the National Board of Medical Examinations (NBME). The specialty composition of the USMLE and NBME taskforce members involved in the creation of examination content is currently unknown., Methods: Using the 2021 USMLE and 2021 NBME Committees and Task Forces documents, we determined each member's board-certified primary specialty and involvement in test material development committees who we dubbed "test writers". Total active physicians by primary specialty were recorded from the 2020 Physician Specialty Data Report published by the Association of American Medical Colleges (AAMC). Descriptive statistics and chi-square analysis were used to analyze the cohorts., Results: The USMLE and NBME test writer primary specialty composition was found to be significantly different compared to the US active physician population (USMLE χ 2 [32] = 172, p < .001 and NBME χ 2 [32] = 200, p < .001). Only nineteen specialties were represented within USMLE test writers, with three specialties being proportionally represented. Two specialties were represented within NBME test writers. Obstetrics and Gynecology physicians were proportionally represented in USMLE but not within NBME test writers. Internal Medicine (IM) accounts for the largest percentage of all USMLE test writers (60/197, 30%) with an excess representation of 31 individuals., Conclusions: There is an imbalance in the specialty representation of USMLE and NBME test writers compared to the US active physician population. These findings may have implications for the unbiased and accurate portrayal of topics in such national examinations; thus, future investigation is warranted., (© 2024. The Author(s).)

Abstract: Letters of reference (LORs) are a common component of the application process for residency training programs. With the United States Medical Licensing Examination Step 1 transitioning to pass/fail grading and with the increasing use of holistic review, the potential role of LORs is rising in importance. Among some key benefits are the ability to provide a broader and more holistic view of applicants, which can include highlighting elements of experiences or skills that could be missed in their application, as well as providing a third-party assessment of the applicant external to their rotation experiences. However, LORs also face issues, including variation in quality, challenges with comparability, and risk of bias. In this article, the authors discuss the unique benefits, limitations, and best practice recommendations for LORs in academic medicine. The authors also discuss future directions, including the role of artificial intelligence, unblinded, and co-created LORs., (Copyright © 2024 the Association of American Medical Colleges.)

This paper reports the findings of a Canada based multi-institutional study designed to investigate the relationships between admissions criteria, in-program assessments, and performance on licensing exams. The study's objective is to provide valuable insights for improving educational practices across different institutions. Data were gathered from six medical schools: McMaster University, the Northern Ontario School of Medicine University, Queen's University, University of Ottawa, University of Toronto, and Western University. The dataset includes graduates who undertook the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1) between 2015 and 2017. The data were categorized into five distinct sections: demographic information as well as four matrices: admissions, course performance, objective structured clinical examination (OSCE), and clerkship performance. Common and unique variables were identified through an extensive consensus-building process. Hierarchical linear regression and a manual stepwise variable selection approach were used for analysis. Analyses were performed on data set encompassing graduates of all six medical schools as well as on individual data sets from each school. For the combined data set the final model estimated 32% of the variance in performance on licensing exams, highlighting variables such as Age at Admission, Sex, Biomedical Knowledge, the first post-clerkship OSCE, and a clerkship theta score. Individual school analysis explained 41-60% of the variance in MCCQE1 outcomes, with comparable variables to the analysis from of the combined data set identified as significant independent variables. Therefore, strongly emphasising the need for variety of high-quality assessment on the educational continuum. This study underscores the importance of sharing data to enable educational insights. This study also had its challenges when it came to the access and aggregation of data. As such we advocate for the establishment of a common framework for multi-institutional educational research, facilitating studies and evaluations across diverse institutions. This study demonstrates the scientific potential of collaborative data analysis in enhancing educational outcomes. It offers a deeper understanding of the factors influencing performance on licensure exams and emphasizes the need for addressing data gaps to advance multi-institutional research for educational improvements., (© 2024. The Author(s), under exclusive licence to Springer Nature B.V.)

Introduction: Reports on the effects of changing the United States Medical Licensing Exam (USMLE) Step 1 examination scoring to pass/fail are evolving in the medical literature. This Council of Academic Family Medicine Educational Research Alliance family medicine clerkship directors' study seeks to describe family medicine clerkship directors' perceptions on the impact of incorporation of Step 1 pass/fail score reporting on students' family medicine clerkship performance., Methods: Ninety-six clerkship directors responded (56.8% response rate). After exclusion of Canadian schools, we analyzed 88 clerkship directors' responses from US schools. We used descriptive statistics for demographics and responses to survey questions. We used ꭓ2 analysis to determine statistically significant associations between survey items., Results: Clerkship directors did not observe changes in students' overall clinical performance after Step 1 pass/fail scoring (60.8%). Fifty percent of clerkship directors reported changes in Step 1 timing recommendations in the past 3 years. Reasons included curriculum redesign (30.5%), COVID (4.5%), change in Step 1 to pass/fail (11.0%), and other reasons (3.7%). Forty-five percent of these clerkship directors did not observe a change in students' clinical medical knowledge after Step 1 went to pass/fail. Eighty-four percent of these clerkship directors did not compare student performance on clerkship standardized exams before and after Step 1 score changes. We found no significant relationship between Step 1 timing and student performance., Conclusions: This study represents an early description of family medicine clerkship directors' perceived observations of the impact of Step 1 scoring changes on student performance. Continued investigation of the effects of USMLE Step 1 pass/fail scoring should occur.

Context: Failure of students to pass the National Medical Licensure Examination (NMLE) is a major problem for universities and the health system in Japan. To assist students at risk for NMLE failure as early as possible after admission, this study investigated the time points (from the time of admission to graduation) at which predictive pass rate (PPR) can be used to identify students at risk of failing the NMLE., Methods: Seven consecutive cohorts of medical students between 2012 and 2018 (n = 637) at the Gifu University Graduate School of Medicine were investigated. Using 7 variables before admission to medical school and 10 variables after admission, a prediction model to obtain the PPR for the NMLE was developed using logistic regression analysis at five time points, i.e., at admission and the end of the 1st, 2nd, 4th, and 6th grades. All students were divided into high (PPR < 95%) and low (PPR ≥ 95%) risk groups for failing the NMLE at the five time points, respectively, and the movement between the groups during 6 years in school was simulated., Results: Medical students who passed the NMLE had statistically significant factors at each of the 5 time points, and the number of significant variables increased as their grade in school advanced. In addition, two factors extracted at admission were also selected as significant variables at all other time points. Especially, age at entry had a consistent and significant effect during medical school., Conclusions: Risk analysis based on multiple variables, such as PPR, can inform more effective intervention compared to a single variable, such as performance in the mock exam. A longer prospective study is required to confirm the validity of PPR., (© 2024. The Author(s).)

Background: Over the past 2 years, researchers have used various medical licensing examinations to test whether ChatGPT (OpenAI) possesses accurate medical knowledge. The performance of each version of ChatGPT on the medical licensing examination in multiple environments showed remarkable differences. At this stage, there is still a lack of a comprehensive understanding of the variability in ChatGPT's performance on different medical licensing examinations., Objective: In this study, we reviewed all studies on ChatGPT performance in medical licensing examinations up to March 2024. This review aims to contribute to the evolving discourse on artificial intelligence (AI) in medical education by providing a comprehensive analysis of the performance of ChatGPT in various environments. The insights gained from this systematic review will guide educators, policymakers, and technical experts to effectively and judiciously use AI in medical education., Methods: We searched the literature published between January 1, 2022, and March 29, 2024, by searching query strings in Web of Science, PubMed, and Scopus. Two authors screened the literature according to the inclusion and exclusion criteria, extracted data, and independently assessed the quality of the literature concerning Quality Assessment of Diagnostic Accuracy Studies-2. We conducted both qualitative and quantitative analyses., Results: A total of 45 studies on the performance of different versions of ChatGPT in medical licensing examinations were included in this study. GPT-4 achieved an overall accuracy rate of 81% (95% CI 78-84; P<.01), significantly surpassing the 58% (95% CI 53-63; P<.01) accuracy rate of GPT-3.5. GPT-4 passed the medical examinations in 26 of 29 cases, outperforming the average scores of medical students in 13 of 17 cases. Translating the examination questions into English improved GPT-3.5's performance but did not affect GPT-4. GPT-3.5 showed no difference in performance between examinations from English-speaking and non-English-speaking countries (P=.72), but GPT-4 performed better on examinations from English-speaking countries significantly (P=.02). Any type of prompt could significantly improve GPT-3.5's (P=.03) and GPT-4's (P<.01) performance. GPT-3.5 performed better on short-text questions than on long-text questions. The difficulty of the questions affected the performance of GPT-3.5 and GPT-4. In image-based multiple-choice questions (MCQs), ChatGPT's accuracy rate ranges from 13.1% to 100%. ChatGPT performed significantly worse on open-ended questions than on MCQs., Conclusions: GPT-4 demonstrates considerable potential for future use in medical education. However, due to its insufficient accuracy, inconsistent performance, and the challenges posed by differing medical policies and knowledge across countries, GPT-4 is not yet suitable for use in medical education., Trial Registration: PROSPERO CRD42024506687; https://www.crd.york.ac.uk/prospero/display&#95;record.php?RecordID=506687., (©Mingxin Liu, Tsuyoshi Okuhara, XinYi Chang, Ritsuko Shirabe, Yuriko Nishiie, Hiroko Okada, Takahiro Kiuchi. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 25.07.2024.)

Context: Osteopathic (Doctor of Osteopathic Medicine [DO]) medical students account for more than 25 % of all medical students in the United States., Objectives: This study examined the predictive validity of Medical College Admission Test (MCAT) total scores and cumulative undergraduate grade point averages (UGPAs) for performance on the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA) Level 1 and Level 2-CE (Cognitive Evaluation) licensure examinations administered by the National Board of Osteopathic Medical Examiners (NBOME). Additionally, the study examined the degree to which MCAT total scores and UGPAs provide comparable prediction of student performance by key sociodemographic variables., Methods: This study involved a collaborative effort between the Association of American Medical Colleges (AAMC), the American Association of Colleges of Osteopathic Medicine (AACOM) and the NBOME. Data were examined for 39 accredited DO-granting medical schools in the United States during the 2017 application cycle. Researchers utilized three regression models that included MCAT total scores, cumulative UGPA, and combined MCAT total scores and cumulative UGPA to determine predictive validity. Researchers also examined the comparability of prediction for sociodemographic variables by examining the differences between observed and predicted error for both scores and pass/fail success rates., Results: Medium to large correlations were discernible between MCAT total scores, UGPA, and COMLEX-USA examination outcomes. For both COMLEX-USA Level 1 and Level 2-CE scores and pass/fail outcomes, MCAT scores alone provided superior predictive value to UGPA alone. However, MCAT scores and UGPA utilized in conjunction provided the best predictive value. When predicting both licensure examination scores and pass/fail outcomes by sociodemographic variables, all three models provided comparable predictive accuracy., Conclusions: Findings from this comprehensive study of DO-granting medical schools provide evidence for the value-added benefit of taking MCAT scores and UGPA into consideration, particularly when these measures are utilized in conjunction. Further, findings provide evidence indicating that individuals from different sociodemographic backgrounds who enter medical school with similar MCAT scores and UGPA perform similarly on licensure examination outcome measures., (© 2024 the author(s), published by De Gruyter, Berlin/Boston.)

Background: The National Medical Licensing Examination (NMLE) is the only objective, standardized metric to evaluate whether a medical student possessing the professional knowledge and skills necessary to work as a physician. However, the overall pass rate of NMLE in our hospital in 2021 was much lower than that of Peking Union Medical College Hospital, which was required to be further improved., Methods: To find the reasons for the unsatisfactory performance in 2021, the quality improvement team (QIT) organized regular face-to-face meetings for in-depth discussion and questionnaire, and analyzed the data by "Plato analysis" and "Brainstorming method". After finding out the reasons, the "Plan-Do-Check-Action" (PDCA) cycle was continued to identify and solve problems, which included the formulation and implementation of specific training plans by creating the "Gantt charts", the check of effects, and continuous improvements from 2021 to 2022. Detailed information about the performance of students in 2021 and 2022, and the attendance, assessment, evaluation and suggestions from our hospital were provided by the relevant departments, and the pass rate-associated data was collected online., Results: After the PDCA plan, the pass rate of NMLE in our hospital increased by 10.89% from 80.15% in 2021 to 91.04% in 2022 (P = 0.0109), with the pass rate of skill examination from 95.59% in 2021 to 99.25% in 2022 (P = 0.0581) and theoretical examination from 84.5% in 2021 to 93.13% in 2022 (P = 0.027). Additionally, the mean scores of all examinees increased with the theoretical examination score increasing from 377.0 ± 98.76 in 2021 to 407.6 ± 71.94 in 2022 (P = 0.004)., Conclusions: Our results showed a success application of the PDCA plan in our hospital which improved the pass rate of the NMLE in 2022, and the PDCA plan may provide a practical framework for future medical education and further improve the pass rate of NMLE in the next year., (© 2024. The Author(s).)

Objective: To investigate how the shift of the United States Medical Licensing Examination (USMLE) Step 1 to a Pass/Fail (P/F) scoring system impacts the perceptions of Urology Program Directors (PDs) on evaluating urology residency applicants., Methods and Materials: A cross-sectional survey was sent to 117 PDs, including questions about program characteristics, perceptions of shelf scores and medical school rank post-transition, beliefs about the predictive value of Step 1 and Step 2 Clinical Knowledge (CK) scores for board success and residency performance, and changes in applicant parameter ranking., Results: Forty-five PDs (38% response rate) participated. Notably, 49% favored releasing quantitative clerkship grades, and 71% valued medical school rank more. Opinions on Step 1 scores' correlation with board success were split (49% agreed), and 44% endorsed Step 2 CK scores' connection to board performance. As predictors of good residents, only 9% and 22% considered Step 1 and Step 2 CK scores, respectively, indicative. Clerkship grades and Urology rotation recommendation letters maintained significance, while research experience gained importance. Step 2 CK scores' importance rose but did not match Step 1 scores' previous significance., Conclusion: The transition to P/F for USMLE Step 1 adds intricacies to urology residency selection, exposing PDs' uncertainties regarding clerkship grades and the relevance of medical school rank. This research underscores the dynamic nature of urology residency admissions, emphasizing the increasing importance of research in evaluating applicants and a diminishing emphasis on volunteering and leadership., Competing Interests: Declaration of Competing Interest The authors declare that they have no conflict of interest., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Objectives: This study assessed the content of US Medical Licensing Examination question banks with regard to out-of-hospital births and whether the questions aligned with current evidence., Methods: Three question banks were searched for key words regarding out-of-hospital births. A thematic analysis was then utilized to analyze the results., Results: Forty-seven questions were identified, and of these, 55% indicated a lack of inadequate, limited, or irregular prenatal care in the question stem., Conclusions: Systematic studies comparing prenatal care in out-of-hospital births versus hospital births are nonexistent, leading to the potential for bias and adverse outcomes. Adjustments to question stems that accurately portray current evidence are recommended.

Background: The learning objectives in the current cross-sectional subject "Rehabilitation, Physical Medicine, Naturopathic Medicine" have been revised as part of the further development of the National Competency-Based Catalogue of Learning Objectives for Medicine (NKLM) to its new version 2.0. Since the NKLM is designed as an interdisciplinary catalogue, a subject assignment seemed necessary from the point of view of various stakeholders. Thus, the German Association of Scientific Medical Societies (AWMF) and the German medical faculties initiated a subject assignment process. The assignment process for the subject "Physical and Rehabilitative Medicine, Naturopathic Medicine" (PRM-NHV; according to the subject list of the first draft of the planned novel medical license regulations from 2020) is presented in this paper., Material and Methods: The AWMF invited its member societies to participate in the assignment of learning objectives of chapters VI, VII, and VIII of the NKLM 2.0 to the individual subjects to which they consider to contribute in teaching. For "PRM-NHV", representatives of the societies for rehabilitation sciences (DGRW), physical and rehabilitation medicine (DGPRM), orthopaedics and traumatology (DGOU), as well as for naturopathy (DGNHK) participated. In a structured consensus process according to the DELPHI methodology, the learning objectives were selected and consented. Subsequently, subject recommendations were made by the AWMF for each learning objective., Results: From the NKLM 2.0, a total of 100 competency-based learning objectives of chapters VII and VIII for the subject "PRM-NHV" were consented by the representatives of the involved societies for presentation on the NKLM 2.0 online platform., Conclusions: In the context of the revision process of medical studies in Germany and under the umbrella of the AWMF and the German medical faculties, a broad consensus of competency-based learning objectives in the subject "PRM-NHV" could be achieved. This provides an important orientation for all medical faculties both for the further development of teaching in the cross-sectional subject "Rehabilitation, Physical Medicine, Naturopathic Medicine" according to the 9th revision of the medical license regulations, which has been valid for twenty years, and for the preparation of the corresponding subjects in the draft bill of the novel license regulations., Competing Interests: Die Autor*innen geben an, dass kein Interessenkonflikt besteht.Zitierweise für diesen ArtikelPhys Med Rehab Kuror 2023; DOI: 10.1055/a-2118-0633, (Thieme. All rights reserved.)

Background: Evaluation of students' learning strategies can enhance academic support. Few studies have investigated differences in learning strategies between male and female students as well as their impact on United States Medical Licensing Examination® (USMLE) Step 1 and preclinical performance., Methods: The Learning and Study Strategies Inventory (LASSI) was administered to the classes of 2019-2024 (female (n = 350) and male (n = 262)). Students' performance on preclinical first-year (M1) courses, preclinical second-year (M2) courses, and USMLE Step 1 was recorded. An independent t-test evaluated differences between females and males on each LASSI scale. A Pearson product moment correlation determined which LASSI scales correlated with preclinical performance and USMLE Step 1 examinations., Results: Of the 10 LASSI scales, Anxiety, Attention, Information Processing, Selecting Main Idea, Test Strategies and Using Academic Resources showed significant differences between genders. Females reported higher levels of Anxiety (p < 0.001), which significantly influenced their performance. While males and females scored similarly in Concentration, Motivation, and Time Management, these scales were significant predictors of performance variation in females. Test Strategies was the largest contributor to performance variation for all students, regardless of gender., Conclusion: Gender differences in learning influence performance on STEP1. Consideration of this study's results will allow for targeted interventions for academic success., (© 2024. The Author(s).)

Efforts are being made to improve the time effectiveness of healthcare providers. Artificial intelligence tools can help transcript and summarize physician-patient encounters and produce medical notes and medical recommendations. However, in addition to medical information, discussion between healthcare and patients includes small talk and other information irrelevant to medical concerns. As Large Language Models (LLMs) are predictive models building their response based on the words in the prompts, there is a risk that small talk and irrelevant information may alter the response and the suggestion given. Therefore, this study aims to investigate the impact of medical data mixed with small talk on the accuracy of medical advice provided by ChatGPT. USMLE step 3 questions were used as a model for relevant medical data. We use both multiple-choice and open-ended questions. First, we gathered small talk sentences from human participants using the Mechanical Turk platform. Second, both sets of USLME questions were arranged in a pattern where each sentence from the original questions was followed by a small talk sentence. ChatGPT 3.5 and 4 were asked to answer both sets of questions with and without the small talk sentences. Finally, a board-certified physician analyzed the answers by ChatGPT and compared them to the formal correct answer. The analysis results demonstrate that the ability of ChatGPT-3.5 to answer correctly was impaired when small talk was added to medical data (66.8% vs. 56.6%; p = 0.025). Specifically, for multiple-choice questions (72.1% vs. 68.9%; p = 0.67) and for the open questions (61.5% vs. 44.3%; p = 0.01), respectively. In contrast, small talk phrases did not impair ChatGPT-4 ability in both types of questions (83.6% and 66.2%, respectively). According to these results, ChatGPT-4 seems more accurate than the earlier 3.5 version, and it appears that small talk does not impair its capability to provide medical recommendations. Our results are an important first step in understanding the potential and limitations of utilizing ChatGPT and other LLMs for physician-patient interactions, which include casual conversations., Competing Interests: The authors have declared that no competing interests exist., (Copyright: © 2024 Safrai, Azaria. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Context: The National Board of Osteopathic Medical Examiners (NBOME) administers the Comprehensive Osteopathic Medical Licensing Examination of the United States (COMLEX-USA), a three-level examination designed for licensure for the practice of osteopathic medicine. The examination design for COMLEX-USA Level 3 (L3) was changed in September 2018 to a two-day computer-based examination with two components: a multiple-choice question (MCQ) component with single best answer and a clinical decision-making (CDM) case component with extended multiple-choice (EMC) and short answer (SA) questions. Continued validation of the L3 examination, especially with the new design, is essential for the appropriate interpretation and use of the test scores., Objectives: The purpose of this study is to gather evidence to support the validity of the L3 examination scores under the new design utilizing sources of evidence based on Kane's validity framework., Methods: Kane's validity framework contains four components of evidence to support the validity argument: Scoring, Generalization, Extrapolation, and Implication/Decision. In this study, we gathered data from various sources and conducted analyses to provide evidence that the L3 examination is validly measuring what it is supposed to measure. These include reviewing content coverage of the L3 examination, documenting scoring and reporting processes, estimating the reliability and decision accuracy/consistency of the scores, quantifying associations between the scores from the MCQ and CDM components and between scores from different competency domains of the L3 examination, exploring the relationships between L3 scores and scores from a performance-based assessment that measures related constructs, performing subgroup comparisons, and describing and justifying the criterion-referenced standard setting process. The analysis data contains first-attempt test scores for 8,366 candidates who took the L3 examination between September 2018 and December 2019. The performance-based assessment utilized as a criterion measure in this study is COMLEX-USA Level 2 Performance Evaluation (L2-PE)., Results: All assessment forms were built through the automated test assembly (ATA) procedure to maximize parallelism in terms of content coverage and statistical properties across the forms. Scoring and reporting follows industry-standard quality-control procedures. The inter-rater reliability of SA rating, decision accuracy, and decision consistency for pass/fail classifications are all very high. There is a statistically significant positive association between the MCQ and the CDM components of the L3 examination. The patterns of associations, both within the L3 subscores and with L2-PE domain scores, fit with what is being measured. The subgroup comparisons by gender, race, and first language showed expected small differences in mean scores between the subgroups within each category and yielded findings that are consistent with those described in the literature. The L3 pass/fail standard was established through implementation of a defensible criterion-referenced procedure., Conclusions: This study provides some additional validity evidence for the L3 examination based on Kane's validity framework. The validity of any measurement must be established through ongoing evaluation of the related evidence. The NBOME will continue to collect evidence to support validity arguments for the COMLEX-USA examination series., (© 2024 the author(s), published by De Gruyter, Berlin/Boston.)

Background: We are currently experiencing a third digital revolution driven by artificial intelligence (AI), and the emergence of new chat generative pre-trained transformer (ChatGPT) represents a significant technological advancement with profound implications for global society, especially in the field of education., Methods: The aim of this study was to see how well ChatGPT performed on medical school exams and to highlight how it might change medical education and practice. Recently, OpenAI's ChatGPT (OpenAI, San Francisco; GPT-4 May 24 Version) was put to the test against a significant Polish medical specialization licensing exam (PES), and the results are in. The version of ChatGPT-4 used in this study was the most up-to-date model at the time of publication (GPT-4). ChatGPT answered questions from June 28, 2023, to June 30, 2023., Results: ChatGPT demonstrates notable advancements in natural language processing models on the tasks of medical question answering. In June 2023, the performance of ChatGPT was assessed based on its ability to answer a set of 120 questions, where it achieved a correct response rate of 67.1%, accurately responding to 80 questions., Conclusions: ChatGPT may be used as an assistance tool in medical education. While ChatGPT can serve as a valuable tool in medical education, it cannot fully replace human expertise and knowledge due to its inherent limitations.

Competing Interests: Competing interests: Brent Thoma reports honoraria and travel support from the Royal College of Physicians and Surgeons of Canada. Sandra Monteiro reports grants and travel support from McMaster University, and consulting fees from Touchstone Institute and Aquifer. Heather Water reports honoraria, salary support and travel support from McMaster University, honoraria from the Foundation for Medical Practice Education. Teresa Chan has received research grants, honoraria, salary support and support for attending meetings from McMaster University, the Royal College of Physicians and Surgeons of Canada, the Government of Ontario and Physician Services Incorporated, University of British Columbia, Northern Ontario School of Medicine, International Association of Medical Science Educators, Catholic University of Korea START Center for Medical Simulation, Texas Children’s Hospital, Baylor College of Medicine, Taipei Veterans General Hospital, Harvard Medical School and the University of Northern British Columbia, Academic Life in Emergency Medicine Limited Liability Company and the Canadian Association of Emergency Physicians. She also reports sitting on the board of the Canadian Institute for Translation of Knowledge and Education. No other competing interests were declared.

Purpose: The United States Medical Licensing Examination (USMLE) recently announced 2 policy changes: shifting from numeric score reporting on the Step 1 examination to pass/fail reporting and limiting examinees to 4 attempts for each Step component. In light of these policies, exam measures other than scores, such as the number of examination attempts, are of interest. Attempt limit policies are intended to ensure minimum standards of physician competency, yet little research has explored how Step attempts relate to physician practice outcomes. This study examined the relationship between USMLE attempts and the likelihood of receiving disciplinary actions from state medical boards., Method: The sample population was 219,018 graduates from U.S. and Canadian MD-granting medical schools who passed all USMLE Step examinations by 2011 and obtained a medical license in the United States, using data from the NBME and the Federation of State Medical Boards. Logistic regressions estimated how attempts on Steps 1, 2 Clinical Knowledge (CK), and 3 examinations influenced the likelihood of receiving disciplinary actions by 2018, while accounting for physician characteristics., Results: A total of 3,399 physicians (2%) received at least 1 disciplinary action. Additional attempts needed to pass Steps 1, 2 CK, and 3 were associated with an increased likelihood of receiving disciplinary actions (odds ratio [OR]: 1.07, 95% confidence interval [CI]: 1.01, 1.13; OR: 1.09, 95% CI: 1.03, 1.16; OR: 1.11, 95% CI: 1.04, 1.17, respectively), after accounting for other factors., Conclusions: Physicians who took multiple attempts to pass Steps 1, 2 CK, and 3 were associated with higher estimated likelihood of receiving disciplinary actions. This study offers support for licensure and practice standards to account for physicians' USMLE attempts. The relatively small effect sizes, however, caution policy makers from placing sole emphasis on this relationship., (Copyright © 2021 by the Association of American Medical Colleges.)

Summary: The United States Medical Licensing Examination announced the changing of Step 1 score reporting from a three-digit number to pass/fail beginning on January 1, 2022. Plastic surgery residency programs have traditionally used United States Medical Licensing Examination Step 1 scores to compare plastic surgery residency applicants. Without a numerical score, the plastic surgery residency application review process will likely change. This article discusses advantages, disadvantages, and steps forward for residency programs related to the upcoming change. The authors encourage programs to continue to seek innovative methods of objectively and holistically evaluating applications., (Copyright © 2021 by the American Society of Plastic Surgeons.)

Competing Interests: The author certifies that neither he, nor any members of his family, has funding or commercial associations (consultancies, stock ownership, equity interest, patent/licensing arrangements, etc.) that might pose a conflict of interest in connection with the submitted article. All ICMJE Conflict of Interest Forms for the author and Clinical Orthopaedics and Related Research® editors and board members are on file with the publication and can be viewed on request.

Competing Interests: The authors have no conflicts of interest to disclose.

Phenomenon: Internal medicine physicians in the United States must pass the American Board of Internal Medicine Internal Medicine Maintenance of Certification (ABIM IM-MOC) examination as part of their ABIM IM-MOC requirements. Many of these physicians use an examination product to help them prepare, such as e-Learning products, including the ACP's MKSAP, UpToDate, and NEJM Knowledge+, yet their effectiveness remains largely unstudied. Approach: We compared ABIM IM-MOC examination performance among 177 physicians who attempted an ABIM IM-MOC examination between 2014-2017 and completed at least 75% of the NEJM Knowledge+ product prior to the ABIM IM-MOC examination and 177 very similar matched control physicians who did not use NEJM Knowledge+. Our measures of ABIM IM-MOC exam performance for NEJM Knowledge+ users were based on the results of the first attempt immediately following the NEJM Knowledge+ use and for non-users were based on the applicable matched examination performance. The three dichotomous examination performance outcomes measured on the first attempt at the ABIM IM-MOC examination included pass rate, scoring in the upper quartile, and scoring in the lower quartile. Findings: Use of NEJM Knowledge+ was associated with a regression adjusted 10.6% (5.37% to 15.8%) greater likelihood of passing the MOC examination (p < .001), 10.7% (2.61% to 18.7%) greater likelihood of having an examination score in the top quartile (p = .009), and -10.8% (-16.8% to -4.86%) lower likelihood of being in the bottom quartile of the MOC examination (p < .001) as compared to similar physicians who did not use NEJM Knowledge+. Insight: Physicians who used NEJM Knowledge+ had better ABIM IM-MOC exam performance. Further research is needed to determine what aspects of e-Learning products best prepare physicians for MOC examinations.