Search

The Extended matching Questions (EMQs), or R-type questions, are format of selected-response. The validity evidence for this format is crucial, but there have been reports of misunderstandings about validity. It is unclear what kinds of evidence should be presented and how to present them to support their educational impact. This review explores the pattern and quality of reporting the sources of validity evidence of EMQs in health professions education, encompassing content, response process, internal structure, relationship to other variables, and consequences. A systematic search in the electronic databases including MEDLINE via PubMed, Scopus, Web of Science, CINAHL, and ERIC was conducted to extract studies that utilize EMQs. The framework for a unitary concept of validity was applied to extract data. A total of 218 titles were initially selected, the final number of titles was 19. The most reported pieces of evidence were the reliability coefficient, followed by the relationship to another variable. Additionally, the adopted definition of validity is mostly the old tripartite concept. This study found that reporting and presenting validity evidence appeared to be deficient. The available evidence can hardly provide a strong validity argument that supports the educational impact of EMQs. This review calls for more work on developing a tool to measure the reporting and presenting validity evidence.

Introduction: Addressing systemic bias in medical school assessment is an urgent task for medical education. This paper outlines recommendations on topic areas for further research on systemic bias, developed from a workshop discussion at the 2023 annual meeting of the Society of Directors of Research in Medical Education., Materials and Methods: During the workshop, directors engaged in small-group discussions on guidelines to address bias in assessment practices following a proposed categorization of 'Do's,' 'Don'ts,' and 'Don't knows' and listed their insights using anonymous sticky notes, which were shared and discussed with the larger group of participants. The authors performed a content analysis of the notes through deductive and inductive coding. We reviewed and discussed our analysis to reach consensus., Results: The workshop included 31 participants from 28 institutions across the US and Canada, generating 51 unique notes. Participants identified 23 research areas in need of further study. The inductive analysis of proposed research areas revealed four main topics: 1) The role of interventions, including pre-medical academic interventions, medical-education interventions, assessment approaches, and wellness interventions; 2) Professional development, including the definition and assessment of professionalism and professional identity formation; 3) Context, including patient care and systemic influences; and 4) Research approaches., Discussion: While limited to data from a single workshop, the results offered perspectives about areas for further research shared by a group of directors of medical education research units from diverse backgrounds. The workshop produced valuable insights into the need for more evidence-based interventions that promote more equitable assessment practices grounded in real-world situations and that attenuate the effects of bias.

While objective clinical structured examination (OSCE) is a worldwide recognized and effective method to assess clinical skills of undergraduate medical students, the latest Ottawa conference on the assessment of competences raised vigorous debates regarding the future and innovations of OSCE. This study aimed to provide a comprehensive view of the global research activity on OSCE over the past decades and to identify clues for its improvement. We performed a bibliometric and scientometric analysis of OSCE papers published until March 2024. We included a description of the overall scientific productivity, as well as an unsupervised analysis of the main topics and the international scientific collaborations. A total of 3,224 items were identified from the Scopus database. There was a sudden spike in publications, especially related to virtual/remote OSCE, from 2020 to 2024. We identified leading journals and countries in terms of number of publications and citations. A co-occurrence term network identified three main clusters corresponding to different topics of research in OSCE. Two connected clusters related to OSCE performance and reliability, and a third cluster on student's experience, mental health (anxiety), and perception with few connections to the two previous clusters. Finally, the United States, the United Kingdom, and Canada were identified as leading countries in terms of scientific publications and collaborations in an international scientific network involving other European countries (the Netherlands, Belgium, Italy) as well as Saudi Arabia and Australia, and revealed the lack of important collaboration with Asian countries. Various avenues for improving OSCE research have been identified: i) developing remote OSCE with comparative studies between live and remote OSCE and issuing international recommendations for sharing remote OSCE between universities and countries; ii) fostering international collaborative studies with the support of key collaborating countries; iii) investigating the relationships between student performance and anxiety.

Objective Structured Clinical Examinations (OSCE) is an objective assessment approach that enables the evaluation of students' practical skills by simulating real clinical scenarios. This scale is specifically designed to evaluate the attributes and characteristics of the OSCE from the perspective of health sciences students, thereby filling a crucial gap in the existing assessment methods. This research conducted a methodological design involving 220 students from the faculties of medicine and nursing between 2022 and 2023. Data were gathered using a "Descriptive Form" and "OSCEPS." Factor analyses were performed for research and validation to ascertain the scale's factorial structure. The factorial analysis was used to test the structural validity of the scale. A structure consisting of 19 items was developed. The Cronbach's alpha coefficient was 0.943. Fit indices resulting from the study were RMSEA = 0.084; GFI = 0.839; CFI = 0.902; IFI = 0.902; RFI = 0.824; NFI = 0.843; TLI = 0.889. This study proves that the OSCEPS is a highly reliable and valid scale with strong psychometric properties. It effectively assesses health sciences students' perceptions of OSCE, thereby enhancing the credibility and usefulness of this assessment tool., (© 2024 The Author(s). Nursing & Health Sciences published by John Wiley & Sons Australia, Ltd.)

Objective: Pharmacists are often the last line of defense from medical errors caused by inaccurate calculations. Effective teaching and assessment of pharmaceutical calculations is essential in preparing students for successful pharmacy careers. This study aimed to elucidate the potential benefit of self-testing practice questions on final examination performance in a first-year pharmaceutical calculations course., Methods: One-hundred and sixteen students across the class of 2026 and 2027 were given access to 110 online practice calculation questions eight days prior to the final examination. Retrospective analysis using Pearson's Correlation Coefficient and an Unpaired t-test was used to assess the effect of self-study practice questions on exam performance., Results: A correlation between higher quiz scores and enhanced final examination scores was observed for both the class of 2026 and 2027. A greater number of attempts on practice quiz questions correlated with a higher score on the final examination for the class of 2026, but not the class of 2027. Also, an earlier first access date was associated with higher final examination scores specifically for the class of 2026., Conclusion: This retrospective study was conducted to evaluate the use of practice calculation questions on final examination performance, and results reveal that the utilization of practice calculation questions positively correlates with improved final examination performance, notably observed in the class of 2026 but not in 2027. These findings suggest the potential efficacy of this preparatory method across various pharmaceutical courses and other calculation-based disciplines internationally., Competing Interests: Declaration of competing interest There are no financial disclosures or conflicts of interest., (Copyright © 2024 The Authors. Published by Elsevier Inc. All rights reserved.)

Background: Traditionally, the assessment of student nursing skills relies heavily on lecturer evaluations. However, lecturers, as the primary evaluators, may be influenced by their individual subjective preferences, experience, and knowledge background. This limitation needs thorough consideration to improve students' assessment skills and ensure that assessment tasks are more objective and accurate. Peer assessment appears to take it into account. Currently, improving the quality of peer assessment in Basic Nursing skills remains a challenge for educators., Aim: To assess the effectiveness of peer assessment as applied to the basic nursing skills examination process and suggestions for improvement, to understand nursing students' perceptions of it, and to explore the feasibility of peer assessment as an alternative to single faculty assessment as perceived by nursing students., Design: Qualitative research., Settings: Conducted in a clinical skills simulation laboratory at a university in Wuxi, Jiangsu Province, China., Participants: 23 Second-year nursing students applying peer assessment to Basic Nursing course exams., Methods: Used face-to-face, semi-structured interviews, and thematic analysis to analyze data., Results: Three major themes and nine subthemes were identified: (1) Positive sentiments towards peer evaluation (e.g., enhancing the learning capacity, increasing emotional intelligence, and improving the objectivity and fairness of nursing skills assessment); (2) Challenges encountered in the peer evaluation process (e.g., inconsistent scoring criteria, exacerbate anxiety, and triggering team conflict); (3) Suggestions for improving peer evaluation (e.g., applying the design of peer assessment exam content, incorporation of lecturer evaluation, and refinement and materialization of marking criteria)., Conclusions: In this study, while the positive impact is evident, the challenges faced by nursing students and future recommendations for peer assessment application are critical. This study provides valuable insights into the potential of peer assessment to enhance nursing education and provides recommendations for optimizing peer assessment strategies for meaningful integration into Nursing Education practice., Competing Interests: Declaration of competing interest The authors declare that they have no competing interests, including financial and non-financial interests., (Copyright © 2024. Published by Elsevier Ltd.)

Background: This review aimed to analyze the effectiveness of the teaching and assessment methodologies used to develop fundamental clinical skills in pharmacist who are training in patient care., Methods: SCOPUS, EMBASE, ERIC, and PubMed were searched for original studies that featured randomized controlled trials as the study design published until March 2024. The search and extraction process followed PRISMA Guidelines., Results: The database search resulted in 2954 articles, of which 14 met the inclusion criteria. Four studies developed and tested interactive web-based software as the teaching methodologies. Eight studies applied simulation to their teaching and/or evaluation strategies. Two articles used high fidelity simulation, and the remaining studies used standardized patients associated with other teaching and evaluation techniques. The simulation methodologies were more effective than the conventional ones in three studies. In the other studies, the interventions were as effective or better than the control, albeit there no meaningful differences between the methods. In the studies that focused on the assessment methods, immediate feedback was preferred by students over delayed feedback. Additionally, the tested assessment tool, General Level Framework, proposed a pragmatic assessment from which the individual's training needs were identified., Conclusion: Few studies involved the objective quantification of learning beyond pre- and post-intervention knowledge tests. Proper assessment in pharmaceutical education requires expansion beyond the administration of student satisfaction, self-efficacy research tools, and knowledge assessments, and should encompass an examination of clinical performance and critical thinking., Competing Interests: Declaration of competing interest The authors declare that they have no conflicts interests. The funders had no role in the study design, data collection and analysis, decision on the publication or preparation of the manuscript., (Copyright © 2024 Elsevier Inc. All rights reserved.)

Background: Training of standardized patients (SPs) for national high-stakes OSCE helps to ensure a reliable assessment of student performance in various clinical scenarios. However, workflow protocols to train SPs vary. Medical schools adopt specific measures to ensure standardization., Approach: We present a development workflow of the SPs' training framework for high-stakes OSCE using a volunteer SPs' bank. Our approach was guided by the social learning theory. Three educators and 17/20 (85%) members of our volunteer SPs' bank worked in a collaborative partnership on the construction of pedagogical content of the training framework comprising three 2-hour sessions. Since SPs have to demonstrate acquired behaviors, intended learning outcomes used the words "apply", "perform" and "participate.", Evaluation: A principal part of the evaluation was the achievement of intended learning outcomes by the SPs during 3 formative OSCEs. Seventeen SPs, 356 fourth year medical students and 60 examiners participated. Quantitative and qualitative data were collected by post-session questionnaires and analyzed using descriptive statistics and thematic analysis. Twelve examiners evaluated a mean of 29.7+/-0.14 SD patient-student encounters. In total, 15/16 SPs (93.8%) considered the contact with students as easy and 4 SPs (31%) reported the experience as stressful. Two themes emerged from the free-text comments: "Gaining experience as SP" and "Concerns for evaluated students.", Implication: The proposed SPs' training framework for high-stakes OSCE may be useful for other medical disciplines and health professions initiating SP-based assessment programs. The strategy of development and evaluation are outlined to guide a successful application of the curriculum standards., (© 2024 The Author(s). The Clinical Teacher published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Context: One central consideration in health professions education (HPE) is to ensure we are making sound and justifiable decisions based on the assessment instruments we use on health professionals. To achieve this goal, HPE assessment researchers have drawn on Kane's argument-based framework to ascertain the validity of their assessment tools. However, the original four-inference model proposed by Kane - frequently used in HPE validation research - has its limitations in terms of what each inference entails and what claims and sources of backing are housed in each inference. The under-specification in the four-inference model has led to inconsistent practices in HPE validation research, posing challenges for (i) researchers who want to evaluate the validity of different HPE assessment tools and/or (ii) researchers who are new to test validation and need to establish a coherent understanding of argument-based validation., Methods: To address these identified concerns, this article introduces the expanded seven-inference argument-based validation framework that is established practice in the field of language testing and assessment (LTA). We explicate (i) why LTA researchers experienced the need to further specify the original four Kanean inferences; (ii) how LTA validation research defines each of their seven inferences and (iii) what claims, assumptions and sources of backing are associated with each inference. Sampling six representative validation studies in HPE, we demonstrate why an expanded model and a shared disciplinary validation framework can facilitate the examination of the validity evidence in diverse HPE validation contexts., Conclusions: We invite HPE validation researchers to experiment with the seven-inference argument-based framework from LTA to evaluate its usefulness to HPE. We also call for greater interdisciplinary dialogue between HPE and LTA since both disciplines share many fundamental concerns about language use, communication skills, assessment practices and validity in assessment instruments., (© 2024 The Author(s). Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Background: This study aimed to evaluate the performance of GPT-3.5, GPT-4, GPT-4o and Google Bard on the United States Medical Licensing Examination (USMLE), the Professional and Linguistic Assessments Board (PLAB), the Hong Kong Medical Licensing Examination (HKMLE) and the National Medical Licensing Examination (NMLE)., Methods: This study was conducted in June 2023. Four LLMs (Large Language Models) (GPT-3.5, GPT-4, GPT-4o and Google Bard) were applied to four medical standardized tests (USMLE, PLAB, HKMLE and NMLE). All questions are multiple-choice questions and were sourced from the question banks of these examinations., Results: In USMLE step 1, step 2CK and Step 3, there are accuracy rates of 91.5%, 94.2% and 92.7% provided from GPT-4o, 93.2%, 95.0% and 92.0% provided from GPT-4, 65.6%, 71.6% and 68.5% provided from GPT-3.5, and 64.3%, 55.6%, 58.1% from Google Bard, respectively. In PLAB, HKMLE and NMLE, GPT-4o scored 93.3%, 91.7% and 84.9%, GPT-4 scored 86.7%, 89.6% and 69.8%, GPT-3.5 scored 80.0%, 68.1% and 60.4%, and Google Bard scored 54.2%, 71.7% and 61.3%. There was significant difference in the accuracy rates of four LLMs in the four medical licensing examinations., Conclusion: GPT-4o performed better in the medical licensing examinations than other three LLMs. The performance of the four models in the NMLE examination needs further improvement., Clinical Trial Number: Not applicable., Competing Interests: Declarations. Ethics approval and consent to participate: Not applicable. Informed consent: Informed consent were not required. Institutional review board: This study does not involve human or animal subjects and therefore did not require approval from the Institutional Ethical Board. Competing interests: The authors declare no competing interests., (© 2024. The Author(s).)

An international medical graduate (IMG) is a doctor who has received their basic medical qualification from a medical school located in a different country from that in which they practice or intend to practice. IMGs are known to face difficulties in their working lives, including differential attainment in assessment. The objective of this review is to map key concepts and types of evidence in academic and gray literature relating to international medical graduates' experiences of clinical competency assessment and to identify knowledge gaps on this topic by systematically searching, selecting, and synthesizing existing knowledge. All studies will relate to IMGs. The concept of interest will be IMGs' experiences of assessment. The context will be postgraduate, licensing or credentialing medical assessments of clinical competence. This review will be conducted in accordance with the Joanna Briggs Institute (JBI) methodology for scoping reviews. Seven electronic databases will be searched for literature published between 2009 and 2024: the Australian Education Index, British Education Index, ERIC, PubMed, PsycINFO, Scopus, and SocINDEX. Gray literature will be searched using Google, Google Scholar, and published reports from postgraduate training bodies and medical licensing organizations. Documents will be independently screened, selected, and extracted by two researchers using a piloted data-extraction tool. Data will be analyzed and presented in tables and in a narrative format. Trial resgistration: Scoping review registration: Open Science Framework: https://osf.io/8gdm7., Competing Interests: The authors have declared that no competing interests exist., (Copyright: © 2024 Hynes et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Licensure exams play a crucial role in ensuring the competence of individuals entering a profession, thereby safeguarding the public and maintaining the quality and integrity of the profession. In Sweden, dentists educated outside the European Union seeking to practise dentistry must undergoa re-certification process. The re-certification process includes a theoretical examination where pass marks are set using a three-level Angoff method. This study aimed to determine the consistency of the Angoff ratings using independent panels at two Swedish universities. Two cohorts of panellists were included in the study: one reference and one external. The reference panel was responsible for rating the upcoming theoretical examinations in the proficiency test, which were used to set the pass mark. The external panel, recruited from a dental school at a university in another region in Sweden, provided ratings after the examinations. Three examinations during 2019-2020 were included in this study (267 items in total). There was a strong correlation (ρ ≥ 0.70, p < .001) between the ratings of the two independent panels, with no significant differences in item ratings across the full exams, dental disciplines, and professional qualifications analysed. This suggests that the three-level Angoff method reliably produces similar standards for assessing the competence of the minimally qualified dentist across different institutions. The expectations of the minimally qualified but still acceptable dentist were comparable between the two independent panels across the three theoretical examinations explored. The alignment between the panels indicates valid, reliable standards across institutions, despite the independent syllabi of the two study programmes. However, while there is an alignment, differences in ratings remain. Consequently, involving multiple institutions in future standard-setting processes could help ensure that the standards reflect a broader range of educational practices, supporting the credibility of licensure examinations., Competing Interests: The authors have declared that no competing interests exist., (Copyright: © 2024 Dalum et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Certification exams are often the least understood aspect of certification processes. This article delves into the development of certification exams, highlighting their crucial role in assessing the knowledge and skills required for effective job performance. It explores how exams are meticulously crafted to align with specific job tasks, knowledge areas, and skill sets within various nursing specialties. Through collaborative efforts between practicing nurses and exam developers, these exams are designed to uphold validity, reliability, and fairness standards. Ultimately, certification exams are pivotal in evaluating nurses' competencies in vital aspects of clinical practice, facilitated by ongoing partnerships between nurses and certifying bodies. [ J Contin Educ Nurs. 2024;55(11):513-516.] .

Objective: This study aims to evaluate the impact of redesigning an entrustable professional activities (EPAs) assessment tool on the accuracy of student performance assessment within pharmacy education., Methods: The study used retrospective programmatic data for students on clinical rotations over a 3-year period and compared entrustment levels assigned by preceptors with suggested entrustment levels. This tool was redesigned to separate formative EPA feedback from final grade determination. Data were analyzed using chi-squared tests to identify trends in students ABOVE, AT, or BELOW the suggested entrustment levels. Additionally, to account for intercohort variability, the relationship between students ABOVE the suggested level of entrustment and postgraduate metrics was examined., Results: After the implementation of the revised tool, there was a significant decrease (-3%) in the percentage of students scoring ABOVE the suggested entrustment levels and an increase in the percentage of students scoring AT (+1%) or BELOW (+2%) the suggested entrustment levels. Changes were also observed in individual patient care settings, with a decrease in grade inflation and an increase in accurate assessments. North American Pharmacist Licensure Examination (NAPLEX) pass rates, residency match rates, and grade point average did not correlate with entrustment levels., Conclusion: The redesigned EPA assessment tool demonstrated a decrease in grade inflation resulting in more accurate assessments. The tool's focus on holistic grading and narrative descriptors contributed to better alignment between preceptor assessment and school-suggested achievement levels. This study suggests that EPA assessments in pharmacy education could benefit from a stronger emphasis on formative feedback and the use of holistic assessment methods for final grade determinations. The findings underscore the potential advantages of considering a separation between EPA scoring and final grades, prompting the Academy to explore their assessment practices to better reflect student performance in clinical experiences., Competing Interests: Declaration of Competing Interest None declared., (Copyright © 2024 American Association of Colleges of Pharmacy. Published by Elsevier Inc. All rights reserved.)

Objective: This study aims to test the accuracy of large language models (LLMs) in answering standardized pharmacy examination practice questions., Methods: The performance of 3 LLMs (generative pretrained transformer [GPT]-3.5, GPT-4, and Chatsonic) was evaluated on 2 independent North American Pharmacist Licensure Examination practice question sets sourced from McGraw Hill and RxPrep. These question sets were further classified into binary question categories of adverse drug reaction (ADR) questions, scenario questions, treatment questions, and select-all questions. Python was used to run χ 2 tests to compare model and question-type accuracy., Results: Of the 3 LLMs tested, GPT-4 achieved the highest accuracy, with 87% accuracy on the McGraw Hill question set and 83.5% accuracy on the RxPrep question set. In comparison, GPT-3.5 had 68.0% and 60.0% accuracy on those question sets, respectively, and Chatsonic had 60.5% and 62.5% accuracy on those question sets, respectively. All models performed worse on select-all questions compared with non-select-all questions (GPT-3: 42.3% vs 66.2%; GPT-4: 73.1 vs 87.2%; Chatsonic: 36.5% vs 71.6%). GPT-4 had statistically higher accuracy in answering ADR questions (96.1%) compared with non-ADR questions (83.9%)., Conclusion: Our study found that GPT-4 outperformed GPT-3.5 and Chatsonic in answering North American Pharmacist Licensure Examination pharmacy licensure examination practice questions, particularly excelling in answering questions related to ADRs. These results suggest that advanced LLMs such as GPT-4 could be used for applications in pharmacy education., Competing Interests: Declaration of Competing Interest A.E. reports a relationship with National Institute of Aging that includes: funding grants. The remaining authors declare no competing interests., (Copyright © 2024 American Association of Colleges of Pharmacy. Published by Elsevier Inc. All rights reserved.)

Background: This article is the last of a four-part series to guide educators on the construction and evaluation of multiple-choice test items in the post-licensure environment. Previous articles in this series described the problem and the mechanics of test item construction and evaluation., Method: A replicable strategy for evaluating the organizational process for constructing multiple-choice test questions is provided. Steps taken to create change are described; work tools are provided., Results: Guidance and training are needed to create multiple-choice test questions. Many educators have not had training in item construction. Educators welcomed training. Personalized mentorship resulted in improvement. Asynchronous learning alone was helpful and well received and improved self-perceived knowledge, yet fell short of achieving competence., Conclusion: Voluntary training may not be adequate to assure enculturation of best practices without accountability standards and monitoring. Future research is indicated to assess the situation and provide national standards for adoption within health care organizations. [ J Contin Educ Nurs. 2024;55(11):535-542.] .

Purpose: Objective structured clinical examinations (OSCE) are a valuable assessment within healthcare education, as they provide the opportunity for students to demonstrate clinical competency, but can be resource intensive to provide faculty graders. The purpose of this study was to determine how overall OSCE scores compared between faculty, peer, and self-evaluations within a Doctor of Pharmacy (PharmD) curriculum., Methods: This study was conducted during the required nonprescription therapeutics course. Seventy-seven first-year PharmD students were included in the study, with 6 faculty members grading 10-15 students each. Students were evaluated by 3 graders: self, peer, and faculty. All evaluators utilized the same rubric. The primary endpoint of the study was to compare the overall scores between groups. Secondary endpoints included interrater reliability and quantification of feedback type based on the evaluator group., Results: The maximum possible score for the OSCE was 50 points; the mean scores for self, peer, and faculty evaluations were 43.3, 43.5, and 41.7 points, respectively. No statistically significant difference was found between the self and peer raters. However, statistical significance was found in the comparison of self versus faculty (p = 0.005) and in peer versus faculty (p < 0.001). When these scores were correlated to a letter grade (A, B, C or less), higher grades had greater similarity among raters compared to lower scores. Despite differences in scoring, the interrater reliability, or W score, on overall letter grade was 0.79, which is considered strong agreement., Conclusions: This study successfully demonstrated how peer and self-evaluation of an OSCE provides a comparable alternative to traditional faculty grading, especially in higher performing students. However, due to differences in overall grades, this strategy should be reserved for low-stakes assessments and basic skill evaluations., Competing Interests: Declaration of competing interest We declare no conflicts of interest or financial interests that the authors or members of their immediate families have in any product or service discussed in the manuscript, including grants (pending or received), employment, gifts, stock holdings or options, honoraria, consultancies, expert testimony, patents, and royalties., (Copyright © 2024. Published by Elsevier Inc.)

Objective: Certifications in psychiatric-mental health nursing promote safe practice by psychiatric-mental health nurse practitioners (PMHNPs) and nurses (PMHNs) and help protect the public from harm. This protection begins with the development of an examination that meets rigorous national education, practice, and accreditation standards and reflects PMHNPs' or PMHNs' clinical practice. Achievement and maintenance of a certification is a journey that involves a commitment to lifelong learning and the improvement of the field of psychiatric-mental health nursing through involvement in the examination process., Methods: This discussion paper outlines the role nurses can play in the development of certification examinations. It describes the process of developing an effective certification examination, including the role of standards, accrediting bodies, and content experts; determining necessary tasks, knowledge, and skills; surveying practitioners to validate information; writing test questions; and ongoing analysis of examination content. The Psychiatric-Mental Health Nurse Practitioner (across the lifespan) Certification (PMHNP-BC) is presented as an example of the process., Results: This discussion paper raises awareness of how certification exams are developed, PMHNPs participate in certification development, and volunteering promotes career development., Conclusion: The PMHNP-BC examination is based on education, practice, and certification accreditation standards and reflects current clinical practice. PMHNPs can (a) point to the rigor of certification as an indication of the quality of care they deliver, (b) volunteer to participate in the examination process to ensure examination rigor, and (c) advance their careers through the development and application of a valuable skill set., Competing Interests: Declaration of Conflicting InterestsThe authors declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: TM, KC, and JK are employed by the American Nurses Credentialing Center, which administers certification examinations. CG, JR, and JS are members of the Commission on Board Certification for the American Nurses Credentialing Center.

Introduction: Research in various areas indicates that expert judgment can be highly inconsistent. However, expert judgment is indispensable in many contexts. In medical education, experts often function as examiners in rater-based assessments. Here, disagreement between examiners can have far-reaching consequences. The literature suggests that inconsistencies in ratings depend on the level of performance a to-be-evaluated candidate shows. This possibility has not been addressed deliberately and with appropriate statistical methods. By adopting the theoretical lens of ecological rationality, we evaluate if easily implementable strategies can enhance decision making in real-world assessment contexts., Methods: We address two objectives. First, we investigate the dependence of rater-consistency on performance levels. We recorded videos of mock-exams and had examiners (N=10) evaluate four students' performances and compare inconsistencies in performance ratings between examiner-pairs using a bootstrapping procedure. Our second objective is to provide an approach that aids decision making by implementing simple heuristics., Results: We found that discrepancies were largely a function of the level of performance the candidates showed. Lower performances were rated more inconsistently than excellent performances. Furthermore, our analyses indicated that the use of simple heuristics might improve decisions in examiner pairs., Discussion: Inconsistencies in performance judgments continue to be a matter of concern, and we provide empirical evidence for them to be related to candidate performance. We discuss implications for research and the advantages of adopting the perspective of ecological rationality. We point to directions both for further research and for development of assessment practices., (© 2024. The Author(s).)

Purpose: Written assessment comments are needed to archive feedback and inform decisions. Regrettably, comments are often impoverished, leaving performance-relevant information undocumented. Research has focused on content and supervisor's ability and motivation to write it but has not sufficiently examined how well the undocumented information lends itself to being written as comments. Because missing information threatens the validity of assessment processes, this study examined the performance information that resists being written., Method: Two sequential data collection methods and multiple elicitation techniques were used to triangulate unwritten assessment comments. Between November 2022 and January 2023, physicians in Canada were recruited by email and social media to describe experiences with wanting to convey assessment information but feeling unable to express it in writing. Fifty supervisors shared examples via survey. From January to May 2023, a subset of 13 participants were then interviewed to further explain what information resisted being written and why it seemed impossible to express in writing and to write comments in response to a video prompt or for their own "unwritable" example. Constructivist grounded theory guided data collection and analysis., Results: Not all performance-relevant information was equally writable. Information resisted being written as assessment comments when it would require an essay to be expressed in writing, belonged in a conversation and not in writing, or was potentially irrelevant and unverifiable. In particular, disclosing sensitive information discussed in a feedback conversation required extensive recoding to protect the learner and supervisor-learner relationship., Conclusions: When documenting performance information as written comments is viewed as an act of disclosure, it becomes clear why supervisors may feel compelled to leave some comments unwritten. Although supervisors can be supported in writing better assessment comments, their failure to write invites a reexamination of expectations for documenting feedback and performance information as written comments on assessment forms., (Copyright © 2024 the Association of American Medical Colleges.)

Background and Purpose: Nursing student retention is essential to meet workforce demands. Jeffreys' Nursing Student Retention Student Perception Appraisal-Revised (SPA-R1) has been used extensively to understand factors that impact retention. Psychometric testing of the SPA-R1 contributes to greater confidence in the instrument's reliability and validity. Methods: Item response theory and specifically, the single parameter polytomous Rasch model was used as a framework for fit statistic testing and rating scale diagnostics of the SPA-R1. This was a secondary analysis of a convenience sample of undergraduate prelicensure nursing students. The setting for the previous study was virtual, and the study period was 2022. Results: The model item characteristic curves for the 27 items of the SPA-R1 have similar shapes and are clustered in proximity. Overall, there are three clusters of items evident in the Rasch standardized residual contrast. The Rasch scale diagnostics indicated that the scale appropriately monotonically increases. However, there is a greater than 5 logit distance between does not apply and severely restricts, between severely restricts and moderately restricts, and between does not restrict or support and moderately supports. These large threshold distances indicate that additional steps in the scale may be warranted. The items cover the mid-range of the amount of retention perceptions; however, there are no items that represent the highest magnitude of the perceived amount of influence on retention. Conclusions: This study contributes further evidence to support the validity and reliability of the SPA-R1. We recommend adding steps to the scale, removing the does not apply response option, and considering scoring by three domains or clusters., (© Copyright 2024 Springer Publishing Company, LLC.)

Background: Although digital health is essential for improving health care, its adoption remains slow due to the lack of literacy in this area. Therefore, it is crucial for health professionals to acquire digital skills and for a digital competence assessment and accreditation model to be implemented to make advances in this field., Objective: This study had two objectives: (1) to create a specific map of digital competences for health professionals and (2) to define and test a digital competence assessment and accreditation model for health professionals., Methods: We took an iterative mixed methods approach, which included a review of the gray literature and consultation with local experts. We used the arithmetic mean and SD in descriptive statistics, P values in hypothesis testing and subgroup comparisons, the greatest lower bound in test diagnosis, and the discrimination index in study instrument analysis., Results: The assessment model designed in accordance with the competence content defined in the map of digital competences and based on scenarios had excellent internal consistency overall (greatest lower bound=0.91). Although most study participants (110/122, 90.2%) reported an intermediate self-perceived digital competence level, we found that the vast majority would not attain a level-2 Accreditation of Competence in Information and Communication Technologies., Conclusions: Knowing the digital competence level of health professionals based on a defined competence framework should enable such professionals to be trained and updated to meet real needs in their specific professional contexts and, consequently, take full advantage of the potential of digital technologies. These results have informed the Health Plan for Catalonia 2021-2025, thus laying the foundations for creating and offering specific training to assess and certify the digital competence of such professionals., (©Francesc Saigí-Rubió, Teresa Romeu, Eulàlia Hernández Encuentra, Montse Guitert, Erik Andrés, Elisenda Reixach. Originally published in JMIR Medical Education (https://mededu.jmir.org), 17.10.2024.)

Introduction: Validity is frequently conceptualized in health professions education (HPE) assessment as an argument that supports the interpretation and uses of data. However, previous work has shown that many validity scholars believe argument and argumentation are relatively lacking in HPE. To better understand HPE's discourse around argument and argumentation with regard to assessment validity, the authors explored the discourses present in published HPE manuscripts., Methods: The authors used a bricolage of critical discourse analysis approaches to understand how the language in influential peer reviewed manuscripts has shaped HPE's understanding of validity arguments and argumentation. The authors used multiple search strategies to develop a final corpus of 39 manuscripts that were seen as influential in how validity arguments are conceptualized within HPE. An analytic framework drawing on prior research on Argumentation Theory was used to code manuscripts before developing themes relevant to the research question., Results: The authors found that the elaboration of argument and argumentation within HPE's validity discourse is scant, with few components of Argumentation Theory (such as intended audience) existing within the discourse. The validity as an argument discourse was legitimized via authorization (reference to authority), rationalization (reference to institutionalized action), and mythopoesis (narrative building). This legitimation has cemented the validity as an argument discourse in HPE despite minimal exploration of what argument and argumentation are., Discussion: This study corroborates previous work showing the dearth of argument and argumentation present within HPE's validity discourse. An opportunity exists to use Argumentation Theory in HPE to better develop validation practices that support use of argument., Competing Interests: Erik Driessen is the Editor-in-Chief for Perspectives on Medical Education. He was removed from all editorial processes relating to the handling of this submission., (Copyright: © 2024 The Author(s).)

Background: The creation of large language models (LLMs) such as ChatGPT is an important step in the development of artificial intelligence, which shows great potential in medical education due to its powerful language understanding and generative capabilities. The purpose of this study was to quantitatively evaluate and comprehensively analyze ChatGPT's performance in handling questions for the National Nursing Licensure Examination (NNLE) in China and the United States, including the National Council Licensure Examination for Registered Nurses (NCLEX-RN) and the NNLE., Objective: This study aims to examine how well LLMs respond to the NCLEX-RN and the NNLE multiple-choice questions (MCQs) in various language inputs. To evaluate whether LLMs can be used as multilingual learning assistance for nursing, and to assess whether they possess a repository of professional knowledge applicable to clinical nursing practice., Methods: First, we compiled 150 NCLEX-RN Practical MCQs, 240 NNLE Theoretical MCQs, and 240 NNLE Practical MCQs. Then, the translation function of ChatGPT 3.5 was used to translate NCLEX-RN questions from English to Chinese and NNLE questions from Chinese to English. Finally, the original version and the translated version of the MCQs were inputted into ChatGPT 4.0, ChatGPT 3.5, and Google Bard. Different LLMs were compared according to the accuracy rate, and the differences between different language inputs were compared., Results: The accuracy rates of ChatGPT 4.0 for NCLEX-RN practical questions and Chinese-translated NCLEX-RN practical questions were 88.7% (133/150) and 79.3% (119/150), respectively. Despite the statistical significance of the difference (P=.03), the correct rate was generally satisfactory. Around 71.9% (169/235) of NNLE Theoretical MCQs and 69.1% (161/233) of NNLE Practical MCQs were correctly answered by ChatGPT 4.0. The accuracy of ChatGPT 4.0 in processing NNLE Theoretical MCQs and NNLE Practical MCQs translated into English was 71.5% (168/235; P=.92) and 67.8% (158/233; P=.77), respectively, and there was no statistically significant difference between the results of text input in different languages. ChatGPT 3.5 (NCLEX-RN P=.003, NNLE Theoretical P<.001, NNLE Practical P=.12) and Google Bard (NCLEX-RN P<.001, NNLE Theoretical P<.001, NNLE Practical P<.001) had lower accuracy rates for nursing-related MCQs than ChatGPT 4.0 in English input. English accuracy was higher when compared with ChatGPT 3.5's Chinese input, and the difference was statistically significant (NCLEX-RN P=.02, NNLE Practical P=.02). Whether submitted in Chinese or English, the MCQs from the NCLEX-RN and NNLE demonstrated that ChatGPT 4.0 had the highest number of unique correct responses and the lowest number of unique incorrect responses among the 3 LLMs., Conclusions: This study, focusing on 618 nursing MCQs including NCLEX-RN and NNLE exams, found that ChatGPT 4.0 outperformed ChatGPT 3.5 and Google Bard in accuracy. It excelled in processing English and Chinese inputs, underscoring its potential as a valuable tool in nursing education and clinical decision-making., (© Zelin Wu, Wenyi Gan, Zhaowen Xue, Zhengxin Ni, Xiaofei Zheng, Yiyi Zhang. Originally published in JMIR Medical Education (https://mededu.jmir.org).)

Background: Multiple-choice test questions are among the main measures of knowledge used by educators in the postlicensure environment; however, these tests are often constructed in the absence of guidelines or the means to evaluate examination quality after administration., Method: Previously, guidance was provided on constructing quality test items. Here we present instruction for professional development specialists to use postadministration test data for an item analysis, providing insight on test flaws and opportunities for iterative examination improvement., Results: The topics of item difficulty, index of discrimination, and distractor analysis are covered for independent analysis, and topics such as reliability are addressed for those who have access to a formal program., Conclusion: Three levels of strategies are described: using a learning management system for item analysis, using free open-source software, and using a minimal standards method of evaluating test items. [ J Contin Educ Nurs . 2024;55(10):487-492.] .

Background: Nursing clinical competence assessment has acquired a special relevance at the undergraduate and postgraduate levels in recent years. In this context, the Objective Structured Clinical Assessment (OSCA) has emerged as a valid and feasible method of assessing nursing competence. The Satisfaction with Nursing Skill Examination: Objective Structured Clinical Assessment (SINE-OSCA) scale is a valid and reliable 10-item measure that has been developed to evaluate nursing students' satisfaction with the OSCA in the Australian context. Given the importance that OSCA has gained in Spain, it is necessary to validate this tool to be used in one of the most spoken languages in the world., Objectives: The purpose of the study was to carry out a modification of the SINE-OSCA, cross-cultural adaptation and a psychometric analysis of the new S-OSCA with Spanish nursing students., Design: A multicenter study of questionnaire development and validation was carried out in 2023 in four Spanish university nursing centers. The study was carried out in 3 phases: design, pilot implementation, and construct validation., Participants/setting: The total population of students from these centers amounted to 1350 students. The final sample consisted of 364 nursing students, selected by convenience sampling., Methods: The process of translation and cultural adaptation of SINE-OSCA to the Spanish population was carried out following the guidelines proposed by Beaton et al. Content validation, Internal consistency and temporal reliability were evaluated., Results: The S-OSCA presents values in the psychometric indicators (V AIKEN, Bland-Altman diagram, and IVC Lawshe) that exceed the cut-off values established even considering the lower limit of the confidence intervals. This spanish version of the SINE-OSCA has a Cronbach's alpha value that is slightly higher than that reported for that original version (0.928 CI 95 % (0.913-0.94)). Regarding temporal reliability, the S-OSCA scale was completed in 40 nursing students at two times separated by an interval of 15 days. The Intraclass Correlation Coefficient (ICC) obtained was 0.974 CI 95 % (0.952-0.986)., Conclusions: The S-OSCA instrument proves to be robust enough to guarantee the quality of its results up to 15 days post-OSCA., Competing Interests: Declaration of competing interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 The Authors. Published by Elsevier Ltd.. All rights reserved.)

Aim: The main objective of this study was to collaboratively develop a competency-based assessment template for oncology nurses., Background: The care of oncology patients and their families requires very specific knowledge and skills, as well as that nurses are trained not only in theoretical and practical knowledge but also in competencies. There is a lack of specific tools to assess competencies in oncology and it is important to include students in the learning process., Design: This study has a Participatory Action Research., Methods: A focus group was conducted for data collection. The process consisted of 4 phases: 1- Identification of the problem; 2- Planning of the change; 3- Realization of the change; 4- Effect of the change and reflection. A purposive sampling where all the students of the master's degree in oncology nursing who wished to participate voluntarily were selected. A total of 23 students participated. The information was recorded and fully transcribed. Triangulation of researchers was used during data analysis., Results: One of the most important aspects included in the new tool was the need for a comprehensive approach to people with oncological diseases and their relatives. The students emphasized the need to primarily address the physical, psychological and social dimensions in this approach., Conclusion: This study developed a competency-based assessment template for oncology nurses, which includes the assessment of knowledge, skills and attitudes. The findings could help other oncology nurse educators or professionals to evaluate their training sessions and even their own knowledge in a healthcare setting., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024 The Authors. Published by Elsevier Ltd.. All rights reserved.)

Objective: This study aimed to develop a prompt engineering procedure for test question mapping and then determine the effectiveness of test question mapping using Chat Generative Pre-Trained Transformer (ChatGPT) compared to human faculty mapping., Methods: We conducted a cross-sectional study to compare ChatGPT and human mapping using a sample of 139 test questions from modules within the Integrated Pharmacotherapeutics course series. The test questions were mapped by 3 faculty members to both module objectives and the Accreditation Council for Pharmacy Education Standards 2016 (Standards 2016) to create the "correct answer". Prompt engineering procedures were created to facilitate mapping with ChatGPT, and ChatGPT mapping results were compared with human mapping., Results: ChatGPT mapped test questions directly to the "correct answer" based on human consensus in 68.0% of cases, and the program matched with at least one individual human response in another 20.1% of cases for a total of 88.1% agreement with human mappers. When humans fully agreed with the mapping decision, ChatGPT was more likely to map correctly., Conclusion: This study presents a practical use case with prompt engineering tailored for college assessment or curriculum committees to facilitate efficient test questions and educational outcomes mapping., Competing Interests: Declaration of Competing Interest Editorial Advisory Board member for AJPE - T.J.M. If there are other authors, they declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2024. Published by Elsevier Inc.)

We address defining and identifying students with dyslexia within the context of multi-tier systems of support (MTSS). We review proposed definitions of dyslexia, evidence for proposed definitional attributes, and emphasize the role of instructional response in identifying students with dyslexia. We identify dyslexia as individuals with specific deficits in reading and spelling single words combined with inadequate response to evidence-based instruction. We propose a hybrid identification process in which assessment is utilized within school-wide MTSS allowing for integration of routinely collected progress monitoring data as well integrating with more formal diagnostic measures. This proposed "hybrid" method demonstrates strong evidence for valid decision-making and directly informs instruction. We close proposing a revised definition of dyslexia that incorporates these elements., (© 2024. The Author(s), under exclusive licence to The International Dyslexia Association.)

Phenomenon: This article reports the under-researched presentation of demographic, social, and economic diversity in medical school examination questions. Approach:  The present study audited 3,566 pre-clinical and clinical multiple-choice and short answer examination questions in the same year (2018) from three medical schools in two continents to review the diversity of patients portrayed. The audit was based on an extension of Critical Race Theory beyond race and ethnicity to include pertinent social determinants of health. Findings:  Patients were presented in 1,537 (43.1%) of the audited examination questions. Apart from age (89.4%) and binary genders (93.9%), other diversity characteristics were rarely portrayed (ethnicity 7.2%, relationship status 1.9%, sexual identity 1.1%, socio-economic status 0.5%, geographic residence 0.1%, disability 0.1%), or not at all (non-binary genders; residency status; religion/spirituality). Insights:  While presenting excessive and unnecessary patient characteristics in examination questions should be avoided, the absence of many diversity aspects may reduce examination authenticity and defeat the teaching of diversity in medicine. Medical schools should consider a routine audit and reasonable improvement of the diversity features of patients in examination questions to support teaching and learning activities addressing patients' diversity.

Performance assessments for student pharmacists are required by the Accreditation Council for Pharmacy Education to monitor student knowledge, retention, and professional skill sets. Near the end of their Advanced Pharmacy Practice Experiences (APPEs), fourth-year pharmacy students (P4s) at the University of Texas at Austin College of Pharmacy must pass a milestone exam (P4ME) to graduate. The P4ME contains four sections: adult medicine, ambulatory care, community, and hospital. If a student fails one or more sections of the initial multiple-choice P4ME, they must retake that specific section(s) on the makeup P4ME. If a student fails the makeup multiple-choice P4ME, they participate in a case based oral exam designed to determine whether they meet minimal competencies to pass the P4ME. This report outlines the process undertaken by the faculty to develop a high-stakes P4 milestone oral exam. While only a few students have participated in the process, outcomes differentiated P4s eligible for graduation versus those who needed to repeat an APPE. Lessons learned about the oral exam process include minimizing the number of faculty in attendance, shortening the exam duration, and standardizing the exam process. Positive aspects include remediation led by a subject matter expert and implementation of a unique rubric to assess student performance. Overall, the authors find the P4 oral milestone exam to be a feasible assessment tool for students who perform poorly on the initial and makeup multiple-choice P4MEs., Competing Interests: Declaration of competing interest There is no similar work that has been published or currently under review by the authors. The authors do not have any conflict of interest and meet the full authorship criteria. The authors do not have any financial disclosures or funding disclosures, and the authors do not have any conflicts of interest., (Copyright © 2023. Published by Elsevier Inc.)

Objectives: This study assessed the relationship between academic performance and National Licensing Examination (NLE) scores of Ethiopian anaesthetists and proposed academic performance thresholds that predict failing the NLE., Design: A retrospective cross-sectional study., Setting: Academic performance measures were collected from eight universities and matched to total and subdomain NLE scores for anaesthetists. Four universities were first generation (oldest), three were second generation and one was third generation (newest)., Participants: 524 anaesthetists., Measures: Selected course grades, yearly (Y1-Y4) and cumulative grade point averages (GPAs) and in-school exit exam status were correlated with NLE scores. Linear regression was used to identify predictors of NLE score. Receiver operating characteristic curves were plotted to estimate GPA thresholds that predict failing the NLE., Results: Total and general surgery anaesthesia subdomain NLE scores showed moderate to strong linear relationships with most academic performance indicators. However, correlations between the rest of the subdomain NLE scores and composite course scores were either small or negligible (r<0.5). For every 1-point increase in a 4-point cumulative GPA and composite senior year practicum course score, the first-attempt NLE score increased by 12.61 (95% CI 10.03 to 15.19; p<0.0001) and 3.98 (95% CI 1.77 to 6.19; p=0.001) percentage points, respectively. Those entering university directly from high school and those attending a third-generation university would, on average, score 2.79 (95% CI 0.23 to 5.35; p=0.033) and 4.34 (95% CI 1.42 to 7.26; p=0.004) higher NLE scores compared with nurse entrants and first-generation university graduates, respectively. On a 4-point grading system, the thresholds for failing the NLE were below 2.68 for year-1 GPA, 2.62 for year-2 GPA, 2.72 for year-3 GPA, 3.04 for year-4 GPA and 2.87 for cumulative GPA., Conclusions: The Ethiopian anaesthetist's NLE score modestly correlated with most academic performance indicators. Based on pass/fail thresholds, the NLE could assist training programmes in improving NLE pass rates., Competing Interests: Competing interests: None declared., (© Author(s) (or their employer(s)) 2024. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.)

Background: ChatGPT, a recently developed artificial intelligence (AI) chatbot, has demonstrated improved performance in examinations in the medical field. However, thus far, an overall evaluation of the potential of ChatGPT models (ChatGPT-3.5 and GPT-4) in a variety of national health licensing examinations is lacking. This study aimed to provide a comprehensive assessment of the ChatGPT models' performance in national licensing examinations for medical, pharmacy, dentistry, and nursing research through a meta-analysis., Methods: Following the PRISMA protocol, full-text articles from MEDLINE/PubMed, EMBASE, ERIC, Cochrane Library, Web of Science, and key journals were reviewed from the time of ChatGPT's introduction to February 27, 2024. Studies were eligible if they evaluated the performance of a ChatGPT model (ChatGPT-3.5 or GPT-4); related to national licensing examinations in the fields of medicine, pharmacy, dentistry, or nursing; involved multiple-choice questions; and provided data that enabled the calculation of effect size. Two reviewers independently completed data extraction, coding, and quality assessment. The JBI Critical Appraisal Tools were used to assess the quality of the selected articles. Overall effect size and 95% confidence intervals [CIs] were calculated using a random-effects model., Results: A total of 23 studies were considered for this review, which evaluated the accuracy of four types of national licensing examinations. The selected articles were in the fields of medicine (n = 17), pharmacy (n = 3), nursing (n = 2), and dentistry (n = 1). They reported varying accuracy levels, ranging from 36 to 77% for ChatGPT-3.5 and 64.4-100% for GPT-4. The overall effect size for the percentage of accuracy was 70.1% (95% CI, 65-74.8%), which was statistically significant (p < 0.001). Subgroup analyses revealed that GPT-4 demonstrated significantly higher accuracy in providing correct responses than its earlier version, ChatGPT-3.5. Additionally, in the context of health licensing examinations, the ChatGPT models exhibited greater proficiency in the following order: pharmacy, medicine, dentistry, and nursing. However, the lack of a broader set of questions, including open-ended and scenario-based questions, and significant heterogeneity were limitations of this meta-analysis., Conclusions: This study sheds light on the accuracy of ChatGPT models in four national health licensing examinations across various countries and provides a practical basis and theoretical support for future research. Further studies are needed to explore their utilization in medical and health education by including a broader and more diverse range of questions, along with more advanced versions of AI chatbots., (© 2024. The Author(s).)

Background: The ability of an expert's item difficulty ratings to predict test-taker actual performance is an important aspect of licensure examinations. Expert judgment is used as a primary source of information for users to make prior decisions to determine the pass rate of test takers. The nature of raters involved in predicting item difficulty is central to set credible standards. Therefore, this study aimed to assess and compare raters' prediction and actual Multiple-Choice Questions' difficulty of the undergraduate medicine licensure examination (UGMLE) in Ethiopia., Method: 815 examinees' responses to 200 Multiple-Choice Questions (MCQs) were used in this study. The study also included experts' item difficulty ratings of seven physicians who participated in the standard settings of UGMLE. Then, analysis was conducted to understand experts' rating variation in predicting the actual difficulty levels of examinees. Descriptive statistics was used to profile the mean rater's and actual difficulty value for MCQs, and ANOVA was used to compare the mean differences between raters' prediction of item difficulty. Additionally, regression analysis was used to understand the interrater variations in item difficulty predictions compared to the actual difficulty. The proportion of variance of actual difficulty explained from rater prediction was computed using regression analysis., Results: In this study, the mean difference between raters' prediction and examinees' actual performance was inconsistent across the exam domains. The study revealed a statistically significant strong positive correlation between the actual and predicted item difficulty in exam domains eight and eleven. However, a non-statistically significant very weak positive correlation was reported in exam domains seven and twelve. The multiple comparison analysis showed significant differences in mean item difficulty ratings between raters. In the regression analysis, experts' item difficulty ratings of the UGMLE had 33% power in predicting the actual difficulty level. The regression model also showed a moderate positive correlation (R = 0.57) that was statistically significant at F (6, 193) = 15.58, P = 0.001., Conclusion: This study demonstrated the complex process for assessing the difficulty level of MCQs in the UGMLE and emphasized the benefits of using experts' ratings in advance. To ensure the exams maintain the necessary reliable and valid scores, raters' accuracy on the UGMLE must be improved. To achieve this, techniques that align with the evolving assessment methodologies must be developed., (© 2024. The Author(s).)

Validity has long held a venerated place in education, leading some authors to refer to it as the "sine qua non" or "cardinal virtue" of assessment. And yet, validity has not held a fixed meaning; rather it has shifted in its definition and scope over time. In this Eye Opener, the authors explore if and how current conceptualizations of validity fit a next era of assessment that prioritizes patient care and learner equity. They posit that health profession education's conceptualization of validity will change in three related but distinct ways. First, consequences of assessment decisions will play a central role in validity arguments. Second, validity evidence regarding impacts of assessment on patients and society will be prioritized. Third, equity will be seen as part of validity rather than an unrelated concept. The authors argue that health professions education has the agency to change its ideology around validity, and to align with values that predominate the next era of assessment such as high-quality care and equity for learners and patients., Competing Interests: The authors have no competing interests to declare., (Copyright: © 2024 The Author(s).)

Standardized tests intend to reduce information asymmetry by providing a common and objective measure of students' academic performance. The basic assumption underlying standardized testing is that differences in student performance on standardized tests should be attributed primarily to differences in the quality of education received by students. However, there is evidence that environmental factors can affect standardized test scores, which may result in anomalous observations or outliers that show a distortion of student performance. In this regard, the exclusion of Spain from PISA 2018 is particularly interesting as Spanish data met PISA 2018 Technical Standards but showed implausible student-response behavior. The aim of this paper is to complement the OECD's analysis of Spain's exclusion from PISA 2018 by exploring the potential reasons behind the outlier results, focusing on the Madrid region., Competing Interests: The authors have declared that no competing interests exist., (Copyright: © 2024 van Grieken et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.)

Aim: This paper aims to report on two modifications made to improve the measurement precision of the Ms. Olsen test. Specifically, three items were added to the Ms. Olsen test and an extended scoring was applied to some items., Design: The competence assessment had a quantitative cross-sectional design comprising of test results from 111 Registered Nurses in four municipalities in South-Eastern Norway., Methods: The Rasch model was applied to evaluate the measurement properties in four versions: the Ms. Olsen test with 19 items and the same with three added items, and the Ms. Olsen test with 19 items with an expanded scoring for the original and expanded version, respectively., Results: The person separation indexes were improved from 0.50 to 0.62. Other measurement properties were not alternating between the four versions; all had shortcomings in terms of targeting (person measure means 3.02-3.87) and unidimensionality (% t-test >5% 9.01%-13.51%)., Conclusion: The clinical relevance and relatively short time spent on testing makes the Ms. Olsen test a reasonable choice and a step in the right direction for assessing competence as a means of targeting continuous professional development of nurses throughout their career. Nevertheless, depending on what kind of decisions are to be made, reliability might still be too low and further development is suggested., Public Contribution: The initiative for the competence assessment came from nursing leaders in the four municipalities involved. The municipalities were represented in all stages of the research process through co-author LS, that is, design, data collection, data analysis, writing of the manuscript as well as dissemination of the results to the four municipalities., Implications for Practice: The Ms. Olsen test exemplifies a move towards objective assessments in nursing. Precise and reliable measurements are essential to support the ongoing professional development of nurses., (© 2024 The Author(s). International Journal of Older People Nursing published by John Wiley & Sons Ltd.)

This paper reports the findings of a Canada based multi-institutional study designed to investigate the relationships between admissions criteria, in-program assessments, and performance on licensing exams. The study's objective is to provide valuable insights for improving educational practices across different institutions. Data were gathered from six medical schools: McMaster University, the Northern Ontario School of Medicine University, Queen's University, University of Ottawa, University of Toronto, and Western University. The dataset includes graduates who undertook the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1) between 2015 and 2017. The data were categorized into five distinct sections: demographic information as well as four matrices: admissions, course performance, objective structured clinical examination (OSCE), and clerkship performance. Common and unique variables were identified through an extensive consensus-building process. Hierarchical linear regression and a manual stepwise variable selection approach were used for analysis. Analyses were performed on data set encompassing graduates of all six medical schools as well as on individual data sets from each school. For the combined data set the final model estimated 32% of the variance in performance on licensing exams, highlighting variables such as Age at Admission, Sex, Biomedical Knowledge, the first post-clerkship OSCE, and a clerkship theta score. Individual school analysis explained 41-60% of the variance in MCCQE1 outcomes, with comparable variables to the analysis from of the combined data set identified as significant independent variables. Therefore, strongly emphasising the need for variety of high-quality assessment on the educational continuum. This study underscores the importance of sharing data to enable educational insights. This study also had its challenges when it came to the access and aggregation of data. As such we advocate for the establishment of a common framework for multi-institutional educational research, facilitating studies and evaluations across diverse institutions. This study demonstrates the scientific potential of collaborative data analysis in enhancing educational outcomes. It offers a deeper understanding of the factors influencing performance on licensure exams and emphasizes the need for addressing data gaps to advance multi-institutional research for educational improvements., (© 2024. The Author(s), under exclusive licence to Springer Nature B.V.)

Objective: Pharmacist licensure exam first-time pass rates have declined for several years. Predictors of licensure exam performance, including the Pharmacy College Admission Test, are no longer required, necessitating alternative strategies. Our aim is to assess the relationship between numerical scores on advanced pharmacy practice experience (APPE) exams and pharmacist licensure exams first-time pass rates., Methods: We conducted a single-center, retrospective, observational analysis. APPE examination results between May 2020 and November 2023 were analyzed for association with licensure information from the Texas State Board of Pharmacy. Exam scores were collected for student cohorts graduating from 2021 to 2022 as study cohort data. Correlation between exam scores and probability of successful licensure was assessed using logistic regression. A classification and regression tree analysis identified the most significant threshold. Predictive ability of the best-fit model was prospectively validated using the 2023 graduate cohort., Results: The overall licensure success rate was 80.6% for the study cohort (2021-2022, n = 206). Exam scores were well correlated to the likelihood of licensing success. The most significant threshold was 77.8%. The success rate was 82.5% vs 16.7% for those scoring above and below the threshold, respectively. The observed and predicted licensure success rates were 91.0% and 88.8%, respectively, in the validation cohort (2023, n = 89). The positive and negative prediction values were 94.9% and 40.0%, respectively., Conclusion: The performances on APPE exams were reasonable in predicting the first-time licensure success rate for a graduating class. Our assessment appears promising as a risk-stratification tool for students in gaining successful pharmacist licensure., Competing Interests: Declaration of Competing Interest None declared., (Copyright © 2024 American Association of Colleges of Pharmacy. Published by Elsevier Inc. All rights reserved.)

Abstract: Letters of reference (LORs) are a common component of the application process for residency training programs. With the United States Medical Licensing Examination Step 1 transitioning to pass/fail grading and with the increasing use of holistic review, the potential role of LORs is rising in importance. Among some key benefits are the ability to provide a broader and more holistic view of applicants, which can include highlighting elements of experiences or skills that could be missed in their application, as well as providing a third-party assessment of the applicant external to their rotation experiences. However, LORs also face issues, including variation in quality, challenges with comparability, and risk of bias. In this article, the authors discuss the unique benefits, limitations, and best practice recommendations for LORs in academic medicine. The authors also discuss future directions, including the role of artificial intelligence, unblinded, and co-created LORs., (Copyright © 2024 the Association of American Medical Colleges.)

Demonstration of competency is an essential component of nursing orientation in the practice environment. Standardization of the assessment with a validated instrument supports high-quality care and patient safety. This study engaged six nurse experts to establish content validation for a newly developed competency-based assessment framed by the domains of the national standard for nursing education. Two rounds of rigorous review resulted in a scale content validity index universal agreement calculation of .891, indicating that all items were evaluated as relevant for inclusion in this comprehensive measure of registered nurse clinical competency in the medical-surgical specialty., (Copyright © 2024 Wolters Kluwer Health, Inc. All rights reserved.)

Abstract: The objective structured clinical examination (OSCE) is effective for the evaluation of clinical competence. Studies examining the use of OSCEs in undergraduate mental health nursing education in the United States are limited. A pilot study and a follow-up study were conducted to establish the reliability and validity of a mental health OSCE to evaluate the clinical competence of prelicensure nursing students. International Nursing Association for Clinical Simulation and Learning Standards of Best Practice were used to guide the design and implementation. Results from both studies provide evidence for the use of OSCE in undergraduate mental health nursing education., Competing Interests: The authors have declared no conflict of interest., (Copyright © 2024 National League for Nursing.)

When large achievement tests are conducted regularly, items need to be calibrated before being used as operational items in a test. Methods have been developed to optimally assign pretest items to examinees based on their abilities. Most of these methods, however, are intended for situations where examinees arrive sequentially to be assigned to calibration items. In several calibration tests, examinees take the test simultaneously or in parallel. In this article, we develop an optimal calibration design tailored for such parallel test setups. Our objective is both to investigate the efficiency gain of the method as well as to demonstrate that this method can be implemented in real calibration scenarios. For the latter, we have employed this method to calibrate items for the Swedish national tests in Mathematics. In this case study, like in many real test situations, items are of mixed format and the optimal design method needs to handle that. The method we propose works for mixed-format tests and accounts for varying expected response times. Our investigations show that the proposed method considerably enhances calibration efficiency., (© 2024. The Author(s).)

First impressions can influence rater-based judgments but their contribution to rater bias is unclear. Research suggests raters can overcome first impressions in experimental exam contexts with explicit first impressions, but these findings may not generalize to a workplace context with implicit first impressions. The study had two aims. First, to assess if first impressions affect raters' judgments when workplace performance changes. Second, whether explicitly stating these impressions affects subsequent ratings compared to implicitly-formed first impressions. Physician raters viewed six videos where learner performance either changed (Strong to Weak or Weak to Strong) or remained consistent. Raters were assigned two groups. Group one (n = 23, Explicit) made a first impression global rating (FIGR), then scored learners using the Mini-CEX. Group two (n = 22, Implicit) scored learners at the end of the video solely with the Mini-CEX. For the Explicit group, in the Strong to Weak condition, the FIGR (M = 5.94) was higher than the Mini-CEX Global rating (GR) (M = 3.02, p < .001). In the Weak to Strong condition, the FIGR (M = 2.44) was lower than the Mini-CEX GR (M = 3.96 p < .001). There was no difference between the FIGR and the Mini-CEX GR in the consistent condition (M = 6.61, M = 6.65 respectively, p = .84). There were no statistically significant differences in any of the conditions when comparing both groups' Mini-CEX GR. Therefore, raters adjusted their judgments based on the learners' performances. Furthermore, raters who made their first impressions explicit showed similar rater bias to raters who followed a more naturalistic process., (© 2023. The Author(s), under exclusive licence to Springer Nature B.V.)

Purpose: To assess the Consultation And Relational Empathy (CARE) measure as a tool for examiners to assess medical students' empathy during Objective and Structured Clinical Examinations (OSCEs), as the best tool for assessing empathy during OSCEs remains unknown., Methods: We first assessed the psychometric properties of the CARE measure, completed simultaneously by examiners and standardized patients (SP, either teachers - SP teacher - or civil society members - SP civil society ), for each student, at the end of an OSCE station. We then assessed the qualitative/quantitative agreement between examiners and SP., Results: We included 129 students, distributed in eight groups, four groups for each SP type. The CARE measure showed satisfactory psychometric properties in the context of the study but moderate, and even poor inter-rater reliability for some items. Considering paired observations, examiners scored lower than SPs ( p  < 0.001) regardless of the SP type. However, the difference in score was greater when the SP was a SP teacher rather than a SP civil society ( p  < 0.01)., Conclusion: Despite acceptable psychometric properties, inter-rater reliability of the CARE measure between examiners and SP was unsatisfactory. The choice of examiner as well as the type of SP seems critical to ensure a fair measure of empathy during OSCEs.