Search

Background: Virtual reality (VR) is increasingly used in healthcare education, offering immersive training experiences that are as effective as conventional methods, with benefits like cost-effectiveness, replicating complex scenarios, and reduced need for physical resources. However, the use of VR as an assessment tool is still emerging, particularly in nursing and medical education. The aim of this systematic review was to examine how immersive VR is used as an assessment tool for nursing and medical students. Methods: Embase, PubMed, PsycINFO, Cochrane, CINAHL, and ERIC were searched for articles that assessed nursing and/or medical students using immersive/HMD VR. The data was extracted, and content analysis was performed. Results: Twenty-six studies met the inclusion criteria, investigating VR assessments in various settings mostly emergencies. Assessments focused on core competencies Patient Care such as first triage, Interpersonal and Communication Skills (e.g., interprofessional communication), and Medical Knowledge (e.g., about coma), utilizing a range of assessment methods from knowledge to performance levels. VR was used either as an automated or supporting assessment tool. Practical considerations in VR implementation were also examined, such as hardware and software. Conclusion: The use of VR in medical education assessment shows promise, particularly for emergency scenarios and performance-based tasks related to core competencies such as Patient Care, Interpersonal and Communication Skills, and Medical Knowledge. While this technology offers opportunities to automate assessments and reduce examiner workload, challenges related to software, costs, and feasibility must be addressed. Additionally, aligning learning objectives, teaching methods, and VR assessments through constructive alignment is essential to ensure effective implementation as both a teaching and evaluation tool. [ABSTRACT FROM AUTHOR]

Abstract Background Virtual reality (VR) is increasingly used in healthcare education, offering immersive training experiences that are as effective as conventional methods, with benefits like cost-effectiveness, replicating complex scenarios, and reduced need for physical resources. However, the use of VR as an assessment tool is still emerging, particularly in nursing and medical education. The aim of this systematic review was to examine how immersive VR is used as an assessment tool for nursing and medical students. Methods Embase, PubMed, PsycINFO, Cochrane, CINAHL, and ERIC were searched for articles that assessed nursing and/or medical students using immersive/HMD VR. The data was extracted, and content analysis was performed. Results Twenty-six studies met the inclusion criteria, investigating VR assessments in various settings mostly emergencies. Assessments focused on core competencies Patient Care such as first triage, Interpersonal and Communication Skills (e.g., interprofessional communication), and Medical Knowledge (e.g., about coma), utilizing a range of assessment methods from knowledge to performance levels. VR was used either as an automated or supporting assessment tool. Practical considerations in VR implementation were also examined, such as hardware and software. Conclusion The use of VR in medical education assessment shows promise, particularly for emergency scenarios and performance-based tasks related to core competencies such as Patient Care, Interpersonal and Communication Skills, and Medical Knowledge. While this technology offers opportunities to automate assessments and reduce examiner workload, challenges related to software, costs, and feasibility must be addressed. Additionally, aligning learning objectives, teaching methods, and VR assessments through constructive alignment is essential to ensure effective implementation as both a teaching and evaluation tool.

Abstract Background Interviewers’ judgements play a critical role in competency-based assessments for selection such as the multiple-mini-interview (MMI). Much of the published research focuses on the psychometrics of selection and the impact of rater subjectivity. Within the context of selecting for entry into specialty postgraduate training, we used an interpretivist and socio-constructivist approach to explore how and why interviewers make judgments in high stakes selection settings whilst taking part in an MMI. Methods We explored MMI interviewers’ work processes through an institutional observational approach, based on the notion that interviewers’ judgements are socially constructed and mediated by multiple factors. We gathered data through document analysis, and observations of interviewer training, candidate interactions with interviewers, and interviewer meetings. Interviews included informal encounters in a large selection centre. Data analysis balanced description and explicit interpretation of the meanings and functions of the interviewers’ actions and behaviours. Results Three themes were developed from the data showing how interviewers make professional judgements, specifically by; ‘Balancing the interplay of rules and agency,’ ‘Participating in moderation and shared meaning making; and ‘A culture of reflexivity and professional growth.’ Interviewers balanced the following of institutional rules with making judgment choices based on personal expertise and knowledge. They engaged in dialogue, moderation, and shared meaning with fellow interviewers which enabled their consideration of multiple perspectives of the candidate’s performance. Interviewers engaged in self-evaluation and reflection throughout, with professional learning and growth as primary care physicians and supervisors being an emergent outcome. Conclusion This study offers insights into the judgment-making processes of interviewers in high-stakes MMI contexts, highlighting the balance between structured protocols and personal expertise within a socially constructed framework. By linking MMI practices to the broader work-based assessment literature, we contribute to advancing the design and implementation of more valid and fair selection tools for postgraduate training. Additionally, the study underscores the dual benefit of MMIs—not only as a selection tool but also as a platform for interviewers’ professional growth. These insights offer practical implications for refining future MMI practices and improving the fairness of high-stakes selection processes.

Abstract Background The administration of performance assessments during the coronavirus disease of 2019 (COVID-19) pandemic posed many challenges, especially for examinations employed as part of certification and licensure. The National Assessment Collaboration (NAC) Examination, an Objective Structured Clinical Examination (OSCE), was modified during the pandemic. The purpose of this study was to gather evidence to support the reliability and validity of the modified NAC Examination. Methods The modified NAC Examination was delivered to 2,433 candidates in 2020 and 2021. Cronbach’s alpha, decision consistency, and accuracy values were calculated. Validity evidence includes comparisons of scores and sub-scores for demographic groups: gender (male vs. female), type of International Medical Graduate (IMG) (Canadians Studying Abroad (CSA) vs. non-CSA), postgraduate training (PGT) (no PGT vs. PGT), and language of examination (English vs. French). Criterion relationships were summarized using correlations within and between the NAC Examination and the Medical Council of Canada Qualifying Examination (MCCQE) Part I scores. Results Reliability estimates were consistent with other OSCEs similar in length and previous NAC Examination administrations. Both total score and sub-score differences for gender were statistically significant. Total score differences by type of IMG and PGT were not statistically significant, but sub-score differences were statistically significant. Administration language was not statistically significant for either the total scores or sub-scores. Correlations were all statistically significant with some relationships being small or moderate (0.20 to 0.40) or large (> 0.40). Conclusions The NAC Examination yields reliable total scores and pass/fail decisions. Expected differences in total scores and sub-scores for defined groups were consistent with previous literature, and internal relationships amongst NAC Examination sub-scores and their external relationships with the MCCQE Part I supported both discriminant and criterion-related validity arguments. Modifications to OSCEs to address health restrictions can be implemented without compromising the overall quality of the assessment. This study outlines some of the validity and reliability analyses for OSCEs that required modifications due to COVID.

Abstract Objective Entrustable Professional Activities (EPAs) are increasingly being used in competency-based medical education approaches. A general lack of time in clinical settings, however, prevents supervisors from providing their trainees with adequate feedback. With a willingness for more administrative tasks being low in both trainees and educators, the authors developed a radical user-friendly mobile application based on the EPA concept called “Surg-prEPAred”. Design Surg-prEPAred is designed to collect micro-assessment data for building competency profiles for surgical residents according to their curriculum. The goal of Surg-prEPAred is to facilitate the performance and documentation of workplace-based assessments. Through aggregated data the app generates a personalized competency profile for every trainee. During a pilot run of 4 months, followed by ongoing usage of the application with a total duration of 9 months (August 2019 to April 2020), 32 residents and 33 consultants made daily use of the application as a rating tool. Every rating included knowledge, skills and professional attitudes of the trainees. Before the initiation of the App and after the 9-month trial period trainees and supervisors where both sent questionnaires to evaluate the user friendliness and effectiveness of the App. Results Five hundred ten App based assessments were generated. Out of 40 pre-defined EPAs, 36 were assessed. 15 trainees and 16 supervisors returned the questionnaires and stated the surg-prEPAred App as very valuable, effective and feasible to evaluate trainees in a clinical setting providing residents with an individual competence portfolio to receive precision medical education. Conclusions The authors expectation is that the Surg-prEPAred App will contribute to an improvement of quality of medical education and thus to the quality of patient care and safety. In the future the goal is to have the App become an integral part of the official Swiss surgical curriculum accepted by the Swiss professional surgical society.

Objective: Entrustable Professional Activities (EPAs) are increasingly being used in competency-based medical education approaches. A general lack of time in clinical settings, however, prevents supervisors from providing their trainees with adequate feedback. With a willingness for more administrative tasks being low in both trainees and educators, the authors developed a radical user-friendly mobile application based on the EPA concept called "Surg-prEPAred". Design: Surg-prEPAred is designed to collect micro-assessment data for building competency profiles for surgical residents according to their curriculum. The goal of Surg-prEPAred is to facilitate the performance and documentation of workplace-based assessments. Through aggregated data the app generates a personalized competency profile for every trainee. During a pilot run of 4 months, followed by ongoing usage of the application with a total duration of 9 months (August 2019 to April 2020), 32 residents and 33 consultants made daily use of the application as a rating tool. Every rating included knowledge, skills and professional attitudes of the trainees. Before the initiation of the App and after the 9-month trial period trainees and supervisors where both sent questionnaires to evaluate the user friendliness and effectiveness of the App. Results: Five hundred ten App based assessments were generated. Out of 40 pre-defined EPAs, 36 were assessed. 15 trainees and 16 supervisors returned the questionnaires and stated the surg-prEPAred App as very valuable, effective and feasible to evaluate trainees in a clinical setting providing residents with an individual competence portfolio to receive precision medical education. Conclusions: The authors expectation is that the Surg-prEPAred App will contribute to an improvement of quality of medical education and thus to the quality of patient care and safety. In the future the goal is to have the App become an integral part of the official Swiss surgical curriculum accepted by the Swiss professional surgical society. [ABSTRACT FROM AUTHOR]

Abstract Background A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies. Method A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors’ background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability. Results Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results. Conclusions This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students’ proficiencies rely on score quality, which in turn relies on expert judgements.

Abstract Background Current postgraduate medical training programmes fall short regarding residents’ development of generic competencies (communication, collaboration, leadership, professionalism) and reflective and deliberate practice. Paying attention to these non-technical skills in a structural manner during postgraduate training could result in a workforce better prepared for practice. A development-oriented performance assessment (PA), which assists residents with assessment of performance and deliberately planned learning activities, could potentially contribute to filling this gap. This study aims to explore residents experiences with the PA. Methods We conducted a qualitative interview study with 16 residents from four different medical specialties who participated in the PA, scheduled halfway postgraduate training. The PA was conducted by an external facilitator, a psychologist, and focused specifically on professional development and career planning. Residents were interviewed 6 months after the PA. Data were analysed using the framework method for qualitative analysis. Results Residents found the PA to be of additional value for their training. The overarching merit was the opportunity to evaluate competencies not usually addressed in workplace-based assessments and progress conversations. In addition, the PA proved a valuable tool for assisting residents with reflecting upon their work and formulating their learning objectives and activities. Residents reported increased awareness of capacity, self-confidence and enhanced feelings of career-ownership. An important factor contributing to these outcomes was the relationship of trust with the facilitator and programme director. Conclusion The PA is a promising tool in fostering the development of generic competencies and reflective and deliberate practice. The participating residents, facilitator and programme directors were able to contribute to a safe learning environment away from the busy workplace. The facilitator plays an important role by providing credible and informative feedback. Commitment of the programme director is important for the implementation of developmental plans and learning activities.

Background: A common feature of performance assessments is the use of human assessors to render judgements on student performance. From a measurement perspective, variability among assessors when assessing students may be viewed as a concern because it negatively impacts score reliability and validity. However, from a contextual perspective, variability among assessors is considered both meaningful and expected. A qualitative examination of assessor cognition when assessing student performance can assist in exploring what components are amenable to improvement through enhanced rater training, and the extent of variability when viewing assessors as contributing their individual expertise. Therefore, the purpose of this study was to explore assessor cognition as a source of score variability in a performance assessment of practice-based competencies. Method: A mixed-method sequential explanatory study design was used where findings from the qualitative strand assisted in the interpretation of results from the quantitative strand. Scores from one objective structured clinical examination (OSCE) were obtained for 95 occupational therapy students. Two Generalizability studies were conducted to examine the relative contribution of assessors as a source of score variability and to estimate the reliability of domain and holistic scores. Think-aloud interviews were conducted with eight participants assessing a subset of student performances from the OSCE in which they participated. Findings from the analysis of think-aloud data and consideration of assessors' background characteristics were used to assist in the interpretation of variance component estimates involving assessors, and score reliability. Results: Results from two generalizability analyses indicated the highest-order interaction-error term involving assessors accounted for the second-highest proportion of variance, after student variation. Score reliability was higher in the holistic vs. analytic scoring framework. Verbal analysis of assessors' think-aloud interviews provided evidential support for the quantitative results. Conclusions: This study provides insight into the nature and extent of assessor variability during a performance assessment of practice-based competencies. Study findings are interpretable from the measurement and contextual perspectives on assessor cognition. An integrated understanding is important to elucidate the meaning underlying the numerical score because the defensibility of inferences made about students' proficiencies rely on score quality, which in turn relies on expert judgements. [ABSTRACT FROM AUTHOR]

Abstract Background The demographics of doctors working in the UK are changing. The United Kingdom (UK) has voted to leave the European Union (EU) and there is heightened political discourse around the world about the impact of migration on healthcare services. Previous work suggests that foreign trained doctors perform worse than UK graduates in postgraduate medical examinations. We analysed the prevalence by country of primary medical qualification of doctors who were required to take an assessment by the General Medical Council (GMC) because of performance concerns. Methods This was a retrospective cohort analysis of data routinely collected by the GMC. We compared doctors who had a GMC performance assessment between 1996 and 2013 with the medical register in the same period. The outcome measures were numbers experiencing performance assessments by country or region of medical qualification. Results The rate of performance assessment varied significantly by place of medical qualification and by year; χ 2(17) = 188, p

Background: The demographics of doctors working in the UK are changing. The United Kingdom (UK) has voted to leave the European Union (EU) and there is heightened political discourse around the world about the impact of migration on healthcare services. Previous work suggests that foreign trained doctors perform worse than UK graduates in postgraduate medical examinations. We analysed the prevalence by country of primary medical qualification of doctors who were required to take an assessment by the General Medical Council (GMC) because of performance concerns. Methods: This was a retrospective cohort analysis of data routinely collected by the GMC. We compared doctors who had a GMC performance assessment between 1996 and 2013 with the medical register in the same period. The outcome measures were numbers experiencing performance assessments by country or region of medical qualification. Results: The rate of performance assessment varied significantly by place of medical qualification and by year; &#9572(17) = 188, p < 0.0001, pseudo-R2 = 15%. Doctors who trained outside of the UK, including those trained in the European Economic Area (EEA), were more likely to have a performance assessment than UK trained doctors, with the exception of South African trained doctors. Conclusions: The rate of performance assessment varies significantly by place of medical qualification. This is the first study to explore the risk of performance assessment by individual places of medical qualification. While concern has largely focused on the competence of non-EEA, International Medical Graduates, we discuss implications for how to ensure European trained doctors are fit to practise before their medical licence in the UK is granted. Further research is needed to investigate whether these country effects hold true when controlling for factors like doctors' sex, age, length of time working in the UK, and English language skills. This will allow evidencebased decisions to be made around the regulatory environment the UK should adopt once it leaves the EU. Patients should be reassured that the vast majority of all doctors working in the UK are competent. [ABSTRACT FROM AUTHOR]

Background: Clinical practice guidelines are intended to improve the process and outcomes of patient care. However, their implementation remains a challenge. We designed an implementation strategy, based on peer assessment (PA) focusing on barriers to change in physical therapy care. A previously published randomized controlled trial showed that PA was more effective than the usual strategy "case discussion" in improving adherence to a low back pain guideline. Peer assessment aims to enhance knowledge, communication, and hands-on clinical skills consistent with guideline recommendations. Participants observed and evaluated clinical performance on the spot in a role-play simulating clinical practice. Participants performed three roles: physical therapist, assessor, and patient. This study explored the critical features of the PA program that contributed to improved guideline adherence in the perception of participants. Methods: Dutch physical therapists working in primary care (n = 49) organized in communities of practice (n = 6) participated in the PA program. By unpacking the program we identified three main tasks and eleven subtasks. After the program was finished, a questionnaire was administered in which participants were asked to rank the program tasks from high to low learning value and to describe their impact on performance improvement. Overall ranking results were calculated. Additional semi-structured interviews were conducted to elaborate on the questionnaires results and were transcribed verbatim. Questionnaires comments and interview transcripts were analyzed using template analysis. Results: Program tasks related to performance in the therapist role were perceived to have the highest impact on learning, although task perceptions varied from challenging to threatening. Perceptions were affected by the roleplay format and the time schedule. Learning outcomes were awareness of performance, improved attitudes towards the guideline, and increased self-efficacy beliefs in managing patients with low back pain. Learning was facilitated by psychological safety and the quality of feedback. Conclusion: The effectiveness of PA can be attributed to the structured and performance-based design of the program. Participants showed a strong cognitive and emotional commitment to performing the physical therapist role. That might have contributed to an increased awareness of strength and weakness in clinical performance and a motivation to change routine practice. [ABSTRACT FROM AUTHOR]

Background: Multi-source feedback (MSF) offers doctors feedback on their performance from peers (medical colleagues), coworkers and patients. Researchers increasingly point to the fact that only a small majority of doctors (60-70 percent) benefit from MSF. Building on medical education and social psychology literature, the authors identified several factors that may influence change in response to MSF. Subsequently, they quantitatively studied the factors that advance the use of MSF for practice change. Methods: This observational study was set in 26 non-academic hospitals in the Netherlands. In total, 458 specialists participated in the MSF program. Besides the collation of questionnaires, the Dutch MSF program is composed of a reflective portfolio and a facilitative interview aimed at increasing the acceptance and use of MSF. All specialists who finished a MSF procedure between May 2008 and September 2010 were invited to complete an evaluation form. The dependent variable was self-reported change. Three categories of independent variables (personal characteristics, experiences with the assessments and mean MSF ratings) were included in the analysis. Multivariate regression analysis techniques were used to identify the relation between the independent variables and specialists' reported change in actual practice. Results: In total, 238 medical specialists (response rate 52 percent) returned an evaluation form and participated in the study. A small majority (55 percent) of specialists reported to have changed their professional performance in one or more aspects in response to MSF. Regression analyses revealed that two variables had the most effect on reported change. Perceived quality of mentoring positively influenced reported change (regression coefficient beta = 0.527, p <0.05) as did negative scores offered by colleagues. (regression coefficient beta = -0.157, p<0.05). The explained variance of these two variables combined was 34 percent. Conclusions: Perceived quality of mentoring and MSF ratings from colleagues seem to be the main motivators for the self-reported change in response to MSF by specialists. These insights could leverage in increasing the use of MSF for practice change by investing in the quality of mentors. [ABSTRACT FROM AUTHOR]

Background: Current postgraduate medical training programmes fall short regarding residents’ development of generic competencies (communication, collaboration, leadership, professionalism) and reflective and deliberate practice. Paying attention to these non-technical skills in a structural manner during postgraduate training could result in a workforce better prepared for practice. A development-oriented performance assessment (PA), which assists residents with assessment of performance and deliberately planned learning activities, could potentially contribute to filling this gap. This study aims to explore residents experiences with the PA. Methods: We conducted a qualitative interview study with 16 residents from four different medical specialties who participated in the PA, scheduled halfway postgraduate training. The PA was conducted by an external facilitator, a psychologist, and focused specifically on professional development and career planning. Residents were interviewed 6 months after the PA. Data were analysed using the framework method for qualitative analysis. Results: Residents found the PA to be of additional value for their training. The overarching merit was the opportunity to evaluate competencies not usually addressed in workplace-based assessments and progress conversations. In addition, the PA proved a valuable tool for assisting residents with reflecting upon their work and formulating their learning objectives and activities. Residents reported increased awareness of capacity, self-confidence and enhanced feelings of career-ownership. An important factor contributing to these outcomes was the relationship of trust with the facilitator and programme director. Conclusion: The PA is a promising tool in fostering the development of generic competencies and reflective and deliberate practice. The participating residents, facilitator and programme directors were able to contribute to a safe learning environment away from the busy workplace. The facilitator plays an important role by providing credible and informative feedback. Commitment of the programme director is important for the implementation of developmental plans and learning activities. [ABSTRACT FROM AUTHOR]