Search

Background: Artificial intelligence (AI) is becoming increasingly used in medical education, but our understanding of the validity of AI-based assessments (AIBA) as compared with traditional clinical expert-based assessments (EBA) is limited. In this study, the authors aimed to compare and contrast the validity evidence for the assessment of a complex clinical skill based on scores generated from an AI and trained clinical experts, respectively., Methods: The study was conducted between September 2020 to October 2022. The authors used Kane's validity framework to prioritise and organise their evidence according to the four inferences: scoring, generalisation, extrapolation and implications. The context of the study was chorionic villus sampling performed within the simulated setting. AIBA and EBA were used to evaluate performances of experts, intermediates and novice based on video recordings. The clinical experts used a scoring instrument developed in a previous international consensus study. The AI used convolutional neural networks for capturing features on video recordings, motion tracking and eye movements to arrive at a final composite score., Results: A total of 45 individuals participated in the study (22 novices, 12 intermediates and 11 experts). The authors demonstrated validity evidence for scoring, generalisation, extrapolation and implications for both EBA and AIBA. The plausibility of assumptions related to scoring, evidence of reproducibility and relation to different training levels was examined. Issues relating to construct underrepresentation, lack of explainability, and threats to robustness were identified as potential weak links in the AIBA validity argument compared with the EBA validity argument., Conclusion: There were weak links in the use of AIBA compared with EBA, mainly in their representation of the underlying construct but also regarding their explainability and ability to transfer to other datasets. However, combining AI and clinical expert-based assessments may offer complementary benefits, which is a promising subject for future research., (© 2023 The Authors. Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Introduction: During a health crisis, hospitals must prioritise activities and resources, which can compromise clerkship-based learning. We explored how health crises affect clinical clerkships using the COVID-19 pandemic as an example., Methods: In a constructivist qualitative study, we conducted 22 semi-structured interviews with key stakeholders (i.e. medical students and doctors) from two teaching hospitals and 10 different departments. We used thematic analysis to investigate our data and used stakeholder theory as a sensitising concept., Results: We identified three themes: (1) emotional triggers and reactions; (2) negotiation of legitimacy; and (3) building resilience. Our results suggest that the health crisis accentuated already existing problems in clerkships, such as students' feelings of low legitimacy, constant negotiation of roles, inconsistencies navigating rules and regulations and low levels of active participation. Medical students and doctors adapted to the new organisational demands by developing increased resilience. Students responded by reaching out for guidance and acceptance to remain relevant in the clinical clerkships. Doctors developed a behaviour of closing in and focused on managing themselves and their patients. This created tension between these two stakeholder groups., Conclusion: A health crisis can critically disrupt the hierarchical structure within the clinical clerkships and exacerbate existing conflicts between stakeholder groups. When medical students are not perceived as legitimate stakeholders in clinical clerkships during a health crisis, their attendance is perceived as unnecessary or even a nuisance. Despite increased student proactiveness and resilience, their roles inevitably shift from being doctors-to-be to students-to-be-managed., (© 2022 The Authors. Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Background: Collaborative skills learning in the form of dyad learning compared with individual learning has been shown to lead to non-inferior skills retention and transfer. However, we have limited knowledge on which learning activities improve collaborative skills training and how the number of collaborators may impact skills transfer. We explored the effects of skills training individually, in dyads, triads or tetrads on learning activities during training and on subsequent skills transfer., Methods: In a randomised, controlled study, participants completed a pre-post-transfer-test set-up in groups of one to four. Participants completed 2 hours of obstetric ultrasound training. In the dyad, triad and tetrad group participants took turns actively handling the ultrasound probe. All performances were rated by two blinded experts using the Objective Structured Assessment of Ultrasound Skills (OSAUS) scale and a Global Rating Scale (GRS). All training was video recorded, and learning activities were analysed using the Interactive-Constructive-Active-Passive (ICAP) framework., Results: One hundred one participants completed the simulation-based training, and ninety-seven completed the transfer test. Performance scores improved significantly from pre- to post-test for all groups (p < 0.001, ηp 2  = 0.55). However, group size did not affect transfer test performance on OSAUS scores (p = 0.13, ηp 2  = 0.06) or GRS scores (p = 0.23, ηp 2  = 0.05). ICAP analyses of training activities showed that time spent on non-learning and passive learning activities increased with group size (p < 0.001, ηp 2  = 0.31), whereas time spent on constructive and interactive learning activities was constant between groups compared with singles (p < 0.001, ηp 2  = 0.72)., Conclusion: Collaborative skills learning in groups of up to four did not impair skills transfer despite less hands-on time. This may be explained by a compensatory shift towards constructive and interactive learning activities that outweigh the effect of shorter hands-on time., (© 2022 The Authors. Medical Education published by Association for the Study of Medical Education and John Wiley & Sons Ltd.)

Introduction: Dyad learning occurs when two students work together to acquire new skills and knowledge. Several studies have provided evidence to support the educational rationale for dyad learning in the controlled simulated setting. However, the role of dyad learning in the clinical setting remains uncertain. Unlike the simulated setting, learning in the clinical setting depends on a complex interplay between medical students, doctors, nurses and patients potentially making dyad learning less valuable in clerkships. The objective of this study was to explore how key stakeholders perceive the value of implementing dyad learning during medical students' clinical clerkships., Methods: In a constructivist qualitative study, we conducted 51 semi-structured interviews with 36 key stakeholders involved in dyad learning, including 10 medical students, 12 doctors, five nurses and nine patients. Data were coded inductively using thematic analysis, then coded deductively using stakeholder theory as a theoretical framework., Results: We found that stakeholders generally perceived the educational impact of dyad learning in the clinical setting similarly but disagreed on its value. Students emphasised that dyad learning made them participate more actively during patient encounters and patients did not mind having two students present. Doctors and nurses considered dyad learning disruptive to the balance between service and training and reported that it did not resonate with their perception of good patient care., Conclusion: Dyad learning enables students to be more active during their clinical clerkships, but it easily disrupts the balance between service and training. This disruption may be exacerbated by the shifted balance in priorities and values between different stakeholder groups, as well as by making implicit teaching obligations more explicit for supervising doctors and nurses. Consequently, implementing dyad learning may not be perceived as valuable by doctors and nurses in the clinical setting, regardless of its pedagogical rationale., (© 2020 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Context: A long-standing myth in medical education research is a divide between two different poles: research aiming to advance theory with little focus on practical applications ('ivory tower' research) and practically oriented research aiming to serve educators and decision-makers with little focus on advancing theory ('in-the-trenches' practice). We explored this myth in a sample of randomised medical education studies using Stokes' four-quadrant framework for the classification of research perspective., Methods: We searched MEDLINE, Embase, CINAHL, PsychINFO, ERIC, Web of Science and Scopus for studies in medical education using a randomised design that were published between 1 January 2018 and 31 December 2018. We used Stokes' four-quadrant framework to categorise the studies according to their use of theory, concepts and their justification for practical use. We compared medical education research published in medical education journals and clinical journals., Results: A total of 150 randomised studies were included in the analysis. The largest segment of studies (46.7%) was categorised as use-inspired basic research (Pasteur's Quadrant), closely followed by pure applied research (40.7%, Edison's Quadrant). Only a few studies were categorised as aiming to advance knowledge with no thought for practical educational application (2.0%, Bohr's Quadrant). The proportion of studies that included educational concepts and theory differed according to publication in clinical journals or medical education journals: 40.5% vs 71.8%, respectively, P < .001. There were no differences between journals with regard to the proportion of studies that included a practical educational or clinical rationale (P = .99)., Conclusion: In a large sample of studies using randomised designs, we found no evidence to support the myth that medical education research divides between two singular poles represented by 'ivory tower research' and 'in-the-trenches practice'. We did confirm prevailing assumptions regarding an emphasis on non-theoretical medical education research in clinical journals., (© 2020 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Context: Traditionally, trainees are instructed to practise with as few errors as possible during simulation-based training. However, transfer of learning may improve if trainees are encouraged to commit errors. The aim of this study was to assess the effects of error management instructions compared with error avoidance instructions during simulation-based ultrasound training., Methods: Medical students (n = 60) with no prior ultrasound experience were randomised to error management training (EMT) (n = 32) or error avoidance training (EAT) (n = 28). The EMT group was instructed to deliberately make errors during training. The EAT group was instructed to follow the simulator instructions and to commit as few errors as possible. Training consisted of 3 hours of simulation-based ultrasound training focusing on fetal weight estimation. Simulation-based tests were administered before and after training. Transfer tests were performed on real patients 7-10 days after the completion of training. Primary outcomes were transfer test performance scores and diagnostic accuracy. Secondary outcomes included performance scores and diagnostic accuracy during the simulation-based pre- and post-tests., Results: A total of 56 participants completed the study. On the transfer test, EMT group participants attained higher performance scores (mean score: 67.7%, 95% confidence interval [CI]: 62.4-72.9%) than EAT group members (mean score: 51.7%, 95% CI: 45.8-57.6%) (p < 0.001; Cohen's d = 1.1, 95% CI: 0.5-1.7). There was a moderate improvement in diagnostic accuracy in the EMT group compared with the EAT group (16.7%, 95% CI: 10.2-23.3% weight deviation versus 26.6%, 95% CI: 16.5-36.7% weight deviation [p = 0.082; Cohen's d = 0.46, 95% CI: -0.06 to 1.0]). No significant interaction effects between group and performance improvements between the pre- and post-tests were found in either performance scores (p = 0.25) or diagnostic accuracy (p = 0.09)., Conclusions: The provision of error management instructions during simulation-based training improves the transfer of learning to the clinical setting compared with error avoidance instructions. Rather than teaching to avoid errors, the use of errors for learning should be explored further in medical education theory and practice., (© 2016 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Context: Concerns have been raised over the gap between education theory and practice and how research can contribute to inform decision makers on their choices and priorities. Little is known about how educational theories and research outcomes produced under optimal conditions in highly controlled settings generalise to the real-life education context. One way of bridging this gap is applying the concept of practical trials in medical education. In this paper we elaborate on characteristics of practical trials and based on examples from medical education we discuss the challenges, limitations and future directions for this kind of research., Current State: Practical trials have the overall aim of informing decision makers. They are carried out in real-life settings and are characterised by (i) comparison of viable alternative education strategies, (ii) broad inclusion criteria regarding participants across several settings and (iii) multiple outcome measures with long-term follow-up to evaluate both benefits and risks. Questions posed by practical trials may be proactive in applying theory in the development of educational innovations or reactive to educational reforms and innovations. Non-inferiority or equivalence designs are recommended when comparing viable alternatives and the use of crossover designs, cluster randomisation or stepped wedge trial designs are feasible when studying implementations across several settings. Outcome measures may include variables related to learners, teachers, educational administration, quality of care, patient outcomes and cost., Conclusions: Practical trials in medical education may contribute to bridge the gap between education theory and practice and aid decision makers in making evidence-based choices and priorities. Conducting practical trials is not without challenges and rigorous design and methods must be applied. Of concern is that the practical focus may lead to failure to include a sound theoretical basis in the research questions and the interventions studied, and that authors fail to obtain informed consent from their participants., (© 2016 John Wiley & Sons Ltd and The Association for the Study of Medical Education.)

Objectives: This study is designed to provide an overview of why, how, when and for whom collaborative learning of clinical skills may work in health professions education., Why: Collaborative learning of clinical skills may influence learning positively according to the non-medical literature. Training efficiency may therefore be improved if the outcomes of collaborative learning of clinical skills are superior or equivalent to those attained through individual learning., How: According to a social interaction perspective, collaborative learning of clinical skills mediates its effects through social interaction, motivation, accountability and positive interdependence between learners. Motor skills learning theory suggests that positive effects rely on observational learning and action imitation, and negative effects may include decreased hands-on experience. Finally, a cognitive perspective suggests that learning is dependent on cognitive co-construction, shared knowledge and reduced cognitive load., When and for Whom: The literature on the collaborative learning of clinical skills in health science education is reviewed to support or contradict the hypotheses provided by the theories outlined above. Collaborative learning of clinical skills leads to improvements in self-efficacy, confidence and performance when task processing is observable or communicable. However, the effects of collaborative learning of clinical skills may decrease over time as benefits in terms of shared cognition, scaffolding and cognitive co-construction are outweighed by reductions in hands-on experience and time on task., Conclusions: Collaborative learning of clinical skills has demonstrated promising results in the simulated setting. However, further research into how collaborative learning of clinical skills may work in clinical settings, as well as into the role of social dynamics between learners, is required., (© 2015 John Wiley & Sons Ltd.)

Objective: To provide a model for conducting cost-effectiveness analyses in medical education. The model was based on a randomised trial examining the effects of training midwives to perform cervical length measurement (CLM) as compared with obstetricians on patients' waiting times. (CLM), as compared with obstetricians., Methods: The model included four steps: (i) gathering data on training outcomes, (ii) assessing total costs and effects, (iii) calculating the incremental cost-effectiveness ratio (ICER) and (iv) estimating cost-effectiveness probability for different willingness to pay (WTP) values. To provide a model example, we conducted a randomised cost-effectiveness trial. Midwives were randomised to CLM training (midwife-performed CLMs) or no training (initial management by midwife, and CLM performed by obstetrician). Intervention-group participants underwent simulation-based and clinical training until they were proficient. During the following 6 months, waiting times from arrival to admission or discharge were recorded for women who presented with symptoms of pre-term labour. Outcomes for women managed by intervention and control-group participants were compared. These data were then used for the remaining steps of the cost-effectiveness model., Results: Intervention-group participants needed a mean 268.2 (95% confidence interval [CI], 140.2-392.2) minutes of simulator training and a mean 7.3 (95% CI, 4.4-10.3) supervised scans to attain proficiency. Women who were scanned by intervention-group participants had significantly reduced waiting time compared with those managed by the control group (n = 65; mean difference, 36.6 [95% CI 7.3-65.8] minutes; p = 0.008), which corresponded to an ICER of 0.45 EUR minute(-1) . For WTP values less than EUR 0.26 minute(-1) , obstetrician-performed CLM was the most cost-effective strategy, whereas midwife-performed CLM was cost-effective for WTP values above EUR 0.73 minute(-1) ., Conclusion: Cost-effectiveness models can be used to link quality of care to training costs. The example used in the present study demonstrated that different training strategies could be recommended as the most cost-effective depending on administrators' willingness to pay per unit of the outcome variable., (© 2015 Medical Education Published by John Wiley & Sons Ltd.)

Context: Dyad practice may be as effective as individual practice during clinical skills training, improve students' confidence, and reduce costs of training. However, there is little evidence that dyad training is non-inferior to single-student practice in terms of skills transfer., Objectives: This study was conducted to compare the effectiveness of simulation-based ultrasound training in pairs (dyad practice) with that of training alone (single-student practice) on skills transfer., Methods: In a non-inferiority trial, 30 ultrasound novices were randomised to dyad (n = 16) or single-student (n = 14) practice. All participants completed a 2-hour training programme on a transvaginal ultrasound simulator. Participants in the dyad group practised together and took turns as the active practitioner, whereas participants in the single group practised alone. Performance improvements were evaluated through pre-, post- and transfer tests. The transfer test involved the assessment of a transvaginal ultrasound scan by one of two clinicians using the Objective Structured Assessment of Ultrasound Skills (OSAUS)., Results: Thirty participants completed the simulation-based training and 24 of these completed the transfer test. Dyad training was found to be non-inferior to single-student training: transfer test OSAUS scores were significantly higher than the pre-specified non-inferiority margin (delta score 7.8%, 95% confidence interval -3.8-19.6%; p = 0.04). More dyad (71.4%) than single (30.0%) trainees achieved OSAUS scores above a pre-established pass/fail level in the transfer test (p = 0.05). There were significant differences in performance scores before and after training in both groups (pre- versus post-test, p < 0.01) with large effect sizes (Cohen's d = 3.85) and no significant interactions between training type and performance (p = 0.59). The dyad group demonstrated higher training efficiency in terms of simulator score per number of attempts compared with the single-student group (p = 0.03)., Conclusion: Dyad practice improves the efficiency of simulation-based training and is non-inferior to individual practice in terms of skills transfer., (© 2015 John Wiley & Sons Ltd.)