Search

Competing Interests: Declaration of competing interest Authors RS, JL, LK and MS report no relevant conflicts of interest. JE has received an unrestricted research grant from Medtronic and WL Gore. TR is consulting for COOK and Bentley, has received research grants from COOK, W.L GORE & Associates, Bentley and sits on the advisory board for Artivion. IvH has received honoraria for consulting and training from Medtronic, Tolochenaz, Switzerland

Background: Access to the neuraxial space, including lumbar punctures and neuraxial anaesthesia, is an everyday procedure in clinical practice. Traditionally these procedures rely on manual palpation technique, but ultrasound is a useful tool when patients prove challenging. Presently, there is a lack of evidence-based guidelines for technical skills acquisition, both with and without ultrasound, and likewise, competency assessment approaches vary globally. Accordingly, we aim to assess the current evidence regarding learning and assessment in neuraxial access ± ultrasound, for future educational recommendations., Methods: This scoping review will be conducted in accordance with the Preferred Reporting Items for Systematic and Meta-Analysis (PRISMA) statement, together with the PRISMA Extension for Scoping Reviews. A systematic search strategy will be based on a PICO approach, focusing on physicians, medical students, or nurses being exposed to education, training, or assessment in procedural neuraxial access ± ultrasound. No comparators are obligated, but outcomes should be assessable using the Kirkpatrick four levels of training evaluation. The search will be performed in Cochrane Library, Embase, Medline, Scopus, PubMed, and CINAHL. Independently, two authors will screen the studies and conflicts will be resolved by a third author. Relevant predefined data will be extracted and analysed using a descriptive approach. The quality of the studies will be assessed using the Medical Education Research Study Quality Instrument., Discussion: This scoping review will contribute by presenting gathered evidence of an overview of the different approaches to achieving education and training of technical skills in neuraxial access, and how skills are tested, which could guide research and future recommendations for skills development and assessment., (© 2024 The Author(s). Acta Anaesthesiologica Scandinavica published by John Wiley & Sons Ltd on behalf of Acta Anaesthesiologica Scandinavica Foundation.)

Introduction: Assessments require sufficient validity evidence before their use. The Assessment for Competence in Chest Tube Insertion (ACTION) tool evaluates proficiency in chest tube insertion (CTI), combining a rating scale and an error checklist. The aim of this study was to collect validity evidence for the ACTION tool on a porcine rib model according to the Messick framework., Methods: A rib model, consisting of a porcine hemithorax that was placed in a wooden frame, was used as simulator. Participants were recruited from the departments of surgery, pulmonology, and emergency medicine. After familiarization with the rib model and the equipment, standardized instructions and clinical context were provided. They performed 2 CTIs while being scored with the ACTION tool. All performances were assessed live by 1 rater and by 3 blinded raters using video recordings. Generalizability-analysis was performed and mean scores and errors of both groups on the first performance were compared. A pass/fail score was established using the contrasting groups' method., Results: Nine novice and 8 experienced participants completed the study. Generalizability coefficients where high for the rating scale (0.92) and the error checklist (0.87). In the first CTI, novices scored lower than the experienced group (38.1/68 vs. 47.1/68, P = 0.042), but no difference was observed on the error checklist. A pass/fail score of 44/68 was established., Conclusion: A solid validity argument for the ACTION tool's rating scale on a porcine rib model is presented, allowing formative and summative assessment of procedural skills during training before patient contact., Competing Interests: The authors declare no conflict of interest., (Copyright © 2023 Society for Simulation in Healthcare.)

Background: Over the past decade, ultrasound utilization has increased within anesthesia and intensive care medicine, enhancing patient safety and diagnostic accuracy. However, the frequency of ultrasound usage and operator training in the Nordic countries remain unclear. This project aims to perform a survey on ultrasound availability, daily clinical use, and how ultrasound skills are trained and assessed, among anesthesiologists., Methods: This online cross-sectional survey will include anesthesiologists from the Nordic countries. The survey will adhere to the CROSS checklist. Survey items will be developed based on a formative model with a conceptual model, consisting of three main parts, including demographics, ultrasound machines and use, and skills development and assessment. The clinical relevance of items will be secured by including anesthesiologists of various levels of experience in the development of the survey. Furthermore, experienced researchers in medical education will participate in the development, contributing with relevant medical educational perspectives. Data will be summarized using a non-parametric descriptive approach. A chi-squared test will examine relevant relationships between certain answers., Results: Results will be published in a peer-reviewed journal and presented at relevant scientific conferences and meetings., Conclusion: This study may find a high availability of ultrasound machines and frequent use in the clinical departments. Despite this expected daily use of ultrasound, missing standardized structured skills acquisition and assessment could be uncovered. The results of this study may contribute to mapping various aspects of clinical ultrasound and skills development for further use in research., (© 2024 The Author(s). Acta Anaesthesiologica Scandinavica published by John Wiley & Sons Ltd on behalf of Acta Anaesthesiologica Scandinavica Foundation.)

Purpose: To investigate whether individuals with long-term reduced stereopsis were able to obtain the same level of surgical skills in simulated vitreoretinal surgery on the Eyesi Surgical Simulator as individuals with normal stereopsis., Methods: Twenty-four medical students were recruited and divided into two groups according to their degree of stereopsis: Group 1 (n = 12) included subjects with normal stereopsis (60 arcsec or lower) and Group 2 (n = 12) included subjects with reduced stereopsis (120 arcsec or higher). Stereopsis was tested using the TNO test (Lameris Ootech BV, Nieuwegein, the Netherlands). The participants were trained in virtual reality-simulated vitreoretinal surgery and continuously measured using a test with solid validity evidence and a pre-defined pass-fail score. All data were analysed using the Wilcoxon rank sum test., Results: We observed no differences in overall performance scores for any of the four modules. The participants with reduced stereopsis used 5.8 more attempts in bimanual training (p = 0.04), 8.8 more attempts in removal of posterior hyaloid (p = 0.04), 9.1 more attempts in navigation training (p = 0.20) and 0.3 fewer attempts in removal of internal limiting membrane (p = 0.69)., Conclusion: The final performance scores on the Eyesi Surgical Simulator were independent of the degree of stereopsis. However, the number of attempts to achieve the pre-defined pass-fail score increased significantly with reduced stereopsis in two of four modules. These results indicate that a high degree of stereopsis is not necessary to become proficient in microsurgery but may prolong the learning curve., (© 2024 The Authors. Acta Ophthalmologica published by John Wiley & Sons Ltd on behalf of Acta Ophthalmologica Scandinavica Foundation.)

Purpose: To compare Manual Small Incision Cataract Surgery (MSICS) microsurgical performance in course participants who received virtual reality simulation-based training by either a surgical expert or a non-ophthalmologist instructor., Setting: Copenhagen Academy for Medical Education and Simulation, Copenhagen, Denmark., Design: Randomized controlled trial., Methods: Residents and specialists in ophthalmology with no prior MSICS experience were included to receive virtual reality simulation training in MSICS using the HelpMeSee simulator. The participants were randomly allocated to receive training from either an experienced MSICS surgeon or a non-ophthalmologist, also known as near-peer teaching. The performances of the participants were evaluated at baseline and post-training using a MSICS proficiency-based test with evidence of validity., Results: Thirty participants were included in the study and 29 completed the course. There was no significant difference in final test score between the two groups (p = 0.13). The performance score of both groups of participants increased significantly after receiving the training (p < 0.001). All participants passed the proficiency-based test after receiving the training., Conclusion: We found no significant difference in surgical proficiency-level whether the participants were trained by a surgical expert or a non-ophthalmologist instructor for MSICS in a virtual-reality based setting. The findings of this study suggest that near-peer teaching within microsurgical performance potentially could be applied with teaching outcomes comparable to a surgical expert-instructor., (© 2024 The Author(s). Acta Ophthalmologica published by John Wiley & Sons Ltd on behalf of Acta Ophthalmologica Scandinavica Foundation.)

Background and Purpose:  Orthopedic trainees must be able to perform intraoperative fluoroscopy imaging to assess the surgical result after volar locking plate surgeries of distal radius fractures. Guided by Messick's contemporary validity framework, the aim of our study was to gather evidence of validity for a test of proficiency for intraoperative imaging of a distal radius fracture using a novel immersive virtual reality simulator., Methods:  11 novices and 9 experienced surgeons employed at orthopedic departments completed 2 individual simulator sessions. At each session the participants performed 3 repetitions of an intraoperative fluoroscopic control of a distal radius fracture, consisting of 5 different fluoroscopic views. Several performance metrics were automatically recorded by the simulator and compared between the 2 groups., Results:  Simulator metrics for 3 of the 5 fluoroscopic views could discriminate between novices and experienced surgeons. An estimated composite score based on these 3 views showed good test-retest reliability, ICC = 0.82 (confidence interval 0.65-0.92; P < 0.001). A discriminatory standard was set at a composite score of 6.15 points resulting in 1 false positive (i.e., novice scoring better than the standard), and 1 false negative (i.e., experienced surgeon scoring worse than the standard)., Conclusion:  This study provided validity evidence from all 5 sources of Messick's contemporary validity framework (content, response process, internal structure, relationship with other variables, and consequences) for a simulation-based test of proficiency in intraoperative fluoroscopic control of a distal radius fracture fixated by a volar locking plate.

Background and Aims: Endoscopists' competence can vary widely, as shown in the variation in the adenoma detection rate (ADR). Computer-aided quality assessment (CAQ) can automatically assess performance during individual procedures. In this review we identified and described different CAQ systems for colonoscopy., Methods: A systematic review of the literature was done using MEDLINE, EMBASE, and Scopus based on 3 blocks of terms according to the inclusion criteria: colonoscopy, competence assessment, and automatic evaluation. Articles were systematically reviewed by 2 reviewers, first by abstract and then in full text. The methodological quality was assessed using the Medical Education Research Study Quality Instrument (MERSQI)., Results: Of 12,575 identified studies, 6831 remained after removal of duplicates and 6806 did not pass the eligibility criteria and were excluded, leaving 25 studies, of which 13 studies were included in the final analysis. Five categories of CAQ systems were identified: withdrawal speedometer (7 studies), endoscope movement analysis (3 studies), effective withdrawal time (1 study), fold examination quality (1 study), and visual gaze pattern (1 study). The withdrawal speedometer was the only CAQ system that tested its feedback by examining changes in ADR. Three studies observed an improvement in ADR, and 2 studies did not. The methodological quality of the studies was high (mean MERSQI, 15.2 points; maximum, 18 points)., Conclusions: Thirteen studies developed or tested CAQ systems, most frequently by correlating it to the ADR. Only 5 studies tested feedback by implementing the CAQ system. A meta-analysis was impossible because of the heterogeneous study designs, and more studies are warranted., Competing Interests: Disclosure All authors disclosed no financial relationships., (Copyright © 2024 American Society for Gastrointestinal Endoscopy. Published by Elsevier Inc. All rights reserved.)

Introduction: Medical students have previously been shown to be just as effective for video rating as experts. We want to compare medical students to experienced surgeons as video assessors of simulated robot-assisted radical prostatectomy (RARP) performance., Materials and Methods: Video recordings of three RARP modules on the RobotiX (formerly Simbionix) simulator from a previous study were used. Five novice surgeons, five experienced robotic surgeons, and five experienced robotic surgeons in RARP performed a total of 45 video-recorded procedures. The videos were assessed with the modified Global Evaluative Assessment of Robotic Skills tool as both full-length and an edited edition that only included the first 5 minutes of the procedure., Results: Fifty medical students and two experienced RARP surgeons (ES) performed a total of 680 video ratings of full-length videos and 5-minute videos (2-9 ratings per video). Medical students and ES showed poor agreement for both full-length videos and 5-minute videos (0.29 and -0.13, respectively). Medical students could not discriminate between the skill level of the surgeons in either full-length videos or 5-minute videos ( P = 0.053-0.36 and P = 0.21-0.82), whereas ES could discriminate between novice surgeons and experienced surgeons (full-length, P < 0.001, and 5 minutes, P = 0.007) and intermediate and experienced surgeons (full-length, P = 0.001, and 5 minutes, P = 0.01) in both full-length videos and 5-minute videos., Conclusion: We found that medical students cannot be used to assess RARP because they showed poor agreement with the ES rating for both full-length videos and 5-minute videos. Medical students could not discriminate between surgical skill levels., Competing Interests: The authors declare no conflict of interest., (Copyright © 2023 Society for Simulation in Healthcare.)

The objective of this study is to compare automated performance metrics (APM) and surgical gestures for technical skills assessment during simulated robot-assisted radical prostatectomy (RARP). Ten novices and six experienced RARP surgeons performed simulated RARPs on the RobotiX Mentor (Surgical Science, Sweden). Simulator APM were automatically recorded, and surgical videos were manually annotated with five types of surgical gestures. The consequences of the pass/fail levels, which were based on contrasting groups' methods, were compared for APM and surgical gestures. Intra-class correlation coefficient (ICC) analysis and a Bland-Altman plot were used to explore the correlation between APM and surgical gestures. Pass/fail levels for both APM and surgical gesture could fully distinguish between the skill levels of the surgeons with a specificity and sensitivity of 100%. The overall ICC (one-way, random) was 0.70 (95% CI: 0.34-0.88), showing moderate agreement between the methods. The Bland-Altman plot showed a high agreement between the two methods for assessing experienced surgeons but disagreed on the novice surgeons' skill level. APM and surgical gestures could both fully distinguish between novices and experienced surgeons in a simulated setting. Both methods of analyzing technical skills have their advantages and disadvantages and, as of now, those are only to a limited extent available in the clinical setting. The development of assessment methods in a simulated setting enables testing before implementing it in a clinical setting., (© 2024. The Author(s).)

Background: Surgical skills acquisition is under continuous development due to the emergence of new technologies, and there is a need for assessment tools to develop along with these. A range of neuroimaging modalities has been used to map the functional activation of brain networks while surgeons acquire novel surgical skills. These have been proposed as a method to provide a deeper understanding of surgical expertise and offer new possibilities for the personalized training of future surgeons. With studies differing in modalities, outcomes, and surgical skills there is a need for a systematic review of the evidence. This systematic review aims to summarize the current knowledge on the topic and evaluate the potential use of neuroimaging in surgical education., Methods: We conducted a systematic review of neuroimaging studies that mapped functional brain activation while surgeons with different levels of expertise learned and performed technical and non-technical surgical tasks. We included all studies published before July 1st, 2023, in MEDLINE, EMBASE and WEB OF SCIENCE., Results: 38 task-based brain mapping studies were identified, consisting of randomized controlled trials, case-control studies, and observational cohort or cross-sectional studies. The studies employed a wide range of brain mapping modalities, including electroencephalography, functional magnetic resonance imaging, positron emission tomography, and functional near-infrared spectroscopy, activating brain areas involved in the execution and sensorimotor or cognitive control of surgical skills, especially the prefrontal cortex, supplementary motor area, and primary motor area, showing significant changes between novices and experts., Conclusion: Functional neuroimaging can reveal how task-related brain activity reflects technical and non-technical surgical skills. The existing body of work highlights the potential of neuroimaging to link task-related brain activity patterns with the individual level of competency or improvement in performance after training surgical skills. More research is needed to establish its validity and usefulness as an assessment tool., (© 2024. The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.)

Background and Purpose: Orthopedic trainees frequently perform short antegrade femoral nail osteosynthesis of trochanteric fractures, but virtual reality simulation-based training (SBT) with haptic feedback has been unavailable. We explored a novel simulator, with the aim of gathering validity evidence for an embedded test and setting a credible pass/fail standard allowing trainees to practice to proficiency., Patients and Methods: The research, conducted from May to September 2020 across 3 Danish simulation centers, utilized the Swemac TraumaVision simulator for short antegrade femoral nail osteosynthesis. The validation process adhered to Messick's framework, covering all 5 sources of validity evidence. Participants included novice groups, categorized by training to plateau (n = 14) or to mastery (n = 10), and experts (n = 9), focusing on their performance metrics and training duration., Results: The novices in the plateau group and experts had hands-on training for 77 (95% confidence interval [CI] 59-95) and 52 (CI 36-69) minutes while the plateau test score, defined as the average of the last 4 scores, was 75% (CI 65-86) and 96% (CI 94-98) respectively. The pass/fail standard was established at the average expert plateau test score of 96%. All novices in the mastery group could meet this standard and interestingly without increased hands-on training time (65 [CI 46-84] minutes)., Conclusion: Our study provides supporting validity evidence from all sources of Messick's framework for a simulation-based test in short antegrade nail osteosynthesis of intertrochanteric hip fracture and establishes a defensible pass/fail standard for mastery learning of SBT. Novices who practiced using mastery learning were able to reach the pre-defined pass/fail standard and outperformed novices without a set goal for external motivation.

Aim: There is currently an increased focus on competency-based training, in which training and assessment play a crucial role. The aim of this systematic review is to create an overview of hands-on training methods and assessment tools for appendicectomy and colon and rectal surgery procedures using either an open, laparoscopic or robot-assisted approach., Method: A systematic review of Medline, Embase, Cochrane and Scopus databases was conducted following the PRISMA guidelines. We conducted the last search on 9 March 2023. All published papers describing hands-on training, evaluation of performance data and development of assessment tools were eligible. The quality of studies and the validity evidence of assessment tools are reported., Results: Fifty-one studies were identified. Laparoscopic assessment tools are abundant, but the literature still lacks good-quality assessment tools for open appendicectomy, robotic colectomy and open rectal surgery. Overall, there is a lack of discussion regarding the establishment of pass/fail standards and the consequences of assessment. Virtual reality simulation is used more for appendicectomy than colorectal procedures. Only a few of the studies investigating training were of acceptable quality. There is a need for high-quality studies in open and robotic-assisted colon surgery and all approaches to rectal surgery., Conclusion: This review provides an overview of current training methods and assessment tools and identifies where more research is needed based on the quality of the studies and the current validity evidence., (© 2024 The Authors. Colorectal Disease published by John Wiley & Sons Ltd on behalf of Association of Coloproctology of Great Britain and Ireland.)

Objective: A "PROficiency based StePwise Endovascular Curricular Training" (PROSPECT) has proven its superiority over traditional training in a randomised controlled trial to acquire basic endovascular skills outside theatre, but real life adherence is low. This study aimed to compare the original distributed training format, where trainees learn at their own pace, with a massed training format offering the same content within a limited time span while exempt from clinical duties. Secondly, long term skills retention was evaluated., Methods: A multicentre, prospective study allocated participants to the distributed D-PROSPECT or to a massed, compact version (C-PROSPECT) based on logistics such as travel time, participant and instructor availability. A multiple choice question (MCQ) test (max. score 20) tested cognitive skills. Technical skills were assessed using a global rating scale (GRS) (max. score 55), examiner's checklist (max. score 85), and validated simulator metrics. Data were collected pre- and post-programme and at three, six, and 12 months after programme completion., Results: Over four years and in two countries, D-PROSPECT was implemented in two centres and C-PROSPECT in three. A total of 22 participants completed D-PROSPECT with a 41% dropout rate, and 21 completed C-PROSPECT with 0% dropout rate. All participants showed significant improvement for all performance parameters after programme completion: MCQ test (median 14.5 vs. 18; p < .001), GRS (median 20 vs. 41; p < .001), examiner's checklist (median 49 vs. 78.5; p < .001), and simulation metrics (p < .001). Scores of C- or D-PROSPECT participants were not significantly different. No significant differences were seen between groups during the retention period., Conclusion: PROSPECT significantly improves the quality of simulated endovascular performances using a massed or distributed training format. A massed training format of PROSPECT may be preferred to decrease dropout during standardised training to obtain basic endovascular skills in existing surgical curricula., (Copyright © 2023 European Society for Vascular Surgery. Published by Elsevier B.V. All rights reserved.)

Background: Many junior doctors must prepare to manage acutely ill patients in the emergency department. The setting is often stressful, and urgent treatment decisions are needed. Overlooking symptoms and making wrong choices may lead to substantial patient morbidity or death, and it is essential to ensure that junior doctors are competent. Virtual reality (VR) software can provide standardized and unbiased assessment, but solid validity evidence is necessary before implementation., Objective: This study aimed to gather validity evidence for using 360-degree VR videos with integrated multiple-choice questions (MCQs) to assess emergency medicine skills., Methods: Five full-scale emergency medicine scenarios were recorded with a 360-degree video camera, and MCQs were integrated into the scenarios to be played in a head-mounted display. We invited 3 groups of medical students with different experience levels to participate: first- to third-year medical students (novice group), last-year medical students without emergency medicine training (intermediate group), and last-year medical students with completed emergency medicine training (experienced group). Each participant's total test score was calculated based on the number of correct MCQ answers (maximum score of 28), and the groups' mean scores were compared. The participants rated their experienced presence in emergency scenarios using the Igroup Presence Questionnaire (IPQ) and their cognitive workload with the National Aeronautics and Space Administration Task Load Index (NASA-TLX)., Results: We included 61 medical students from December 2020 to December 2021. The experienced group had significantly higher mean scores than the intermediate group (23 vs 20; P=.04), and the intermediate group had significantly higher scores than the novice group (20 vs 14; P<.001). The contrasting groups' standard-setting method established a pass-or-fail score of 19 points (68% of the maximum possible score of 28). Interscenario reliability was high, with a Cronbach α of 0.82. The participants experienced the VR scenarios with a high degree of presence with an IPQ score of 5.83 (on a scale from 1-7), and the task was shown to be mentally demanding with a NASA-TLX score of 13.30 (on a scale from 1-21)., Conclusions: This study provides validity evidence to support using 360-degree VR scenarios to assess emergency medicine skills. The students evaluated the VR experience as mentally demanding with a high degree of presence, suggesting that VR is a promising new technology for emergency medicine skills assessment., (©Marie Høxbro Knudsen, Niklas Breindahl, Tor-Salve Dalsgaard, Dan Isbye, Anne Grethe Mølbak, Gerhard Tiwald, Morten Bo Søndergaard Svendsen, Lars Konge, Joanna Bergström, Tobias Todsen. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 06.06.2023.)

Background: Basic thoracic ultrasound is being used more frequently by clinicians in several settings due to its high diagnostic accuracy for many common causes of respiratory failure and dyspnoea. However, ultrasound examinations are operator-dependent, and sufficient competences are needed to obtain high sensitivity and specificity of examinations. Additionally, it is crucial for ultrasound operators to perceive the competence to interpret the images and integrate them into the patient history and other examinations. This study aims to explore and gather validity evidence for an objective structured clinical examination test of basic thoracic ultrasound competences and establish a pass/fail score., Methods: An expert panel created the test which included two theoretical and five practical stations representing cases with different diagnoses that cause respiratory symptoms and which are possible to establish by basic thoracic ultrasound. Twenty-five participants with different levels of experience in basic thoracic ultrasound completed the test. Data of the test scores were used for item analysis, and exploring validity evidence was done according to Messick's framework which is recommended. The contrasting groups' standard setting method was used to establish a pass/fail score., Results: The summarised internal consistency reliability was high with a Cronbach's alpha of 0.87. The novice group (n = 4) had a mean test score of 42 ± 10.1 points, the intermediate group (n = 8) scored 79.1 ± 8.1 points, and the experienced group (n = 13) 89.0 ± 6.2 points (one-way ANOVA, p < 0.001). A pass/fail score of 71 points was thus derived (maximum test score = 105 points)., Conclusion: We developed a test for the assessment of clinical competences in basic thoracic ultrasound with solid validity evidence, and a pass/fail standard with no false positives or false negatives., (© 2023. The Author(s).)

Objectives: To investigate the validity of the Interventional Ultrasound Skills Evaluation (IUSE) tool for assessment of procedural competence in ultrasound-guided procedures in a clinical environment, including a pass/fail score., Methods: Novices and experienced radiologists were recruited from four hospitals and were observed and assessed while performing ultrasound-guided procedures. Performances were assessed using the IUSE tool by two independent raters. Validity evidence was gathered in accordance with Messick's framework: response process was ensured by standardisation of written rater instructions. Internal structure was explored using Cronbach's alpha for internal consistency reliability; inter-rater reliability was calculated as Pearson's r independently across all ratings, and test-retest reliability was reported using Cronbach's alpha. Relationship to other variables was investigated by comparing performances of the participants in each group. Consequences evidence was explored by calculating a pass/fail standard using the contrasting groups method., Results: Six novices and twelve experienced radiologists were enrolled. The IUSE tool had high internal consistency (Cronbach's alpha = 0.96, high inter-rater reliability (Pearson's r = 0.95), and high test-retest reliability (Cronbach's alpha = 0.98), and the mean score was 33.28 for novices and 59.25 for experienced with a highly significant difference (p value < 0.001). The pass/fail score was set at 55 resulting in no false positives or false negatives., Conclusions: Validity evidence from multiple sources supports the use of the IUSE tool for assessment of competence in ultrasound-guided procedures in a clinical environment and its use in high-stakes assessment such as certification. A credible pass/fail criterion was established to inform decision-making., Key Points: • A multi-site validity investigation established that the Interventional Ultrasound Skills Evaluation (IUSE) tool can be used to assess procedural competence in ultrasound-guided procedures. • Validity evidence was gathered according to Messick's framework validity from the following sources: response process, internal structure, relationship to other variables, and consequences evidence. • The IUSE tool can be used for both formative and summative assessment, and a credible pass/fail score was established to help inform decision-making such as certification., (© 2022. The Author(s), under exclusive licence to European Society of Radiology.)

Objective: This systematic review aims to examine the use of standard-setting methods in the context of simulation-based training of surgical procedures., Summary of Background: Simulation-based training is increasingly used in surgical education. However, it is important to determine which level of competency trainees must reach during simulation-based training before operating on patients. Therefore, pass/fail standards must be established using systematic, transparent, and valid methods., Methods: Systematic literature search was done in 4 databases (Ovid MEDLINE, Embase, Web of Science, and Cochrane Library). Original studies investigating simulation-based assessment of surgical procedures with the application of a standard setting were included. Quality of evidence was appraised using GRADE., Results: Of 24,299 studies identified by searches, 232 studies met the inclusion criteria. Publications using already established standard settings were excluded (N = 70), resulting in 162 original studies included in the final analyses. Most studies described how the standard setting was determined (N = 147, 91%) and most used the mean or median performance score of experienced surgeons (n = 65, 40%) for standard setting. We found considerable differences across most of the studies regarding study design, setup, and expert level classification. The studies were appraised as having low and moderate evidence., Conclusion: Surgical education is shifting toward competency-based education, and simulation-based training is increasingly used for acquiring skills and assessment. Most studies consider and describe how standard settings are established using more or less structured methods but for current and future educational programs, a critical approach is needed so that the learners receive a fair, valid, and reliable assessment., Competing Interests: The authors declare no conflicts of interest., (Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.)

Background: Competence in transurethral resection of bladder tumors (TURB) is critical in bladder cancer management and should be ensured before independent practice. Objective: To develop an assessment tool for TURB and explore validity evidence in a clinical context. Design, Setting, and Participants: From July 2019 to March 2021, a total of 33 volunteer doctors from three hospitals were included after exemption from the regional ethics committee (REG-008-2018). Participants performed two TURB procedures on patients with bladder tumors. A newly developed assessment tool (Objective Structured Assessment for Transurethral Resection of Bladder Tumors Skills, OSATURBS) was used for direct observation assessment (DOA), self-assessment (SA), and blinded video assessment (VA). Outcome Measurements and Statistical Analysis: Cronbach's alpha and Pearson's r were calculated for across items internal consistency reliability, inter-rater reliability, and test-retest reliability. Correlation between OSATURBS scores and the operative experience was calculated with Pearson's r and a pass/fail score was established. Differences in assessment scores were explored with paired t -test and independent samples t -test. Results and Limitations: The internal consistency reliability across items Cronbach's alpha was 0.94 ( n  = 260, p  < 0.001). Inter-rater reliability was 0.80 ( n  = 64, p  < 0.001). Test-retest correlation was high, r  = 0.71 ( n  = 32, p  < 0.001). Relationship with TURB experience was high, r  = 0.71 ( n  = 32, p  < 0.001). Pass/fail score was 19 points. DOAs were strongly correlated with video ratings ( r  = 0.85, p  < 0.001) but with a significant social bias with lower scores for inexperienced and higher scores for experienced participants. Participants tended to overestimate their own performances. Conclusions: OSATURBS tool for TURB can be used for assessment of surgical proficiency in the clinical setting. DOA and SA are biased, and blinded VA of TURB performances is advised. Clinical Trials NCT03864302.

Introduction: Simulation training at home improves access to training, but motivation can be difficult to maintain. Dyad training could keep trainees motivated. This study aimed to examine the effect of self-regulated training of basic surgical skills in pairs versus individually., Methods: One hundred one medical doctors were included in this prospective, mixed-method, simulation-based study. Participants were randomized to train individually or in pairs during a 6-week course in open surgical skills, consisting of didactic instructions and self-directed training at home. Trainees kept a training log and filled in a questionnaire. Skills were tested before and after the course. Tests were rated by an expert using the Objective Structured Assessment of Technical Skills Global Rating Scale., Results: Ninety-seven doctors completed the study. We found no differences in test score between dyad and individual trainees. Dyad trainees compared with individual trainees improved by 7.23 points (intercept estimate) versus 6.94 points, respectively (P = 0.881). Dyad trainees trained less frequently [7.3 times (intercept estimate) vs. 12.1 times, P < 0.001, but for longer intervals compared with individual trainees (68 minutes vs. 38 minutes)]. Dyad trainees reported benefits such as having a sparring partner, receiving feedback, corrections, increased motivation, having fun, and more structured training sessions. However, the coordination of training sessions was difficult., Conclusions: No differences were found between dyad trainees and individual trainees in improvement of surgical skills. Off-site dyad training led to fewer, but longer training sessions, which may have negatively impacted the effectiveness of training. Dyad trainees perceived dyad training beneficial., Competing Interests: The authors declare no conflict of interest., (Copyright © 2021 Society for Simulation in Healthcare.)

Contrast-enhanced ultrasound (CEUS) is an imaging modality applied in a broad field of medical specialties for diagnostic uses, guidance during biopsy procedures and ablation therapies and sonoporation therapy. Appropriate training and assessment of theoretical and practical competencies are recommended before practicing CEUS, but no validated assessment tools exist. This study was aimed at developing a theoretical multiple-choice question-based test for core CEUS competencies and gathering validity evidence for the test. An expert team developed the test via a Delphi process. The test was administered to medical doctors with varying CEUS experience, and the results were used to evaluate test items, internal-consistency reliability, ability to distinguish between different proficiency levels and to establish a pass/fail score. Validity evidence was gathered according to Messick's framework. The final test with 47 test items could distinguish between operators with and without CEUS experience with acceptable reliability. The pass/fail score led to considerable risk of false positives and negatives. The test may be used as an entry test before learning practical CEUS competencies but is not recommended for certification purposes because of the risk of false positives and negatives., Competing Interests: Conflict of interest disclosure A.B. is an advisory board member for General Electrics, Inventiva and Boehringer Ingelheim? The remaining authors have nothing to disclose., (Copyright © 2021 The Authors. Published by Elsevier Inc. All rights reserved.)

Purpose: To investigate which patient-related outcomes are dependent on the experience of the cataract surgeon., Methods: The study was designed as a prospective observational study. Novice (<150 surgeries performed) and experienced (>1000 surgeries performed) cataract surgeons from the Department of Ophthalmology, Rigshospitalet - Glostrup and Nordsjaellands Hospital - Hillerød were included in the study. Patients operated by the included surgeons were examined preoperatively, 1 day, 3 days, and 3 weeks after standard, noncomplicated cataract surgery. Primary outcomes were change in central corneal thickness and endothelial cell loss. Secondary outcomes were best-corrected visual acuity, intraocular pressure, aqueous flare, central macular thickness, and surgical complications., Results: Surgery performed by novice surgeons resulted in significantly lower visual acuity (mean -3.6 letters (Early Treatment Diabetic Retinopathy Study (ETDRS)); 95% CI: -7.3; -0.4, p = 0.03) and greater corneal thickness (mean 26.7 µm; 95% CI: 6.8; 46.6, p = 0.01) on the first day postoperative than surgery performed by experienced surgeons., Conclusion: The experience of the cataract surgeon affected visual acuity and central corneal thickness in the immediate postoperative period. In the future, these patient-related outcomes may be used to assess the technical proficiency of surgical trainees and investigate the effect of different training programs., (© 2020 Acta Ophthalmologica Scandinavica Foundation. Published by John Wiley & Sons Ltd.)

Purpose: Competency-based education relies on the validity and reliability of assessment scores. Generalizability (G) theory is well suited to explore the reliability of assessment tools in medical education but has only been applied to a limited extent. This study aimed to systematically review the literature using G-theory to explore the reliability of structured assessment of medical and surgical technical skills and to assess the relative contributions of different factors to variance., Method: In June 2020, 11 databases, including PubMed, were searched from inception through May 31, 2020. Eligible studies included the use of G-theory to explore reliability in the context of assessment of medical and surgical technical skills. Descriptive information on study, assessment context, assessment protocol, participants being assessed, and G-analyses was extracted. Data were used to map G-theory and explore variance components analyses. A meta-analysis was conducted to synthesize the extracted data on the sources of variance and reliability., Results: Forty-four studies were included; of these, 39 had sufficient data for meta-analysis. The total pool included 35,284 unique assessments of 31,496 unique performances of 4,154 participants. Person variance had a pooled effect of 44.2% (95% confidence interval [CI], 36.8%-51.5%). Only assessment tool type (Objective Structured Assessment of Technical Skills-type vs task-based checklist-type) had a significant effect on person variance. The pooled reliability (G-coefficient) was 0.65 (95% CI, .59-.70). Most studies included decision studies (39, 88.6%) and generally seemed to have higher ratios of performances to assessors to achieve a sufficiently reliable assessment., Conclusions: G-theory is increasingly being used to examine reliability of technical skills assessment in medical education, but more rigor in reporting is warranted. Contextual factors can potentially affect variance components and thereby reliability estimates and should be considered, especially in high-stakes assessment. Reliability analysis should be a best practice when developing assessment of technical skills., (Copyright © 2021 by the Association of American Medical Colleges.)

Background: Valid and reliable assessment of skills is essential for improved and evidence-based training concepts. In a recent study, we presented a novel tool to assess procedural skills in resuscitative endovascular balloon occlusion of the aorta (REBOA), REBOA-RATE, based on international expert consensus. Although expert consensus is a strong foundation, the performance of REBOA-RATE has not been explored. The study aimed to examine the reliability and validity of REBOA-RATE., Methods: This was an experimental simulation-based study. We enrolled doctors with three levels of expertise to perform two REBOA procedures in a simulated scenario of out-of-hospital cardiac arrest. Procedures were video-recorded, and videos were blinded and randomized. Three clinical experts independently rated all procedures using REBOA-RATE. Data were analyzed using Messick's framework for validity evidence, including generalizability analysis of reliability and determination of a pass/fail standard., Results: Forty-two doctors were enrolled: 16 novices, 13 anesthesiologists, and 13 endovascular experts. They all performed two procedures, yielding 84 procedures and 252 ratings. The REBOA-RATE assessment tool showed high internal consistency (Cronbach's α = 0.95) and excellent interrater reliability (intraclass correlation coefficient, 0.97). Assessment using one rater and three procedures could ensure overall reliability suitable for high-stakes testing (G-coefficient >0.80). Mean scores (SD) for the three groups in the second procedure were as follows: novices, 32% (24%); anesthesiologists, 55% (29%); endovascular experts, 93% (4%) (p < 0.001). The pass/fail standard was set at 81%, which all experts but no novices passed., Conclusion: Data strongly support the reliability and validity of REBOA-RATE, which successfully discriminated between all experience levels. The REBOA-RATE assessment tool requires minimal instruction, and one rater is sufficient for reliable assessment. Together, these are strong arguments for the use of REBOA-RATE to assess REBOA skills, allowing for competency-based training and certification concepts., Level of Evidence: Diagnostic test, no or poor gold standard, level V., (Copyright © 2021 Wolters Kluwer Health, Inc. All rights reserved.)

Background: Placing a peripheral vein catheter can be challenging due to several factors, but using ultrasound as guidance increases the success rate. The purpose of this review is to investigate the knowledge already existing within the field of education in ultrasound-guided peripheral vein catheter placement and explore the efficacy and clinical impact of different types of education., Methods: In accordance with PRISMA-guidelines, a systematic search was performed using three databases (PubMed, EMBASE, CINAHL). Two reviewers screened titles and abstracts, subsequently full-text of the relevant articles. The risk of bias was assessed using the Cochrane Collaboration risk of bias assessment tool and the New Ottawa scale., Results: Of 3409 identified publications, 64 were included. The studies were different in target learners, study design, assessment tools, and outcome measures, which made direct comparison difficult. The studies addressed a possible effect of mastery learning and found e-learning and didactic classroom teaching to be equally effective., Conclusion: Current studies suggest a potential benefit of ultrasound guided USG-PVC training on success rate, procedure time, cannulation attempts, and reducing the need for subsequent CVC or PICC in adult patients. An assessment tool with proven validity of evidence to ensure competence exists and education strategies like mastery learning, e-learning, and the usage of color Doppler show promising results, but an evidence-based USG-PVC-placement training program using these strategies combined is still warranted.

Objectives: To develop a generic and objective tool for assessing competence in percutaneous ultrasound-guided procedures., Methods: Interventional ultrasound experts from the Nordic countries were invited to participate in a three-round Delphi process. A steering committee was formed to manage the process. In round 1, the experts were asked to suggest all aspects to consider when assessing competence in US-guided procedures. Suggestions were analyzed and condensed into assessment items. In round 2, the expert panel rated these items on a 1-5 scale and suggested new items. Items with a mean rating of ≤ 3.5 were excluded. In round 3, the expert panel rated the list items and suggested changes to the items., Results: Twenty-five experts were invited, and response rates in the three rounds were 68% (17 out of 25), 100% (17 out of 17), and 100% (17 out of 17). The three-round Delphi process resulted in a 12-item assessment tool, using a five-point rating scale. The final assessment tool evaluates pre-procedural planning, US technique, procedural technique, patient safety, communication, and teamwork., Conclusions: Expert consensus was achieved on a generic tool for assessment of competence in percutaneous ultrasound-guided procedures-the Interventional Ultrasound Skills Evaluation (IUSE). This is the initial step in ensuring a valid and reliable method for assessment of interventional US skill., Key Points: • Through a Delphi process, expert consensus was achieved on the content of an assessment tool for percutaneous ultrasound-guided procedures-the Interventional Ultrasound Skills Evaluation (IUSE) tool. • The IUSE tool is comprehensive and covers pre-procedural planning, US technique, procedural technique, patient safety, communication, and teamwork. • This is an important step in ensuring valid and reliable assessment of interventional US skills.

Background: The aim of this study was to analyze learning curves and competency gains of novice vascular trainees when performing open aortic repair in a simulation-based environment., Methods: This was a prospective study of 16 vascular trainees performing infrarenal open aortic repair on an inanimate abdominal aortic aneurysm simulator with pulsatile pressure and flow. Each participant performed 4 procedures as a primary surgeon while getting structured feedback by a supervising experienced vascular surgeon. All sessions were video recorded and were anonymously and independently assessed by 3 rater-trained experts on an online platform using the newly validated open abdominal aortic aneurysm repair of technical expertise assessment tool. All supervisor interferences and procedure time was noted., Results: Reliability between raters was excellent (intraclass correlation coefficient = 0.92). Participants' mean scores almost doubled during the course between the first (13.4, 95% confidence interval [CI], 6.8-20) and fourth session (29.8, 95% CI, 26.3-33.3) with a mean difference of 14.6 (P < 0.001). Supervisor interference also decreased significantly from mean 3.0 (95% CI, 1.5-3.6) in the first to 0.7 (95% CI, 0.4-1.0) in the fourth session (P = 0.004). Procedure time decreased with a mean of 24 minutes: from 81 min (95% CI, 71.8-90.3) to 57 min (95% CI, 51.1-63.2, P < 0.001). There was a significant negative correlation between procedure time and the Open Abdominal Aortic Aneurysm Repair of Technical Expertise score (Pearson's r = -0.72, P < 0.01). Only half of the participants passed the pass/fail score of 27.7 points during the course., Conclusions: Novice vascular trainees achieve skills and competencies in open aortic repair in a simulated setting with dedicated supervision and feedback and can become ready for supervised surgery on real patients. Learning rates are individual, and it is important to construct training programs with emphasis on proficiency and not merely attending a course., (Copyright © 2020. Published by Elsevier Inc.)

Background: Motor skills have been identified as a useful measure to evaluate competency in bronchoscopy. However, no automatic assessment system of motor skills with a clear pass/fail criterion in flexible bronchoscopy exists., Objectives: The objective of the study was to develop an objective and automatic measure of motor skills in bronchoscopy and set a pass/fail criterion., Methods: Participants conducted 3 bronchoscopies each in a simulated setting. They were equipped with a Myo Armband that measured lower arm movements through an inertial measurement unit, and hand and finger motions through electromyography sensors. These measures were composed into an objective and automatic composite score of motor skills, the motor bronchoscopy skills score (MoBSS)., Results: Twelve novices, eleven intermediates, and ten expert bronchoscopy operators participated, resulting in 99 procedures available for assessment. MoBSS was correlated with a higher diagnostic completeness (Pearson's correlation, r = 0.43, p < 0.001) and a lower procedure time (Pearson's correlation, r = -0.90, p < 0.001). MoBSS was able to differentiate operator performance based on the experience level (one-way ANOVA, p < 0.001). Using the contrasting groups' method, a passing score of -0.08 MoBSS was defined that failed 30/36 (83%) novice, 5/33 (15%) intermediate, and 1/30 (3%) expert procedures., Conclusions: MoBSS can be used as an automatic and unbiased assessment tool for motor skills performance in flexible bronchoscopy. MoBSS has the potential to generate automatic feedback to help guide trainees toward expert performance., (© 2021 S. Karger AG, Basel.)

Purpose: Reliable assessment of surgical skills is vital for competency-based medical training. Several factors influence not only the reliability of judgments but also the number of observations needed for making judgments of competency that are both consistent and reproducible. The aim of this study was to explore the role of various conditions-through the analysis of data from large-scale, simulation-based assessments of surgical technical skills-by examining the effects of those conditions on reliability using generalizability theory., Method: Assessment data from large-scale, simulation-based temporal bone surgical training research studies in 2012-2018 were pooled, yielding collectively 3,574 assessments of 1,723 performances. The authors conducted generalizability analyses using an unbalanced random-effects design, and they performed decision studies to explore the effect of the different variables on projections of reliability., Results: Overall, 5 observations were needed to achieve a generalizability coefficient > 0.8. Several variables modified the projections of reliability: increased learner experience necessitated more observations (5 for medical students, 7 for residents, and 8 for experienced surgeons), the more complex cadaveric dissection required fewer observations than virtual reality simulation (2 vs 5 observations), and increased fidelity simulation graphics reduced the number of observations needed from 7 to 4. The training structure (either massed or distributed practice) and simulator-integrated tutoring had little effect on reliability. Finally, more observations were needed during initial training when the learning curve was steepest (6 observations) compared with the plateau phase (4 observations)., Conclusions: Reliability in surgical skills assessment seems less stable than it is often reported to be. Training context and conditions influence reliability. The findings from this study highlight that medical educators should exercise caution when using a specific simulation-based assessment in other contexts.

Objective: The aim of this study was to gather validity evidence for the Assessment of basic Vascular Ultrasound Expertise (AVAUSE) tool, and to establish a pass/fail score for each component, to support decisions for certification., Methods: A cross sectional validation study performed during the European Society for Vascular Surgery's annual meeting. Validity evidence was sought for the theoretical test and two practical tests based on Messick's framework. The participants were vascular surgeons, vascular surgical trainees, sonographers, and nurses with varying experience levels. Five vascular ultrasound experts developed the theoretical and two practical test components of the AVAUSE tool for each test component. Two stations were set up for carotid examinations and two for superficial venous incompetence (SVI) examinations. Eight raters were assigned in pairs to each station. Three methods were used to set pass/fail scores: contrasting groups' method; rater consensus; and extended Angoff., Results: Nineteen participants were enrolled. Acceptable internal consistency reliability (Cronbach's alpha) for the AVAUSE theoretical (0.93), carotid (0.84), and SVI (0.65) practical test were shown. In the carotid examination, inter-rater reliability (IRR) for the two rater pairs was good: 0.68 and 0.78, respectively. The carotid scores correlated significantly with years of experience (Pearson's r = 0.56, p = .013) but not with number of examinations in the last five years. For SVI, IRR was excellent at 0.81 and 0.87. SVI performance scores did not correlate with years of experience and number of examinations. The pass/fail score set by the contrasting groups' method was 29 points out of 50. The rater set pass/fail scores were 3.0 points for both carotid and SVI examinations and were used to determine successful participants. Ten of 19 participants passed the tests and were certified., Conclusion: Validity evidence was sought and established for the AVAUSE comprehensive tool, including pass/fail standards. AVAUSE can be used to assess competences in basic vascular ultrasound, allowing operators to progress towards independent practice., (Copyright © 2020 European Society for Vascular Surgery. Published by Elsevier B.V. All rights reserved.)

Purpose: To compare manual and robot-assisted vitreoretinal surgery using a virtual-reality surgical simulator., Methods: Randomized controlled crossover study. Ten experienced vitreoretinal surgeons and 10 novice ophthalmic surgeons were included. The participants were randomized to start with either manual or robot-assisted surgery. Participants completed a test session consisting of three vitreoretinal modules on the Eyesi virtual-reality simulator. The automated metrics of performance supplied by the Eyesi simulator were used as outcome measures. Primary outcome measures were time with instruments inserted (seconds), instrument movement (mm), and tissue treatment (mm)., Results: Robot-assisted surgery was slower than manual surgery for both novices and vitreoretinal surgeons, 0.24 SD units (P = 0.024) and 0.73 SD units (P < 0.001), respectively. Robot-assisted surgery allowed for greater precision in novices and vitreoretinal surgeons, -0.96 SD units (P < 0.001) and -0.47 SD units (P < 0.001), respectively. Finally, novices using robot-assisted surgery inflicted less tissue damage when compared with that using manual surgery, -0.59 SD units (P = 0.009)., Conclusion: At the cost of time, robot-assisted vitreoretinal surgery seems to improve precision and limit tissue damage compared with that of manual surgery. In particular, the performance of novice surgeons is enhanced with robot-assisted vitreoretinal surgery.

Purpose: To develop and investigate an Eyesi simulator-based test for the more experienced cataract surgeon for evidence of validity., Methods: The study was a prospective interventional cohort study and carried out at the Copenhagen Academy for Medical Education and Simulation. The Eyesi Simulator was used for the test which was developed by three expert cataract surgeons. Ten cataract surgeons (>250 surgeries performed) and ten ophthalmic residents performed two repetitions of the test. The test consisted of four modules: Iris Expansion Ring insertion - level 1, Iris Expansion Ring extraction - level 2, Capsulorhexis - level 3 and Anterior Vitrectomy - level 6., Results: Internal consistency reliability showed Cronbach's alpha of 0.63. Test-retest reliabilities were significant for Iris Expansion Ring extraction - level 2 (p = 0.012) and Capsulorhexis - level 3 (p = 0.018). Differences between the two groups were only significant in both repetitions for the Iris Expansion Ring extraction - level 2 (p < 0.001 and p = 0.041, respectively). Furthermore, we found a statistically significant difference between the mean module scores for novices and the more experienced surgeons for Iris Expansion Ring insertion - level 1 (p = 0.021) and Capsulorhexis - level 3 (p = 0.019) in the first repetition., Conclusion: The investigated modules show evidence of validity within several aspects of Messick's framework. However, the evidence is not strong enough to apply the test for certification purposes of cataract surgeons, but the modules may still be relevant in the training of advanced cataract surgical procedures., (© 2020 The Authors. Acta Ophthalmologica published by John Wiley & Sons Ltd on behalf of Acta Ophthalmologica Scandinavica Foundation.)

Purpose: Teaching technical skills through the use of guided discovery learning (GDL) is an ongoing topic of research. In this approach, learners practice and struggle before receiving formal instruction. This has shown promise in other domains of learning, yet in the realm of procedural skills, clarity is still needed. This study seeks to address these gaps by investigating efficacy and mechanisms relating to application for a GDL approach in teaching basic surgical skills., Method: In 2018, young surgical trainees (N = 16) undertook a 6-week open surgical course applying the principles of GDL, each lesson beginning with a discovery phase before subsequent instruction and practice. A concurrent triangulation mixed-methods approach was used with direct observation and collection of semistructured interviews using a framework designed from productive failure literature. At the end of the course, all participants took a conceptual knowledge test and a performance-based skills test. Performance on the skills test was rated using global ratings and checklists., Results: The GDL cohort outperformed the historical cohort on the written exam (F [1,65] = 4.96, P = .029, d = .62), as well as on the summative suturing test (F [1,65] = 6.23, P = .015, d = .68). Furthermore, 3 main themes that highlight the mechanisms and mediators of efficient GDL were: (1) building conceptual knowledge, (2) motivating self-regulated learning, and (3) the type of skill and psychological safety., Conclusions: GDL can be an efficient approach to teaching procedural skills. Implications for future research and curricular design are discussed.

Contrast-enhanced ultrasound (CEUS) is dependent on the observer's competencies, and thus, appropriate education and testing of individual competency is essential. The current international curriculum in CEUS is methodologically weak and lacks validated competency assessment tests. In this study, we aimed to develop a syllabus for core competencies in CEUS using the Delphi process and an international panel of experts. A core writing group constructed an initial draft of the syllabus based on the existing literature. Eight international experts in CEUS or medical education were recruited as Delphi panelists. The draft underwent iterative and anonymous Delphi rounds until a pre-defined level of consensus was reached. The final syllabus consisted of 16 items, which were indexed in two main domains and in procedural order. An expert-generated proposal for a syllabus of core CEUS competencies has been constructed via the Delphi process and may serve as framework for future development of a CEUS curriculum, including competency assessment tests., (Copyright © 2020 World Federation for Ultrasound in Medicine & Biology. All rights reserved.)

Background: Airway management is a defining skill for anaesthesiologists. Anaesthesiologists must maintain and update these crucial skills throughout their career, but how this is best achieved remains unclear. This study aimed to clarify anaesthesiologists' procedural volume, confidence in airway management and their current and preferred future educational strategies., Methods: A questionnaire was developed consisting of 28 items exploring essential skills in airway management. All anaesthesiologists in the Capital Region of Denmark were invited to participate., Results: The response rate was 84% (240/285). Most anaesthesiologists felt competent to a high or very high degree in basic airway management. Anaesthesiologists from anaesthesia felt confident to a significantly higher degree than those working in the intensive care unit (ICU) regarding the practical aspects of airway management in both the anticipated difficult airway (93% vs 73%, P < .001) and the unanticipated difficult airway (81% vs 61%, P = .002). Both groups performed most of the key advanced techniques ≤4 times yearly, whereas anaesthesiologists from the ICU had a lower and less diverse procedural volume than those working in anaesthesia. The anaesthesiologists preferred training through their daily clinical work, hands-on workshops, and scenario-based simulation training. However, a large discrepancy was identified between the current and the desired level of training., Conclusion: The anaesthesiologists felt competent to a high or very high degree in basic airway management but the current procedural volume in advanced airway management causes concern for skill maintenance. Furthermore, we found a gap between the current and the desired level of supplemental training., (© 2020 The Acta Anaesthesiologica Scandinavica Foundation. Published by John Wiley & Sons Ltd.)

Objective: The aims of this study were to develop a procedure specific assessment tool for open abdominal aortic aneurysm (AAA) repair, gather validity evidence for the tool and establish a pass/fail standard., Methods: Validity was studied based on the contemporary framework by Messick. Three vascular surgeons experienced in open AAA repair and an expert in assessment and validation within medical education developed the OPEn aortic aneurysm Repair Assessment of Technical Expertise (OPERATE) tool. Vascular surgeons with varying experiences performed open AAA repair in a standardised simulation based setting. All procedures were video recorded with the faces anonymised and scored independently by three experts in a mutual blinded setup. The Angoff standard setting method was used to establish a credible pass/fail score., Results: Sixteen novices and nine experienced open vascular surgeons were enrolled. The OPERATE tool achieved high internal consistency (Cronbach's alpha .92) and inter-rater reliability (Cronbach's alpha .95) and was able to differentiate novices and experienced surgeons with mean scores (higher score is better) of 13.4 ± 12 and 25.6 ± 6, respectively (p = .01). The pass/fail score was set high (27.7). One novice passed the test while six experienced surgeons failed., Conclusion: Validity evidence was established for the newly developed OPERATE tool and was able to differentiate between novices and experienced surgeons providing a good argument that this tool can be used for both formative and summative assessment in a simulation based environment. The high pass/fail score emphasises the need for novices to train in a simulation based environment up to a certain level of competency before apprenticeship training in the clinical environment under the tutelage of a supervisor. Familiarisation with the simulation equipment must be ensured before performance is assessed as reflected by the low scores in the experienced group's first attempt., (Copyright © 2020 European Society for Vascular Surgery. Published by Elsevier B.V. All rights reserved.)

Purpose: Direct ophthalmoscopy can reveal systemic, neurologic and ophthalmic conditions, but is poorly mastered among young physicians. A theoretical test is needed to measure effect of educational interventions. We developed and gathered validity evidence for a multiple-choice questionnaire (MCQ)-based theoretical test in direct ophthalmoscopy., Methods: The MCQ was developed by interviewing experts. Then, validity evidence was evaluated using Messick's validity framework. Content was ensured by inviting the experts to contribute in a Delphi-like process. Response process was ensured by piloting and by streamlining all instructions. Then, the test was taken by ophthalmologists and by medical students without experience in direct ophthalmoscopy. Results were used to evaluate internal structure (item quality analysis and internal consistency), relations to other variables (correlation of test scores to experience level) and consequences (establishment of pass-fail score and the consequences of its use)., Results: The first phase of the study yielded 100 MCQs. In second phase, we identified that 60 items fulfilled predefined relevance and item quality requirements. These items demonstrated very high internal consistency (Cronbach's alpha = 0.95), significantly discriminated medical students from specialists (p < 0.001, independent samples t-test) and the established pass-fail score of 50 (83%) correct answers resulted in no false positives (students passing) and no false negatives (specialists failing). A Decision study identified that sampling 15 items suffice for certification., Conclusion: We developed and validated an MCQ-based theoretical test in direct ophthalmoscopy that enables an evidence-based approach to measuring, evaluating and certifying the theoretical knowledge necessary for direct ophthalmoscopy., (© 2019 Acta Ophthalmologica Scandinavica Foundation. Published by John Wiley & Sons Ltd.)

Objectives: A prospective observational trial to develop and gather validity evidence using Messick's framework for a simulator-based test in TURB. Methods: Forty-nine doctors were recruited from urology departments (Herlev/Gentofte University Hospital, Rigshospitalet Copenhagen University Hospital and Zealand University Hospital Roskilde) and enrolled from April to September 2018. The TURB Mentor™ virtual reality (VR) simulator was assessed at an expert meeting selecting clinically relevant cases and metrics. Test sessions were done on identical simulators at two university hospitals in Denmark. All participants performed three TURB procedures on the VR simulator. Simulator metrics were analysed with analysis of variance (ANOVA) and metrics with the ability to discriminate between groups were combined in a total simulator score. Finally, a pass/fail score was identified using the contrasting groups' method. Results: Eleven simulator metrics were found eligible and four had significant discrimination ability between competency levels: resected pathology (%) ( p  = 0.008); cutting in bladder wall ( n ) ( p  = 0.004); time (s) ( p  = 0.034); and inspection of the bladder wall (%) ( p  = 0.002). The internal structure of the total simulator score [(resected pathology*inspection of the bladder wall)/time] was high with the intraclass correlation coefficient, Cronbach's alpha = 0.85. The mean total simulator score was significantly lower in the novice group than in the intermediate, 15.9 and 25.6, respectively (mean difference = 9.7, p  = 0.011) and experienced group, 30.6 (mean difference = 14.7, p  < 0.001). A pass/fail score of 22 was identified. Conclusion: We found validity evidence for a newly developed VR simulator-based test and establised a pass/fail score identifying surgical skills in TURB. The TURBEST test can be used in a proficiency-based TURB simulator training programme for accreditation prior to supervised procedures on patients.

Purpose: To investigate the correlation between performance on a virtual reality simulator and real-life cataract surgical performance., Setting: Nine ophthalmology departments in Denmark and Copenhagen Academy for Medical Education and Simulation, Copenhagen, Denmark., Design: Prospective multicenter study., Methods: Cataract surgeons with different experience levels were included. The participants performed 3 consecutive video-recorded phacoemulsification surgeries that were rated by masked raters using the Objective Structured Assessment of Cataract Surgical Skills (OSACSS) scoring system. Thereafter, the participants performed a previously validated test on an Eyesi virtual reality simulator. Primary outcomes were the mean OSACSS score from all 3 surgeries and the simulator score from the participants' first repetition of the performance test., Result: Nineteen surgeons participated. There was a statistically significant correlation between the simulator performance score and the mean OSACSS score across all experience levels, with a Pearson correlation of 0.65 (P = .003, R 2  = 0.42)., Conclusion: Simulator performance was significantly correlated with real-life cataract surgical performance., (Copyright © 2019 ASCRS and ESCRS. All rights reserved.)

Background: Simulation-based mastery training may improve clinical performance. The aim of this study was to determine the effect of simulation-based mastery training on clinical performance in abdominal diagnostic ultrasound for radiology residents., Method: This study was a multicenter randomized controlled trial registered at clinicaltrials.gov (identifier: NCT02921867) and reported using the Consolidated Standards of Reporting Trials (CONSORT) statement. Twenty radiology residents from 10 different hospitals were included in the study. Participants were randomized into two groups: (1) simulator-based training until passing a validated test scored by a blinded reviewer or (2) no intervention prior to standard clinical ultrasound training on patients. All scans performed during the first 6 weeks of clinical ultrasound training were scored. The primary outcome was performance scores assessed using Objective Structured Assessment of Ultrasound Skills (OSAUS). An exponential learning curve was fitted for the OSAUS score for the two groups using non-linear regression with random variation. Confidence intervals were calculated based on the variation between individual learning curves., Results: After randomization, eleven residents completed the simulation intervention and nine received standard clinical training. The simulation group participants attended two to seven training sessions using between 6 and 17 h of simulation-based training. The performance score for the simulation group was significantly higher for the first 29 scans compared to that for the non-simulation group, such that scores reached approximately the same level after 49 and 77 scans, respectively., Conclusion: We showed improved performance in diagnostic ultrasound scanning on patients after simulation-based mastery learning for radiology residents., Trial Registration: NCT02921867 KEY POINTS: • Improvement in scanning performance on patients is seen after simulation-based mastery learning in diagnostic abdominal ultrasound. • Simulation-based mastery learning can prevent patients from bearing the burden of the initial steep part of trainees' learning curve.

This paper describes transfer of skills obtained from training with the EyeSI virtual reality simulator of ophthalmic surgery to real-life surgical performance. Skills in real-life phacoemulsification surgery were assessed by systematic blinded evaluation of surgical videos based on the OSACCS system. Nineteen Danish cataract surgeons with varying clinical experience levels had their cataract surgery skills evaluated before and after completing a standardized mastery learning program on the EyeSI. It was found that transfer of skills could be demonstrated only for surgeons with a real-life experience of less than 75 completed, independent cases. We could not demonstrate transfer of skills from the EyeSI cataract module to the EyeSI vitreoretinal module, so each subspecialty seems to require specific training. Finally, the discriminative power of EyeSI simulation between emerging surgeons and experts was found to reside only in the first training sessions. The EyeSI simulator in its current state of development, and our implementation of it, seems to require further development before it can be used as a tool to select residents for surgical training and to re-certify more senior surgeons.

Background: Competency-based training has gained ground in surgical training and with it assessment tools to ensure that training objectives are met. Very few assessment tools are available for evaluating performance in thoracoscopic procedures. Video recordings would provide the possibility of blinded assessment and limited rater bias. This study aimed to provide validity evidence for a newly developed and dedicated tool for assessing competency in Video-Assisted Thoracoscopic Surgery (VATS) lobectomy., Methods: Participants with varying experience with VATS lobectomy were included from different countries. Video recordings from participants' performance of a VATS right upper lobe lobectomy on a virtual reality simulator were rated by three raters using a modified version of a newly developed VATS lobectomy assessment tool (the VATSAT) and analyzed in relation to the unitary framework (content, response process, internal structure, relation to other variables, and consequences of testing)., Results: Fifty-three participants performed two consecutive simulated VATS lobectomies on the virtual reality simulator, leaving a total of 106 videos. Content established in previously published studies. Response process Standardized data collection was ensured by using an instructional element, uniform data collection, a special rating program, and automatic generation of the results to a database. Raters were carefully instructed in using the VATSAT, and tryout ratings were carried out. Internal structure Inter-rater reliability was calculated as intra-class correlation coefficients, to 0.91 for average measures (p < 0.001). Test/re-test reliability was calculated as Pearson's r of 0.70 (p < 0.001). G-coefficient was calculated to be 0.79 with two procedures and three raters. By performing D-theory was found that either three procedures rated by two raters or five procedures rated by one rater were enough to reach an acceptable G-coefficient of ≥ 0.8. Relation to other variables Significant differences between groups were found (p < 0.001). The participants' VATS lobectomy experience correlated significantly to their VATSAT score (p = 0.016). Consequences of testing The pass/fail score was found to be 14.9 points by the contrasting groups' method, leaving five false positive (29%) and six false negatives (43%)., Conclusion: Validity evidence was provided for the VATSAT according to the unitary framework. The VATSAT provides supervisors and assessors with a procedure-specific assessment tool for evaluating VATS lobectomy performance and aids with the decision of when the trainee is ready for unsupervised performance.

Background: Nontechnical skills are important for safe and efficient surgery. Teams performing video-assisted thoracoscopic surgery (VATS) lobectomy express that it is of utmost importance to have a shared mental model (SMM) of the patient, current situation, and team resources. However, these SMMs have never been explored in a clinical setting. The aim of this observational study was to measure the similarity of SMMs within teams performing VATS lobectomy., Methods: In this national, multicenter study, SMMs of teams performing VATS lobectomy (n = 64) were measured by preoperative and postoperative questionnaires that were completed by all team members (n = 172). Participants' responses were compared within each team to explore SMMs of risk assessment, familiarity, technical skills, nontechnical skills, and problems., Results: Analysis showed poor agreement between team members with respect to risk assessment, but higher levels of agreement were found for assessments of familiarity, technical skills, and nontechnical skills within the team (Cronbach's alpha = 0.90), most notably for surgical subteams (ie, surgeon plus assistant surgeon plus surgical nurses). During the surgical procedure, the most frequent problems were related to anesthesia, and these were most often recognized by the surgeons. The operating room nurses were the least aware of each other's and the surgeons' problems., Conclusions: Significant variation exists in the SMMs among VATS team members, with poor agreement regarding the patient and current situation, but better agreement with respect to team resources. Focus on preoperative and perioperative team reflexivity, in addition to explicit communication within unfamiliar teams, may provide opportunities to enhance SMMs, with possible downstream effects on team performance., (Copyright © 2019 The Society of Thoracic Surgeons. Published by Elsevier Inc. All rights reserved.)

Background: Trocar pigtail catheter thoracentesis (TPCT) is a common procedure often performed by junior physicians. Simulation-based training may effectively train physicians in the procedure prior to performing it on patients. An assessment tool with solid validity evidence is necessary to ensure sufficient procedural competence., Objectives: Our study objectives were (1) to collect evidence of validity for a newly developed pigtail catheter assessment tool (Thoracentesis Assessment Tool [ThorAT]) developed for the evaluation of TPCT performance and (2) to establish a pass/fail score for summative assessment., Methods: We assessed the validity evidence for the ThorAT using the recommended framework for validity by Messick. Thirty-four participants completed two consecutive procedures and their performance was assessed by two blinded, independent raters using the ThorAT. We compared performance scores to test whether the assessment tool was able to discern between the two groups, and a pass/fail score was established., Results: The assessment tool was able to discriminate between the two groups in terms of competence level. Experienced physicians received significantly higher test scores than novices in both the first and second procedure. A pass/fail score of 25.2 points was established, resulting in 4 (17%) passing novices and 1 (9%) failing experienced participant in the first procedure. In the second procedure 9 (39%) novices passed and 2 (18%) experienced participants failed., Conclusions: This study provides a tool for summative assessment of competence in TPCT. Strong validity evidence was gathered from five sources of evidence. A simulation-based training program using the ThorAT could ensure competence before performing thoracentesis on patients., (© 2019 S. Karger AG, Basel.)

Background: Thoracic ultrasound (TUS) has a high diagnostic accuracy for many common pulmonary diseases, but theoretic knowledge in sonographic physics, thoracic anatomy and physiology, and sonopathologic patterns is required to develop competence., Objectives: The aims of the study were to develop and gather validity evidence for a theoretical test in TUS and to establish a pass/fail standard., Methods: Content was provided based on expert interviews, leading to the creation of 113 initial multiple-choice question (MCQ) items. Consensus was reached on 92 proceeding items through a Delphi process, and items were presented to physicians with different knowledge and experience in TUS. Answers were used for item statistics in order to select the items with the most optimal item discrimination and difficulty (i.e., level I items) to be included in the final test. Mean scores of the novice, intermediate and experienced groups were compared, and a pass/fail score was established using the contrasting groups' standard setting method., Results: Item statistics revealed 38 level I items, of which 30 were selected to be included in the final test. The internal consistency was high (Cronbach's alpha = 0.88). Differences in mean scores were 8.6 points (p < 0.001), 6.3 points (p = 0.01), and 14.9 points (p < 0.001) between novices and intermediates, intermediates and experienced, and novices and experienced, respectively. A pass/fail standard of 20 points was established., Conclusion: The established MCQ test can distinguish between physicians with different levels of competence in TUS and enables an objective, evidence-based approach for assessing the theoretical knowledge of trainees undergoing an educational programme in TUS., (© 2019 S. Karger AG, Basel.)

Background: The use of robotic surgery for minimally invasive procedures has increased considerably over the last decade. Robotic surgery has potential advantages compared to laparoscopic surgery but also requires new skills. Using virtual reality (VR) simulation to facilitate the acquisition of these new skills could potentially benefit training of robotic surgical skills and also be a crucial step in developing a robotic surgical training curriculum. The study's objective was to establish validity evidence for a simulation-based test for procedural competency for the vaginal cuff closure procedure that can be used in a future simulation-based, mastery learning training curriculum., Methods: Eleven novice gynaecological surgeons without prior robotic experience and 11 experienced gynaecological robotic surgeons (> 30 robotic procedures) were recruited. After familiarization with the VR simulator, participants completed the module 'Guided Vaginal Cuff Closure' six times. Validity evidence was investigated for 18 preselected simulator metrics. The internal consistency was assessed using Cronbach's alpha and a composite score was calculated based on metrics with significant discriminative ability between the two groups. Finally, a pass/fail standard was established using the contrasting groups' method., Results: The experienced surgeons significantly outperformed the novice surgeons on 6 of the 18 metrics. The internal consistency was 0.58 (Cronbach's alpha). The experienced surgeons' mean composite score for all six repetitions were significantly better than the novice surgeons' (76.1 vs. 63.0, respectively, p < 0.001). A pass/fail standard of 75/100 was established. Four novice surgeons passed this standard (false positives) and three experienced surgeons failed (false negatives)., Conclusion: Our study has gathered validity evidence for a simulation-based test for procedural robotic surgical competency in the vaginal cuff closure procedure and established a credible pass/fail standard for future proficiency-based training.

Background: Specific assessment tools can accelerate trainees' learning through structured feedback and ensure that trainees attain the knowledge and skills required to practice as competent, independent surgeons (competency-based surgical education). The objective was to develop an assessment tool for video-assisted thoracoscopic surgery (VATS) lobectomy by achieving consensus within an international group of VATS experts., Method: The Delphi method was used as a structured process for collecting and distilling knowledge from a group of internationally recognized VATS experts. Opinions were obtained in an iterative process involving answering repeated rounds of questionnaires. Responses to one round were summarized and integrated into the next round of questionnaires until consensus was reached., Results: Thirty-one VATS experts were included and four Delphi rounds were conducted. The response rate for each round were 68.9% (31/45), 100% (31/31), 96.8% (30/31), and 93.3% (28/30) for the final round where consensus was reached. The first Delphi round contained 44 items and the final VATS lobectomy Assessment Tool (VATSAT) comprised eight items with rating anchors: (1) localization of tumor and other pathological tissue, (2) dissection of the hilum and veins, (3) dissection of the arteries, (4) dissection of the bronchus, (5) dissection of lymph nodes, (6) retrieval of lobe in bag, (7) respect for tissue and structures, and (8) technical skills in general., Conclusion: A novel and dedicated assessment tool for VATS lobectomy was developed based on VATS experts' consensus. The VATSAT can support the learning of VATS lobectomy by providing structured feedback and help supervisors make the important decision of when trainees have acquired VATS lobectomy competencies for independent performance.

Background: Competence in video-assisted thoracoscopic surgery lobectomy has previously been established on the basis of numbers of procedures performed, but this approach does not ensure competence. Specific assessment tools, such as the newly developed video-assisted thoracoscopic surgery lobectomy assessment tool, allow for structured and objective assessment of competence. Our aim was to provide validity evidence for the video-assisted thoracoscopic surgery lobectomy assessment tool., Methods: Video recordings of 60 video-assisted thoracoscopic surgery lobectomies performed by 18 thoracic surgeons were rated using the video-assisted thoracoscopic surgery lobectomy assessment tool. All 4 centers of thoracic surgery in Denmark participated in the study. Two video-assisted thoracoscopic surgery experts rated the videos. They were blinded to surgeon and center., Results: The total internal consistency reliability Cronbach's alpha was 0.93. Inter-rater reliability between the 2 raters was Pearson's r = 0.71 (P < .001). The mean video-assisted thoracoscopic surgery lobectomy assessment tool scores for the 10 procedures performed by beginners were 22.1 (standard deviation [SD], 8.6) for the 28 procedures performed by the intermediate surgeons, 31.2 (SD, 4.4), and for the 20 procedures performed by experts 35.9 (SD, 2.9) (P < .001). Bonferroni post hoc tests showed that experts were significantly better than intermediates (P < .008) and beginners (P < .001). Intermediates' mean scores were significantly better than beginners (P < .001). The pass/fail standard calculated using the contrasting group's method was 31 points. One of the beginners passed, and 2 experts failed the test., Conclusions: Validity evidence was provided for a newly developed assessment tool for video-assisted thoracoscopic surgery lobectomy (video-assisted thoracoscopic surgery lobectomy assessment tool) in a clinical setting. The discriminatory ability among expert surgeons, intermediate surgeons, and beginners proved highly significant. The video-assisted thoracoscopic surgery lobectomy assessment tool could be an important aid in the future training and certification of thoracic surgeons., (Copyright © 2018 The American Association for Thoracic Surgery. Published by Elsevier Inc. All rights reserved.)

Competency-based medical education relies on the use of assessment tools, which can describe and/or measure medical competencies and are supported by validity evidence. The exploration of validity evidence ensures, that the assessment tool not only measures what is intended, but also that the interpretation and decisions made are fair and just. In this review, we have described a contemporary approach to validity used in Kane's framework. We have also used an example to illuminate, how the exploration of validity evidence can be done in a scientific and systematic manner.