Collecting Validity Evidence for Simulation-Based Assessment of Point-of-Care Ultrasound Skills

Objectives The aim of this study was to examine the validity of a simulator test designed to evaluate focused assessment with sonography for trauma (FAST) skills. Methods Participants included a group of ultrasound novices (n = 25) and ultrasound experts (n = 10). All participants had their FAST skills assessed using a virtual reality ultrasound simulator. Procedural performance on the 4 FAST windows was assessed by automated simulator metrics, which received a passing or failing score. The validity evidence for these simulator metrics was examined by a stepwise approach according to the Standards for Educational and Psychological Testing. Metrics with validity evidence were included in a simulator test, and the reliability of test scores was determined. Finally, a pass/fail level for procedural performance was established. Results Of the initial 55 metrics, 34 (61.8%) had validity evidence (P