Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study. Volume 1. Summary Report

Distributed Mission Operations (DMO) training consists of multiplayer networked environments enabling warfighting training on higher-order individual and team-oriented skills. Surprisingly, only sparse DMO training effectiveness literature can be found and very few studies contain objective data. The dataset used in this research represents the largest DMO effectiveness dataset known to exist today (76 teams/384 pilots on over 3,000 engagements), containing 33 months' worth of multi-faceted DMO data, including objective data from the simulators, multiple participant surveys, subject matter expert (SME) ratings of performance, and knowledge structure tests. Observed performance differences between the pre- and post-test mirror-image point-defense assessment sessions served as the primary basis for the evaluation. Results were dramatic: On the post-test, 58.33% fewer enemy strikers reached their target and there were 54.77% fewer F-16 mortalities. Furthermore, there were corroborating significant improvements from the numerous measured skill metrics (e.g., weapons employment), SME expert observer ratings, and participant self-report opinion ratings. These converging results provide substantial evidence that pilots become much more proficient on key aspects of combat mission objectives as a function of training within the simulator. Finding highly significant performance differences across multiple datasets between the pre- and post-tests with a combat-ready participant pool in a complex task/environment forms a formidable argument that DMO training yields considerable within-simulator warfighter competency improvement. In this report, we summarize the different dataset classes, overview the primary hypotheses and results associated with each, and discuss the convergence of the datasets to illustrate the "big picture" DMO training effectiveness. As such, more detailed hypotheses, analyses, and discussions are contained in separate reports (Vols. II through V).
This is the first volume of a five-volume report, Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study. Prepared in collaboration with Air Force Research Laboratory, Warfighter Readiness Research Division, Mesa, AZ.