Search

Large Force Exercises such as Red Flag in the United States and Pitch Black in Australia require significant investments in resources and personnel. Participating units may spend months preparing for an LFE to ensure that warfighters receive the greatest training benefit from this investment. Local area training, however, cannot replicate the most demanding aspects of LFEs. Air Forces in the United Kingdom, Sweden, and Australia have used distributed simulation training to complement live-fly exercises to prepare for LFEs. In this panel presentation, the speakers will describe how training exercises using distributed simulation were structured and conducted to meet specific training goals. The panel will conclude with presentations on how detailed analysis of training needs is necessary to structure simulator scenarios and how future training exercises could be made more effective., Presented at the European Simulation Interoperability Workshop, held in Ottawa, Canada, 2010. Prepared in cooperation with Defence Science and Technology Laboratory, Fareham, UK; Swedish Defence Research Agency, Stockholm, Sweden; and Defence Science and Technology Organization, Fishermans Bend, Australia.

Training programs for warfighters have become increasingly dependent on the use of simulation systems, made necessary by the growing costs of live flight training. The transition to a greater dependence on simulators has required an increase in concerns about the ability of simulators to provide effective training. Research in the area of simulator fidelity has focused primarily on training capabilities, evaluating current systems for strengths and weaknesses. The evaluation of current systems is an essential component in providing the best possible training environment to pilots; however, the impact of improvements is of equal importance. An informed decision about which changes will have the greatest benefit for warfighter readiness would provide a means of maximizing the return on the investment in the development of the system and creating the most efficient and effective training environment possible. The current work presents a process by which system deficiencies and potential solutions are identified and then integrated into quantitative models based on multiple factors. The factors included in the model are potential for improvement, training gaps, extent to which solutions address deficiencies, material costs of solutions, and time costs of solutions. A project manager and engineers provided feedback in three areas: the extent to which a solution addresses a deficiency, the cost of the technology, and the engineering hours involved in the change. Three models were computed and compared across solutions, revealing the utility of the empirical process. The models included in the comparison varied the weights within and across factors, establishing the importance of determining appropriate weightings. The process and algorithm presented in the current work have far reaching practical applications as a tool for assisting in the decision-making process during development and improvement of training technologies., Presented at the Fall Simulation Interoperability Workshop held in Orlando, FL in Sep 2008. Published in the Proceedings of the Fall Simulation Interoperability Workshop, paper 08F-SIW-007 p11-17, 2008.

During the development and enhancement of simulator systems, new technologies are incorporated to improve desired capabilities. The degree to which changes to the simulator system improve training capabilities in anticipated areas is of primary interest; however, an evaluation of benefits from newly incorporated technologies should consider the impact on training capabilities in other, less directly related areas. The current study utilized methodology developed to identify and evaluate the fidelity and deficiencies of a simulation system. The process involved utilizing Mission Essential Competencies and Subject Matter Experts to determine the training capabilities of a simulation system. The work was applied to the development of a Deployable Tactical Trainer as a method of providing feedback on the system. Data from the process were obtained throughout the development of the simulator, and the current study presents a comparison of the evaluation of the system before and after a meaningful upgrade to the technology of the simulation system. At issue is determining whether or not the changes in the training capabilities were as expected. Incidental changes to training capabilities of the system were identified and are discussed, presenting an overall evaluation of the expected versus unexpected effects on the system. Implications for the importance of building a global awareness of the simulator system are discussed. The growing dependence on simulation systems for training increases the impact of changes to the system on pilot readiness., Presented at the AIAA Modeling and Simulation Technologies Conference and Exhibit held in Honolulu, HI on 18-21 Aug 2008. Published in the Proceedings of the AIAA Modeling and Simulation Technologies Conference and Exhibit, paper no. AIAA 2008-6681, 2008.

This technical paper contains the contributions of the Air Force Research Laboratory, Human Effectiveness Directorate, Warfighter Readiness Research Division (AFRL/RHA) to the 2008 Interservice/Industry Training, Simulation, and Education Conference (I/ITSEC). I/ITSEC is the premiere event of its kind in the world of training, modeling, and simulation. The 2008 conference theme was: Learn. Train. Win! The conference included multiple presentations of previously unpublished papers, as well as tutorials and special events--all selected by an extensive peer review process. This paper contains four AFRL/RHA paper presentations and the special I/ITSEC edition of the AFRL/RHA newsletter, Fight's On. PAPERS: Montijo et al. Training Interventions for Reducing Flight Mishaps; Eidman & Kam. Computer-Generated Forces for Joint Close Air Support and Live Virtual Constructive Training; Rowe et al. Assessing High-Fidelity Training Capabilities using Subjective and Objective Tools; and Fitzgerald et al. Student Flight Instructor Competencies, Attached papers were presented at the 2008 Interservice/Industry Training Simulation, and Education Conference in Orlando FL on 1-4 Dec 2008. The original document contains color images. All DTIC reproductions will be in black and white.

Researchers have attempted to measure pilot knowledge and changes in knowledge, in both simulated and live-fly events. However, measurement in these training environments has been more successful in measuring overall flight performance outcomes rather than on underlying changes in knowledge. Research to assess changes in pilots' knowledge as a result of training is underway at the Air Force Research Laboratory (AFRL) in Mesa, Arizona, using the Pathfinder Network Scaling technique. The Pathfinder method uses individual judgments of the relationships between concepts/constructs in a domain as a basis to develop an empirically derived representation of knowledge about the concepts/constructs. These representations can be compared and changes in representation can be quantified to assess the impact of an intervention on knowledge. Previous research has demonstrated the value of Pathfinder for assessing the impact of both education and training interventions in domains such as computer programming. At AFRL, pilots, as part of a week-long 4-ship F-16 Distributed Mission Operations (DMO) training research program, participated in a Pathfinder study to asses F-16 pilot understanding of complex combat mission constructs/concepts critical to mission performance. The objective was to assess training effects that are more fundamental and process-orientated. This paper will report findings from a sample of 71 F-16 pilots who vary in experience level. Our results will be discussed both in terms of practical utility of the Pathfinder technique as a measurement methodology and in terms of knowledge measurement as a criterion for evaluating training., Prepared in cooperation with Arizona State University, Meza, Arizona. Presented at the Interservice/Industry Training, Simulation and Education Conference held in Orlando, Florida on 26-29 November, 2007.

The importance of fidelity in simulation training has been established as a key factor in training effectiveness. Design and development of simulation systems is dealt with commonly as an iterative procedure in which the goal is to improve on the training effectiveness of the system. Improvements are made based on methods of identifying and resolving deficiencies. Improvements must then be considered relative to available resources to determine how to prioritize the multiple options. The current paper presents a flexible methodology for evaluating the relative effects of multiple deficiencies on a simulation system, utilizing information from warfighters about the effectiveness of a Deployable Tactical Trainer (DTT). First, subject matter experts were asked to evaluate the fidelity of the current DTT system across a set of 199 experiences and emergency procedures (EP). Next, the same subject matter experts identified deficiencies of the current system. Then the warfighters evaluated the 199 experiences and EPs for which deficiency had the most adverse affect on training. A composite score was computed for each deficiency, using weighted variables accounting for (a) the amount of improvement possible and (b) the importance of training for each experience. Deficiency scores provide a means of comparing the relative effect of each deficiency on warfighter training. The impact of utilizing multiple sources of information is presented through a comparison of the different decisions that would result from using partial information or alternative weighting methods. The proximal implications of using the proposed methodology to have the greatest impact on improvement are discussed. The distal implications of the impact the improvements have on pilot perceptions, and ultimately, on objective pilot performance measures, are also discussed., Presented at the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) held in Orlando, FL on 26-29 Nov 2007. Published in the Proceedings of the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), Paper 7171, 2007.

This study fills a void in the literature on skill decay by incorporating a cognitively complex task and an extended nonuse period. Using 192 paid participants who trained for approximately 17 hours on a command-and-control microworld simulation, we examined the effectiveness of distributed versus massed practice and post-acquisition observational rehearsal in minimizing skill decay and facilitating adaptive transfer and reacquisition after an 8-week nonuse period. We also examined how individual differences in ability and motivation predicted retention, transfer, and reacquisition as well as participation in voluntary post-acquisition observational rehearsal. Results showed that distributed practice had a positive effect on skill retention and post-observational rehearsal had a positive effect on transfer. Although some individual differences (e.g., self-efficacy) were consistently associated with retention, transfer, and reacquisition, overall levels of prediction were relatively weak, especially for transfer. Differences in motivation rather than ability differentiated individuals who participated in voluntary rehearsal from those who did not., Prepared in collaboration with Air Force Research Laboratory (AFRL), Mesa, AZ. Prepared in cooperation with Texas A&M University and University of Oklahoma. Sponsored in part by Defense Advanced Research Projects Agency (DARPA).

The United States Air Force Research Laboratory (AFRL), Human Effectiveness Directorate, Warfighter Readiness Research Division, in conjunction with the United States Air Force Major Command, Air Combat Command (ACC), has for a number of years pursued a program of research whose focus is the rational integration of networked flying, flying-related, and command and control simulators into current training via Distributed Mission Operations (DMO). The Mission Essential Competency (MEC) work analysis methodology was developed as one facet of this DMO initiative. The MEC approach addresses multiple AFRL/ACC DMO program needs. Among other purposes, MECs enable the determination of training requirements and the appropriate mix of live operation and virtual training media, allow ACC to identify the "value" of DMO and provide justification for funding, and directly enable the construction of air combat simulation scenarios for which validated measures can be developed. While much has already been written about the MECs, one particular unaddressed topic relates to the nature of this effort relative to other work analysis and competency methods. How are MECs different from other methods of work analysis? What outcomes does the MEC approach produce that are unique? Is the MEC process rigorous enough to be considered a work analysis method, as Industrial/Organizational Psychologists understand the terminology and process? To address these questions, the authors take a historical-comparative approach. First, they consider the origin and nature of modern job analysis. Second, they do the same for the competency movement. Third, they review in some detail how MECs are developed and used. Fourth and finally, they discuss MECs as a job analytic technique, including some thoughts on the validity of the method., Prepared in cooperation with Aptima, Inc., Woburn, MA.

Over the past several years, the UK Defence Science and Technology Laboratory (DSTL) and the US Air Force Research Laboratory (AFRL) have been involved in research to develop training and assessment methods for use in mission training via distributed simulation. As part of these efforts, we developed competency-based training and rehearsal scenarios and data collection instrumentation for routinely delivering and assessing distributed training events. The most recent collaborative study, named "Red Skies" involved extending our work to include field assessments of the training benefits derived from involvement in a simulation-based distributed mission training event and subsequent live flying at a Coalition Red Flag exercise event at Nellis Air Force Base, Nevada in the US. This event was the largest Red Flag event ever hosted, and presented a number challenges for the study and for data collection. This paper presented the methods and results from the distributed simulation preparation, which involved connecting simulators in Bedford UK with those of the Warfighter Readiness Research Division of the AFRL, located in Mesa, Arizona. It also presents results from the subsequent live fly and evaluation in the Coalition Red Flag exercise, which included demonstrating our first and successful attempt to follow coalition pilots from a Distributed Mission Operations (DMO) training event to the Red Flag exercise and to collect data while the live fly event was underway. Outcome data is presented and discussed. We close with a discussion of the implications of our work to date as a way forward for future training events and how the methods and process developed for Red Skies can allow us to quantify and demonstrate training benefits from distributed simulation training for live Operations, Pres. at the MSG-35 Symposium on The Effectiveness of Modeling and Simulation-From Anecodotal to Substantive Evidence, held in Warsaw Poland on 13-14 Oct 2005. Published in RTO-MSG-035, Paper 13, p13-5 - 13-14, 2005.

In today's fast paced operational environment it is almost impossible to evaluate new approaches and capabilities on operational aircraft in active duty squadrons. Further, obtaining range facilities and other aircraft to provide a tactically rich environment for test and evaluation of new concepts and capabilities is both time consuming and very expensive. Finally, many new weapon system capabilities cannot be evaluated in live fly environments as the actual systems do not exist in hardware form yet. Experimentation by expert fighter pilots using simulation facilities in the US and Sweden has resulted in improved tactical employment and training. We will present results from several controlled comparative studies we have undertaken with operational pilots and controllers examining new weapons characteristics and instrumentation for operational aircraft; live, virtual, and constructive data linking and tactics training; precision engagement tactics and system requirements; and embedding human performance assessment in simulation environments and on live aircraft. We will also highlight research currently underway with the Swedish Air Force which is examining and validating simulation technology and methods for integrating Swedish Air Force into North Atlantic Treaty Organization (NATO/OTAN) peacekeeping support operations., Presented at the NATO RTO Modelling and Simulation Conference held in Rome, Italy on 5-6 Oct 2006, and Koblenz, Germany on 7-8 Oct 2006. Published in Transforming Training and Experimentation through Modelling and Simulation, Meeting Proceedings RTO-MP-MSG-045, Paper 19. ISBN 92-837-0075-9 and ISBN 978-92-837-0075-3 . Note: Power Point link inoperable, slides begin on page 14. See also ADM002053. The original document contains color images.

Due to infrequent training while deployed, warfighters skills can decay, thus creating a training gap. Deployed training has historically been hindered by restrictions to live-fly training opportunities due to factors such as ops-tempo, airspace/range restrictions, security issues, alert requirements, and wartime rules of engagement. In order to maintain high proficiency and readiness levels, changes are needed in standard training programs while warfighters are deployed. The United States Air Force's Distributed Mission Operations (DMO) concept has become critical to warfighter training across all mission areas. Despite an increased reliance on DMO training, a deployable DMO training capability does not exist to provide the critical training opportunities previously unavailable during extended deployments. Moreover, while researchers have discussed and described the tradespace associated with varying levels of physical and functional fidelity, opportunities to conduct controlled studies to examine and quantify how much of what kind of fidelity for skills improvement and maintenance are few and far between. This presentation will describe a series of research studies underway at the Warfighter Readiness Research Division in Mesa Arizona designed to assess within-training and training transfer to live operations for both individual and tactical teams, with varying levels of fidelity in the training environment. These studies are examining a variety of specific parameters associated with achieving and maintaining combat readiness and proficiency. We conclude with discussion of our longer-term plan to demonstrate and quantify the impact of deployed training and rehearsal environment on transfer to the operational environment and to subsequent retraining intervals., Presented at the RTO Human Factors and Medicine Panel (HFM) Workshop held in United States Military Academy, West Point, NY on 13-15 Jun 2006 and published in proceedings of the same as key note address 3. See also ADM002024, Virtual Media for Military Applications (Media virtuels pour les applications militaires) (CD-ROM), RTO-MP-HFM-136, AC/323(HFM-136)TP/68. The original document contains color images.

Historically USAF Combat Air Forces (CAF) aircrews have trained under an event-based system, maintaining Combat Mission Ready status by accomplishing specified numbers of sortie types and events during the annual training cycle. Advances in training analysis and design and performance measurement are enabling a revolution in training philosophy and a transition to a competency-based training methodology. Competency-based training places emphasis on the required proficiency rather than the number of times the mission has been performed. The transition requires answers to two basic but vital questions: "What do we measure and how do we measure it?" Air Combat Command (ACC) and the US Air Force Research Laboratory (AFRL) have teamed to answer both questions. ACC's Flight Operations Division (ACC/DOT) and the AFRL Warfighter Readiness Research Division, pioneered work in Mission Essential Competencies (MECs -- Note that the phrase Mission Essential Competencies, and associated acronyms have been Service Marked.) that provided the "what" -- what knowledge and skills do operational aircrew require to succeed in a complex combat environment. MEC-based research syllabi and new performance oriented debriefing capabilities have produced dramatic increases in warfighter performance in very short periods of time and are key to answering the "how" question. The implementation of "what" and "how" result in training that is both adaptive to the immediate needs of the training audience and scalable in application from the individual to multi-team collective training. At the end of the day the CAF will season combat aviators quicker and be able to sustain higher levels of overall combat readiness.

This paper describes part of an ongoing research program designed to integrate both objective and observer-provided data to develop comprehensive tools for assessing and diagnosing pilot performance in complex and dynamic training and rehearsal environments. The goal is to provide a probabilistic capability to assess pilot knowledge and skill competencies and to provide results to instructors for their use in the remediation of performance and the identification of "gaps" that remain. The development process and efforts to date are reported., Task Order (T.O.) 34. Presented at the Behavior Representation in Modeling and Simulation (BRIMS) Conference, held in Mesa, AZ, on 12-15 May 2003. Published in the Proceedings of the Behavior Representation in Modeling and Simulation (BRIMS) Conference, 2003. Prepared in collaboration with Air Force Research Laboratory (AFRL), Mesa, AZ.

The work in this report focuses on the user acceptance of the Air Force Research Laboratory, Human Effectiveness Directorate, Warfighter Readiness Research Division (AFRL/HEA) Distributed Mission Operations (DMO) training research site which contains four high-fidelity F-16 simulators, an Airborne Warning and Control System (AWACS) simulator, a threat generation system, and a full complement of brief/debrief technologies. The 327 F-16 pilots and the 49 AWACS users based their opinions on experiences obtained during consistent local area network five-day training research syllabi containing over 40 total 4vX scenarios, primarily air-to-air with some air-to-ground. Examining the user ratings for opinions about the DMO environment revealed that across all 58 rated statements, both F-16 and AWACS participants generally like the environment. "I would recommend this training experience to other pilots/controllers" was rated by all but one of 49 controllers and all but 16 of 327 pilots with the highest rating possible of "Strongly Agree." Performing a content analysis on their open-ended responses to what they felt was most beneficial about the environment, F-16 pilots most frequently wrote comments relating to "realistic qualities," while AWACS operators most frequently wrote positive comments regarding the "scenarios." The second most written comment for both the F-16 and AWACS operators related to positive skill acquisition. Finally, when asked to rate to what extent the Mesa DMO system provides the 45 different F-16 critical air-to-air experiences (defined by the Mission Essential Competency process), the pilots rated that 38 experiences (84%) could be obtained "to a moderate extent" or higher, more than all seven other environments surveyed., This report is the fourth volume of a five-volume report, Distributed Operations Within-Simulator Training Effectiveness Baseline Study. Prepared in cooperation with L3 Communications, Mesa, AZ and Air Force Research Laboratory, Warfighter Readiness Research Division, Mesa, AZ.

To provide an indirect assessment of learning for the overall Distributed Mission Operations (DMO) Within Simulator Training Effectiveness Baseline Study as described in Volume I, Summary Report, of AFRL-HE-AZ-TR-2006-0015, the current work examined pilots who participated in a Pathfinder data collection methodology both before and after five days of DMO training. The Pathfinder methodology is a qualitative/quantitative method that can be used to assess if the pilots' underlying knowledge structures (i.e., their understanding) of air combat may have changed significantly as a function of DMO training. A total of 144 F-16 pilots rated the relatedness of 105 pairs of air combat concepts. Analyzing the before/after DMO training Pathfinder results by flight qualification or Viper flight position revealed remarkably similar results across all matrices. The most remarkable attribute of these networks is their stability; the pilots, regardless of demographic classification or before/after DMO training, view the relationships between the air combat concepts in very similar ways. We postulate that the stability of the networks most likely indicates that the 15 chosen concepts are at a high level of abstraction and reflect the shared knowledge in the general F-16 pilot population. As such, the concepts are likely not sensitive enough (i.e., detailed enough) to reflect differences in understanding resulting from five days of DMO training., This is the last volume of a five-volume report, Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study. Prepared in collaboration with Air Force Research Laboratory, Warfighter Readiness Research Division, Mesa, AZ.

The current work documented in this report is the subject matter expert (SME) rating data from a large study examining the within-simulator learning benefits of Distributed Mission Operations (DMO) training as described in AFRL-HE-AZ-TR-2006-0015 Vol I, Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study: Summary Report. That study examined 76 participating F-16 four-ship teams in week-long DMO training exercises and compared beginning-of-week to end-of-week performance on mirror-image air combat scenarios. As a major assessment component to the overall study, the current work reports data we collected from SME ratings of pilot performance. Two SME rating methods were employed -- real-time ratings (i.e., done during mission execution) and SME ratings captured later using a scientific "blind" protocol. Comparing SME ratings of performance on the mirror-image scenarios revealed highly significant performance increases as a function of DMO training. SME real-time and blind ratings were both found to be significantly higher for end-of-week scenarios. The fact that the real-time and blind rater results corroborated one another provides strong support that performance greatly improved as a function of DMO training. Lastly, though the real-time ratings are not as scientific as the blind ratings, the fact that the blind ratings revealed very similar results to the real-time ratings greatly increases our confidence for continuing to use real-time ratings to assess performance. Compared to implementing an ongoing blind rating system, this justification to use real-time ratings creates a significant logistical, time, and financial savings for future research., This is the third volume of a five-volume report, Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study. Prepared in cooperation with The Boeing Co., Mesa, AZ and Air Force Research Laboratory, Warfighter Readiness Research Division, Mesa, AZ.

Distributed Mission Operations (DMO) training consists of multiplayer networked environments enabling warfighting training on higher-order individual and team-oriented skills. Surprisingly, only sparse DMO training effectiveness literature can be found and very few studies contain objective data. The dataset used in this research represents the largest DMO effectiveness dataset known to exist today (76 teams/384 pilots on over 3,000 engagements), containing 33 months' worth of multi-faceted DMO data, including objective data from the simulators, multiple participant surveys, subject matter expert (SME) ratings of performance, and knowledge structure tests. Observed performance differences between the pre- and post-test mirror-image point-defense assessment sessions served as the primary basis for the evaluation. Results were dramatic: On the post-test, 58.33% fewer enemy strikers reached their target and there were 54.77% fewer F-16 mortalities. Furthermore, there were corroborating significant improvements from the numerous measured skill metrics (e.g., weapons employment), SME expert observer ratings, and participant self-report opinion ratings. These converging results provide substantial evidence that pilots become much more proficient on key aspects of combat mission objectives as a function of training within the simulator. Finding highly significant performance differences across multiple datasets between the pre- and post-tests with a combat-ready participant pool in a complex task/environment forms a formidable argument that DMO training yields considerable within-simulator warfighter competency improvement. In this report, we summarize the different dataset classes, overview the primary hypotheses and results associated with each, and discuss the convergence of the datasets to illustrate the "big picture" DMO training effectiveness. As such, more detailed hypotheses, analyses, and discussions are contained in separate reports (Vols. II through V)., This is the first volume of a five-volume report, Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study. Prepared in collaboration with Air Force Research Laboratory, Warfighter Readiness Research Division, Mesa, AZ.

The current work reports only the objective data from AFRL-HE-AZ-TR-2006-0015, Volume I, Distributed Mission Operations Within-Simulator Training Effectiveness: Summary Report, but here we expand the reporting of objective data both in depth and breadth. We examined F-16 pilots participating in week-long Distributed Mission Operation (DMO) training exercises and compared extensive computer-collected data between beginning-of-week and end-of-week pilot performance on mirror-image scenarios. The DMO research environment in Mesa, AZ consisted of four high-fidelity F-16 simulators and one high-fidelity Airborne Warning and Control System simulator. Participating F-16 teams flew over 40 total scenarios according to a five-day syllabus, book-ended on Monday and Friday by mirror-image point defense air combat benchmark scenarios. Seven mission outcome measures were found to be significantly better on Friday than Monday: A 58.33% decrease in enemy strikers reaching their target, 38.10% greater distance from the base the F-16s disposed of the strikers, 54.77% fewer F-16 mortalities, 75.26% more enemy striker kills (before reaching base), 6.82% higher proportion of Viper Advanced Medium Range Air-to-Air Missile (AMRAAM) shots resulting in a kill, 51.60% lower proportion of enemy Alamo missile shots resulting in a kill, and a highly impressive 314.21% increase in an overall summary scoring scheme developed by subject matter experts. Significant trends were also found for a number of other metrics assessing skills. Of all the measures investigated in the current work, not a single offensive/defensive trade-off was observed, which significantly strengthens our conclusion that significant within-simulator learning took place., This is the second volume of a five-volume report, Distributed Mission Operations Within-Simulator Training Effectiveness Baseline Study. See Volume 1, AD

The Air Force Research Laboratory (AFRL), Warfighter Readiness Research Division, is conducting research to measure and track warfighter performance of knowledge and skills from and individual level to the command and control (C2) level, within both high-fidelity distributed simulation environments and live training environments. One critical development is a performance measurement system, the Performance Evaluation Tracking System (PETS), which captures data necessary for competency-based assessment and evaluations, end-user performance feedback, simulator technology developer validation, and for researcher and program manager evaluation of training techniques and technologies. PETS is comprised of various components that allow integration into focused benchmark studies, large-scale distributed coalition operations, and live-fly training scenarios. This paper describes the application of PETS during the multinational Warfighter Alliance in a Virtual Environment (First WAVE) coalition simulation event; discuss the lessons learned, the impact of different levels of abstraction and representational levels of correlation in the data, and the challenges facing both researchers and operational personnel for both producting and using performance data and methods., Presented at the 2005 European Simulation and Modeling Conference held in Porto, Portugal on 24-26 October 2005. Published in the Proceedings of the 2005 European Simulation and Modeling Conference, p354-366, October 2005. Prepared in collaboration with L-3 Communications, Mesa, AZ, contract F41624-97-d-5000. Also includes briefing charts.

Intelligence researchers traditionally focus their attention on the individual level and overlook the role of intelligence at the interindividual level. This research investigated the interplay of the effects of intelligence at the individual and interindividual levels by manipulating the intelligence-based composition of dyadic training teams. Using a sample of 176 young adult males and a complex computer-based criterion task, homogeneous and heterogeneous dyadic training teams were created based on intelligence scores, and both team and individual performance were assessed throughout 10 h of training. Results indicated a strong additive influence of intelligence on team performance and a slightly positive nonadditive effect in uniformly high (HH)-ability teams. Trainees' individual skill acquisition was strongly correlated with the performance of their teams. However, nonadditive partner effects were observed such that high-ability trainees acquired significantly more skill when paired with high-ability partners instead of low-ability partners, but low-ability trainees benefited very little from being paired with high-ability partners., Published in Intelligence, v33 p39-65, 2005. Prepared in cooperation with Texas A&M University, College Station, TX. Prepared in collaboration with U.S. Air Force Research Laboratory (AFRL), Mesa, AZ.

In this study, we created homogeneous and heterogeneous dyadic training teams on the basis of g to examine how team composition affects the acquisition and performance of a complex skill at both team and individual levels. Specifically, 176 young adult males completed 10 hours of training on a complex skill involving strong cognitive and psychomotor demands. Participants practiced in pairs and completed tests of both team and individual performance. We found a strong additive influence of ability on team performance. Uniformly high-ability teams outperformed mixed-ability teams, who in turn outperformed uniformly low-ability teams. At the individual level, high-ability trainees acquired significantly more skill when paired with other high-ability partners instead of low-ability partners; however, low-ability trainees benefited only slightly from being paired with high-ability partners., Prepared in cooperation with Texas A&M University, College Station, TX.

This study reports the analyses of data collected from an evaluation effort for 2 Mission Ready Technician "MRT" training programs for C-141 transport and F-16 fighter aircraft crew chiefs. We obtained ratings from over 100 trainees in each program, as well as from their trainers and supervisors, both during training and in the field via survey. The goal of this research was to explore the criterion space set up for this evaluation. Whereas past evaluation research has explored task difficulty, frequency, and importance, this research explores an expanded criterion space, including task confidence, task performance, task difficulty, and task frequency. Descriptive statistics, predictive regressions, and exploratory factor analyses are reported. We conclude that the data show a similar factor structure for both aircraft and that MRT frequency of task performance and confidence ratings are highly predictive of field performance. A major implication is that one way to optimize the effectiveness of training is to emphasize the development of trainee confidence at a relatively micro level, such as the task level., Published in Military Psychology, v15(1), p59-76, 2003.

This paper highlights the development and lessons learned from a vignette-based approach to knowledge assessment. Our initial development which is based on Situational Judgment Inventory (SJI) and Job Knowledge Inventory (JKI) research, used an open-ended paper-based assessment instrument, referred to as Situation Assessment and Action Selection (SAAS), to examine pilots assessment of air-to-air situations as well as their opinions on appropriate courses of action. Scoring of pilot responses was challenging. One limiting factor in using open-ended responses is the time and effort required to score them. We are exploring the use of automated scoring of the responses, beginning with Latent-Semantic Analysis (LSA). Successful LSA scoring would greatly enhance the utility of the method and support the next phase of development. The next phase of development is intended to be a more automated version of the instrument, referred to as the Air Superiority Knowledge Assessment System (ASKAS). Results from our evaluation of SAAS are presented and discussed. Lessons learned and a rationale for developing a multimedia-based assessment system is discussed. Finally, key features of ASKAS are described with respect to their potential for helping researchers and practitioners assess the impact of DMT on pilots knowledge and understanding., Published in Proceedings of the 2002 Interservice/Industry Training, Simulation, and Education Conference, held in Orlando, FL on 2-5 Dec 2002.

Air combat scenarios present unique problems where the solution is not obvious. An advanced networked simulation environment can be used to train high-level cognitive air combat skills such as problem solving. A problem in the development of principled and construct oriented assessment is related to determining the level of specificity of the assessment. We present a detailed discussion of the definition of critical competencies associated with combat mission problem solving performance, and we describe our approach to develop and implement an embedded performance assessment system that maps outcomes to these competencies. Finally, we discuss the implications for our approach for comprehensive assessment and discuss our goals for an evaluation of the competency-based approach to assessment., Published in the Journal of Computers in Human Behavior v18 n6, p773-82, 2002.

Educators and trainers generally agree that reflection is a desirable activity for learners to engage in, but has proven to be relatively difficult to promote. In this report, we have described a specific situation in which systematic efforts to promote reflection achieved only limited success. A list of factors we believe individually and collectively affect the degree to which reflection occurs was presented with the acknowledgement that there is scant research to support our beliefs. Clearly, research is needed if we are to develop a proven set of strategies for promoting the reflection on action attempted in this effort. The author's reflection of reflection leads to two conclusions. First, that reflection does indeed result in enhanced learning. And secondly, that our understanding of how to tap this powerful strategy is quire limited. Hopefully, this report will prompt additional interest in the topic and eventually result in a larger body of research and theory that can be applied across a variety of settings. Clearly, advances in both theory and in development and testing of practical strategies for promoting reflection is needed if we are to obtain the benefits claimed for this important form of what is probably a uniquely human behavior.

The military continues to face substantial reductions in training and operational resources. Of particular concern are reductions in the budget for in-residence initial skills and follow-on training courses. Although advanced training technologies offer opportunities for reducing the time required for a given course while maintaining levels of proficiency, technology is not a solution in and of itself. This report presents results from a recent application of a career field education and training planning simulation capability to identify cost-effective opportunities for the introduction of advanced training technology to reduce time-to-train and overall training costs. The report also discusses the importance of life-cycle modeling of entire career fields and important cost and utility factors for deriving total training costs and relative return-on-investment timelines. Implications for large-scale implementation of advanced training technology for restructuring future training are highlighted and discussed.

As noted in A Theory of Behavior in Organizations , by J.C. Naylor, R.D. Pritchard, and D.R. Ilgen's (1980), we extend their theory of behavior in organizations to explain work team productivity, and test it using time-series analysis on data from a large-scale study of teams. Priority scores for 37 indicators of productivity across five work teams over 23 months were used to predict month-by-month changes in productivity for each of the 37 team products. The results show that team productivity improvements can be explained by feedback including priority scores derived from nonlinear contingency functions of the product indicators. Furthermore, teams which initially performed more poorly benefited the most from the priority score feedback. Goal setting positively affected productivity gain and did so consistently across work teams, after the effects of priority feedback and the interaction of work team with priority feedback were controlled. Implications for team performance strategies are discussed., Prepared in cooperation with Texas A&M Univ., College Station, TX.

The 42nd Annual Conference of the International Military Testing Association met in Edinburgh, Scotland, on 7-9 Nov 2000. Representatives of various armed services of the US and other nations.This conference fosters improved personnel assessment for behavioral measurement, occupational analysis, manpower analysis, simulation models, training programs, selection methodology, survey and feedback systems. It promotes cooperation in the exchange of assessment procedures and instruments, and promotes assessment of military personnel as a scientific adjunct to modern military personnel management. Attached are three papers prepared by personnel of the Air Force Research Laboratory's Warfighter Readiness Research Division, relevant to the objectives of this conference which are included in the conference proceedings., Presented at the Annual Conference of the International Military Testing Association (42nd) held in Edinburgh, Scotland on 7-9 November 2000. Pub. in Proceedings of the Annual Conference of the International Military Testing Association (42nd), p481-307, 2000.

There is a critical need for the development of a general utility software package which would permit both policy capturing and policy specifying analyses through modeling of individual people characteristics and job properties (work states). Such software should be functional at multiple levels of abstraction: at an operational level, for person-job matching, and at an organizational planning level for executive assessment of alternative strategies for mission accomplishment or adaptation to changes. Planners and decision-makers need to be able to evaluate optimal ways of allocating available people to existing or anticipated work, and individuals need to order their preferences based on available or expected work and possible training. The needed software system would extend traditional person-job match technology to become a multi-level simulation and analysis system which could be potentially very useful in fuzzy budgeting or programming, and in planning for military to civilian conversions (individuals, units, bases, etc.). We strongly advocate an expeditious research & development effort for the design and tryout of such a software system., Presented at the Annual Conference of the Military Testing Association (35th) held in Williamsburg, VA in Nov 1993.