Search

The utility and effectiveness of simulation-based training of air combat skill has been supported by a long research history. However, despite this success, there has been much debate on the effectiveness of gradesheets that have been used as the primary measure of air combat skill. This study assessed the effectiveness of a Gradesheet that has been used to measure air combat team performance in Distributed Mission Operations (DMO). It was hypothesized that the DMO Gradesheet is an effective measure of air combat team performance, and is therefore sensitive to differences in team performance over time during training, among performance indicators, and across team experience levels. Between August 2000 and December 2001, 32 teams of F-16 pilots from various U.S. Air Force operational units participated in a week of structured DMO training. Training consisted of teams flying four networked high-fidelity F-16 simulators in scenarios against multiple constructed threats. On all scenarios of the training syllabus, air combat Subject Matter Experts (SMEs) graded the performance of the teams on 40 indicators that make constitute the DMO Gradesheet. Analyses of the data found that aggregate mean graded team performance increased over a week of DMO training. However, for all indicators, mean graded performance increased linearly and at approximately the same rate, suggesting that the Gradesheet captured only a single component of air combat performance. Furthermore, team experience did not moderate change in graded performance over missions as expected, undermining the claim that the Gradesheet measured even general performance. Results suggest that the DMO Gradesheet as used in the current study lacked the sensitivity, validity, and reliability desired in a measure of air combat performance. Future directions in the development of subjective and objective measures of air combat performance are discussed. (8 tables, 21 figures, 24 refs.), The original document contains color images.

Training specialists and subject-matter experts at the Air Force Research Laboratory's Mesa Research Site have been characterizing Distributed Mission Training (DMT) scenarios in terms of specific learning objectives linked to mission-essential competencies and to the underlying knowledge, skills, and experiences that are required for successful combat performance. As part of scenario characterization, we have identified the mission characteristics and levels of those characteristics that are important for complexity indexing. Techniques for determining overall scenario complexity and for relating scenario characteristics to mission essential competencies have been developed. This report presents results from a validation study comparing the new, analytically based complexity methodology with an empirically based approach. In addition, applications of the new assessment to both F-15 and F-16 weapon system capability and scenario characteristics to learning objectives while controlling overall complexity are discussed together with plans for developing DMT instructor support systems.

It has been consistently demonstrated that the active interlocked modeling (AIM) protocol, a dyadic training protocol developed by Shebilske, Regian, Arthur, and Jordan (1992), achieves a 100% increase in training efficiency over a control individual-trainee based condition. The purpose of the present study was to replicate this finding as well as to investigate its robustness in terms of skill loss and reacquisition. Thus, we compared the AIM-dyad training protocol to a standard individual protocol regarding the acquisition of a complex skill (i.e., Space Fortress: cf Mane & Donchin 1989). More importantly, skill loss after an 8-week non-practice interval and skill re-acquisition was also investigated. Despite half as much hands-on practice, the performance of dyadic trainees did not differ from that of individuals on tests of skill acquisition. loss and re-acquisition. These findings provide strong support and justification for the ongoing use of innovative dyadic protocols for the training of pilots and navigators in both military an non-military settings.

The military environment is constantly changing and restructuring. Manpower, personnel, and training (MPT) planning and management is increasingly crucial to maintaining the mission readiness of the forces. The Department of Defense needs an integrated MPT planning and management system. We believe that a unified theory of work is needed to provide a framework and to guide and focus related research and development. This unified theory of work will connect theories of human traits and states, theories of task and job characteristics, theories of job/task performance, and perhaps theories of organizational behavior. For example, Mitchell and Driskill (1986, August), in Optimizing Integrated Personnel System Training Decisions and Development, presented at the State-of-the-Art Applications of Job Analysis: Integrated Personnel Systems, a symposium conducted at the American Psychological Association convention, in Washington, DC, proposed a theory that relates training to task performance, via a series of intervening and exogenous variables. Such a theory could be extended to encompass individual differences among workers and tasks, as well as key organizational and environmental variables. This paper explores key issues associated with a unified theory approach to MPT modeling and decision making. Further, basic research and development needs required for such an integrated approach are highlighted and discussed.

The Air Force Research Laboratory (AFRL) in Mesa, AZ, has an ongoing program of research on training in a 4-ship F-16 Distributed Mission Training (DMT) system. A team of four pilots comes to the laboratory for a week-long training exercise and fly together as a 4-ship team on several missions designed to provide exposure to a range of combat scenarios. Missions involve briefing and debriefing sessions in addition to simulator time. Several projects are underway to assess the training effectiveness and compare different training methods. Pilots rated the relatedness of all pairs of 21 concepts from the domain of air-to-air engagements both before and after training. Measures of the internal consistency (Coherence) of the ratings and Pathfinder networks were derived from the ratings. Data from a group of the most experienced pilots (experts) provided a point of reference for the less-experienced pilots. There was significant correlation between experience level and similarity to the expert reference group at the beginning of the week supporting the general validity of the measurement methods. There was a significant negative correlation between experience level and change in similarity to experts from pre- to post training ratings. There was a significant difference in mean Coherence between the experts and novices at the beginning of the week but not at the end of the week. These measurement methods appear to provide a basis for evaluating conceptual change. These assessment methods should prove to be a useful adjunct to performance-based methods of assessing training., Air Force Research Laboratory Technical Monitor: Dr Winston Bennett Jr., 480-988-6561 X-297, DSN 474-6297 This paper was presented at the Interservice/Industry Training Systems and Education Conference, held in Orlando FL, Dec 2001.

Previous research on Distributed Mission Training (DMT) shows that pilots and Airborne Warning and Control System (AWACS) weapons directors rate DMT as highly effective for training multiship, multibogey air combat. DMT exercises were also used as opportunities for pilots participating in Flight Lead Upgrade (FLUG) training to gain experience in planning, briefing, leading, and debriefing 4-ship missions in an intensive air-to-air threat environment. We describe a four-phase research program at the Air Force Research Laboratory, Warfighter Training Research Division (AFRL/HEA) in Apr 99 to assess the effectiveness of using DMT to augment FLUG training. We reviewed training records at one F-16 base to identify 4-ship FLUG missions that would benefit most from DMT experience and establish baseline rates for sorties repeated due to noneffective upgrading pilot proficiency. A 5-day DMT-FLUG protocol was developed targeting these missions. DMT-FLUG training exercises were conducted over one year. During these exercises, upgrading pilots led several missions of increasing complexity using the AFRL 4-ship DMT testbed in Mesa AZ with AWACS weapons controllers from AFRL s research facility at Brooks AFB TX. We assessed transfer to aircraft training through review of training records and interviews with both upgrading pilots and their instructors. Eight out of 12 upgrading pilots participating in DMT-FLUG successfully completed the program without any repeated missions, one pilot repeated one mission, two transferred out of fighters, and one pilot was still in training. We discuss pilots and AWACS weapons directors performance within DMT and identify mission tasks most appropriate for DMT., Original contains color plates: All DTIC reproductions will be in black and white.

Evaluating the impact of organizational interventions and change requires that systematic and reliable information be available at multiple levels of analysis. However, most organizations intervene and initiate change at one level (e.g., the individual or workgroup in a shop) but measure success in terms of effectiveness and outputs at a more aggregate level (e.g., the division, department, or wing). Research evaluating organizational interventions and change has typically focused on organizational levels, in and of themselves, without an analysis of the linkages between levels or processes and the interdependencies amongst outputs from - one level or process as inputs to other levels and processes. Moreover, these evaluations have, for the most part, failed to quantitatively examine the interaction among the levels or among the key elements within each level. With respect to organizational assessment and process change-related activities, these interdependencies must be clearly defined and quantified if benefits from change are to be examined systematically (Cascio, 1996; Van Maanen, 1979). Furthermore, researchers (e.g., Bennett, Ruck, & Huffcutt, 1992; and Bennett, Yadrick, & Perrin, 1996) have argued that these linkages might play a critical role in moderating the ability to detect change at subsequent levels. This paper describes a preliminary taxonomy for the assessment of change in organizational restructuring efforts. The need for cross-level metrics of success and the potential use of linkages and interdependencies for organizational and process diagnosis and prescription will be described and discussed.

The present paper summarizes a research project focusing on ways to improve the usefulness of organization level outcome measures of unit readiness/ effectiveness. Toward this goal, a measurement approach using unit level outcome measures is presented. The measurement system presented adapts and extends the performance distribution assessment approach proposed by Kane (1986; 1992). We demonstrate that, while originally used with subjective performance judgments, the system is readily adapted to regularly collected unit level outcomes. A vital concern for the Air Force is the maintenance of mission capability and readiness. A crucial mechanism for the maintenance of mission readiness is personnel training. Of tremendous importance to the design, implementation, and revision of training throughout the Air Force, as with any organization, is the ability to evaluate the effectiveness of training interventions. Specifically, the effective evaluation of any training intervention is crucial to informed decision making regarding the intervention. Central to effective training evaluation is the standard or criteria against which the training is evaluated. In addition, the comprehensive evaluation of training interventions mandates the use of multiple criterion measures., Prepared in collaboration with Texas A&M Univ., College Station, TX.

Occupational researchers and analysts continue to search for new ways to meet the needs of the manpower, personnel, and training (MPT) communities they serve. Recent efforts seem to be moving away from task-based occupational analysis toward more global measures of knowledges, skills, and abilities (KSAs) and job content. It is alleged that traditional task-based occupational analysis is too labor intensive, too costly, too cumbersome, and too static to meet the emerging and rapidly changing needs of a business process- and team-based approach to the organization of work activities (Casio, 1995; May, 1996 a-c). More and more, the trend has been toward relying on the judgments and opinions of a few subject matter experts (SMEs) on a handful of broad work dimensions or requirements to make important policy decisions.

This report documents the process and results from a feasibility study and evaluation of computer-based occupational data collection. The study was accomplished the Air Force Occupational Measurement Squadron in collaboration with the Human Resources Directorate of the Armstrong Laboratory and the Royal Australian Air Force. The feasibility study was conducted to assess the operational and logistical problems involved with the development, implementation, and evaluation of computer-based job and occupational data collection methods. This study included an evaluation of the quality and functional equivalence of the computer-based survey data versus traditional paper-and-pencil (scanable) forms, the availability of needed computer equipment in the field, and an assessment of the costs and logistics involved with disk-based mail out surveying. Two Air Force specialties (Information Management and Aircraft Fuel Systems) were included in the study; each AFS was sampled for 1,000 cases plus 50 task difficulty and 50 training emphasis factor ratings. While some problems were encountered in the study, particularly with the task factor rating data collection, final job inventory return rates of computer-disk surveys were quite comparable to paper-and-pencil forms. Analysis of the task information for both AFSs indicated that computer-based survey respondents tended to systematically check and rate more tasks, a result expected from previous research. Analysis of overall group job descriptions revealed few differences in percent members performing (PMP) or relative percent time spent (PTS) rankings. Correlations between paper-and-pencil and computer-based sample group mean vectors were in the high 90's for PMP and PTS factors as well as for the rank ordering of tasks on these factors. Overall, computer-based occupational surveys appear to be comparable, feasible, and economical.

A robust finding in the literature on skill maintenance is that extended skill retention intervals lead to decreased task performance. We investigated three potential moderators of the skill retention interval (SRI) - task performance decrement relationship: (a) degree of initial skill learning, (b) performer aptitude, and (c) degree of task difficulty. Results in eight samples of enlisted U.S. Air Force personnel (total N = 1,544) strongly supported a negative SRI - task performance relationship, but support for moderator effects was mixed. Implications for skill maintenance in actual work settings are discussed.

Data obtained on over 1,500 first-term U.S. Air Force enlisted personnel indicated that work sample administrators' global ratings of work sample performance substantially reflect actual ratee behavior in the work sample, and not potentially biasing factors (e.g., race, gender, amount of recent experience), supporting the "folk wisdom" that work samples are high-fidelity and valid measures of performance.

The present study summarizes research focusing on ways to improve the usefulness of organizational level measures of unit readiness/effectiveness through the evaluation of numerous aircraft maintenance related measures of performance. In addition, a measurement approach using unit level outcome measures is presented, which adapts and extends the performance distribution assessment approach proposed by Kane (1986; 1992). It is demonstrated that, while originally used with subjective performance judgements, the system is readily adapted to regularly collected unit level outcomes. An important characteristic of the measurement system presented a focus on the range of performance observed, which considers the fluctuation or variability in performance as well as the level of performance. In addition, the system incorporates a relativistic scaling of performance information. That is, performance is expressed as a ratio of measured performance to some benchmark distribution'. This benchmark distribution may represent established standards, expected, or previously attained levels of performance. This scaling process serves to express actual performance in terms relative to some previously established range of performance. The representation of performance in distributional form along with relativistic scaling has several important advantages to traditional measurement approaches. It allows for an assessment of the consistency of performance ant the extent to which negatively valued outcomes are avoided, which serves to facilitate the comparison and combination of data across diverse performance measures. The current study presents a demonstration of the proposed measurement system with aircraft maintenance data. Preliminary results indicate that this approach does in fact have the potential to improve the utility of or animation level criterion measures.

An augmented framework for training criteria based on Kirkpatricks' (1959a, 1959b, 1960a, 1960b) model divides training reactions into affective an utility reactions; and learning into posttraining measures of learning, retention, and behavior/skill demonstration. Meta-analysis results among criteria using this framework include the finding of substantial reliabilities across training criteria and reasonable convergence among subdivisions of criteria within a larger level. Utility-type reaction measures were more strongly related to learning or on-the-job performance (transfer) than affective-type reaction measures. Moreover. utility-type reaction measures were stronger correlates of transfer than were measures of immediate or retained learning. These latter findings support recent concurrent thinking regarding use of reactions in training (e.g., Warr & Bunce, 1995). Implications for choosing and developing training criteria are discussed.

Traditionally, the same type of task analysis procedures has been used for all types of jobs. Recent changes in the workplace include the increased focus on the cognitive demands of tasks and the increased use of work teams to accomplish these tasks. Researchers have discussed the implications of these trends for task analysis procedures in organizations today. Cascio (1995) noted an organizational change away from task-based work to more of a process. Jobs are no longer defined by a limited number of tasks, requiring the employee to accomplish a range of tasks that may change over time. As a higher proportion of jobs are focusing on troubleshooting activities, cognitive task analysis may be more appropriate for identifying strategies involved in effective performance.

This report provides a brief overview of three methods, or general approaches, to judging the usefulness of training. The first method or approach is the traditional evaluation approach of training effectiveness evaluation. This method or approach centers on estimations of training effect size and the determination of the statistical significance of those training effects. Thus, in this first category standard pre(post analyses and control group comparisons are included. Next traditional training utility analysis, or training utility evaluation is reviewed here, costs and benefits of training are always contrasted in some way in determining traditional utility analysis. This branch of evaluation traces its roots back to Brogden & Taylor (1950), who discussed the need to examine the dollar criterion. Finally, anticipatory training evaluation is discussed Anticipatory training evaluation examines what kinds of training will have the greatest effectiveness and utility, given a variety of parameters and choices. The primary tool in anticipatory evaluation is Multi-Attribute Utility analysis (MAU). As opposed to training effectiveness and training utility evaluation, MAU is designed explicitly as a decision tool. It can be used most effectively as an anticipatory evaluation, the results of which facilitates planning for training. A detailed example of the development and application of MAU is described in this report, since it is the least well known of the three approaches to researchers and practitioners in the training are. In explicating this expanded view of training evaluation, this report attempts to represent state-of-the-art understanding and research; thus current issues like risk and uncertainty in input and output evaluation and utility indices, and problems in transfer of learned skills to the job, are addressed in appropriate contexts.

Estimates of the cost of providing training, in forms ranging from classroom instruction to on-the-job training, are needed to support decisions about who gets trained, when, where, and on what skills. To counter the myriad of uncontrollable factors that may obscure the relationship between manpower, personnel, and training policy changes and organizational outcomes, an organizational simulation of Air Force occupations called the Training Impact Decisions System (TIDES) was developed. An important first step in obtaining these occupation-level outcome estimates in TIDES is to identify groups of tasks with similar knowledge and skill requirements, because economies will be realized when these tasks are trained at the same time. This report compares results from using two different methods to identify groups of Air Force Occupational Survey tasks where these training economies would occur, including methods based on subject matter experts' judgments and statistical clustering using task coperformance. The results from two field applications indicated that the statistical methods could replicate much of the structure of the experts' clusters, and so, could be used to facilitate the process of identifying these task groups. Use of these methods to form task clusters which could be used to support a broad range of training and personnel decisions is also discussed.

Using Campbell's (1990) theory of job performance, the present study sought to identify the factors underlying the job performance of international assignees and their relative importance. Four alternative models of international assignee job performance were also evaluated. Subjects were 338 international assignees from diverse countries (nationality) and organizations, assigned to diverse countries, and performing diverse jobs. Although eight factors were identified as hypothesized, the content and labels of these factors differed from those postulated by the theory. The eight job performance factors in descending order of importance were Flexibility, Family Situation, Management/Administration, Integrity, Effort, Tolerance, Cross-Cultural Interest, and Openness. Factor importance ratings were also influenced by position (managerial/non-managerial status), organizational type, international assignees' nationality, the country to which they were assigned, and cultural distance (similarity between nationality and assigned country). Finally, although military personnel were excluded from this study since they typically are assigned to places where their culture is present in some form (e.g., they are assigned to a U.S. military base), the factors identified as being important for international assignees, in general, should also be considered as part of the assignment process and in job design and redesign for military personnel in foreign postings. The importance of these factors should also be considered when developing the cultural training and education programs for military personnel prior to their assignment within non-US host countries. The likelihood that these factors may serve as predictors of the success of a military member in a foreign assignment in a manner similar to that of non-military personnel must be considered and addressed.

This report documents recent research and application of a decision support system (DSS) developed by the Armstrong Laboratory, Human Resources Directorate. This system is the Training Impact Decision System (TIDES). The TIDES is a computer-based DSS specifically designed to support key planners and decision makers (functional managers at the Air Staff or Major Command, training managers, etc.) for a career specialty (occupation). The TIDES was developed to extend task-based job analysis to permit more systematic consideration of personnel utilization factors, training costs, resource requirements, training capacities of units (base-level organizations) and managers preferences in determining the optimal allocation of personnel and training resources., Prepared in cooperation with Metrica Inc., San Antonio, TX.

Historically, some of the most significant research and development (R&D) in occupational analysis (OA) has been accomplished by military organizations for military applications. Several of the most important OA methodologies were developed under military sponsorship. Examples include functional job analysis, worker-oriented approaches (the Position Analysis Questionnaire, the General Work Inventory, etc.), and the Task Inventory/Comprehensive Occupational Data Analysis Programs (CODAP) approach. Significant advances are continuing to be made in occupational analysis for military applications. Many military organizations are major users of these technologies. The major methods developed for military applications are used in the civilian world as well, in both the public and private sectors. This report highlights recent research advances and applications of military OA technology. It is designed to bring military psychologists up to date concerning OA technology for military applications and the impact that military occupational analysis is having on civilian practice. First, the history of occupational analysis R&D and application in military organizations is reviewed, to establish the context for ensuing chapters covering current military OA R&D results and applications. The various chapters address OA in both U. S. and allied nations' service organizations. In addition, civilian applications of military OA technology are discussed. Finally, future directions for OA are highlighted. The potential use of OA methods for modeling organizational structure, and for informing organizational interventions (e.g., job enlargement, training, technostructural change), is also discussed. The final chapter includes a proposed research agenda for future military occupational analysis R&D based on comments received from symposium discussants., Prepared in collaboration with The Institute for Job and Occupational Analysis, San Antonio, TX.

With the increased popularity of meta-analysis, a number of articles have questioned and highlighted the role that judgment calls play in the implementation and, ultimately, outcomes of studies that use this technique. In the absence of standardized data analyses packages, coupled with the wide range of options available to meta-analysts, the current study investigated the effect of choice of data analysis programs on meta-analysis study outcomes. The objective of this Monte Carlo study was to investigate the extent to which four commonly used Schmidt and Hunter validity generalization-based meta-analysis software programs, all based on the same conceptual and theoretical assumptions, produced identical outcomes when used to analyze the same dataset. The results indicate that while there were some differences in values obtained from the programs, these differences tended to be very small, typically occurring in the fourth and sometimes fifth decimal place, and did not influence the meta- analytic outcomes. Finally, differences in the features and capabilities of each of the programs are presented and discussed., Prepared in collaboration with Texas A and M Univ., College Station, and Bradley Univ., Peoria, IL.