Search

Evaluation of aircrew performance serves the critical functions of assessing the qualifications of individual pilots and, under newer proficiency-based training programs, providing data for modifying training programs. We apply psychometric methods to assessing and improving the quality of evaluation of aircrew performance. Quality evaluations require human judges to recognize and discriminate changes in performance levels (sensitivity) and map these observations onto the appropriate grade-scale values (accuracy). We define statistical measures for both of these properties. A distinction is made between referent-rater reliability (RRR) and traditional interrater reliability, and we argue that RRR more meaningfully measures evaluators' grading performance and has clearer training implications. We also discuss the implementation of training and calibration sessions that are intended to help improve evaluators' ratings of aircrew performance. We offer several practical guidelines for designing and conducting these sessions. [ABSTRACT FROM AUTHOR]

Two instructors rated the crew performance of helicopter crews (N = 45) who flew a simulated mission in a full-motion simulator. The instructors were given dimension training and behavior-observation training before the ratings. Before making their ratings, instructors completed an observation form as they watched videotapes of the crews and then completed a form that helped link behaviors to dimensions. For their ratings, instructors recorded or evaluated crew behaviors using 3 types of items: (a) specific crew behaviors in response to scenario events (e.g., whether the crew kept out of icing conditions), (b) evaluations of crew responses to scenario events (e.g., overall handling of the icing problem), and (c) crew resource management (CRM) dimensions for the entire scenario (e.g., evaluations of decision making). Results showed (a) both interjudge agreement and internal consistency were high for evaluations of crew responses to scenario events, (b) interjudge agreement was low but internal consistency was high on CRM items and scales, and (c) interjudge agreement was high but internal consistency was low for specific observable behaviors. The results for the evaluations of crew responses to scenario events were very encouraging and showed reliability over time and over crews. Suggestions for improving the reliability for the other 2 item types are provided. [ABSTRACT FROM AUTHOR]

The assessment of pilot performance relies on systematic observation and assessment by a single trained rater or instructor/evaluator (I/E). Because of the importance of aviation safety, it is imperative that the rating and evaluation processes used by these I/Es yield reliable and valid data. This article describes a case study focused on improving the reliability and validity of crew assessment at a commercial carrier. The process for improving reliability involved the evaluation of current training methods, the construction and evaluation of 5 metrics for assessing interrater reliability, and a standardized process for using these metrics to train I/Es. A separate set of interventions was developed and implemented to improve structural validity. Data collected from 2 fleets in this airline during a 3-year period indicate aspects of reliability and validity that should be the focal points of continuing I/E training. [ABSTRACT FROM AUTHOR]

As the workforce increasingly ages, it will be critical to find ways of accommodating the decrements in abilities that come with age. Finding accommodations will be of particular concern for occupations such as airline pilot, in which small errors may result in fatal accidents. This study examined the perceived performance of older pilots, if information on ergonomic redesign interventions had a significant effect on attitudes toward ergonomic redesign interventions for older pilots, and how perceived performance and type of information interacted to affect attitudes toward ergonomic redesign interventions for older pilots. Participants in the study included 147 flight instructors and 24 student pilots. Results and implications for future research are discussed. [ABSTRACT FROM AUTHOR]