Search

The paper proposes that industrial-organizational (I-O) psychology will benefit greatly from expanding our research focus from predominantly individual differences to studying organizational differences. We argue here that an increased organizational frame of reference on variables of interest to I-O psychology (e.g., selection, job design, performance management (PM), work motivation) is important because it will enhance our understanding of organizational behavior and make I-O research more effective in practice. After noting some organizational-level research already being done, several examples are provided for how an organizational mindset and methods can provide new insights into traditional areas of I-O effort. Also discussed is how methodological issues that may have constrained the study of organizational differences in the past and the potential new issues such research may yield can be addressed. We conclude that the future maintenance and enhancement of the I-O psychology brand as a science–practice profession requires enhanced attention to the organization level of analysis as our frame of reference for research.

This article reviews the history of performance management (PM), beginning with performance evaluation. We discuss various strategies that have been used to enable accurate ratings as well as cognitive processes and contextual factors that have been shown to significantly impact ratings. We raise questions about the concept of true performance and whether raters can be enabled and motivated to make accurate ratings. We progress to discussing more structured and comprehensive PM processes that typically involve cascading goals, goal setting, competency modeling, evaluation of behavior and results, and implementation. These systems have proven to be tedious and low-value, resulting in unprecedented, wide-spread experimentation with innovative practices to move companies away from heavy PM processes to simpler, cost-effective strategies that actually drive performance. These have ranged from abandoning ratings to implementing innovations in goal-setting, real-time feedback, coaching, and PM behavior change. Directions for future research and practice are discussed.

This chapter focuses on implications of the changing work environment for performance management (PM) design and practice. The chapter begins by discussing how technology has created disruptive change and hypercompetition, which have left organizations searching for how to create competitive advantage, concluding that agility is essential for their long-term success. Given the limited research in this area, a large-scale research program was undertaken (as described in the first part of this chapter) to provide an evidence-based definition of agility and to understand what organizational conditions lead to it and which outcomes result from it. The research showed, remarkably, that companies with high organizational agility delivered up to five times higher financial performance than others. It was learned that six organizational conditions enabled agility, two of which were team-level factors that are particularly relevant for PM. These were interconnected performance and real-time performance drivers (i.e., agile goals, real-time feedback, and solving performance problems). In the second part of this chapter, implications of these factors for PM design and practice are discussed.

This chapter points out that 360 Feedback processes are often asked to do too many things at once. It examines the implications for using 360 Feedback as a predictor versus using it as a criterion measure to support various strategic talent decisions in organizations. Because 360s are quite flexible as a methodology, the chapter emphasizes the importance of proactively defining a clear purpose for the process and illustrates the critical link between purpose and design of the process. The chapter explores specific ways in which 360 processes should differ when being used as a predictor or as a criterion. Finally, the chapter describes design considerations that apply to both uses of 360 Feedback and articulates specific guardrails that will impact the quality and validity of any 360 program.

Despite years of research and practice, dissatisfaction with performance appraisal is at an all-time high. Organizations are contemplating changes to their performance management systems, the most controversial of which is whether to eliminate performance ratings. The pros and cons of retaining performance ratings were the subject of a lively, standing-room-only debate at the 2015 Society for Industrial and Organizational Psychology conference in Philadelphia (Adler, 2015). Given the high interest in this topic, this article recaps the points made by the panelists who participated in the debate. The arguments for eliminating ratings include these: (a) the disappointing interventions, (b) the disagreement when multiple raters evaluate the same performance, (c) the failure to develop adequate criteria for evaluating ratings, (d) the weak relationship between the performance of ratees and the ratings they receive, (e) the conflicting purposes of performance ratings in organizations, (f) the inconsistent effects of performance feedback on subsequent performance, and (g) the weak relationship between performance rating research and practice in organizations. The arguments for retaining ratings include (a) the recognition that changing the rating process is likely to have minimal effect on the performance management process as a whole, (b) performance is always evaluated in some manner, (c) “too hard” is no excuse for industrial–organizational (I-O) psychology, (d) ratings and differentiated evaluations have many merits for improving organizations, (e) artificial tradeoffs are driving organizations to inappropriately abandon ratings, (f) the alternatives to ratings may be worse, and (g) the better questions are these: How could performance ratings be improved, and are we conducting the entire performance management process properly? The article closes with questions organizational members have found useful for driving effective performance management reform.

Recently a revolution has taken place in organizations around the world to transform their performance management systems from burdensome chores into a valuable business practices. Many high-profile companies have announced they are getting rid of the dreaded performance reviews and replacing them with ongoing coaching and feedback. Although these cases are inspiring other organizations to contemplate change, many are left with more questions than answers. While many fads and quick fixes have been proposed to answer these questions, little research exists to support them. This book provides a practical and evidence-based guide for building a performance management approach that actually improves performance. It cuts through the hype and gives actionable advice, useful tools, and real-world examples for organizations to build the business case for change, plan the transformation, design the new system, and implement the change effectively. Featuring research findings as well as concrete strategies from organizations that have proven successful, this book provides a roadmap for meaningful change. It will be of interest to professionals and scholars interested in evidence-based performance management and the challenges facing organizations.

In spite of numerous attempts over decades to improve performance management (PM) systems, PM is viewed as more broken than ever, with managers and employees seeing it as a burdensome activity that is of little value. Yet, the behaviors that PM is meant to achieve are in fact important drivers of engagement and performance. So where is the disconnect? The problem is that formal PM systems have reduced PM to intermittent steps and processes that are disconnected from day-to-day work and behaviors that actually drive performance: communicating ongoing expectations, providing informal feedback in real time, and developing employees through experience. To deliver on its promise, PM needs to shift from focusing on the formal system to focusing on the PM behaviors that matter every day. We describe a 5-step PM reform process that helps organizations achieve this change and that shows promise for increasing satisfaction and positive outcomes from PM processes. Central to the intervention is that organizational members need to intentionally practice and solidify effective PM behavior through a structured, on-the-job, experiential learning intervention that yields meaningful behavior change. The change-management and training interventions discussed here provide a model for organizational culture and behavior change efforts beyond PM.

This chapter reviews the debate that has surrounded measuring workplace behavior versus results and discusses why the measurement of results has become increasingly popular. It proposes methods for developing results measures for individuals and teams and the associated challenges. The chapter also reviews the concept of cascading goals and provides guidelines for developing individual and team objectives, which are thought to be an important precursor to achieving organizationally relevant results. It discusses evaluation methods that should facilitate accurate and fair measurement of the results employees achieve, using a combination of objective and subjective measures. The chapter also discusses individual difference constructs that are likely to predict performance results. Measuring results relies on identifying performance objectives that state the outcomes an employee is expected to achieve in sufficient, measurable detail such that it is clear whether or not the objectives have been met.

Although extensive research and practice have focused on understanding and improving performance management systems in organizations, the formula for effective performance management remains elusive. We propose that a significant part of the problem is that performance management has been reduced to prescribed steps within formal administrative systems that are disconnected from the day-to-day activities that determine performance management effectiveness (e.g., communicating clear work expectations, setting short-term objectives and deadlines, and providing continual guidance). We argue that interventions to improve performance management should cease their exclusive focus on reinventing formal system features. Although well-developed tools and systems can facilitate performance management, these alone do not yield effective performance management. In lieu of making further changes to formal performance management systems, we argue for devoting more attention to improving manager–employee communication and aspects of the manager–employee relationship and propose an approach we believe holds promise for improving performance management processes in organizations.

In our recent focal article, we proposed that a fundamental change is needed in how performance management is implemented and viewed. Rather than attempting to improve the effectiveness of performance management by changing features of the formal system (e.g., steps, tools, and processes), we recommended increased focus on improving manager–employee communication and aspects of the manager–employee relationship that are foundational for effective performance management. Our article generated a number of thoughtful and engaging commentaries. In this commentary, we respond to several themes raised in the commentaries and relate them to the recommendations we made for improving performance management processes in organizations.

In our recent focal article, we proposed that a fundamental change is needed in how performance management is implemented and viewed. Rather than attempting to improve the effectiveness of performance management by changing features of the formal system (e.g., steps, tools, and processes), we recommended increased focus on improving manager–employee communication and aspects of the manager–employee relationship that are foundational for effective performance management. Our article generated a number of thoughtful and engaging commentaries. In this commentary, we respond to several themes raised in the commentaries and relate them to the recommendations we made for improving performance management processes in organizations.

This research further investigates an 8-dimension taxonomy of adaptive performance developed by Pulakos, Arad, Donovan, and Plamondon (2000) and the usefulness of past experience, interest, and self-efficacy predictors developed to measure these same 8 dimensions for predicting adaptive job performance. Participants in the concurrent, criterion-related validation study included 739 military personnel. They completed the 3 adaptability measures as well as more traditional cognitive and noncognitive predictors. Supervisors of the study participants rated their adaptive job performance. Results showed support for the 8-dimension model of adaptability. In addition, cognitive ability, personality, and the new adaptability predictors were shown to predict adaptive performance, with some past experience items adding incremental validity beyond the more traditional cognitive ability and personality measures. Results are discussed.

The purpose of this research was to develop a taxonomy of adaptive job performance and examine the implications of this taxonomy for understanding, predicting, and training adaptive behavior in work settings. Two studies were conducted to address this issue. In Study 1, over 1,000 critical incidents from 21 different jobs were content analyzed to identify an 8-dimension taxonomy of adaptive performance. Study 2 reports the development and administration of an instrument, the Job Adaptability Inventory, that was used to empirically examine the proposed taxonomy in 24 different jobs. Exploratory factor analyses using data from 1,619 respondents supported the proposed 8-dimension taxonomy from Study 1. Subsequent confirmatory factor analyses on the remainder of the sample (n = 1,715) indicated a good fit for the 8-factor model. Results and implications are discussed.

A procedure for developing alternate test forms that are parallel in the sense that scores on the different forms have similar means, standard deviations, and factor structures is described and applied to a bio-data inventory and a situational judgment test. Careful consideration of item-by-item parallelism during development resulted in alternate forms that were parallel at the item level. Further, comparison with a biodata test form comprised of items randomly selected from a pool of biodata items revealed that for the types of measures described here it may be necessary to produce parallel forms of each item to create alternate forms that are parallel in the way in which Cronbach (1947) originally defined parallelism.

Previous research on the effects of bias in criterion-related validation research has typically involved the use of statistical corrections for halo, leniency, and/or central tendency. We present arguments that likability and similarity of raters to ratees may constitute a form of predictor-related criterion bias. One cannot investigate this form of bias without clear understanding of method, predictor, and criterion constructs and careful direct measurement of each. Measurement and theorizing of method constructs is rarely, if ever, undertaken in criterion-related validation work. The results of a criterion-related validation of the use of quantitative and verbal ability and interview and role-play simulations to predict the performance of 372 federal investigative agents are reported. Using the all-Y LISREL model (Williams & Anderson, 1994), we found that likability and similarity factors were related to interview and role play measures. However, none of these potential "biases" affected both predictor and criterion constructs, hence there was no effect on the estimates of the relationships between the predictors and criteria. Limitations with respect to the generalizability of these results to criterion-related research in which performance data are not as carefully collected as well as advantages and disadvantages of more traditional regression and correlational analyses are discussed.

This research explores two strategies for reducing adverse impact and their effects on criterion-related validity. The first strategy involved assessing a broad range of both cognitive and noncognitive abilities that were identified via a thorough job analysis using a Biodata Measure, Situational Judgment Test, Structured Interview, and Verbal Ability Test. The second strategy involved developing alternative measures of verbal ability that more closely mirrored the behavioral requirements of the job than traditional multiple-choice measures of this construct. Specifically, the validity and subgroup differences associated with a video-administered writing test and a written problem exercise were compared to a traditional multiple-choice verbal ability measure. Predictor and criterion data were collected from 456 participants (259 White, 100 African American, and 97 Hispanic). Although both strategies resulted in reductions in subgroup differences, somewhat larger reductions were observed for the broad-base...

This research extends the work of Hunter (1983) and Borman, White, Pulakos, and Oppler (1991) on models of supervisory performance ratings. A rating model that included measures of cognitive ability, practical intelligence, job knowledge, task proficiency, achievement orientation, and performance ratings was proposed and evaluated. A major purpose of this research was to evaluate the fit of the proposed model for different race and gender subgroups as well as for peer and supervisor raters. The models were tested on a sample of 456 professionals in a large government agency. Overall, results of the LISREL analyses showed no statistically significant differences in the fit of the model for the different race or gender subgroups examined. Significant differences in the fit of the model and variance accounted for in the peer versus supervisor groups were observed.

This research investigated empirical issues regarding the validity of individual interviewers’(N= 62) ratings collected after a structured interview. Each interviewer rated an average of 25 interviewees. One hypothesis examined but not supported was that systematic interviewer errors will attenuate interview validity when data are aggregated across interviewers. Also investigated was the validity of ratings averaged across interviewers compared to consensus ratings; consensus ratings were shown to have significantly but probably not practically higher validities. Third, a meta-analysis of individual interviewer validities revealed that all of the variance in validities could be attributed to sampling error. Results and implications are discussed.

This research compared the validity of two different types of structured interview questions (i.e., experience-based and situational) under tightly controlled conditions. The experience-based interview questions required that 108 study participants relate how they had handled situations in the past requiring skills and abilities necessary for effective performance on the job. Situational questions, administered to another group of 108 study participants, provided interviewees with hypothetical job-relevant situations and asked them how they would respond if they were confronted with these problems. The experience-based interview questions yielded higher levels of validity than the situational questions. Additional analyses showed that the interview added incrementally to the prediction of performance beyond the variance accounted for by a cognitive ability test. There were small differences in subgroup performance (White, Black, Hispanic, male, and female) on the experience-based interview, though it was equally valid for all subgroups.

Use of a structural equations analysis to estimate the degree of predictor-related criterion bias in selection research is illustrated. For the hypothetical data considered in this paper, a likability methods factor produced significant changes in estimates of the structural parameters relating the predictors to the criterion. Regression analyses using two methods factors as covariates yielded similar results. The advantages of employing the more complicated LISREL analysis versus more familiar regression analysis are discussed and some suggestions for future research using the structural equations approach are presented.

In this chapter, we examine issues relevant to incorporating trainability and adaptive performance into selection research. We adopt the definition of adaptive performance suggested by Pulakos et al. (2000) that specified eight dimensions defining this construct. One of these dimensions, leaning new tasks, technology, and procedures, was used to define trainability. We then examine recent models of adaptive performance and training to identify likely predictors of adaptability and trainability and propose a method for determining when and where these criteria should be included and explicitly predicted in selection research. We examine the pros and cons associated with different criterion measures and recommend that typical rating measures potentially supplemented by lower-fidelity work sample measures be incorporated in selection research. Finally, we discuss gaps in the current literature and recommend areas for future research.

This research explores the role of early supervisory experience and cognitive ability in first-line supervisor performance. Similar to F. L. Schmidt, J.E. Hunter, and A. N. Outerbridge's (1986) study of nonsupervisors, this research tested structural models hypothesizing relationships among supervisory experience, cognitive ability, supervisory knowledge and proficiency, and performance ratings, using a sample of 570 second-tour soldiers. The Schmidt et al. model with an additional ability → experience path provided the best fit. The significant ability → experience path was interpreted as indicating that demonstrated ability contributes to soldiers being given the opportunity to obtain supervisory experience. Experience had a greater impact on supervisor proficiency than on supervisor knowledge. Ability had a greater impact on supervisor knowledge than on proficiency

Three methods of assessing subgroup bias in performance measurement commonly found in the literature are identified. After a review of these approaches, findings are reported from analyses of data collected in the United States Army's Project A (J. P. Campbell, 1987). Correlations between nonrating performance measures and supervisor ratings were generally not mederated by race, but correlations between nonrating indicators of negative performance and ratings assigned by peers were. In addition, significant interactions between rater and ratee race on performance ratings were not eliminated when variance in the nonrating measures was removed from the ratings provided by Black and White raters. Conclusions about the magnitude and nature of bias in supervisor and peer ratings are discussed

Proposed and evaluated in this research were causal models that included measures of cognitive ability, job knowledge, task proficiency, two temperament constructs (achievement and dependability), awards, problem behavior, and supervisor ratings. The models were tested on a sample of 4362 US Army enlisted personel in nine different jobs.