Search

This paper assesses the reliability and validity of cognitive and socioemotional skills measures and investigates the correlation between schooling, skills acquisition, and labor earnings. The primary data from Pakistan incorporates two innovations related to measurement and sampling. On measurement, the paper develops and implements a battery of instruments intended to capture cognitive and socioemotional skills among young adults. On sampling, the paper uses a panel that follows respondents from their original rural locations in 2003 to their residences in 2018, a period over which 38 percent of the respondents left their native villages. In terms of their validity and reliability, our skills measures compare favorably to previous measurement attempts in low- and middle-income countries. The following are documented in the data: (a) more years of schooling are correlated with higher cognitive and socioemotional skills; (b) labor earnings are correlated with cognitive and socioemotional skills as well as years of schooling; and (c) the earnings-skills correlations depend on respondents' migration status. The magnitudes of the correlations between schooling and skills on the one hand and earnings and skills on the other are consistent with a widespread concern that such skills are under-produced in the schooling system. [This report was prepared by the World Bank Group's Development Research Group, Development Economics. Funding was provided by RISE and World Bank's Strategic Research Program Fund.]

The author discusses valid and reliable ways to assign grades in an academic course in any discipline. "Validity" means the accuracy of a grade's reflection of student learning and achievement. "Reliability" concerns the degree of random error that might be present and affect validity. First the author defines a grade as a measure of achievement, then identifies problems with existing grading practices and outlines principles that lead to valid grading. Grading criteria, which should be carefully and thoughtfully selected, are reviewed and categorized as "highly recommended," "controversial," and "inappropriate." Whichever grading method instructors choose should accurately measure their students' achievement, because a grade represents a contract between instructor and student that accurately reflects a particular level of learning. Finally, the author discusses grade inflation.

Correlation analysis has been used widely by researchers and analysts when analyzing large-scale assessment data. Limit research provided reliable methods to estimate various correlations and their standard errors with the complex sampling design and multiple plausible values taken into account. This report introduces the methodology used by the wCorr R package (Emad & Bailey, 2017) for computing the Pearson, Spearman, polyserial, and polychoric correlations, with and without weights applied. The methodology treats tetrachoric correlation as a specific case of the polychoric correlation and biserial correlation as a specific case of the polyserial correlation. Simulation evidence is presented to show correctness of the methods, including an examination of the bias and consistency.

Data from student ratings of instruction (SRI) are used ubiquitously as a key element in providing instructors with valuable feedback and evaluators with critical student input. Nonetheless, calls for the elimination of SRI continue to appear in academic journals and higher education periodicals. This paper brings to bear the huge body of research on SRI to which so many academics and institutions have contributed. Some of the most egregiously erroneous statements about SRI are rebutted with brief reviews of the readily available compelling evidence. Although some faculty frustrations about misuse of SRI are valid, we argue that inclusion of student voice is critical. Students can provide useful feedback because they have firsthand experience over multiple occasions of what actually occurred in the classroom. Recommendations are made for best practices in using SRI as one of many sources for improving and evaluating teaching.

Initiatives to increase leadership accountability coupled with efforts to promote data-driven leadership have led to widespread adoption of instruments to assess school leaders. In this paper we present a decision matrix that practitioners and researchers can use to facilitate instrument selection. Our decision matrix focuses on the psychometric properties of the instrument; the model of leadership used to construct the instrument; the feasibility of implementation; and the extent to which feedback lends itself to changes in behavior (actionability). We apply this decision matrix to two of the most prominent leadership feedback instruments, CALL and VAL-ED. We find substantial similarities between the instruments, yet each has unique strengths and drawbacks. We discuss the implications of the comparative strengths and drawbacks of the two instruments and how they might be most effectively deployed to improve leadership practices.

Teacher and principal evaluation systems now emerging in response to federal, state and/or local policy initiatives typically require that a component of teacher evaluation be based on multiple performance metrics, which must be combined to produce summative ratings of teacher effectiveness. Districts have utilized three common approaches to combine these multiple performance measures, all of which introduce bias and/or additional prediction error that was not present in the performance measures originally. This paper investigates whether the bias and error introduced by these approaches erodes the ability of evaluation systems to reliably identify high- and low-performing teachers. The analysis compares the expected differences in long-term teacher value-added among teachers identified as high- or low-performing under these three approaches, using simulated data based on estimated inter-correlations and reliability of measures in the Gates Foundation's Measures of Effective Teaching project. Based on the results of our simulation exercise presented here, we conclude these approaches can undermine the evaluation system's objectives in some contexts. Depending on the way these performance measures are actually combined to categorize teacher performance, the additional error and bias can be large enough to undermine the district's objectives. An appendix presents: (1) Correlation Matrix; and (2) Estimated Coefficients under the Error-Minimizing Approach.

For many teachers, the classroom observation has been the only opportunity to receive direct feedback from another school professional. As such, it is an indispensable part of every teacher evaluation system. Yet it also requires a major time commitment from teachers, principals, and peer observers. To justify the investment of time and resources, a classroom observation should be both accurate and reliable. In this paper, the authors evaluate the accuracy and reliability of school personnel in performing classroom observations. The authors also examine different combinations of observers and lessons observed that produce reliability of 0.65 or above when using school personnel. They asked principals and peers in Hillsborough County, Florida, to watch and score videos of classroom teaching for 67 teacher-volunteers using videos of lessons captured during the 2011-12 school year. Each of 129 observers provided 24 scores on lessons provided to them, yielding more than 3,000 video scores for this analysis. The authors briefly summarize seven key findings: (1) Observers rarely used the top or bottom categories ("unsatisfactory" and "advanced") on the four-point observation instrument; (2) Compared to peer raters, administrators differentiated more among teachers; (3) Administrators rated their own teachers 0.1 points higher than administrators from other schools and 0.2 points higher than peers; (4) Although administrators scored their own teachers higher, their rankings were similar to the rankings produced by others outside their schools; (5) Allowing teachers to choose their own videos generated higher average scores. However, the relative ranking of teachers was preserved whether videos were chosen or not; (6) When an observer formed a positive (or negative) impression of a teacher in the first several videos, that impression tended to linger; and (7) There are a number of different ways to ensure reliability of 0.65 or above. The authors conclude by discussing the implications for the design of teacher evaluation systems in practice. (Contains 7 figures, 10 tables, and 21 footnotes.)

This paper reports on a study of the current state of the art in teaching assessment. The major goal of the study was to examine a sample of assessment systems and then develop a specification for a state-of the art performance assessment system to be used for human capital management functions. The authors hope was that this specification would (a) provide a guidepost for working on a coherent instructional vision and devising methods to assess the alignment of actual instruction with that vision and (b) help states and districts think about how they want to develop their own teaching competency model and what assessment approaches fit best with different uses of this model. To that end, the paper concludes with an initial specification for a high-quality, multiuse assessment system and a preliminary road map for developing such a system. To inform their thinking, they began by reviewing several prominent assessment systems. The eight systems selected for this study are: (1) The "PRAXIS III" teacher performance assessment, developed by the Educational Testing Service for use in teacher licensure; (2) The Performance Assessment for California Teachers ("PACT"), developed by a consortium of California teacher training institutions, led by Stanford University, and used for initial teacher licensure; (3) The Formative Assessment System Continuum of Teacher Development ("FAS Continuum"), developed by the New Teacher Center at the University of California, Santa Cruz; (4) The Framework for Teaching, as originally developed by Charlotte Danielson ("original FFT"); (5) The version of the FFT adapted for teacher evaluation by the Cincinnati Public Schools ("Cincinnati TES"); (6) The teacher evaluation process used by the National Institute for Excellence in Teaching's Teacher Advancement Program ("TAP"); (7) The assessment system developed for certification by the National Board for Professional Teaching Standards ("NBPTS"); and (8) The Classroom Assessment Scoring System ("CLASS") developed by Robert Pianta and his colleagues at the Center for Advanced Study of Teaching and Learning at the University of Virginia. Based on their review of the competencies covered by the eight systems, their assessment procedures, and the evidence for the reliability and validity of assessment scores, they propose a preliminary set of specifications that states or districts may want to consider when deciding what approach to take to their own assessment of teaching. Developing a teaching assessment for use in a strategic human capital management system is a complex undertaking. The authors outline a process that they believe can help districts and states think about how to get started and what resources are needed. Using Value-Added and Instructional Practice Measures Together for Human Capital Management Decisions is appended. (Contains 7 tables, 1 figure and 10 footnotes.)

In this paper we report on work estimating the stability of value-added estimates of teacher effects, an important area of investigation given public interest in workforce policies that implicitly assume effectiveness is a stable attribute within teachers. The results strongly reject the hypothesis that teacher performance is completely stable within teachers over long periods of time, but estimates suggest that a component of performance appears to persist within teachers, even over a ten-year panel. We also find that little of the changes in teacher effectiveness estimates within teachers can be explained by observable characteristics. The Influence of Bias is appended. (Contains 6 tables, 4 figures and 41 footnotes.)

For Americans to engage in a global information society, it is critical that they have access to high-speed, high-bandwidth Internet, meaning broadband. Network connectivity opens up a wealth of possibilities to K-12 educators. While it has the potential to result in fundamental changes in teaching methods, it can definitely be used to enhance already effective teaching methods. However, schools must have confidence in their network infrastructure before network connectivity will be integrated into the classroom. Networks must be reliable and quick; and, if they do not function, as expected and technical support is not readily available, then educators will not use them. Schools face challenges to the integration of broadband in teaching and learning in the areas of access, skills, policy, and motivation. Last mile support is essential and all stakeholders must work together to address the main issues facing the improvement of broadband in schools. A solution to these situations is on-site support to streamline and enable effective use of broadband-enabled technology. For most schools, the school librarian, in their roles as school leaders, can expertly direct this "last mile" implementation by providing the technology coordination, support, and leadership necessary to address access issues from desktop to district. (Contains 2 tables and 4 figures.) [This report was produced by the Florida State University PALM (Partnerships Advancing Library Media) Center. Support for this paper was provided by Gale Cengage Learning.]

This paper attempts to summarize the conclusions of the major reviews of the literature on student ratings of teaching. It is an update of a paper by the same name published as IDEA Paper No. 20 from the Center for Faculty Evaluation and Development in 1988. Viewing student ratings as data rather than evaluations may help to put them in proper perspective. Studies have considered the multidimensionality of student ratings and their reliability and validity. They have been compared to student learning outcomes, the self-ratings of the instructor, and the ratings of others, and possible sources of bias have been studied. There are probably more studies of student ratings than of all the other data used to evaluate college teaching combined, and there are certainly enough studies to allow some conclusions. In general, student ratings tend to be statistically reliable, valid, and relatively free from bias and need for control. Nevertheless, they are only one source of data about teaching and must be used with multiple sources of data to make judgments about all the components of teaching. (Contains 67 references.) (SLD)

This report summarizes efforts by the state of New York to ensure the accuracy and integrity of data reported by school districts. The Office of the State Comptroller (OSC) has taken the lead in reviewing these data, focusing on the uses of attendance and enrollment data and making recommendations for improvement in state systems and local practices. Recent investigative reports described serious problems with educational data reported by the New York City school system, including incidents of fraud and abuse. These reports support findings in OSC audits and reinforce the need to improve procedures and internal controls for school-district data. School district data submissions provide the information used in the state's school district "report cards" and other statewide reports that evaluate school performance and provide accountability. The lack of adequate controls of these data is important as a matter of accountability, but also in regard to compulsory attendance laws. The report provides a summary of two audits and a followup review of district data, along with recommended changes in procedures. The recommendations encompass a broad range of data-integrity issues and provide a comprehensive framework for improvement. An appendix contains information on the impact of attendance and enrollment on state-aid payments. (RJM)

Data from the Work History and Attitudes survey of the Social Change and Economic Life research initiative (SCELI) enquiry of 1986-1989 and the first wave of the British Household Panel Survey (BHPS) were analyzed to determine their continuity and comparability with regard to employee attitudes in general and job satisfaction and work centrality in particular. The study revealed a high degree of comparability between the two data sets and in the results obtained when they are used. However, several difficulties in comparing the two data sets were also identified. The SCELI was credited with providing good indicators of both human capital ("own-skill") and job skill and for providing extensive details about the technical milieu of work and the work organization, which were deemed good control variables. BHPS was found to offer excellent data on human capital, particularly on recently acquired technical training and vocational and other education. However, BHPS's direct measures of job skill proved minimal. An imputation process was suggested as one way of resolving this difficulty. The material offered by BHPS on partnership and household structures was concluded to be so admirable as to partly compensate for its lack of workplace and organizational data. (Thirteen figures/tables are included. Five additional tables and a comment on the expressive work ethic are appended.) (Author/MN)

This report provides empirical results of attempts to achieve consistency of estimates between two National Center for Education Statistics (NCES) surveys, the 1993-94 Private School Survey (PSS) and the Schools and Staffing Survey (SASS). Comparisons are made among statistical and computational procedures that may achieve the desired consistency between estimates. An overview describes the problem of consistency and why it matters. Sections 2 through 4 of this report explore the adjustment alternatives, with the main methods described in Section 2. These include the modified generalized least squares (GLS) approach evaluated in a pilot study and the variation of the GLS proposed by I. Olkin (1958). The third section presents results for nine independent applications for each category of the nine-category NCES typology for private schools. Section 4 presents conclusions, summarizing the work done, and making some recommendations. The new Olkin approach represents some improvements on the GLS approach previously tested, but results are not as encouraging as had been hoped. Section 5 presents references, and the algorithms used are illustrated in an appendix (Section 6). (Contains 36 tables, 40 figures, and 49 references.) (SLD)

This report provides empirical results of attempts to achieve consistency of estimates between two National Center for Education Statistics (NCES) surveys. These surveys are the 1991- 92 Private School Survey (PSS) and the Private School Component of the 1990-91 Schools and Staffing Survey (SASS). Consistency was sought in the numbers of schools, teachers, and students from these two sources. Comparisons are made among statistical and computational procedures that might serve to bring about the desired consistency. The complex nature of the survey designs is considered, and definitional problems are explored. Consequences of discrepancies and adjustments are discussed. The results of tests of nine typologies are presented. For some typologies (Catholic Parochial, Catholic Private, Conservative Christian, and Other Affiliated) the operational assessment shows the impact of adjustment on the SASS at least did no apparent harm, and an independent assessment shows that it might have been beneficial to accuracy. For three other typologies (Other Unaffiliated, Nonsectarian Regular, and Nonsectarian Special) the Generalized Least Squares (GLS) adjustment may have caused severe problems. Overall, results suggest that the use of a modified GLS reweighting, even when beneficial, does not make much of a positive difference beyond achieving consistency with PSS. Other methods appear needed to take advantage of the opportunity offered by having both surveys fielded for the same year. An appendix discusses a media GLS method. (Contains 19 tables, 36 figures, and 23 references.) (SLD)

The purpose of this study was to provide an analysis and evaluation of composite variables in the National Education Longitudinal Study of 1988 (NELS:88) and School and Staffing Survey (SASS) in a way that provides guidance to the staff of the National Center for Education Statistics (NCES) in the more effective use of survey resources. The study provides ways of understanding the contributions made by individual survey items through an appraisal of the contributions they make to composite measures of which they are a part. It also shows how composite variables may provide more reliable measures of the concepts of interest than do individual survey items, and how they permit the more effective summarization and communication of survey results. The composite variables examined include: (1) self-concept and locus of control; (2) socioeconomic status; and (3) school climate. Additional variables designated by NCES and variables found to be of interest during the course of the study were also analyzed. The guiding question was whether a particular variable contributed to or detracted from the efficacy of a composite variable. The strategy followed in the study used several factor analyses followed by calculation of reliability and validity estimates on a given composite variable. While the field costs of surveys have escalated, the computer has made analysis comparatively quick and cheap, making it cost effective to conduct a variety of analyses of survey items and their composites. A summary of the analyses is included. (SLD)

A study was conducted by New York's Broome Community College (BCC) to compare student-reported parental income with income reported by the students' parents on their 1982 tax records. Student reports of parental income were obtained from a survey administered to students entering BCC in fall 1983. The college had parents' federal income tax records on file for the 126 students from this group who applied for financial aid. Six variables were chosen as being potentially useful in explaining the accuracy of reports of parental income: age, sex, average grade in high school, education of father, occupation of father, and intended major. The sample included only one married and one black student, who were subsequently removed from the sample. Compared with similar studies on the subject, students in the BCC study were more likely to estimate their parents' income accurately: 37.3% checked the accurate income class on the entering-student survey; 29.4% underreported income; and 33.3% overreported it. Over 65% of the respondents were within $6,000 of an accurate response. Accurate responses were more likely to be given by females, students whose fathers had a low job status and/or a high school education or less, students with a B average in high school, and students enrolled in a non-transfer major. The study findings suggested that students who apply for financial aid are more accurate in estimating their parents' income than the general population of students, and that students from disadvantaged backgrounds report more accurately than students from higher socioeconomic backgrounds. The most important limitation of this study is that the all-white sample is not representative of the heterogeneous student population of community colleges. (KP)

The methodology of five national surveys designed to estimate the scale and scope of training was analyzed to identify the reasons underlying the variability and unreliability of estimates of education and training. The five surveys were as follows: May Triennial Adult Education Supplement to the Current Population Survey, National Longitudinal Survey of Youth, Survey of Income and Program Participation, National Longitudinal Survey of High School Class of 1972, and High School and Beyond. The analysis focused on the following: fundamental purpose of each survey, population sampled, goodness of fit, and survey structure/design. The analysis established that the five surveys do not measure fully what they purport to measure. It was concluded, however, that when taken collectively, they provided an invaluable insight into the nature of work-related education and training in the United States. The following are recommendations for improving the surveys' reliability and capacity for measuring the scale and scope of education and training: (1) "bundle" questions focusing on incidence of work-related education and training with questions dealing explicitly with work and the skills needed for it; (2) include matched samples of employment establishments and employees; and (3) develop more explicit questions focusing on training credentials and certificates. (Contains 16 figures/tables and 19 references.) (MN)

The Walsh and Carballo evaluation of the effectiveness of transitional bilingual education programs in five Massachusetts communities, has the following flaws: (1) the sample of school districts studied suffered from "self-selection bias"; (2) the sample does not include a single large, urban school district; (3) the student samples analyzed are much too small to allow for any conclusions; (4) there is no statistical analysis of the data nor control for pre-existing differences between groups; and (5) the wrong comparison is conducted. In elaborating on those flaws, the report first describes five models of how to instruct children who do not speak English. It then summarizes the requirements of a methodological sound study and maintains that the studies cited by Walsh and Carballo do not follow those rules. Finally, the above five flaws are discussed in detail. A bibliography is included. (PS)

Peer evaluation research was reviewed from three major perspectives--validity, methodology, and situational factors. Most of the studies focused on either concurrent or predicitve validity in a military training situation. Evaluation criteria included leadership potential, promotion potential, personality traits, and supervisory skills. Substantial validity was generally found, with correlation coefficients in the .30 to .50 range. Reliability and validity of different evaluation methods (rating, ranking, nominations, and combinations of these techniques) did not vary substantially. Evaluation methods did, however, differ in feasibility and acceptability, the latter largely a function of familiarity with the evaluation procedure and perceived difficulty. Situational factors have documented and potential effects which developers and users of peer evaluation should recognize. These factors include group size, informal group structure, demographic characteristics, group boundaries, hierarchical characteristics, friendship, length of association, and types of interaction. Although many issues surrounding peer evaluation or associate evaluation remain unresolved, evidence suggests they are a powerful tool in discriminating complex human behavior. (Author/CP)