Search

Many states and school districts in the United States use standardized test scores to compute annual measures of student achievement progress and then use school-level averages of these growth measures for various reporting and diagnostic purposes. These aggregate growth measures can vary consequentially from year to year for the same school, complicating their use and interpretation. We develop a method, based on the theory of empirical best linear prediction, to improve the accuracy and stability of aggregate growth measures by pooling information across grades, years, and tested subjects for individual schools. We demonstrate the performance of the method using both simulation and application to 6 years of annual growth measures from a large, urban school district. We provide code for implementing the method in the package schoolgrowth for the R environment.

Errors-in-variables (EIV) regression is a standard method for consistent estimation in linear models with error-prone covariates. The Stata commands eivreg and sem both can be used to compute the same EIV estimator of the regression coefficients. However, the commands do not use the same methods to estimate the standard errors of the estimated regression coefficients. In this article, we use analysis and simulation to demonstrate that standard errors reported by eivreg are negatively biased under assumptions typically made in latent-variable modeling, leading to confidence interval coverage that is below the nominal level. Thus, sem alone or eivreg augmented with bootstrapped standard errors should be preferred to eivreg alone in most practical applications of EIV regression.

Summary A common problem in educational evaluation is estimating causal effects of interventions from non-experimental data on students. Scores from standardized achievement tests often are used to adjust for differences in background characteristics of students in different non-experimental groups. An open question is whether, and how, these adjustments should account for the errors in test scores as measures of latent achievement. The answer depends on what information was used to assign students to non-experimental groups. Using a case-study of estimating teacher effects on student achievement, we develop two novel empirical tests about what information is used to assign students to teachers. We demonstrate that assignments are influenced by both information that is unobserved by the researcher, and error prone test scores. We develop a model that is appropriate for this complex selection mechanism and compare its results with common simpler estimators. We discuss implications for the broader problem of causal modelling with error prone confounders.

Background:Analysis of covariance (ANCOVA) is commonly used to adjust for potential confounders in observational studies of intervention effects. Measurement error in the covariates used in ANCOVA models can lead to inconsistent estimators of intervention effects. While errors-in-variables (EIV) regression can restore consistency, it requires surrogacy assumptions for the error-prone covariates that may be violated in practical settings.Objectives:The objectives of this article are (1) to derive asymptotic results for ANCOVA using EIV regression when measurement errors may not satisfy the standard surrogacy assumptions and (2) to demonstrate how these results can be used to explore the potential bias from ANCOVA models that either ignore measurement error by using ordinary least squares (OLS) regression or use EIV regression when its required assumptions do not hold.Results:The article derives asymptotic results for ANCOVA with error-prone covariates that cover a variety of cases relevant to applications. It then uses the results in a case study of choosing among ANCOVA model specifications for estimating teacher effects using longitudinal data from a large urban school system. It finds evidence that estimates of teacher effects computed using EIV regression may have smaller bias than estimates computed using OLS regression when the data available for adjusting for students’ prior achievement are limited.

In a school context, learning is usually detected by repeated measurements of the skill of interest through a sequence of specially designed tests; in particular, this is the case with tracking improvement in oral reading fluency in elementary school children in the U.S. Results presented in this paper suggest that it is possible and feasible to detect improvement in oral reading fluency using data collected during children's independent reading of a book using the Relay Reader™ app. We are thus a step closer to the vision of having a child read for the story, not for a test, yet being able to unobtrusively assess their progress in oral reading fluency.

This article proposes a flexible extension of the Fay–Herriot model for making inferences from coarsened, group-level achievement data, for example, school-level data consisting of numbers of students falling into various ordinal performance categories. The model builds on the heteroskedastic ordered probit (HETOP) framework advocated by Reardon, Shear, Castellano, and Ho by allowing group parameters to be modeled with regressions on group-level covariates, and residuals modeled using the flexible exponential family of distributions recommended by Efron. We demonstrate that the alternative modeling framework, termed the “Fay–Herriot heteroskedastic ordered probit” (FH-HETOP) model, is useful for mitigating some of the challenges with direct maximum likelihood estimators from the HETOP model. We conduct a simulation study to compare the costs and benefits of several methods for using the FH-HETOP model to estimate group parameters and functions of them, including posterior means, constrained Bayes estimators, and the “triple goal” estimators of Shen and Louis. We also provide an application of the FH-HETOP model to math proficiency data from the Early Childhood Longitudinal Study. Code for estimating the FH-HETOP model and conducting supporting calculations is provided in a new package for the R environment.

This article considers the application of the simulation-extrapolation (SIMEX) method for measurement error correction when the error variance is a function of the latent variable being measured. Heteroskedasticity of this form arises in educational and psychological applications with ability estimates from item response theory models. We conclude that there is no simple solution for applying SIMEX that generally will yield consistent estimators in this setting. However, we demonstrate that several approximate SIMEX methods can provide useful estimators, leading to recommendations for analysts dealing with this form of error in settings where SIMEX may be the most practical option.

Matching estimators are commonly used to estimate causal effects in nonexperimental settings. Covariate measurement error can be problematic for matching estimators when observational treatment groups differ on latent quantities observed only through error-prone surrogates. We establish necessary and sufficient conditions for matching and weighting with functions of observed covariates to yield unconfounded causal effect estimators, generalizing results from the standard (i.e., no measurement error) case. We establish that in common covariate measurement error settings, including continuous variables with continuous measurement error, discrete variables with misclassification, and factor and item response theory models, no single function of the observed covariates computed for all units in a study is appropriate for matching. However, we demonstrate that in some circumstances, it is possible to create different functions of the observed covariates for treatment and control units to construct a va...

Student Growth Percentiles (SGPs) increasingly are being used in the United States for inferences about student achievement growth and educator effectiveness. Emerging research has indicated that SGPs estimated from observed test scores have large measurement errors. As such, little is known about “true” SGPs, which are defined in terms of nonlinear functions of latent achievement attributes for individual students and their distributions across students. We develop a novel framework using latent regression multidimensional item response theory models to study distributional properties of true SGPs. We apply these methods to several cohorts of longitudinal item response data from more than 330,000 students in a large urban metropolitan area to provide new empirical information about true SGPs. We find that true SGPs are correlated 0.3 to 0.5 across mathematics and English language arts, and that they have nontrivial relationships with individual student characteristics, particularly student race/ethnicity and absenteeism. We evaluate the potential of using these relationships to improve the accuracy of SGPs estimated from observed test scores, finding that accuracy gains even under optimal circumstances are modest. We also consider the properties of SGPs averaged to the teacher level, widely used for teacher evaluations. We find that average true SGPs for individual teachers vary substantially as a function of the characteristics of the students they teach. We discuss implications of our findings for the estimation and interpretation of SGPs at both the individual and aggregate levels.

In the “individually randomized group treatment” (IRGT) experimental design, individuals are first randomly assigned to a treatment arm or a control arm, but then within each arm, are grouped together (e.g., within classrooms/schools, through shared case managers, in group therapy sessions, through shared doctors, etc.) to receive services. In the common setting where the second-stage grouping is not experimentally controlled, we demonstrate that the standard error estimators for the estimated treatment effects from both random and fixed effects models are biased because parameters required to compute the standard errors are not identifiable from the data. We discuss implications for designing and analyzing IRGTs and provide suggestions for mitigating the bias.

Standardized tests are often designed to provide only a snapshot of test takers' knowledge, skills, or abilities at a single point in time. Sometimes, however, they are expected to serve more demanding functions, one of them is assessing change in knowledge, skills, or ability over time because of learning effects. The latter is the case for the newly developed TOEFL Junior® Standard test, which measures improvement in young learners' proficiency in English as a foreign language. In this study, we used nonexperimental-repeated measures data from approximately 4,600 students from multiple countries to examine the extent to which observed patterns in within-individual changes in test scores were consistent with changes in underlying language proficiency because of learning. Because most students were actively participating in English language learning programs, the time interval between test administrations, which varied among students, served as a proxy for the extent of English language learning opportunities. We used hierarchical linear models to model growth in test performance as a function of the time interval between test administrations and found a positive, statistically significant relationship; that is, test takers with longer intervals between retesting exhibited greater gains than did test takers who retested at shorter intervals. The estimated relationship for the total score corresponded to between .16 and .24 test standard deviations of growth per year, depending on model specification. The findings are robust to sensitivity analyses that explore potential biasing factors. Overall, the findings are consistent with the hypothesis that the TOEFL Junior Standard test is capable of reflecting change in English language proficiency over time.

Student growth percentiles (SGPs) express students' current observed scores as percentile ranks in the distribution of scores among students with the same prior-year scores. A common concern about SGPs at the student level, and mean or median SGPs (MGPs) at the aggregate level, is potential bias due to test measurement error (ME). Shang, vanIwaarden, and Betebenner (SVB; this issue) develop a simulation-extrapolation (SIMEX) approach to adjust SGPs for test ME. In this paper, we use a tractable example in which different SGP estimators, including SVB's SIMEX estimator, can be computed analytically to explain why ME is detrimental to both student-level and aggregate-level SGP estimation. A comparison of the alternative SGP estimators to the standard approach demonstrates the common bias-variance tradeoff problem: estimators that decrease the bias relative to the standard SGP estimator increase variance, and vice versa. Even the most accurate estimator for individual student SGP has large errors of roughly 19 percentile points on average for realistic settings. Those estimators that reduce bias may suffice at the aggregate level but no single estimator is optimal for meeting the dual goals of student- and aggregate level inferences.

The introduction and establishment of alien species is one of the many profound influences of ongoing anthropogenic global environmental change. Invasion biology has emerged as the interdisciplinary study of the patterns, processes and consequences of the redistribution of biodiversity across all environments and spatio-temporal scales. The modern discipline hinges on the knowledge that biological invasions cannot be defined and studied solely by their final outcome of establishing alien species but rather as a sequential series of stages, or barriers, that all alien species transit: the 'invasion pathway'. Some of the most important influences for a species transiting these sequential stages (i.e. transport, introduction, establishment and spread) are event-level effects, which vary independently of species and location, such as the number of individuals released in any given location (propagule pressure). The number of studies of biological invasions has increased exponentially over the past two decades, and we now have a significant body of research on different aspects of the invasion process. In particular, the hierarchical nature of the invasion pathway has lent itself strongly to modern statistical methods in hierarchical modelling. Now, the science behind invasive species management must continue to develop innovative ways of using this accumulated knowledge for delivering actionable management procedures.

This article suggests two alternative statistical approaches for estimating student growth percentiles (SGP). The first is to estimate percentile ranks of current test scores conditional on past test scores directly, by modeling the conditional cumulative distribution functions, rather than indirectly through quantile regressions. This would remove the need for post hoc procedures required to ensure monotonicity of the estimated quantile functions, and for inversion of those functions to obtain SGP. We provide a brief empirical example demonstrating this approach and its potential benefits for handling discreteness of the observed test scores. The second suggestion is to estimate SGP directly from longitudinal item-level data, using multidimensional item response theory models, rather than from test scale scores. This leads to an isomorphism between using item-level data from one test to make inferences about latent student achievement, and using item-level data from multiple tests administered over time ...

Ratings of teachers' instructional practices using standardized classroom observation instruments are increasingly being used for both research and teacher accountability. There are multiple instruments in use, each attempting to evaluate many dimensions of teaching and classroom activities, and little is known about what underlying teaching quality attributes are being measured. We use data from multiple instruments collected from 458 middle school mathematics and English language arts teachers to inform research and practice on teacher performance measurement by modeling latent constructs of high-quality teaching. We make inferences about these constructs using a novel approach to Bayesian exploratory factor analysis (EFA) that, unlike commonly used approaches for identifying factor loadings in Bayesian EFA, is invariant to how the data dimensions are ordered. Applying this approach to ratings of lessons reveals two distinct teaching constructs in both mathematics and English language arts: (1) quality of instructional practices; and (2) quality of teacher management of classrooms. We demonstrate the relationships of these constructs to other indicators of teaching quality, including teacher content knowledge and student performance on standardized tests., Published at http://dx.doi.org/10.1214/15-AOAS833 in the Annals of Applied Statistics (http://www.imstat.org/aoas/) by the Institute of Mathematical Statistics (http://www.imstat.org)

A common goal in conservation planning is to acquire areas that are critical to realizing biodiversity goals in the most cost-effective manner. The way monetary acquisition costs are represented in such planning is an understudied but vital component to realizing cost efficiencies. We sought to design a protected-area network within a forested urban region that would protect 17 birds of conservation concern. We compared the total costs and spatial structure of the optimal protected-area networks produced using three acquisition-cost surrogates (area, agricultural land value, and tax-assessed land value). Using the tax-assessed land values there was a 73% and 78% cost savings relative to networks derived using area or agricultural land value, respectively. This cost reduction was due to the considerable heterogeneity in acquisition costs revealed in tax-assessed land values, especially for small land parcels, and the corresponding ability of the optimization algorithm to identify lower-cost parcels for inclusion that had equal value to our target species. Tax-assessed land values also reflected the strong spatial differences in acquisition costs (US$0.33/m(2)-$55/m(2)) and thus allowed the algorithm to avoid inclusion of high-cost parcels when possible. Our results add to a nascent but growing literature that suggests conservation planners must consider the cost surrogate they use when designing protected-area networks. We suggest that choosing cost surrogates that capture spatial- and size-dependent heterogeneity in acquisition costs may be relevant to establishing protected areas in urbanizing ecosystems.