Search

Mixed-effects models are an analytic technique for modeling repeated measurement or nested data. This paper explains the logic of mixed-effects modeling and describes two examples of mixed-effects analyses using R. The intended audience of the paper is psychologists who specialize in cognitive development research. Therefore, the concepts and examples covered will focus primarily on repeated-measurement data resulting from studies in which participants respond to multiple items or trials. However, many of the concepts and examples we cover will likely be of use to readers outside this area of psychology. Finally, we discuss recommendations for dealing with practical challenges, suggest approaches for troubleshooting, and provide guidance on reporting results from mixed-effects models.

Background: Women and people of color continue to be underrepresented in many STEM fields and careers. Many studies have linked societal biases against the mathematical abilities of women and people of color to this underrepresentation, as well as to earlier measures of mathematical confidence and performance. Recent studies have shown that teachers may unintentionally have biases that reflect those in broader society. Yet, many studies on teachers' reports of students' abilities use data in the field--not experimental data--and thus often cannot say if the findings reflect bias or actual differences. The few experimental studies conducted suggest bias against the abilities of girls and students of color, but the prior work has limitations, which we seek to address (e.g., local samples, no exploration of moderators, no preregistration). Methods: In this preregistered experiment of 458 teachers across the U.S., we randomly assigned gender- and race-specific names to solutions to math problems, then asked teachers to rate the correctness of the solution, as well as the student's math ability and effort. Teachers also completed scales reflecting their own beliefs and dispositions, which we then assessed how those beliefs/dispositions moderated their biases. We used multilevel modeling to account for the nested data structure. Results: Consistent with our preregistered hypotheses, when the solution was not fully correct, findings suggest teachers thought boys had higher ability, even though the same teachers did not report differences in the correctness of the solution or perceived effort. Moreover, teachers who reported that gender disparities no longer exist in society were particularly likely to underestimate girls' abilities. Although findings revealed no evidence of racial bias on average, teachers' math anxiety moderated their ability judgments of students from different races, albeit with only marginal significance; teachers with high math anxiety tended to assume that White students had higher math ability than students of color. Conclusions: The present research identifies teachers' beliefs and dispositions that moderate their gender and racial biases. This experimental evidence sheds new light on why even low-performing boys consistently report higher math confidence and pursue STEM--namely, their teachers believe they have higher mathematical ability.

Although numerous survey-based studies have found that students who identify as lesbian, gay, bisexual, or questioning (LGBQ) have elevated risk for many negative academic, disciplinary, psychological, and health outcomes, the validity of the types of data on which these results rest have come under increased scrutiny. Over the past several years, a variety of data-validity screening techniques have been used in attempts to scrub data sets of "mischievous responders," youth who systematically provide extreme and untrue responses to outcome items and who tend to falsely report being LGBQ. We conducted a preregistered replication of Cimpian et al. with the 2017 Youth Risk Behavior Survey to (1) estimate new LGBQ-heterosexual disparities on 20 outcomes; (2) test a broader, mechanistic theory relating mischievousness effects with a feature of items (i.e., item response-option extremity); and (3) compare four techniques used to address mischievous responders. Our results are consistent with Cimpian et al.'s findings that potentially mischievous responders inflate LGBQ-heterosexual disparities, do so more among boys than girls, and affect outcomes differentially. For example, we find that removing students suspected of being mischievous responders can cut male LGBQ-heterosexual disparities in half overall and can completely or mostly eliminate disparities in outcomes including fighting at school, driving drunk, and using cocaine, heroin, and ecstasy. Methodologically, we find that some methods are better than others at addressing the issue of data integrity, with boosted regressions coupled with data removal leading to potentially very large decreases in the estimates of LGBQ-heterosexual disparities, but regression adjustment having almost no effect. While the empirical focus of this article is on LGBQ youth, the issues discussed are relevant to research on other minority groups and youth generally, and speak to survey development, methodology, and the robustness and transparency of research.

Women are underrepresented in many science, technology, engineering, and mathematics (STEM) majors and in some non-STEM majors (e.g., philosophy). Combining newly gathered data on students' perceptions of college major traits with data from the Education Longitudinal Study of 2002 (ELS:2002), we find that perceived gender bias against women emerges as the dominant predictor of the gender balance in college majors. The perception of the major being math or science oriented is less important. We replicate these findings using a separate sample to measure college major traits. Results suggest the need to incorporate major-level traits in research on gender gaps in college major choices and the need to recognize the impact of perceptions of potential gender discrimination on college major choices.

Effectively educating the large English learner population requires policymakers to ensure developmentally appropriate settings and services throughout the time students are learning English, as well as during their transition to fluent English proficient status--a process termed "reclassification." Using longitudinal student-level data from two U.S. states (N = 107,549), the authors implemented recent advances in multi-site regression discontinuity designs to assess the effects of reclassification policies across districts. They found that reclassification decisions are heavily influenced by state criteria; however, there is considerable variability across districts in the extent of state-level influence. The authors also found robust evidence of between-district heterogeneity in the effects of reclassification on subsequent achievement and graduation. They discuss the implications of these findings for reclassification policies and future research on the topic. Looking toward the next century of education research, the authors discuss ways that multi-site regression discontinuity designs can be combined with qualitative research to enable policymakers and practitioners to better understand variation in effects of policies across contexts as well as the mechanisms underlying those effects.

Studies using data from the Early Childhood Longitudinal Study-Kindergarten Class of 1998-1999 (ECLS-K:1999) revealed gender gaps in mathematics achievement and teacher perceptions. However, recent evidence suggests that gender gaps have closed on state tests, raising the question of whether such gaps are absent in the ECLS-K:2011 cohort. Extending earlier analyses, this study compares the two ECLS-K cohorts, exploring gaps throughout the achievement distribution and examining whether learning behaviors might differentially explain gaps more at the bottom than the top of the distribution. Overall, this study reveals remarkable consistency across both ECLS-K cohorts, with the gender gap developing early among high achievers and spreading quickly throughout the distribution. Teachers consistently rate girls' mathematical proficiency lower than that of boys with similar achievement and learning behaviors. Gender differences in learning approaches appear to be fairly consistent across the achievement distribution, but girls' more studious approaches appear to have more payoff at the bottom of the distribution than at the top. Questions remain regarding why boys outperform girls at the top of the distribution, and several hypotheses are discussed. Overall, the persistent ECLS-K patterns make clear that girls' early mathematics learning experiences merit further attention. [This article was published in "AERA Open," v2 n4 p1-19 Oct-Dec 2016.]

Studies using data from the Early Childhood Longitudinal Study--Kindergarten Class of 1998-1999 (ECLS-K:1999) revealed gender gaps in mathematics achievement and teacher perceptions. However, recent evidence suggests that gender gaps have closed on state tests, raising the question of whether such gaps are absent in the ECLS-K:2011 cohort. Extending earlier analyses, this study compares the two ECLS-K cohorts, exploring gaps throughout the achievement distribution and examining whether learning behaviors might differentially explain gaps more at the bottom than the top of the distribution. Overall, this study reveals remarkable consistency across both ECLS-K cohorts, with the gender gap developing early among high achievers and spreading quickly throughout the distribution. Teachers consistently rate girls' mathematical proficiency lower than that of boys with similar achievement and learning behaviors. Gender differences in learning approaches appear to be fairly consistent across the achievement distribution, but girls' more studious approaches appear to have more payoff at the bottom of the distribution than at the top. Questions remain regarding why boys outperform girls at the top of the distribution, and several hypotheses are discussed. Overall, the persistent ECLS-K patterns make clear that girls' early mathematics learning experiences merit further attention. [For the corresponding grantee submission, see ED569926.]

Researchers have long endeavored to understand whether teachers' evaluations of their students' mathematical ability or performance are accurate or whether their evaluations reveal implicit biases. To disentangle these factors, in a randomized controlled study (N = 390), we examined teachers' evaluations of 18 mathematical solutions to which gender- and race-specific names had been randomly assigned. Teachers displayed no detectable bias when assessing the correctness of students' solutions; however, when assessing students' mathematical ability, biases against Black, Hispanic, and female students were revealed, with biases largest against Black and Hispanic girls. Specifically, non-White teachers' estimations of students' mathematical ability favored White students (both boys and girls) over students of color, whereas (primarily female) White teachers' estimations of students' mathematical ability favored boys over girls. Results indicate that teachers are not free of bias, and that teachers from marginalized groups may be susceptible to bias that favors stereotype-advantaged groups.

Quantitative research on sexual minority youths (SMYs) has likely contributed to misperceptions about the risk and deviance of this population. In part because it often relies on self-reported data from population-based self-administered questionnaires, this research is susceptible to misclassification bias whereby youths who are not SMYs are classified as such and vice versa. I introduce a taxonomy of classification errors, including seven unique sources, and discuss the potential effects of each error on estimates of youth outcomes. Through examples, I illustrate how researchers can detect and correct such errors and discuss how piecewise attempts to address bias can sometimes unintentionally exacerbate other forms of bias. Finally, I propose a unified conceptual model for addressing the various error sources, urging researchers to take a proactive and integrative approach to data validity by attending to errors at each step in the research process, from questionnaire development to data analysis. Although I focus on errors specific to the topic of sexuality research on youths, this argument can serve as a broader template for the development of taxonomies of error along other dimensions considered in educational research, such as race and disability status.

A valuable extension of the single-rating regression discontinuity design (RDD) is a multiple-rating RDD (MRRDD). To date, four main methods have been used to estimate average treatment effects at the multiple treatment frontiers of an MRRDD: the "surface" method, the "frontier" method, the "binding-score" method, and the "fuzzy instrumental variables" method. This article uses a series of simulations to evaluate the relative performance of each of these four methods under a variety of different data-generating models. Focusing on a two-rating RDD (2RRDD), we compare the methods in terms of their bias, precision, and mean squared error when implemented as they most likely would be in practice--using optimal bandwidth selection. We also apply the lessons learned from the simulations to a real-world example that uses data from a study of an English learner reclassification policy. Overall, this article makes valuable contributions to the literature on MRRDDs in that it makes concrete recommendations for choosing among MRRDD estimation methods, for implementing any chosen method using local linear regression, and for providing accurate statistical inferences.

In a Registered Report, the authors propose a new survey‐bias‐mitigation method—incorporating inverse probability weighting via boosted regression—to better understand lesbian, gay, bisexual, or questioning (LGBQ)‐heterosexual youth risk disparities. This method is tested using the 2019 US Centers for Disease Control and Prevention‐collected Youth Risk Behavior Survey (YRBS) national data, which contain 12,847 observations (ages 12–18 [M = 16, SD = 1.25]; 49.1% male [8.7% LGBQ] and 50.9% female [22.4% LGBQ]; nationally representative regarding race and ethnicity). Looking across 44 outcomes, the authors found that the YRBS contains responses that are potentially biased against LGBQ youth in systematic ways, inflating perceived risk for this group in some outcomes. This potential bias is more pronounced among reported males than among reported females, and it is more pronounced for low‐incidence outcomes. For example, the steroid‐use disparity estimate among reported males reduced by 67%, while the reduction in bullying victimization was small and not statistically significant. The authors discuss robustness of results, the new method, and data policy implications. [ABSTRACT FROM AUTHOR]

A valuable extension of the single-rating regression discontinuity design (RDD) is a multiple-rating RDD (MRRDD). To date, four main methods have been used to estimate average treatment effects at the multiple treatment frontiers of an MRRDD: the “surface” method, the “frontier” method, the “binding-score” method, and the “fuzzy instrumental variables” method. This article uses a series of simulations to evaluate the relative performance of each of these four methods under a variety of different data-generating models. Focusing on a two-rating RDD (2RRDD), we compare the methods in terms of their bias, precision, and mean squared error when implemented as they most likely would be in practice—using optimal bandwidth selection. We also apply the lessons learned from the simulations to a real-world example that uses data from a study of an English learner reclassification policy. Overall, this article makes valuable contributions to the literature on MRRDDs in that it makes concrete recommendations for choosing among MRRDD estimation methods, for implementing any chosen method using local linear regression, and for providing accurate statistical inferences. [ABSTRACT FROM PUBLISHER]