Search

Differential item functioning (DIF) analyses have been used as the primary method in large-scale assessments to examine fairness for subgroups. Currently, DIF analyses are conducted utilizing manifest methods using observed characteristics (gender and race/ethnicity) for grouping examinees. Homogeneity of item responses is assumed denoting that all examinees respond to test items using a similar approach. This assumption may not hold with all groups. In this study, we demonstrate the first application of the latent class (LC) approach to investigate DIF and its sources with heterogeneous (linguistic minority groups). We found at least three LCs within each linguistic group, suggesting the need to empirically evaluate this assumption in DIF analysis. We obtained larger proportions of DIF items with larger effect sizes when LCs within language groups versus the overall (majority/minority) language groups were examined. The illustrated approach could be used to improve the ways in which DIF analyses ...

The purpose of this research is to examine the comparability of mathematics and science scores for students from English language backgrounds (ELB) and non-English language backgrounds (NELB). We examine the relationship between English reading proficiency and performance on mathematics and science assessments in Australia, Canada, the United Kingdom, and the United States. The findings indicate a strong relationship with reading proficiency accounting for up to 43% of the variance in mathematics and up to 79% in science. In all comparisons, ELB students either outperformed NELB students or performed at the same level. However, when statistical adjustments were made for reading proficiency, in both mathematics and science, the score gap between the groups became statistically non-significant in three out of the four countries. These findings point to differences in score meaning in mathematics and science assessments and limitations in comparing performances of ELB and NELB.

Diversity and heterogeneity among language groups have been well documented. Yet most fairness research that focuses on measurement comparability considers linguistic minority students such as English language learners (ELLs) or Francophone students living in minority contexts in Canada as a single group. Our focus in this research is to examine the degree to which measurement comparability, as indicated by differential item functioning (DIF), is consistent for sub-groups among linguistic minority Francophone students in Canada. The findings suggest that the linguistic minority Francophone students who speak French at home and those who do not speak French at home should not be grouped together for investigating measurement comparability or for examining performance gaps. We identified a great degree of differences in DIF identification with a consistency of 7–10% in DIF identification in the separate analyses for the two groups. The findings highlight methodological problems with investigating fairness f...

Minority Francophone students in predominantly English-speaking Canadian provinces tend to perform lower on large-scale assessments of achievement than their Anglophone peers and majority Francophone students in Quebec. This study is the first to apply multilevel modeling methods to examine the extent to which school-level factors may be differentially associated with achievement for minority Francophone students compared to majority Francophone students. Forty-four percent of the variance in science achievement among Francophone schools was explained by the model. After taking into consideration gender as well as student- and school-level socioeconomic status, French-language schools in Ontario and New Brunswick had, on average, lower PISA 2006 science achievement scores compared to schools in Quebec. Findings indicate that differences in performance between linguistic minority and majority Francophone students could not be reduced to differences in socioeconomic status or to the school resource ...

Even if national and international assessments are designed to be comparable, subsequent psychometric analyses often reveal differential item functioning (DIF). Central to achieving comparability is to examine the presence of DIF, and if DIF is found, to investigate its sources to ensure differentially functioning items that do not lead to bias. In this study, sources of DIF were examined using think-aloud protocols. The think-aloud protocols of expert reviewers were conducted for comparing the English and French versions of 40 items previously identified as DIF (N = 20) and non-DIF (N = 20). Three highly trained and experienced experts in verifying and accepting/rejecting multi-lingual versions of curriculum and testing materials for government purposes participated in this study. Although there is a considerable amount of agreement in the identification of differentially functioning items, experts do not consistently identify and distinguish DIF and non-DIF items. Our analyses of the think-aloud protoco...

This chapter examines trends in the reporting of validity evidence across multiple journals and disciplines, including measurement, quality of life research, and medical education. The 15 synthesis chapters from this volume were compared based on the five validity evidence categories presented in the Standards for Educational and Psychological Testing. Major findings from the meta-synthesis suggest that two categories of validity evidence are generally ignored across fields: evidence based on response processes and evidence based on consequences. Additionally, it was found that evidence based on internal structure is exceedingly prevalent in current validation practices, above and beyond other sources of validity evidence. Possible explanations for these findings, as well as implications for future investigations of validity, are discussed.