Search

Rapid response behaviour, a type of test disengagement, cannot be interpreted as a true indicator of the targeted constructs and may compromise score accuracy as well as score validity for interpre...

Comparing group is one of the key uses of large-scale assessment results, which are used to gain insights to inform policy and practice and to examine the comparability of scores and score meaning....

Shortly after the COVID-19 pandemic arrived in the United States, schools across the country had to enact significant, rapid changes to their instructional models, and schools varied widely in their access to the resources needed to support these efforts. Researchers across the U.S. quickly launched surveys, website reviews, and other data-collection methods to document these shifts. In this chapter, we draw on this research to describe the U.S. K-12 educational context, the policies states adopted, the practices and resources schools offered, and the potential effects on students’ academic, social, and emotional learning. In these discussions we draw particular attention to inequities in educational opportunities across schools serving different student populations. We then discuss how different sources of data will be needed to help identify educational needs and mitigate disparities in instruction and learning post-pandemic.

The purpose of this study is to examine item equivalence and score comparability of the Progress in International Reading Literacy Study (PIRLS) 2011 for the Canadian French and English language groups. Two methods of differential item functioning were conducted to examine item equivalence across 13 test booklets designed to assess reading literacy in early years of schooling. Four bilingual reviewers with expertise in reading literacy conducted independent, linguistic, and cultural reviews to identify both the degree of item equivalence and potential sources of differences between language versions of released items. Results indicate that an average of 25% of items per booklet function differentially at the item level. Reviews by experts indicate differences between the two language versions on some items flagged as displaying differential item functioning (DIF). Some of these were identified to have linguistic differences pointing to differential difficulty levels in the two language versions.

In gender differential item functioning (DIF) research it is assumed that all members of a gender group have similar item response patterns and therefore generalizations from group level to subgroup and individual levels can be made accurately. However DIF items do not necessarily disadvantage every member of a gender group to the same degree, indicating existence of heterogeneity of response patterns within gender groups. In this article the impact of heterogeneity within gender groups on DIF investigations was investigated. Specifically, it was examined whether DIF results varied when comparing males versus females, gender × socioeconomic status subgroups and latent classes of gender. DIF analyses were conducted on reading achievement data from the Canadian sample of the Programme of International Student Assessment 2009. Results indicated considerable heterogeneity within males and females and DIF results were found to vary when heterogeneity was taken into account versus when it was not.

Assessing complex constructs such as those discussed under the umbrella of 21st century constructs highlights the need for a principled assessment design and validation approach. In our discussion, we made a case for three considerations: (a) taking construct complexity into account across various stages of assessment development such as the design, scaling, and interpretation aspects; (b) cognitive validity evidence that goes beyond traditional psychometric analyses of response patterns, and (c) cross-cultural validity. We analyze the four articles in this special issue with respect to these three considerations and discuss the kinds of evidence needed to support interpretation of scores from 21st century constructs.

Assessing mathematical literacy of students who have limited proficiency in the language of the test is a critical challenge in mathematics education. Previous research indicates that knowledge and competencies of such students are underestimated. This presents a major validity and fairness problem for assessment. Most efforts addressing fairness and validity issues in assessment of linguistic minority students focus on the test language only. To overcome limitations of single approaches, we examine in this study the interaction between the test language and the student language background by means of multiple methods. Thus, we investigate possible linguistic bias of items flagged as functioning differentially (the result of DIF analyses) by means of (a) two levels of expert analyses and (b) student think-aloud protocols to investigate language effects in published mathematics items from the 2000 to 2003 Programme for International Student Assessment (PISA) administration for students attending French schools in Canada and speaking either French or other languages at home. DIF analyses were conducted to identify items on which students from different home language backgrounds attending French schools achieve differently. The expert panels tended to identify surface characteristics of language that may be responsible for group differences but not for the differential effects detected by differential item functioning (DIF). Student think-aloud protocols in part confirm and in part contradict DIF results, providing insights for the source of the differences. Suggestions are provided for further study.

Differential item functioning (DIF) analyses have been used as the primary method in large-scale assessments to examine fairness for subgroups. Currently, DIF analyses are conducted utilizing manifest methods using observed characteristics (gender and race/ethnicity) for grouping examinees. Homogeneity of item responses is assumed denoting that all examinees respond to test items using a similar approach. This assumption may not hold with all groups. In this study, we demonstrate the first application of the latent class (LC) approach to investigate DIF and its sources with heterogeneous (linguistic minority groups). We found at least three LCs within each linguistic group, suggesting the need to empirically evaluate this assumption in DIF analysis. We obtained larger proportions of DIF items with larger effect sizes when LCs within language groups versus the overall (majority/minority) language groups were examined. The illustrated approach could be used to improve the ways in which DIF analyses ...

The assessment of linguistic minorities often involves using multiple language versions of assessments. In these assessments, comparability of scores across language groups is central to valid comparative interpretations. Various frameworks and guidelines describe factors that need to be considered when developing comparable assessments. These frameworks provide limited information in relation to the development of multiple language versions of assessments for assessing linguistic minorities within countries. To this end, we make various suggestions for the types of factors that should be considered when assessing linguistic minorities. Our recommendations are tailored to the particular constraints potentially faced by various jurisdictions tasked with developing multiple language versions of assessments for linguistic minorities. These challenges include having limited financial and staffing resources to develop comparable assessments and having insufficient sample sizes to perform psychometric analyses ...

The Programme for International Student Assessment (PISA) was designed by the Organisation for Economic Cooperation and Development (OECD) to evaluate the quality, equity, and efficiency of school systems around the world. Specifically, the PISA has assessed 15-year-old students’ reading, mathematics, and science literacy on a 3-year cycle, since 2000. Also, the PISA collects information about how those outcomes are related to key demographic, social, economic, and educational variables. However, the preponderance of reports involving PISA data focus on achievement variables and cross-national comparisons of achievement variables. Challenges in evaluating achievement of students from different cultural and educational settings and data concerning students’ approaches to learning, motivation for learning, and opportunities for learning are rarely reported. A main goal of this themed issue of Teachers College Record (TCR) is to move the conversation about PISA data beyond achievement to also include factors that affect achievement (e.g., SES, home environment, strategy use). Also we asked authors to consider how international assessment data can be used for improving learning and education and what appropriate versus inappropriate inferences can be made from the data. In this introduction, we synthesize the six articles in this issue and themes that cut across them. Also we examine challenges associated with using data from international assessments, like the PISA, to inform education policy and practice within and across countries. We conclude with recommendations for collecting and using data from international assessments to inform research, policy, and teaching and learning.

The purpose of this research is to examine the comparability of mathematics and science scores for students from English language backgrounds (ELB) and non-English language backgrounds (NELB). We examine the relationship between English reading proficiency and performance on mathematics and science assessments in Australia, Canada, the United Kingdom, and the United States. The findings indicate a strong relationship with reading proficiency accounting for up to 43% of the variance in mathematics and up to 79% in science. In all comparisons, ELB students either outperformed NELB students or performed at the same level. However, when statistical adjustments were made for reading proficiency, in both mathematics and science, the score gap between the groups became statistically non-significant in three out of the four countries. These findings point to differences in score meaning in mathematics and science assessments and limitations in comparing performances of ELB and NELB.

Heterogeneity within English language learners (ELLs) groups has been documented. Previous research on differential item functioning (DIF) analyses suggests that accurate DIF detection rates are reduced greatly when groups are heterogeneous. In this simulation study, we investigated the effects of heterogeneity within linguistic (ELL) groups on the accuracy of DIF detection. Heterogeneity within such groups may occur for a myriad of reasons including differential lengths of time residing in English-speaking countries, degrees of exposure to English-speaking environments, and amounts of English instruction. Our findings revealed that at high levels of within-group heterogeneity, DIF detection is at the level of chance, implying that a large proportion of DIF items might remain undetected when assessing heterogeneous populations potentially leading to developing biased tests. Based on our findings, we urge test development organizations to consider heterogeneity within ELL and other heterogeneous focus grou...

Diversity and heterogeneity among language groups have been well documented. Yet most fairness research that focuses on measurement comparability considers linguistic minority students such as English language learners (ELLs) or Francophone students living in minority contexts in Canada as a single group. Our focus in this research is to examine the degree to which measurement comparability, as indicated by differential item functioning (DIF), is consistent for sub-groups among linguistic minority Francophone students in Canada. The findings suggest that the linguistic minority Francophone students who speak French at home and those who do not speak French at home should not be grouped together for investigating measurement comparability or for examining performance gaps. We identified a great degree of differences in DIF identification with a consistency of 7–10% in DIF identification in the separate analyses for the two groups. The findings highlight methodological problems with investigating fairness f...

In this study, we contrast results from two differential item functioning (DIF) approaches (manifest and latent class) by the number of items and sources of items identified as DIF using data from an international reading assessment. The latter approach yielded three latent classes, presenting evidence of heterogeneity in examinee response patterns. It also yielded more DIF items with larger effect sizes and more consistent item response patterns by substantive aspects (e.g., reading comprehension processes and cognitive complexity of items). Based on our findings, we suggest empirically evaluating the homogeneity assumption in international assessments because international populations cannot be assumed to have homogeneous item response patterns. Otherwise, differences in response patterns within these populations may be under-detected when conducting manifest DIF analyses. Detecting differences in item responses across international examinee populations has implications on the generalizability and meani...

Minority Francophone students in predominantly English-speaking Canadian provinces tend to perform lower on large-scale assessments of achievement than their Anglophone peers and majority Francophone students in Quebec. This study is the first to apply multilevel modeling methods to examine the extent to which school-level factors may be differentially associated with achievement for minority Francophone students compared to majority Francophone students. Forty-four percent of the variance in science achievement among Francophone schools was explained by the model. After taking into consideration gender as well as student- and school-level socioeconomic status, French-language schools in Ontario and New Brunswick had, on average, lower PISA 2006 science achievement scores compared to schools in Quebec. Findings indicate that differences in performance between linguistic minority and majority Francophone students could not be reduced to differences in socioeconomic status or to the school resource ...

Context Generalization is a critical concept in all research designed to generate knowledge that applies to all elements of a unit (population) while studying only a subset of these elements (sample). Commonly applied criteria for generalizing focus on experimental design or representativeness of samples of the population of units. The criteria tend to neglect population diversity and targeted uses of knowledge generated from the generalization. Objectives This article has two connected purposes: (a) to articulate the structure and discuss limitations of different forms of generalizations across the spectrum of quantitative and qualitative research and (b) to argue for considering population heterogeneity and future uses of knowledge claims when judging the appropriateness of generalizations. Research Design In the first part of the paper, we present two forms of generalization that rely on statistical analysis of between-group variation: analytic and probabilistic generalization. We then describe a third form of generalization: essentialist generalization. Essentialist generalization moves from the particular to the general in small sample studies. We discuss limitations of each kind of generalization. In the second part of the paper, we propose two additional criteria when evaluating the validity of evidence based on generalizations from education research: population heterogeneity and future use of knowledge claims. Conclusions/Recommendations The proposed criticisms of research generalizations have implications on how research is conducted and research findings are summarized. The main limitation in analytic generalization is that it does not provide evidence of a causal link for subgroups or individuals. In addition to making explicit the uses that the knowledge claims may be targeting, there is a need for some changes in how research is conducted. This includes a need for demonstrating the mechanisms of causality; descriptions of intervention outcomes as positive, negative, or neutral; and latent class analysis accompanied with discriminant analysis. The main criticism of probabilistic generalization is that it may not apply to subgroups and may have limited value for guiding policy and practice. This highlights a need for defining grouping variables by intended uses of knowledge claims. With respect to essentialist generalization, there are currently too few qualitative studies attempting to identify invariants that hold across the range of relevant situations. There is a need to study the ways in which a kind of phenomenon is produced, which would allow researchers to understand the various ways in which a phenomenon manifests itself.

In this study, we investigated differential item functioning (DIF) and its sources using a latent class (LC) modeling approach. Potential sources of LC DIF related to instruction and teacher-related variables were investigated using substantive and three statistical approaches: descriptive discriminant function, multinomial logistic regression, and multilevel multinomial logistic regression analyses. Results revealed that differential response patterns, as indicated by identification of LCs, were most strongly associated with student achievement levels and teacher-related variables rather than manifest characteristics such as gender, test language, and country, which are the focus of typical measurement comparability research. Findings from this study have important implications for measurement comparability and validity research. Evidence of within-group heterogeneity in the test data structure suggests that the identification of DIF and its sources may not apply to all examinees in the group and that me...

International large-scale assessments of achievement often have a large degree of differential item functioning (DIF) between countries, which can threaten score equivalence and reduce the validity of inferences based on comparisons of group performances. It is important to understand potential sources of DIF to improve the validity of future assessments; however, previous attempts to identify sources of DIF have had variable results. This study had two purposes. The first was to apply a confirmatory approach (Poly-SIBTEST) to investigate sources of DIF typically found in international large-scale assessments: adaptation effects and cognitive loadings of items. We conducted three pairwise DIF analyses on Spanish and English versions of the Progress in International Reading Literacy Study 2001 Reader booklet. Results confirmed that item cognitive loadings were a source of differential functioning favoring both England and the United States when compared against Colombia; however, adaptation effects did not...

Even if national and international assessments are designed to be comparable, subsequent psychometric analyses often reveal differential item functioning (DIF). Central to achieving comparability is to examine the presence of DIF, and if DIF is found, to investigate its sources to ensure differentially functioning items that do not lead to bias. In this study, sources of DIF were examined using think-aloud protocols. The think-aloud protocols of expert reviewers were conducted for comparing the English and French versions of 40 items previously identified as DIF (N = 20) and non-DIF (N = 20). Three highly trained and experienced experts in verifying and accepting/rejecting multi-lingual versions of curriculum and testing materials for government purposes participated in this study. Although there is a considerable amount of agreement in the identification of differentially functioning items, experts do not consistently identify and distinguish DIF and non-DIF items. Our analyses of the think-aloud protoco...

Testing is used in decision-making in a variety of contexts worldwide. These include selection and screening of individuals for education programs (including higher education admissions), certification and licensing of professionals, and policymaking in schools and higher education. The consequences of testing-for example, being denied admission to education programs or authorization to practice specific professions-affect individuals, organizations, and jurisdictions. In every testing context, questions arise about the degree to which a test provides accurate and meaningful information for fair decision-making. Also, the use of assessment has been the target of specific criticism, including claims that the high-stakes nature of tests leads to “teaching to the test,” thereby reducing schools’ and districts’ control over K-12 education curricula, and that higher education admission tests lack a focus on subject matter knowledge. This chapter discusses the main areas of criticism and recommends several assessment design requirements that address them.

In this chapter, we articulate the structure and discuss limitations of different forms of generalizations across the spectrum of quantitative and qualitative research. Three main categories of generalization are identified and discussed: analytic, probabilistic (sample to population), and essentialist. We argue for a set of criteria for evaluating research generalization and evidence. Underlying our review is the conviction that it is more important for health researchers to worry about the quality of research evidence than about whether the research is of the quantitative, qualitative, or mixed-method type. Independent of whether the research is qualitative or quantitative, we are as critical of research that overgeneralizes as we are of research that fails to offer generalizations beyond the actual case(s) studied.

In this study, the Canadian English and French versions of the Problem-Solving Measure of the Programme for International Student Assessment 2003 were examined to investigate their degree of measurement comparability at the item- and test-levels. Three methods of differential item functioning (DIF) were compared: parametric and nonparametric item response theory and ordinal logistic regression. Corresponding derivations of these three DIF methods were investigated at the test-level to examine both differential test functioning (DTF) and the correspondence between findings at the item-level with those at the test-level. Item-level findings suggested consistency in DIF detection across methods; however, differences in effect sizes of DIF were found by each method. Test-level results revealed a high degree of consistency across DTF methods. Discrepancies were found between item- and test-level comparability analyses. Item-level analyses suggested moderate to low degrees of comparability, whereas test-level f...

In this study, we examine the degree of construct comparability and possible sources of incomparability of the English and French versions of the Programme for International Student Assessment (PISA) 2003 problem-solving measure administered in Canada. Several approaches were used to examine construct comparability at the test- (examination of test data structure, reliability comparisons and test characteristic curves) and item-levels (differential item functioning, item parameter correlations, and linguistic comparisons). Results from the test-level analyses indicate that the two language versions of PISA are highly similar as shown by similarity of internal consistency coefficients, test data structure (same number of factors and item factor loadings) and test characteristic curves for the two language versions of the tests. However, results of item-level analyses reveal several differences between the two language versions as shown by large proportions of items displaying differential item functioning,...

This paper demonstrates and discusses the use of think aloud protocols (TAPs) as an approach for examining and confirming sources of differential item functioning (DIF). The TAPs are used to investigate to what extent surface characteristics of the items that are identified by expert reviews as sources of DIF are supported by empirical evidence from examinee thinking processes in the English and French versions of a Canadian national assessment. In this research, the TAPs confirmed sources of DIF identified by expert reviews for 10 out of 20 DIF items. The moderate agreement between TAPs and expert reviews indicates that evidence from expert reviews cannot be considered sufficient in deciding whether DIF items are biased and such judgments need to include evidence from examinee thinking processes.

Background Survival rates of children with a chronic illness is at an all-time high. Up to 98% of children suffering from a chronic illness, which may have been considered fatal in the past, now reach early adulthood. It is estimated that as many as 30% of school-aged children are affected by a chronic illness. For this population of children, the prevalence of educational and psychological problems is nearly double in comparison with the general population. Methods This study investigated the educational and psychological effects of childhood chronic illness among 1512 Canadian children (ages 10–15 years). This was a retrospective analysis using data from the National Longitudinal Survey of Children and Youth, taking a cross-sectional look at the relationships between childhood chronic illnesses, performance on a Mathematics Computation Exercise (MCE) and ratings on an Anxiety and Emotional Disorder (AED) scale. Results When AED ratings and educational handicaps were controlled for, children identified with chronic illnesses still had weaker performance on the MCE. Chronic illness did not appear to have a relationship with children's AED ratings. The regression analysis indicated that community type and illness were the strongest predictors of MCE scores. Conclusions The core research implications of this study concern measurement issues that need to be addressed in future large-scale studies. Clinical implications of this research concern the need for co-ordinated services between the home, hospital and school settings so that services and programmes focus on the ecology of the child who is ill.

This commentary first summarizes and discusses the analysis of the two translation processes described in the Oliveira, Colak, and Akerson article and the inferences these researchers make based on their research. In the second part of the commentary, we describe procedures and criteria used in adapting tests into different languages and how they may apply to adaptation of instructional materials. The authors provide a good theoretical analysis of what took place in two translation instances and make an important contribution by taking the first step in providing a systematic discussion of adaptation of instructional materials. Our discussion proposes procedures for adapting instructional materials for examining equivalence of source and target versions of adapted instructional materials. We highlight that many of the procedures and criteria used in examining comparability of educational tests is missing in this emerging research of area.

Abstract: Over the past four decades, there has been much debate on key sources of data in evaluating education, determining school effectiveness, and providing evidence to inform accountability and education planning. Entangled in this debate has been the extent to which large-scale assessments of learning provide valid evidence about the quality of schooling and education in Canada and how they can be used to inform education practice and policy. This article discusses five issues in large-scale assessments that are key for their usefulness and for making valid inferences. Based on recent research on assessment design and validity, the authors offer recommendations for large-scale assessments to better serve the multiple purposes they are intended to serve.

Preface Acknowledgements Contributor Biographies Introduction Part I: Goals of History Education: Models of Historical Cognition and Learning 1 Historical Consciousness in Germany: Concept, Implementation, Assessment Carlos Kolbl & Lisa Konrad 2 The Difficulty of Assessing Disciplinary Historical Reading Abby Reisman 3 Heritage as a Resource for Enhancing and Assessing Historical Thinking: Reflections from the Netherlands Carla van Boxtel, Maria Grever & Stephan Klein 4 Relating Historical Consciousness to Historical Thinking through Assessment Catherine Duquette Commentary 1 Into the Swampy Lowlands of Important Problems Robert B. Bain Part 2: Issues in Designing Assessments of Historical Thinking 5 Assessing for Learning in the History Classroom Bruce VanSledright 6 Historical Thinking, Competencies and their Measurement: Challenges and Approaches Andreas Korber & Johannes Meyer-Hamme 7 A Design Process for Assessing Historical Thinking: The Case of a One-Hour Test Peter Seixas, Lindsay Gibson & Kadriye Ercikan 8 Material-based and Open-ended Writing Tasks for Assessing Narrative Competence among Students Monika Waldis, Jan Hodel, Holger Thunemann, Meik Zulsdorf-Kersting, & Beatrice Ziegler Commentary 2 Historical Thinking: In Search of Conceptual and Practical Guidance for the Design and Use of Assessments of Student Competence Josh Radinsky, Susan R. Goldman, James W. Pellegrino Part 3: Large-scale Assessment of Historical Thinking 9 A Large-Scale Assessment of Historical Knowledge and Reasoning: NAEP U.S. History Stephen Lazer 10 Assessing Historical Thinking in the Redesigned Advanced Placement United States History Course and Exam Lawrence G. Charap 11 Historical Consciousness and Historical Thinking Reflected in Large-scale Assessment in Sweden Per Eliasson, Fredrik Alven, Cecilia Axelsson Yngveus, & David Rosenlund Commentary 3 Assessment of Historical Thinking in Practice Susan M. Brookhart Part 4: Validity of Score Interpretations 12 The Importance of Construct Validity Evidence in History Assessment: What is Often Overlooked or Misunderstood? Pamela Kaliski, Kara Smith, & Kristen Huff 13 Cognitive Validity Evidence for Validating Assessments of Historical Thinking Kadriye Ercikan, Peter Seixas, Juliette Lyons-Thomas, & Lindsay Gibson 14 Measuring Up?: Multiple-Choice Questions Gabriel A. Reich 15 History Assessments of Thinking: An Investigation of Cognitive Validity Mark Smith & Joel Breakstone Commentary 4 The Validity of Historical Thinking Assessments: A Commentary Denis Shemilt

This study contributes to understanding sources of gender differential item functioning (DIF) on mathematics tests. This study focused on identifying sources of DIF and differential bundle functioning for boys and girls on the British Columbia Principles of Mathematics Exam (Grade 12) using a confirmatory SIBTEST approach based on a multidimensional model. Problem solving as a content area was confirmed as a source of gender DIF in favor of boys when the item is presented in the form of a story problem or when the problems are noncontext specific. Patterns and relations content areas produced a mixture of confirmed sources of DIF, with some subtopics favoring the girls and some favoring the boys. In contrast to what might be expected given the findings of previous gender DIF research, this study did not find geometry to be a source of gender DIF. All of the higher cognitive level items favored boys. High levels of DIF were detected in favor of girls on the bundle of computation items in which no equations...

Identifying the sources of differential item functioning (DIF) in international assessments is very challenging, because such sources are often nebulous and intertwined. Even though researchers frequently focus on test translation and content area, few actually go beyond these factors to investigate other cultural sources of DIF. This article introduces the multiple-variable matching method using logistic regression analysis to identify sources of DIF. A case study demonstrates how this methodology identified Extra Lesson Hours After School (ELHAS) as a potential source of DIF between Taiwan and the United States in the Third International Mathematics and Science Study (TIMSS) 1999. DIF is not a fixed character of any test item, nor is a cultural factor an inherent source of DIF. The legitimacy of a source of DIF relies on the specific context and purpose for the cross-country comparison.

In education research, a polar distinction is frequently made to describe and produce different kinds of research: quantitative versus qualitative. In this article, the authors argue against that polarization and the associated polarization of the “subjective” and the “objective,” and they question the attribution of generalizability to only one of the poles. The purpose of the article is twofold: (a) to demonstrate that this polarization is not meaningful or productive for education research, and (b) to propose an integrated approach to education research inquiry. The authors sketch how such integration might occur by adopting a continuum instead of a dichotomy of generalizability. They then consider how that continuum might be related to the types of research questions asked, and they argue that the questions asked should determine the modes of inquiry that are used to answer them.

Multiple scoring is widely used in large-scale assessments. The use of a single response for making multiple inferences as is done in multiple scoring has implications on the validity of these inferences and interpretations based on assessment results. The purpose of this article is to review two types of multiple scoring practices and discuss how multiple scoring affects inferences.

The objective of this research was to examine the comparability of constructs assessed by English and French versions of the Third International Mathematics and Science Study (TIMSS). The differences in constructs observed in our analyses indicate serious limitations of using TIMSS results for making comparisons that use overall performance in mathematics and science. In particular, large differences between constructs were observed between the U.S. and French scales. Such limitations include rank ordering of performance of countries such as the United States and France, as well as conducting research using TIMSS data to compare factors associated with performance. The results from this study point to differences in constructs assessed by TIMSS in different countries and the importance of empirical evidence to support construct comparability before TIMSS results can be meaningfully used for research.