Search

We examined semiotic exchangeability in pop-up glossary translations and illustrations used as supports for second language learners (SLLs) in computer-administered mathematics tests. In a sample of 516 mathematics items, Grades 3-8 and 11, from a large-scale assessment program in the US, test developers identified terms that could be translated and terms that could be illustrated without altering the target constructs. In each item, we examined: number of unique words (w), number of unique terms identified for translation (t), and number of unique terms identified for illustration (i.) Of the 3,186 terms identified for translation or for illustration, only about one sixth were identified both for translation and for illustration. We observed statistically significant "w" and "t" differences attributable to the main effect of depth of knowledge and type of item and statistically significant "i" differences attributable to the main effect of type of item. The correlation of item translatability (T = t/w) and item illustratability (I = i/w) was low for all items combined but higher and statistically significant for some types of items and for greater values of depth of knowledge. We conclude that translation and illustration pop-up glossaries are functionally different sets of semiotic resources that support SLLs in different ways.

In this experimental study, 20 multiple-choice test items from the Massachusetts Grade 5 science test were linguistically simplified, and original and simplified test items were administered to 310 English learners (ELs) and 1,580 non-ELs in four Massachusetts school districts. This study tested the hypothesis that specific linguistic features of test items contributed to construct-irrelevant variance in science test scores of ELs. Simplifications targeted specific linguistic features, to identify those features with the largest impacts on ELs' test performance. Of all the linguistic simplifications used in this study, adding visual representations to answer choices had the largest positive effect on ELs' performance. These findings have significant implications for the design of multiple-choice test items that are fair and valid for ELs.

This paper is concerned with fair and valid testing. It reports on a long-term project aimed at creating visual accessibility resources in mathematics computer-administered test items in a large-scale assessment system. We discuss our experience designing and developing pop-up illustration glossaries (IGs) for second language learners (SLLs) -- students who are not native users of the predominant language in the society in which they live, are typically schooled in that predominant language, and face the challenge of learning content and demonstrating knowledge in a language that is not their first language. An IG is a visual representation of a word/term identified as likely to be linguistically challenging to a substantial proportion of SLL test takers and which appears on the screen when the test taker clicks on that word/term. Because they interact with the user, IGs are more sensitive than other accessibility resources to students' individual needs -- a characteristic that addresses the linguistic heterogeneity of SLL populations. We discuss the extent to which IGs contribute to ensuring valid, fair assessment for SLL students, and the implications of the lessons learned for improving assessment systems.

The National Science Foundation recently recognized that career-life balance in science, technology, engineering, and mathematics (STEM) may present some unique challenges for women of color compared with their White and/or male counterparts, thus negatively impacting retention and advancement for a minority demographic that has long been underrepresented in STEM (National Science Foundation, 2011). However, there is a dearth of literature on this topic. This article advances our understanding of the nature of these challenges by analyzing career-life balance issues for three underrepresented minority women and the impact of these issues on their career trajectories. We explore the experiences of these women, at different stages of pursuing STEM academia, as they encounter career-life balance issues. Using the approach of narrative inquiry, we describe the interplay between domestic realities (e.g., social life, elder care, pregnancy) and the "ideal worker norm," a prevalent gendered standard within STEM of pure dedication to scientific work. Using the lens of "cumulative disadvantage" we examine how the intersection of racial/ethnic and gender identities generated a set of experiences for each participant that positioned her outside that norm, resulting in significant obstacles (e.g., exclusion from professional networks, lack of support, questioning of competence). We describe how continual tension with the ideal worker norm impacted these women's career trajectories, illustrate how their experiences of career-life balance may be different from those who more closely embody the norm, and detail issues that need to be addressed to increase the participation, retention, and advancement of women of color in STEM academia.

We address valid testing for English language learners (ELLs)--students in the United States who are schooled in English while they are still acquiring English as a second language. Also, we address the need for procedures for systematically developing ELL testing accommodations--changes in tests intended to support ELLs to gain access to the content of items. We report our experience developing and investigating illustration-based accommodations for ELLs in terms of five main activities: specification, design, production, evaluation, and revision. Our findings indicate that, to benefit from a given accommodation, ELLs need to possess certain skills that enable them to use it but that are not related to the knowledge assessed. Critical to developing effective accommodations is examining how their properties interact with the characteristics of the students and the characteristics of test items.

We examined the performance of English language learners (ELLs) and non-ELLs on Grade 4 and Grade 5 mathematics content knowledge (CK) and academic language (AL) tests. CK and AL items had different semiotic loads (numbers of different types of semiotic features) and different semiotic structures (relative frequencies of different semiotic modalities). For both linguistic groups, the percentage of items correct was higher for the AL than the CK tests. However, the score gains attributable to instruction were smaller for the AL than the CK tests. CK and AL test scores correlated more highly for non-ELLs than ELLs before instruction. This suggests that, before instruction, the meaning-making system of ELLs was less consolidated than that of non-ELLs, who dealt with the interpretive demands of CK and AL items with similar effectiveness. We discuss the importance of using a semiotic perspective in test design and in interpreting ELL student performance.

We examined how rater language background affects the scoring of short-answer, open-ended test items in the assessment of English language learners (ELLs). Four native English and four native Spanish-speaking certified bilingual teachers scored 107 responses of fourth- and fifth-grade Spanish-speaking ELLs to mathematics items administered in English and Spanish. We examined the mean scores given by raters of different language backgrounds. Also, using generalizability theory, we examined the amount of score variation due to student (the object of measurement) and four sources of measurement error-item, language of testing, rater language background, and rater nested in rater language background. We observed a small, statistically significant difference between mean scores given by raters of different language background and a negligible score variation due to the main and interaction effect of rater. Provided that they are certified bilingual teachers, and regardless of language of testing, raters of different language backgrounds can score ELL responses to short-answer, open-ended items with comparable reliability. (Contains 4 tables, 1 figure, and 5 footnotes.)