Search

Item response theoretic methods are applied to the measurement of achievement of students from various instructional backgrounds. This extended item response theory (IRT) approach serves as a tool for studying instructional bias, or instructional sensitivity. The model maintains the form of an IRT model, but has parameters that quantify the extent of the effect attributed to opportunity to learn (OTL). The technique is applied to detect instructional sensitivity using the Second International Mathematics Study (SIMS) set of 40 core items for eighth-graders in the United States. The SIMS data came from about 280 schools and about 7,000 students measured at the end of spring 1982. The achievement test used contained 180 items in the areas of arithmetic, algebra, geometry, and measurement distributed among four test forms. In the SIMS data, there was considerable heterogeneity in the mathematics instruction experiences of students. The model features parameters estimating the influence of student background and OTL content pertinent to each specific test item on a single latent mathematics ability trait, and the effects of the mathematics ability trait and the item-specific OTL on the difficulties of test items. The analysis indicates that certain test items representing early stages of learning about selected mathematical topics were particularly sensitive to specific instruction. An 18-item list of references is included. (SLD)

This study developed, applied, and evaluated a theory-based method of detecting the underlying causes of differential difficulty. The method was applied to two subgroups taking the Scholastic Aptitude Test-Mathematics (SAT-M), 261 visually impaired students taking Braille forms of the test and 1,985 black students at 3 test administrations. It involved: (1) reviewing the literature to identify possible causes of differential item functioning; (2) forming item categories based on those factors; (3) identifying categories that functioned differentially; (4) assessing the functioning of the items composing deviant categories; and (5) relating item and category functioning. Results were compared to a traditional item-level analysis. In both subgroups, the cluster and traditional methods agreed on the overall extent of differential functioning (substantial in the first group, virtually absent in the second). The method would seem to be applied most productively when a small number of hypotheses can be derived from a reasonably strong research base, overlap among cluster structures can be avoided, and results can be supplemented with experimental studies of protocol analysis. (Contains 27 references, 11 tables, and an appendix of category definitions.) (Author/SLD)

Because the language of a multiple choice test is formal and often unfamiliar, certain linguistic features may lead a test-taker to misconstrue the test instructions, questions, or answers. When this happens, a shared understanding of meaning between tester and test-taker is not present, and the test results are invalid. Although this problem exists for all test-takers, it is more acute for members of non-mainstream speech communities because they are less likely to suspend their normative expectations of discourse. This study was undertaken to identify potential sources of linguistic distortion that might undermine a multiple-choice test's validity for measuring job-relevant knowledge, skills, and abilities among Black English Vernacular speakers in a southern urban setting. Problematic linguistic features identified include: undue syntactic complexity, tense switching, the use of incomplete question stems, plural/possessive ambiguity, and others. Included is a bibliography of significant references on multiple-choice test bias, and a form that can be used when examining test items for sociolinguistic bias. (JL)

The differential performance of black and white college freshmen on a direct measure of writing skills, the essay portion of the American College Testing Program's Collegiate Assessment of Academic Proficiency (CAAP), was studied. The data consisted of responses of a sample of 998 black and 3,727 white examinees to two individual prompts for essays, each requiring 20 minutes of testing time. Each essay was scored by two independent raters. Analysis of the data indicated that black examinees did not perform as well as did white examinees on the essay test. The differences between the populations on the essay were similar in magnitude to differences found on a multiple-choice test of writing skills that also formed part of the CAAP. Results must be interpreted cautiously as the samples, particularly the black sample, were probably atypical. Seven tables present data from the study. An interpretation of the CAAP Writing (Essay) Test Scores is appended. (Author/SLD)

An adaptation of the standardization approach to assessing differential item functioning that applies to all item responses, including omits and not reached, is described. Applications of this method to evaluate differential speededness show that there is evidence of differential speededness for Blacks and Hispanics, but not for Asian Americans. Data from a study by A. P. Schmitt and C. A. Bleistein (1987) and from a recent form of the Scholastic Aptitude Test were used. There may be a dependency between differential speededness and test section location. Differential speededness may be more noticeable when the test section is located at the beginning of a test. Implications of these findings for evaluations of content-related differential item functioning and on differential test-taking strategies are described. (Contains five figures and eight references.) (Author/SLD)

The first of two studies reported examined the factors that predict differences in item responses for black and white matched examinees to analogies on the Graduate Record Examinations (GRE). Data were taken from 13 forms of the GRE Verbal Test, with a median sample size of 21,000 whites and a median sample size of 1,400 blacks for the purpose of computing Differential Item Functioning (DIF) values for each test form. Three factors were found independently to predict differential item responses to 234 GRE analogies: (1) item difficulty; (2) analogy stems with a part/whole relationship; and (3) analogy stems with an "attribute" relationship. A study of 220 Scholastic Aptitude Test (SAT) analogy items found very similar results. For both GRE and SAT analogies, black examinees performed differentially better than matched white examinees on the hard analogy items. Study II used the same forms to explore the importance of item difficulty as a predictor of differential item responses of blacks and whites for other verbal item types (antonyms, sentence completions, and reading comprehension). Item difficulty was found to be an important predictor of the observed differences. Several hypotheses were advanced to account for some of the observed ethnic differences. Appendix A gives sample means and standard deviations. Appendix B lists variable labels and intercorrelations. (Contains 11 tables and 36 references.) (SLD)

This study was conducted to attempt to verify findings from an earlier study by I. M. Lawrence, W. E. Curley, and F. J. McHale (1988) in which differential item functioning (DIF) was examined for females on Reading subscore items from four forms of the Scholastic Aptitude Test-Verbal (SAT-V). The specific focus of this research was on verifying the technical science hypothesis as a contributing factor to DIF for females on science reading comprehension questions and the true-science classification as a contributing factor to DIF for females on sentence completion items. The total male sample for 3 SAT forms was 311,517 and the total female sample was 357,941. Confirmatory evidence for the factors identified in the earlier study is not clear-cut in this study. For reading comprehension items, the presence of technical science material in a reading passage tends to make the corresponding items more difficult for females. However, some items associated with highly technical passages do not function differently for males and females. With respect to sentence completion items, there was partial support for the hypothesis that true-science items are more difficult for females. However, the limited number of items studied does not warrant statements about the differential functioning of sentence completion items. An appendix contains some sample technical science reading comprehension items. (Contains six figures, seven tables, and two references.) (Author/SLD)

A simple technique, developed by A. Phillips (1987) is used to approximate the covariance between the Mantel-Haenszel log-odds-ratio estimator for a 2 x 2 x k table and the sample marginal proportions. These results are then applied to obtain an approximate variance estimate of an adjusted risk difference based on the Mantel-Haenszel odds-ratio estimator. The adjusted risk difference is of potential value in those applications where at least one of the sample rates is descriptive of a relevant population rate. The example applies to the use of the Mantel Haenszel estimator to study the differential difficulty of test questions across groups of examinees. (Contains 11 references.) (Author/SLD)

The Mantel-Haenszel approach for investigating differential item functioning (DIF) was applied to U.S. history items that were administered as part of the National Assessment of Educational Progress (NAEP). DIF analyses were based on the responses of 7,743 students in grade 11. On some items, Blacks, Hispanics, and females performed more poorly than other students, conditional on number-right score. It was hypothesized that this resulted in part from the fact that ethnic and gender groups differed in their exposure to the material included in the assessment. Supplementary Mantel-Haenszel analyses were undertaken in which the number of historical periods studied, as well as score, was used as a conditioning variable. Contrary to expectation, the additional conditioning did not lead to a reduction in the number of DIF items. Both methodological and substantive explanations for this unexpected result were explored. (Contains 12 tables and 4 references.) (Author/SLD)

This review of research studies focused on differences between Hispanic-American and White, non-Hispanic groups on the Scholastic Aptitude Test (SAT). Questions studied were the factors associated with ethnic group mean differences on the SAT; the types of item format or content found differentially easier or more difficult for Hispanics; the predictive validity of the SAT; and the adequacy of Hispanic students' test preparation. Data for the studies were from the 1987 College Board Profiles of College Bound Students and other College Board information. Mean differences between Hispanic and non-Hispanic White students were relatively large and were associated with differences in language background, parental education, high school grades, and type of academic courses taken. The numbers of items showing differential difficulty levels were small and not linked with differences in predictive validity. Overall, tests were slightly less accurate in predicting Hispanic students' success in college than for non-Hispanic Whites. The largest barrier to college access for Hispanic students may be inequity in the availability of guidance counseling. Although evidence concerning the adequacy of college admissions tests for Hispanic students is correlational, data do suggest room for improvement in both tests and test preparation. (Contains 4 tables and 46 references.) (SLD)

Standardization methodology was used to analyze omitted responses of Hispanic examinees on the Scholastic Aptitude Test. Study or focal groups were 2,956 Mexican-Americans, 3,230 Puerto Ricans, and 278,009 White test-takers. Results indicate that both Mexican-Americans and Puerto Rican students omitted fewer items than White students of comparable ability but that Mexican-Americans tended to omit even less. Examination of items with the least differential omit rate by both Hispanic groups indicate that items with true cognates or words with a common root in English and Spanish were omitted differentially less. Some dependency on ability level was observed for the omit patterns of Puerto Rican examinees. (Contains 2 tables, 6 figures, and 18 references.) (Author/SLD)

This study was undertaken to test the hypothesis that items of the Test of English as a Foreign Language (TOEFL) containing reference to American people, places, customs, etc., tend to favor examinees who have spent some time living in the United States. Two samples of examinees were drawn from the March 1987 TOEFL administration, one tested in the United States consisting of 5,799 persons living in the country more than 1 year, and the other consisting of individuals with less than 1 month residence in the United States tested in their native countries (n=21,652 worldwide). Mantel Haenszel analyses were carried out for each of the 146 operational items of the test. In a separate part of the study, five raters were asked to rate test items to determine explicit references to some aspect of America. Ratings were highly reliable, and all but 16 of the 146 items were unanimously judged to have or not to have some reference to Americana. Of the TOEFL items, however, only one gave a consistent advantage, found in every region studied, to persons with U.S. residence experience. No support was provided for the hypothesis that TOEFL items give an advantage to persons who have lived in the United States for some time. Four appendixes contain the distribution of candidates by region, instructions for a pilot study and the formal study, and a display of raters' assignments of Americana to each test item. (Contains 3 figures, 22 tables, and 11 references.) (Author/SLD)

The accuracy of procedures that are used to compare the performance of different groups of examinees on test items obviously depends on the correct classification of members in each examinee group. The significance of this dependence is determined by the sensitivity of the statistical procedure and the proportion of examinees who are unidentified. Since the number of nonrespondents to questions of ethnicity is often of the same order of magnitude as the number of identified members of most minority groups, understanding the effect of nonresponse is crucial to evaluating the validity of a procedure that is used to study differential item functioning (DIF). This study examined the effect of nonresponse to questions of ethnic identity on the measurement of DIF for Scholastic Aptitude Test verbal items using a commonly used modern method, the Mantel-Haenszel procedure (P. W. Holland and D. T. Thayer, 1988). The study considered 94,486 examinees who identified themselves as black or white or did not identify themselves by ethnic group. It was found that efforts to obtain more complete ethnic identifications from the examinees would be rewarded with more accurate DIF analyses. (Contains four tables and eight references.) (Author/SLD)

The stability of bias estimates from J. Schueneman's chi-square method, the transformed Delta method, Rasch's one-parameter residual analysis, and the Mantel-Haenszel procedure, were compared across small and large samples for a data set of 30,000 cases. Bias values for 30 samples were estimated for each method, and means and variances of item bias were computed across all the samples, for comparisons contrasting sample size, sex, and race. The point estimates of item bias, based on 30 replications for each method, were also correlated across random samples, and classification techniques compared the results for agreement. The results showed that none of the methods consistently flagged more or fewer items as biased, though at the larger sample sizes the Mantel-Haenszel and Rasch methods were particularly sensitive at detecting item bias and in high agreement. Reliabilities of the Modified Delta method were generally lower than the others, as were the correlations between Modified Delta and the other indices. The results showed that not until the number of cases in each comparison group reached 1,000 did the reliabilities for any technique approach 0.80. (Contains 5 tables and 22 references. (Author/SLD)