Search

Understanding and accounting for gender performance differences on high stakes examinations has become a particular concern for educational researchers to ensure test fairness for all examinees. In the context of second/foreign language proficiency testing, research suggests that males and females do not react differently at the item level. However, as R. Nandakumar (1993) suggested, items with small but systematic differential item functioning (DIF) may very often go statistically unnoticed, but when combined, they may be detected at the bundle level. Thus a study of differential bundle functioning (DBF) becomes necessary in order to understand more fully the influence of gender on test performance, especially when important, although perhaps subtle, secondary dimensions associated with different testlets have been found in the Test of English as a Foreign Language. In this study of the English Proficiency Test in China, the computer program SIBTEST was used for DIF/DBF analyses and DIMTEST for dimensionality testing. Subjects were 3,160 males and 1,299 females. The results indicate that although the English Proficiency Test did not demonstrate much gender DIF, the SIBTEST and DIMTEST analyses identified and confirmed the presence of the bundle of listening comprehension obviously favoring females and the bundles of grammar and vocabulary and close favoring males slightly. (Contains 3 tables, 2 figures, and 24 references.) (Author/SLD)

This chapter briefly describes some of the methods that test developers and psychometricians have devised to identify item and test bias and some of the challenges they still face. Although it may not be reasonable for classroom teachers to use these methods on a day-to-day basis in constructing tests, the authors argue that it is important for readers to know that these methods are widely used by researchers, professional test developers, and state agencies that develop standardized tests of student achievement. (Contains 12 references.) (GCP)

This study investigated the applicability to computerized adaptive testing (CAT) data of a differential item functioning (DIF) analysis that involves an empirical Bayes (EB) enhancement of the popular Mantel Haenszel (MH) DIF analysis method. The computerized Law School Admission Test (LSAT) assumed for this study was similar to that currently being evaluated for a potential computerized LSAT. In this case, rather than being presented with a single item at a time, test takers are presented with small groups of items, referred to as testlets. The CAT pool for this research consisted of 10 5-item testlets at each of three difficulty levels. The item parameters, which are statistics that describe the various item characteristics such as item difficulty, were specified to resemble those typically observed for the LAST. Using these item-level statistics, responses to the test questions were generated for simulated test takers. These simulations consisted of four conditions that varied in terms of group sample sizes and group ability distributions. Sample sizes for the two test taker groups were either 1,000 or 3,000. The distribution of test taker ability for the two groups was either the same or differed by one standard deviation. Results show the performance of the EB DIF approach to be very promising, even in extremely small samples. The EB estimates tended to be closer to their target values than did ordinary MH statistics; the EB statistics were also more highly correlated with the true DIF values than were the MH statistics. An appendix contains data tables. (Contains 9 figures, 3 tables, and 42 references.) (SLD)

This study investigated the possible impacts of language and curriculum differences on the performance of test items by subpopulations of students. Focusing on Measurement and Geometry items completed by students in French- and English-language schools in Ontario made it possible to explore the differences and to compare the item response theory (IRT) and Mantel-Haenszel (MH) approaches to finding items with differential item functioning (DIF). The tests in this study might have had DIF as a result of translation or curriculum. Data came from the 2001 administration of the School Achievement Indicators Program (SAIP) mathematics assessment. The dataset contained information on 793 13-year-old and 677 16-year-old students from English-language schools and 487 13-year-old and 546 16-year-old students from French-language schools. Thirteen of the 27 items studies were flagged as exhibiting DIF by the MH approach, and 6 of these 13 were also flagged by IRT. Differences in curricula and practice between the French- and English-language schools are very difficult to separate from language differences, and, in fact, results point to the possibility of another source of DIF: different test-taking approaches in the two populations. Results demonstrate the complexity of the factors that contribute to how items are understood and approached by different groups of students. (Contains 5 tables, 7 figures, and 40 references.) (SLD)

This paper studied the usefulness of differential item functioning (DIF) methodology for examining potential mode effects. Although the goal was not to validate the comparability of the assessments per se, it is of interest to speculate why some formats could give rise to differential performance. Data were obtained from two instruments on which validation studies were conducted to determine if scores were comparable across modes of administration. The first instrument was a norm-referenced aptitude test called InView, administered to students in grades 4 to 9 in computer and paper modes for this study. Additional data were obtained from a Test of Adult Basic Education (TABE), also norm-referenced. Test takers were asked about their levels of computer experience and preferences. Two DIF statistics, the Linn-Harnisch procedure (1981) and the nonparametric Standardized Mean Difference were used to assess DIF. Results show that some levels of InView had substantial numbers of items that were flagged for the online comparisons with the standardization study, while others had relatively few. The TABE had two flagged items, and differences between computer-based and standardization groups were throughout the ability range with more apparent differences in the lower portion of the distribution. In general, students were neutral to mode or preferred computer-administered tests. Findings show that DIF methodology presents a well-studied method for examining group differences at the item level that can be used to examine mode of administration differences at the item level. Appendixes contain the student surveys and derivations of the DIF methods. (Contains 10 figures, 4 tables, and 10 references.) (SLD)

Statistical procedures for detecting differential item functioning (DIF) are often used to screen items for construct irrelevant variance. Standard DIF detection procedures focus on only one categorical variables at an aggregated group or one-way level, like gender or ethnicity/race. Building on previous work by P. Hu and N. Dorans (1998), N. Dorans and P. Holland (1993), and Y. Zhang (2001), this study applied a DIF dissection classification scheme to SAT:I Verbal data. Subsequently, the effect of deleting sizable DIF items on reported scores after equipercentile re-equating were explored using data from a spring administration of the SAT for 9,517 test takers in 10 subgroups. By using a "dissection" approach to reference and focal group formations, this two-way classification scheme may yield new and detailed insight into item functioning at the subgroup level. Two hypotheses were studied: (1) whether or not the deletion of sizeable DIF items disadvantageous to a particular subgroup will affect that subgroup the most; and (2) whether or not the effects of item deletion on scores can be predicted by the standardization method. Both hypotheses were predicted by the results of this research. Scaled score differences following item deletion and re-equating varied among subgroups, depending on the DIF effects. Subgroups disadvantaged by the subsequently deleted sizable DIF items gained scaled score points whereas advantaged groups lost. Regression analyses confirmed the second hypothesis. It was also shown that by deleting an item with sizable negative DIF, the focal group might be greatly benefited. Among three item deletion scenarios, DIF effects yielded from the two-way classification scheme showed very little interaction in the majority of cases. (Contains 3 figures, 14 tables, and 15 references.) (Author/SLD)

This study introduced and demonstrated a new methodology for item and test bias studies: moderated differential item functioning (DIF). This technique expands the DIF methodology to incorporate contextual and sociological variables as moderating effects of the DIF. The study explored differential domain functioning (DDF), so that the focus of interpretation for this test is on the "domain" rather than the item. This moderated DDF effect was shown in a multiple choice and constructed response provincial assessment that was designed to match a specific mathematics curriculum. Participants were 45,728 fourth graders, 45,022 seventh graders, and 43,525 tenth graders in British Columbia, Canada. Data were narrowed to create four contrast groups of communities that reflect differences in contextual variables: rural low-income, rural affluent, urban low-income, and urban affluent. Gender DDF was explored using a general liner statistical model. After statistically matching males and females on their mathematical ability, gender DDF was moderated by the contextual variables. Thus, this "moderation" approach allowed the investigation of sociological, community-based contextual variables that may help explain the complex functioning of DIF in large scale testing. The paper advocates a more "sociological" and "ecological" approach to help explain differences in item and test performance. (Contains 22 references.) (Author/SLD)

This study aimed at finding criteria like within-method stability rates or between-method agreement rates that could help to choose a powerful and low-cost differential item function (DIF) detection method. The study tried to verify the within-method stability of item response theory (IRT) based over non-IRT-based procedures in two different cultural contexts. Data from the Third International Mathematics and Science Study for 1995 and 1999 were used to see if the items identified as having translation DIF in 1995 between U.S. and Japanese groups were the same in 1999. Four procedures were used for that purpose: two IRT-based and two non-IRT-based. In each case, absolute and relative criteria were used to classify the strength of the DIF. The study also investigated between-method agreement rates. Results show that the non-IRT-based methods, especially Mantel Haenszel (a low cost method), possessed between-method agreement rates as high as those obtained by IRT-based methods. Also, the stability rates of non-IRT-based methods were found to be very close to the stability rates of IRT-based methods. (Contains 11 tables and 39 references.) (SLD)

This study compared the ability of hierarchical linear modeling (HLM) to detect differential item functioning (DIF) to standard DIF detection methods, such as Rasch difficulty difference. The big advantages to using HLM for DIF detection are that the person abilities so produced are adjusted for any DIF in the items, and the DIF can then be modeled as a function of other predictors at a lower level in the same analysis. Data were simulated, and 180 data sets were produced to compare DIF detection approaches. In most cases, the amount of root mean square error (rmse) for Rasch and HLM is similar, but consistent points of difference occur when the number of people is small and the proportion of people in the focal group is low. In that case, the HLM rmse is larger, and when the number of people is large, the size of the DIF is small, and the proportion of people in the focal group is small, the HLM rmse is smaller. The paper discusses reasons HLM can produce better estimates than conventional methods in some circumstances. (SLD)

A simulation study was conducted to examine the effect of large ability differences using two differential item functioning (DIF) detection procedures, SIBTEST and TESTGRAF. DIF items are hard to identify when group ability differences are large (A. Gotzmann, C. Vandenberghe, and M. Gierl, 2000; R. Hambleton and H. Rogers, 1989). This problem was investigated in the current study for the SIBTEST and TESTGRAF DIF detection procedures. Four ability differences (0.0, -1.0, -1.5, and -2.0) and 8 sample sizes were manipulated in a simulation study. Type I error and power rates were computed. The SIBTEST Type I error rates were inflated at the larger ability differences. Conversely, the TESTGRAF Type I error rates remained low for most ability differences and sample sizes. The SIBTEST power rates remained high, even with larger ability differences. The TESTGRAF power rates dropped as ability differences were introduced. (Contains 4 figures, 11 tables, and 55 references.) (Author/SLD)

The effect of calculator type on student performance on a mathematics examination was studied. Differential item functioning (DIF) methodology was applied to examine group differences (calculator use) on item performance while conditioning on the relevant ability. Other survey questions were developed to ask students the extent to which they used a calculator, the perceived usefulness of the calculator, and how often it was used in the classroom. In addition, content experts were asked to identify whether an item was sensitive to calculator usage, the potential for being useful or distracting, and how the type of calculator would affect student performance. Student test data was obtained from the Tennessee Gateway assessment end-of-course test in Algebra 1. Six forms of the test were spiraled with approximately 7,000 students taking each form. No evidence of pervasive uniform difference in DIF resulting from calculator use was detected in this data. The type of questions in this assessment tended not to be sensitive to differential calculator use and did not impact test performance significantly. DIF was also not evident for students who used calculators versus those who did not. Calculator type, use, and familiarity were associated with differences in the univariate comparison of test scores. For example, students who responded that they used a graphing calculator performed higher than the other groups. Classroom practice and experience with calculators appeared to vary widely. Results from the expert judgment indicate that items with more egregious sensitivity to calculator use could be identified, although experts tended to over-identify items as being sensitive to calculator use. One appendix contains survey items related to calculator use, and the other describes the Linn-Harnisch DIF procedure. (Contains 3 figures, 4 tables, and 15 references.) (SLD)

Maximum usefulness of information provided to teachers and program staff by the enhanced score reports of the Preliminary SAT/National Merit Scholarship Qualifying Test (PSAT/NMSQT) depends on the development of aggregate skill reports; that is, reports that provide skill information aggregated across groups of students, such as grade level within high school. In this study, differential item functioning (DIF) methodology is extended to individual skill performance in order to compare aggregate groups, like schools or states, to the total population matched on an overall score. The results of four DIF detection methods used for this purpose are compared. Mantel-Haenszel and standardization procedures were calculated using binary skill classification data, and polytomous standardization and analysis of covariance (ANCOVA) were calculated using continuous posterior skill mastery probabilities. Results indicate that the ANCOVA approach was very sensitive to differential skill performance between the aggregate group and the population matched on ability. The Mantel-Haenszel, standardization, and polytomous standardization approaches provided consistent results that were less sensitive to differential skill performance. Taken together, these findings suggest that the ANCOVA approach may be too sensitive to differences in skill performance across groups matched on ability. Limitations and directions for future research are discussed. An appendix provides additional information about student skills. (Contains 10 tables and 22 references.) (Author/SLD)

Many students in the United States who are required to take educational tests are not fully proficient in English. To address this problem, a state-mandated testing program created dual language English-Spanish versions of some of their tests. In this study, the psychometric properties of the English and dual language versions of a fourth-grade mathematics test were explored. Analyses of the consistency of test structure across the two forms were conducted using structural equation modeling and multidimensional scaling. Analyses of differential item functioning (DIF) were conducted using Poly-SIBTEST. The results suggest slight structural differences across the two versions of the test. Part of this difference was attributed to overall proficiency differences across the two studied groups, and part was attributed to DIF. The implications of the findings for future research in this area are discussed. (Contains 4 figures, 11 tables, and 28 references.) (Author/SLD)

Holland's Self-Directed Search (SDS) protocols were used with at-risk Hispanic female and male adolescents were examined to determine their sensitivity to this population. The SDS was completed by 89 at-risk Hispanic students (49 females and 33 males) in grade 9 at a nontraditional school for students who have been deemed at risk for gang and domestic violence, substance abuse, and early sexual activity and potentially able to benefit from the school's special programs. The students' summary scores were compared with those of two normative samples in the SDS manual the Hispanic population and the high school population. The SDS protocol scores of the males and the females in the study population differed significantly from those of the students in both normative groups, thus directly supporting previous studies suggesting that Holland's model may not be valid for use with special populations. The study population indicated less preference for most of Holland's themes than did the students in the norm samples. The study population members' order of preference for Holland's themes also differed from that of the students in the norm samples. It was concluded that Holland's model might not be sensitive enough to address the unique hurdles faced by the youth in the study population because of their at-risk status. (Contains 12 references) (MN)

This paper provides an introductory primer on methods for exploring item bias. The purposes of item bias analysis are to investigate whether test scores are affected by different sources of variance in the various subpopulations, and if different sources of variance are found, to determine if an unfair advantage exists. The review discusses three primary methods for detecting item bias: (1) methods based on item response theory (IRT); (2) chi-square methods; and (3) methods based on delta plots. When choosing the best item bias detection methodology, the researcher must consider the application of the results, the availability of software needed, and the practicality of implementing the methodology chosen. The IRT model leads the other item bias methods by being the most theoretically sound, although statistically complex, procedure. Chi-square and delta plot methods are not as theoretically sophisticated yet are more practical and easier to implement. (Contains 3 tables, 3 figures, and 15 references.) (SLD)

One of the most prevalent and controversial accommodations for students with disabilities is an audio presentation of written test material, the "read-aloud" accommodation. This study used test administration data from the Missouri Assessment Program to examine effects of using the read-aloud accommodation on the characteristics of multiple-choice math and reading test items administered to approximately 3200 students in 3rd and 4th grades. Four subject groups were defined: non-accommodated students with no disabilities; non-accommodated students with no disabilities matched in test performance to students with a reading disability; non-accommodated students with a reading disability; and students with a reading disability who took the test with the read-aloud accommodation. Differential item function (DIF) analysis was used to determine numbers of difficult items for each group. Significantly more DIF items were identified for students receiving the read aloud accommodation than for similarly disabled students tested without the accommodation, suggesting that the read-aloud accommodation is ineffective for students with reading disabilities or that a reading test measures the reading construct differently for students with a reading disability than for students without disabilities. Appendices present detailed test results. (Contains 13 references.) (DB)

Computerized adaptive tests (CATs) pose major obstacles to the traditional assessment of differential item functioning (DIF). This paper proposes a modification of the SIBTEST DIF procedure for CATs, called CATSIB. CATSIB matches test takers on estimated ability based on unidimensional item response theory. To control for impact-induced Type I error inflation, the SIBTEST regression correction is shown to have an easily implemented and theoretically justified counterpart in the CAT setting. A simulation study was conducted to evaluate the performance of CATSIB. Simulated test takers were adaptively administered 25 simulated operational items from a pool of 1,000 and were linearly administered 16 simulated pretest items that were evaluated for DIF. The pretest items were designed to represent varying levels of discrimination, difficulty, and amounts of DIF. Sample size varied from 250 to 500 in each group. Simulated levels of impact ranged from 0 to 1 standard deviations difference in mean ability levels. Results show that CATSIB with the regression correction displays impact-induced Type I error inflation. In terms of power, even with as few as 250 test takers in each group, CATSIB had detection rates of 64% or greater for large values of DIF. When sample size was increased to 500 in each group, these power rates increased to more than 90%. CATSIB displayed nearly unbiased estimation under nearly all the simulated conditions. (Contains 7 tables, 1 figure, and 11 references.) (Author/SLD)

The primary purpose of this study was to identify potential sources of gender differential item bias (DIF) in a high school exit examination composed of both selected-response and constructed-response items in the content areas of English, social studies, mathematics, and biology. A secondary purpose was to determine the agreement between the polytomous differential item functioning (DIF) detection methods, the Generalized Mantel-Haenszel (GMH) approach and Poly-Simultaneous Item Bias (Poly-SIB), and their counterparts, the Mantel-Haenszel procedure (MH) and SIB. Data were from four different Alberta Education Diploma Examinations for June and January 1998. The numbers of students that completed each form ranged from 2,328 to 3,386. Results indicate that both GMH and Poly-SIB were comparable to their dichotomous counterparts, MH and SIB, although there were slight differences between MH and GMH. Results about gender DIF support some hypotheses and not others. Males did not outperform females on geometry and mathematical problem solving items. Although more than 50 mathematics items were analyzed, only 8 dichotomous items were flagged. None of the gridded response items were flagged for DIF, and references to stereotypical male or female activities were not identified as DIF items or did not consistently favor one group or the other. While the majority of the dichotomous items favored males, all of the polytomous items favored females. These findings suggest that there may be an item-by-format interaction where females perform better on constructed response items even in measures of quantitative ability. The paper discusses some areas for future research. (Contains 10 tables, 1 figure, and 34 references.) (SLD)

Items on achievement tests are designed to be equivalent in educational testing situations. That is, the information provided to the student is designed to be the same regardless of the ethnicity of the examinee. As a result, students of equal ability would normally be expected to select the same answer regardless of their ethnicity. However, aboriginals and nonaboriginals may use different cognitive processing skills, and consequently may find items differentially easy or difficult depending on which cognitive processing style is elicited by the item. This study used differential bundle functioning (DBF) analyses with cognitive processing theory as the organizing principle to study group differences. Aboriginal and nonaboriginal students were compared on grade-6 mathematics (956 aboriginal and 2,000 nonaboriginal students), science (480 aboriginal and 2,000 nonaboriginal students), and social studies (971 aboriginal and 2,000 nonaboriginal students) achievement tests. Items on these tests were identified by raters trained in cognitive processing theory as eliciting either simultaneous or successive cognitive processing skills. Items were then grouped and tested for DBF using the statistical program SIBTEST. It was hypothesized that differences favoring aboriginals would occur for those bundles eliciting simultaneous processes, while differences favoring nonaboriginals would occur for those bundles eliciting successive processes. When the bundles were tested, performance differences did not favor aboriginals for simultaneous items nor nonaboriginals for successive items. Ability distribution differences between the aboriginal and nonaboriginal samples coupled with small sample sizes for the aboriginal samples were the key limitations of the study. Recommendations for future research are presented. (Contains 2 tables and 32 references.) (Author/SLD)

The purpose of this study was to determine the extent and magnitude of differential item functioning (DIF) between males and females in existing attitudinal data sets. The focus was on the approximate proportion of items that show statistically significant DIF in selected data sets concerning attitude scales, the magnitude of this DIF, and whether the items more often favor males or females. Two methods for detecting DIF, the Mantel-Haenszel (MH) procedure and logistic regression (LR), were used. While more than 70 data sets were used, only 23 met the standards for inclusion in this study. These 23 data sets contained 54 acceptable scales with a total of 42,370 subjects responding to 982 items. Results suggest that these scales may have had more than a few items functioning differentially by gender. Generally DIF, with respect to gender, appeared to be reasonably balanced between items favoring males and those favoring females. However, it cannot be concluded that the combined influence of the items (effect size) was impartial. The magnitude of the DIF was not trivial, being in the range of medium to large. Results also show that both methods of DIF detection yielded very similar results with respect to uniform DIF. Conditions favoring one approach over the other are discussed. An appendix contains brief descriptions of the studies considered. (Contains 1 table and 17 references.) (SLD)

The Mantel-Haenszel (MH) differential item functioning (DIF) parameter for uniform DIF is well defined when item responses follow the two-parameter-logistic (2PPL) item response function (IRF), but not when they follow the three-parameter-logistic (3PL) IRF, the model typically used with multiple choice items. This research report presents a general formulation of the MH DIF population parameter for any IRF and presents results for numerous 3PL uniform DIF conditions. The results indicate that for items of medium or high difficulty, the 2PL DIF parameter formulation can overestimate the 3PL DIF parameter and the MH DIF estimator may exhibit less than expected power to identify even substantial DIF in certain circumstances. Implications of this study on the routine operational task of identifying DIF at the Law School Admission Council are still not known, and may in fact be minimal. However, because some items on the Law School Admission Test (LSAT) are known to exhibit guessing behavior, the results suggest that additional research is warranted. (Contains 2 tables, 5 figures, and 13 references.) (Author/SLD)

Differential item functioning (DIF) analyses are conducted to investigate how items function in various subgroups. The Mantel-Haenszel (MH) DIF statistic is used at the Law School Admission Council and other testing companies. When item functioning can be well-described in terms of a one- or two-parameter logistic item response theory (IRT) model and subgroup differences can be conceptualized as a difference in item difficulty, the MH DIF statistic can be readily understood in terms of the IRT item parameters. When items follow the three-parameter logistic IRT model, however, the relationship between MH DIF statistic and IRT is more complicated, and several competing parameters that relate the statistic to IRT parameters have been proposed. The goal of this study was to investigate various MH DIF parameters to determine which is most appropriate. The parameters were compared with the MH DIF statistic using both simulated and real data from a recent administration of the Law School Admission Test (responses of 20,092 white male test takers). Results suggest that the most appropriate parameter is the one that is theoretically most similar to the MH DIF statistic itself. (Contains 3 figures, 2 tables, and 16 references.) (Author/SLD)

This study was conducted because revisions are being considered to the language questions that currently appear in the Student Descriptive Questionnaire (SDQ) of the Scholastic Assessment Test (SAT). The current test includes two questions about language acquisition: (1) "What language did you learn to speak first (EFL)?" and (2) "What language do you know best (EBL)?" Data were derived from a standard SAT I administration in 1998-1999 at which 192,737 high school juniors and seniors were tested. The study began by considering the effects of using samples derived from the EBL question rather than the current EFL-derived samples for differential item functioning (DIF) analyses. The first analysis was of Mantel Haenszel (MH) FID item statistics, which indicated that additional verbally-loaded mathematics and verbal questions would be flagged as inappropriate if EBL-derived samples were used. SDQ response patterns and scaled score data for several racial/ethnic groups were also examined to see if these data suggested any reasons for concern. Findings suggest that if the EFL question remains as it is now worded, then no change to current SAT I procedures would be necessary. The answers A (English only) and B (English and another language) to the EFL question would continue to define the target population for SAT I DIF analyses. If the EFL questions were dropped from the SDQ, then the analyses in this study would suggest, chiefly as a result of Asian American response patterns, that EBL A is better than EBL A and B to define the target population. However, for the Hispanic group, it would be undesirable to exclude the EBL B group from SAT I DIF analyses. It seems that the EBL question as it currently appears is inappropriate to define the target population for SAT I DIF analyses no matter which responses are used. Some suggestions are offered for additional research. (Contains 11 tables and 4 figures.) (SLD)

Previous studies that investigated the effect of unequal ability distributions on the Type I error (TIE) of the Mantel-Haenszel chi-square test for detecting differential item functioning (DIF) simulated ability distributions that differed only in means. This simulation study suggests that the magnitude of TIE inflation is increased, and the type of items that show inflation are somewhat broadened, under the realistic scenarios of ability distributions that differ not only in means but also in variances or variances alone. There were several conditions for which unequal variances in the ability distributions, either alone, or in combination with unequal means, produced practically important TWI inflation when no such inflation was observed under unequal means alone. This occurred primarily for the hard and medium difficulty items with high discrimination and on the short test with the total sample size of 2,000, but it was also observed for the medium-length test. The most peculiar finding was the aberrant behavior of the highly discriminating hard items, which was the only item to show marked TIE inflation due to unequal variances alone when only rare mild inflation was observed under the combination of unequal variances and unequal means. (Contains 14 tables, 5 figures, and 39 references.) (SLD)

This study assessed the effects of the type of medical curriculum on differential item functioning (DIF) and group differences at the test level in Level 1 of the Comprehensive Osteopathic Medical Licensing Examinations (COMLEX). The study also explored the relationship of the DIF and group differences at the test level. There are generally two types of osteopathic preclinical curriculum: traditional discipline-based and organ system-based. The June 1998 administration of COMLEX was used. This examination involved 800 multiple-choice items and 2,122 students from 17 osteopathic schools. Of these students, 578 studied an organ-system-based curriculum. The study reveals that the directions of the influences of curriculum type were consistent at the test and item levels, but the magnitude of the influences was minimal. At the total examination level, students from the system-based curriculum performed better but with no practical significance. At the item level, the majority of items were immune from the impact of curriculum, while more items of those that had significant DIF favored students in organ-system based curricula. It is concluded that the advantages of students in the system-based curricula on the total examination were due in part to items discriminative against students in the discipline-based curricula. (Contains 2 tables and 11 references.) (SLD)

CATSIB is a differential item functioning (DIF) assessment methodology for computerized adaptive test (CAT) data. Kernel smoothing (KS) is a technique for nonparametric estimation of item response functions. In this study an attempt has been made to develop a more efficient DIF procedure for CAT data, KS-CATSIB, by combining CATSIB with kernel smoothing. A correction for smoothing in boundaries is also implemented. It is hoped that such a methodology could provide a more powerful DIF technique for smaller samples while enhancing the interpretation of local DIF analyses. A simulation study was conducted to investigate the DIF estimation bias of KS-CATSIB in comparison to CATSIB with small samples. Sixteen DIF items varying in difficulty and discrimination were considered for this purpose. A sample of 500 examinees was used in the reference group and a sample of 250 examinees was used in the focal group. Preliminary results show that the correction for smoothing in boundaries, even though effective in reducing the bias in estimation, is still larger for KS-CATSIB in comparison with CATSIB. Therefore, DIF estimates associated with KS-CATSIB are statistically biased and would lead to high Type I error rates. Further modifications of KS-CATSIB are necessary before the program is ready for full implementation. (Contains 3 tables and 13 references.) (Author/SLD)

When a test is translated from a source language to a target language, the result is generally not two psychometrically equivalent tests. If the sources of differential item functioning (DIF) across languages could be predicted, this could have important implications for test adaptations. This study examined item revision as a tool for improving test adaptations. A panel of eight translators and researchers revised the DIF items detected in a previous study of tests administered in Hebrew and Russian (A. Allalouf and others, 1999), based on the reasons for DIF the previous study had identified. The revised items were then administered in six test sessions, and the revised target language items were compared to the original source language items and to the original translation. Results from the 6 sections, each of which included between 5,800 and 7,150 examinees taking the original Hebrew examination, between 1,485 and 2,033 examinees taking the original Russian examination, and between 380 and 840 examinees taking the revised Russian version, show that the revision succeeded in reducing DIF in 27 of the 37 revised items (in 12 to a significant degree). Eight items exhibited increased DIF, which was significant for two items. Results were studied to attempt to determine which sources of DIF and which item types can be revised most effectively. Empirical guidelines for using a panel to reduce DIF are presented. An appendix contains the item revision form. (Contains 8 tables and 14 references.) (SLD)

This study examined gender-based differential item functioning (DIF) on the 10th-grade history achievement test administered as part of the National Education Longitudinal Study of 1988 (NELS:88), which followed 24,599 8th graders into 10th and 12th grades. Several DIF analyses with varying matching criteria were conducted, and the results were supplemented with a survey study that helped validate the interpretations of the underlying causes of DIF. DIF in favor of each gender corresponded to traditional sex role stereotypes; males performed better on "masculine" items, whereas females were advantaged on "feminine" questions. The survey study, conducted with 432 high school juniors and seniors, confirmed that both high school boys and high school girls perceived the items to be sex-typed in the manner predicted by sex-role appropriateness. The findings reveal that the male advantage on this particular test was limited to specific content areas and did not represent a difference in overall proficiency. (Contains 4 tables and 33 references.) (Author/SLD)

The primary objective of this research was to examine the relationship between student achievement in mathematics and pedagogical approach used by middle school mathematics teachers in the United States who participated in the Third International Mathematics and Science Study. In this research, student achievement was explored at the item, rather than test, level with the thought that differences might be found only at this micro level. It was hypothesized that middle school mathematics students whose teachers utilized a more student-centered, or constructivist, pedagogical approach would have a higher probability of obtaining the correct answer to mathematics items that measured conceptual, rather than procedural, understanding. This hypothesis was explicitly tested using differential item functioning analyses. Results support the hypothesis, although not as strongly as had been expected. An appendix contains the teacher survey. (Contains 3 tables, 1 figure, and 10 references.) (Author/SLD)

Data from a 50-item translated test used for certification were used to assess the percentage and type of agreement between the Mantel-Haenszel (MH) and Differential Functioning of Items and Tests (DFIT) techniques for the detection of differential item functioning (DIF). The DFIT procedure flagged 10 of 30 items as exhibiting significant DIF while the MH technique flagged 2 of 30 items for significant DIF. In both methods items were flagged for significant DIF when translation differences appeared in the item stems. The DFIT method was more sensitive in detecting DIF, resulting exclusively from differences in the item answer options. The overall percent agreement between the two techniques for the detection of DIF in this investigation was 20 percent. The MH technique detected 1 of 10 items as exhibiting nonuniform DIF and 1 of 10 items as displaying uniform DIF. The DFIT procedure detected 4 of 10 items as exhibiting nonuniform DIF and 6 of 10 as displaying uniform DIF. Four appendixes contain tables of descriptive statistics and the English and back-translated item versions. (Contains 34 references.) (Author/SLD)

A multilevel approach was proposed for the assessment of differential item functioning and compared with the traditional logistic regression approach. Data from the Comprehensive Osteopathic Medical Licensing Examination for 2,300 freshman osteopathic medical students were analyzed. The multilevel approach used three-level hierarchical generalized linear models. The software HLM for Windows executed the hierarchical linear model (HLM) analysis and the Statistical Analysis System Proc Logistic was used for the conventional logistic regression analysis. It was not surprising to see that HLM was more conservative in identifying DIF in this study. The study demonstrates that the multilevel approach to DIF is meaningful and the results of its use appear reasonable. Implications for use of the two approaches are discussed. (SLD)

Differential item functioning (DIF) may be defined as an item that displays different statistical properties for different groups after the groups are matched on an ability measure. For instance, with binary data, DIF exists when there is a difference in the conditional probabilities of a correct response for two manifest groups. This paper suggests that the occurrence of DIF can be explained by recognizing that the observed data do not reflect a homogeneous population of individuals, but are a mixture of data from multiple latent populations or classes. This conceptualization of DIF hypothesizes that when one observes DIF using the current conceptualization of DIF, it is only to the degree that the manifest groups are represented in the latent classes in different proportions. A Monte Carlo study was conducted to compare various approaches to detecting DIF under this formulation of DIF. Results show that as the latent class proportions became more equal, the DIF detection methods identification rates approached null condition levels. (Contains 6 tables, 3 figures, and 27 references.) (Author/SLD)

New ways are proposed to look at data from the Canadian School Achievement Indicators Program (SAIP) using item response theory (IRT) modeling. The focus is on the traditional test of the SAIP 1997 mathematics study. The test is two-staged in that the first 15 items, of median difficulty, were to be completed beforehand by all students as a "placement" test. Where students were to start the second portion of the test was determined by their scores on the first 15 items. The dimensionality of the scale, the item fit, item bias, and item invariance were considered in applying IRT modeling. The scale obtained reflected the relative ability of the students. Results suggest that mean scale scores can be safely compared between jurisdictions, genders, or age groups. The proposed approach seems reasonably appropriate when a two-stage testing procedure has been used to evaluate students' performance. (Contains 1 table, 5 figures, and 19 references.) (SLD)

An earlier study (D. Koretz, 1997) found that Kentucky had been unusually successful in testing most students with disabilities, but it also found numerous signs of poor measurement, including differential item functioning (DIF) in mathematics, apparently excessive use of accommodations, and implausibly high mean scores for some groups of students with disabilities. This study used newer data to test the stability of the findings over a 2-year period, to extend some of the analyses to additional subject areas, and to compare performance on open-response items to that on multiple-choice items. The researchers analyzed data from students in grades 4, 5, 7, 8, and 11. The inclusiveness of the assessment persisted, and the frequency of special accommodations remained unchanged. The mean performance of elementary school students with disabilities dropped substantially, apparently because of a lessened impact of accommodations on scores. These lower scores appear to be more plausible. The differences in scores between disabled and nondisabled students tended to be larger on multiple-choice components in the early grades. Across grades, the effects of accommodations were stronger on the open-response items. Results provide some evidence that score accommodations change the dimensionality of the assessment. DIF was found in both test formats. Further research and more detailed data are needed to clarify the reasons for the findings. (Contains 19 tables, 15 figures, and 10 references.) (SLD)

This study explores methods for detecting gender-based differential item functioning (DIF) on the 12th grade multiple-choice and constructed-response science tests administered as part of the National Education Longitudinal Study of 1988 (NELS:88). Several combinations of conditioning variables were explored for DIF detection on both tests, and results were supplemented with evidence from interviews of 25 students who completed the test items. On both tests, DIF in favor of males was exhibited primarily on items that involved visualization and that called upon knowledge and experiences acquired outside of school. The findings reveal that neither content nor format alone explained the patterns of male and female performance, and that an investigation of response processes may provide valuable additional information about the nature of gender differences in science achievement. (Contains 7 tables, 4 figures, and 56 references.) (Author/SLD)

This paper provides a review of procedures for detection of differential item functioning (DIF) for item response theory (IRT) and observed score methods for the graded response model. In addition, data from a test anxiety scale were analyzed to examine the congruence among these procedures. Data from Nasser, Takahashi, and Benson (1997) were reanalyzed for purposes of this study. The data were obtained from participants' responses to an Arabic Version of Sarason's (1984) Reactions to Test (RTT) scale. The sample consisted of 421 tenth graders from two Arab high schools in the central district of Israel. Results indicated stronger agreement within IRT methods and within observed score methods than between these two sets of DIF detection methods. A discussion is included focusing on reasons for these similarities and differences. Results of this study can provide useful information about the relationships to expect between various DIF detection methods. (Contains 10 tables and 49 references.) (Author/SLD)

Type I error rates of the likelihood ratio test for the detection of differential item functioning (DIF) in the partial credit model were investigated using simulated data. The partial credit model with four ordered performance levels was used to generate data sets of a 30-item test for samples of 300 and 1,000 simulated examinees. Three different combinations of sample sizes of reference and focal group comparisons were simulated under two different ability matching conditions. One hundred replications of DIF detection comparisons were simulated for each of six conditions. Type I error rates of the likelihood ratio for all six conditions were with theoretically expected values at each of the nominal alpha levels considered. (Contains 25 references.) (Author/SLD)

In many testing situations, differential item functioning (DIF) is a potentially serious problem. It occurs when a test item appears to be easier for one group of examinees than another even after controlling for overall skill level. Differential person functioning (DPF) can occur when "items" can be considered raters and the persons are the objects being rated. This paper introduces the notion of DIF with object-rater data and transposed person-item matrices. The discussion only considers methods for DIF or DPF detection using a binary coding of raters and right-wrong coding of subjects on cognitive tests. Three examples are used: (1) data from a standard setting session of a test of counseling knowledge and skills using an Angoff variation with 9 expert judges or raters; (2) ratings by 14 committee members of 25 research proposals; and (3) data from a mathematics achievement test taken by 384 sixth graders. These examples serve to illustrate the potential utility of DPF, or the application of DIF to rating data and transposed data from cognitive or diagnostic assessment. Notions incorporated in DPF might be especially useful in the area of diagnostic testing. (Contains 4 figures and 17 references.) (SLD)

Conventional two-group differential item functioning (DIF) analysis for dichotomous items is extended to factorial DIF analysis for polytomous items where multiple grouping factors with multiple groups in each are jointly analyzed. By adopting the formulation of general linear models, item parameters across all possible groups are treated as a dependent variable and the factors as independent variables. These item parameters are then reparameterized as a set of grand item parameters and sets of DIF parameters representing main and interaction effects of the factors on the items. Results of simulation studies show that the parameters of the proposed modeling could be satisfactorily recovered. A real data set of 10 polytomous items with 1,924 subjects was analyzed. Applications and implications of the proposed modeling are addressed. (Contains 3 tables, 4 figures, and 18 references.) (Author)

This study examined the reliability of three methods for detecting differential item functioning (DIF) (i.e., the Mantel-Haenszel method, the standardization method, and the logistic regression method) applied to achievement test data. In addition, the study examined the influences of different sources of error variance, including examinee, occasion, and curriculum sampling on the magnitude of the reliability of the different DIF detection methods. Three datasets were assembled from the 1992 spring and fall standardization administration of the Iowa Tests of Basic Skills, and these were manipulated to control for error variance sources. Results indicated that the Mantel-Haenszel and standardization methods were more reliable in detecting DIF than the logistic regression method. The data also indicated that controlling the error variance of curriculum sampling slightly increased the reliability of DIF detection while controlling for error variance due to examinee sampling gives confusing results. (Contains 13 tables, 8 figures, and 12 references.) (Author/SLD)

This paper introduces logistic regression as a viable alternative when the researcher is faced with variables that are not continuous. If one is to use simple regression, the dependent variable must be measured on a continuous scale. In the behavioral sciences, it may not always be appropriate or possible to have a measured dependent variable on a continuous scale. Logistic regression, a technique derived from logit modeling or logit analysis, has been the analysis of choice in many areas of research in this situation. Logistic regression, like the other regression analyses still looks at the relationship between the variables of interest as the core focus of the analysis, but it uses the concept of the odds ratio as its measure of association. An example illustrates the interpretation of coefficients using the odds ratio. Traditionally the chi-square statistic has been used to test for independence between variables. In logistic regression, the likelihood-ratio-chi-squared statistic (G-squared) is important in comparing the observed and expected frequencies of the variable of interest to assess the goodness of fit of the model. Logistic regression has been used in many areas of research in the behavioral sciences, and can be used in detecting differential item functioning. It is a part of statistical software packages, and an example is included of use of this viable and efficient tool for statistical analysis. (Contains 5 tables, 1 figure, and 18 references.) (SLD)

This study compares logistic discriminant function analysis for differential item functioning (DIF) with a technique for the detection of DIF that is based on item response theory rather than the Mantel-Haenszel procedure. In this study, the areas between the two item characteristic curves, also called the item characteristic curve method is compared with the logistic discriminant function analysis method for each item and the entire test. Data from a state-level performance-based assessment program for approximately 16,000 examinees are used. Substantial differences are found between the items identified as having DIF using the item characteristic curve method and the logistic discriminant function method. Possible reasons for these differences are discussed. Since a clear determination about the best method to determine DIF is not apparent, manufactured data should be used in a Monte Carlo study to eliminate design aspects of the assessment that confused these results. Appendixes contain plots of item characteristic curves and illustrative figures. (Contains 7 plots, 7 figures, and 13 references.)(SLD)

The fifth-grade and eighth-grade science items on a state performance assessment were compared for differential item functioning (DIF) due to gender. The grade 5 sample consisted of 8,539 females and 8,029 males and the grade 8 sample consisted of 7,477 females and 7,891 males. A total of 30 fifth grade items and 26 eighth grade items were examined for DIF using the Multilog software package. There was substantial DIF on certain items: in the grade 5 data, four items exhibited the largest DIF. Females performed better on these items, none of which involved using an algorithm. Interestingly three of the four items with large DIF were in the section on salinity, a topic that perhaps is differentially interesting to males and females. In addition, these four items were embedded in real contexts, and each involved considerable writing. Results also show that there is considerably less DIF in the grade 5 science items than in the grade 8 science items. DIF indices for grade 8 range from 0.02 to 0.56, while for grade 5 they range from 0.02 to 0.33. Adolescence, which affects eighth graders more strongly than fifth graders, may emphasize gender differences because of the development of secondary sex characteristics. Implications of these findings are discussed. (Contains 32 references, two tables, and four figures.) (SLD)

The conventional two-group differential item functioning (DIF) analysis is extended to an analysis of variance-like (ANOVA-like) DIF analysis where multiple factors with multiple groups are compared simultaneously. Moreover, DIF is treated as a parameter to be estimated rather than simply a sign to be detected. This proposed approach allows the investigation of the effects of DIF on items more thoroughly. Results of simulation studies show that the parameters of the proposed models were recovered very well. A real data set with 10 dichotomous items was analyzed. Implications and applications are discussed. (Contains 4 tables, 2 figures, and 10 references.) (Author/SLD)