An Empirical Bayes Enhancement of Mantel-Haenszel DIF Analysis for Computer-Adaptive Tests. LSAC Research Report Series.

This study investigated the applicability to computerized adaptive testing (CAT) data of a differential item functioning (DIF) analysis that involves an empirical Bayes (EB) enhancement of the popular Mantel Haenszel (MH) DIF analysis method. The computerized Law School Admission Test (LSAT) assumed for this study was similar to that currently being evaluated for a potential computerized LSAT. In this case, rather than being presented with a single item at a time, test takers are presented with small groups of items, referred to as testlets. The CAT pool for this research consisted of 10 5-item testlets at each of three difficulty levels. The item parameters, which are statistics that describe the various item characteristics such as item difficulty, were specified to resemble those typically observed for the LAST. Using these item-level statistics, responses to the test questions were generated for simulated test takers. These simulations consisted of four conditions that varied in terms of group sample sizes and group ability distributions. Sample sizes for the two test taker groups were either 1,000 or 3,000. The distribution of test taker ability for the two groups was either the same or differed by one standard deviation. Results show the performance of the EB DIF approach to be very promising, even in extremely small samples. The EB estimates tended to be closer to their target values than did ordinary MH statistics; the EB statistics were also more highly correlated with the true DIF values than were the MH statistics. An appendix contains data tables. (Contains 9 figures, 3 tables, and 42 references.) (SLD)