Back to Search Start Over

How Much g Is in the Distractor? Re-Thinking Item-Analysis of Multiple-Choice Items

Authors :
Boris Forthmann
Natalie Förster
Birgit Schütze
Karin Hebbecker
Janis Flessner
Martin T. Peters
Elmar Souvignier
Source :
Journal of Intelligence, Vol 8, Iss 1, p 11 (2020)
Publication Year :
2020
Publisher :
MDPI AG, 2020.

Abstract

Distractors might display discriminatory power with respect to the construct of interest (e.g., intelligence), which was shown in recent applications of nested logit models to the short-form of Raven’s progressive matrices and other reasoning tests. In this vein, a simulation study was carried out to examine two effect size measures (i.e., a variant of Cohen’s ω and the canonical correlation RCC) for their potential to detect distractors with ability-related discriminatory power. The simulation design was adopted to item selection scenarios relying on rather small sample sizes (e.g., N = 100 or N = 200). Both suggested effect size measures (Cohen’s ω only when based on two ability groups) yielded acceptable to conservative type-I-error rates, whereas, the canonical correlation outperformed Cohen’s ω in terms of empirical power. The simulation results further suggest that an effect size threshold of 0.30 is more appropriate as compared to more lenient (0.10) or stricter thresholds (0.50). The suggested item-analysis procedure is illustrated with an analysis of twelve Raven’s progressive matrices items in a sample of N = 499 participants. Finally, strategies for item selection for cognitive ability tests with the goal of scaling by means of nested logit models are discussed.

Details

Language :
English
ISSN :
20793200
Volume :
8
Issue :
1
Database :
Directory of Open Access Journals
Journal :
Journal of Intelligence
Publication Type :
Academic Journal
Accession number :
edsdoj.14d3c6b534aa9ba9b77d36d6930dc
Document Type :
article
Full Text :
https://doi.org/10.3390/jintelligence8010011