Rater judgments and word difficulty: Conceptualizing the substantive validity of the VST

The substantive component of construct validity requires a confrontation between empirical test results and content relevance. The Vocabulary Size Test (VST) has been extensively validated in terms of empirical results. Less is known, however, about expert judgments of content relevance. The VST was constructed and validated according to the principle that frequency underlies vocabulary acquisition. This does not mean, however, that the two are equivalent. To better understand the construct of word difficulty as it is measured in VSTs, the results of two Rasch analyses, one on the VST, the other a Multi Facet Rasch Model (MFRM) of a questionnaire distributed to education practitioners were correlated (n = 80, r = 0.67, p < 0.001, 95% CI = [0.53, 0.77]). Semi-structured interviews were then conducted to explore how practitioners understand the concept of word difficulty. Findings indicate that word difficulty is understood to encompass more than frequency, validating recent research into the predictive power of lexical sophistication variables.