Expertise, Domains, and the Consensual Assessment Technique

The Consensual Assessment Technique (CAT) argues that the most valid judgments of the creativity are those of the combined opinions of experts in the field. Yet who exactly qualifies as an expert to evaluate a creative product such as a short story? This study examines both novice and expert judgments of student short fiction. Results indicate a need for caution in using non-expert raters. Although there was only a small (but statistically significant) difference between experts' and novices' mean ratings, the correlation between the two sets of ratings was just 0.71. Experts were also far more consistent in their ratings compared to novices, whose level of inter-rater reliability was potentially problematic. (Contains 1 footnote.)