Quantifying the Quality Difference between L1 and L2 Essays: A Rating Procedure with Bilingual Raters and L1 and L2 Benchmark Essays

It is the consensus that, as a result of the extra constraints placed on working memory, texts written in a second language (L2) are usually of lower quality than texts written in the first language (L1) by the same writer. However, no method is currently available for quantifying the quality difference between L1 and L2 texts. In the present study, we tested a rating procedure for enabling quality judgments of L1 and L2 texts on a single scale. Two main features define this procedure: 1) raters are bilingual or near native users of both the L1 and L2; 2) ratings are performed with L1 and L2 benchmark texts. Direct comparisons of observed L1 and L2 scores are only warranted if the ratings with L1 and L2 benchmarks are parallel tests and if the ratings are reliable. Results showed that both conditions are met. Effect sizes ("Cohen's d") indicate that, while score variances are large, there is a relatively large added L2 effect: in the investigated population, L2 text scores were much lower than L1 text scores. The tested rating procedure is a promising method for cross-national comparisons of writing proficiency. (Contains 9 tables, 1 figure and 3 notes.)