Search

The study utilised a fine-grained diagnostic checklist to assess first-year undergraduates in Hong Kong and evaluated its validity and usefulness for diagnosing academic writing in English. Ten English language instructors marked 472 academic essays with the checklist. They also agreed on a Q-matrix, which specified the relationships among the checklist items and five writing subskills. This conceptual Q-matrix was refined iteratively via fitting a psychometric model (i.e. the reduced reparameterised unified model) to empirical data (i.e. the checklist marks) through the computer program Arpeggio Suite. The final Q-matrix was found to be valid and useful; it had far fewer parameters but greater power to discriminate masters and non-masters of academic writing skills. The study found that the cognitive diagnostic model (CDM)-based skill diagnosis could identify the strengths and weaknesses in the five writing subskills for students across three proficiency levels and could provide richer and finer ...

Most studies on holistic scoring procedures adopt a rater perspective, focusing on raters and textual features; few studies adopt a test-taker perspective. This study investigated test-taker perceptions of impression management strategies for holistically scored essay tests and estimated the extent to which such perceptions can predict essay scores. A total of 886 Chinese test-takers took two essay tests and completed a perception questionnaire, where they rated the importance of impression management strategies versus the target language and writing skills for helping them achieve good essay scores. Four raters marked the essay papers holistically; each essay paper received two independent ratings. Questionnaire items were analysed to verify scale reliability and to explore the underlying structure of each perception scale. The analysis identified two opposing factors underlying test-takers’ perceptions of the impression management strategies, which represented a risk-taking approach and a defensive approach to managing rater impressions. Regression analyses found the defensive approach consistently predicted essay scores, i.e. test-takers assigning higher values to the defensive approach achieved significantly higher scores than test-takers who assigned it lower values. Corpus-based textual analyses found defensive writers writing longer essays and committing fewer linguistic errors, while risk-takers used slightly more sophisticated words and sentences but also committed more errors. The findings were discussed in terms of their implications for validity and washback.

This study utilized Structural Equation Modeling to investigate the washback mechanism, focusing on two design aspects of an English language proficiency test: component weighting (weight assigned to different test papers) and testing methods (item format), and their washback on test preparation. Two months before taking the test, a large sample of test-takers (N = 1000) were surveyed regarding their perceptions of the two design aspects and their test preparation activities. Their official test scores were collected when they were available. Data was analyzed to estimate the washback effects of perceived changes on test-taker time management and approaches to test preparation, and their test performance. The study found that test-takers spent more time on the papers with higher weight and less on those with lower weight. Reporting component scores seemed unable to adjust this tendency. Meanwhile, favorable perception of test validity was associated with a higher level of engagement in both desirable language learning activities and focused test preparation (drilling and cramming). This suggests that favorable perceptions may not be able to reduce negative washback, but may be able to promote positive ones.

This study examined test takers’ perception of assessment demand and its impact on the measurement of intended constructs. More than 800 test takers took a pre- and a posttest of College English Test Band 4 and filled in a perception questionnaire to report the skills they perceive as necessary for answering the test. The study found test takers perceived language skills and test-taking skills as equally necessary. Perception of linguistic knowledge negatively affected performance on the reading test, while perceptions of test-taking skills positively affected performance on the reading test and the essay test. Test takers’ perception of assessment seemed to affect the measurement of the intended construct, although the effects are marginal. The study also found instrumentally oriented test takers and test takers with higher starting English ability tend to perceive the test as more demanding, such perception, in turn, contributed to better test performance.