Search

We examine the effects of computer-based versus paper-based assessment of critical thinking skills, adapted from English (in the U.S.) to Chinese. Using data collected based on a random assignment between the two modes in multiple Chinese colleges, we investigate mode effects from multiple perspectives: mean scores, measurement precision, item functioning (i.e. item difficulty and discrimination), response behavior (i.e. test completion and item omission), and user perceptions. Our findings shed light on assessment and item properties that could be the sources of mode effects. At the test level, we find that the computer-based test is more difficult and more speeded than the paper-based test. We speculate that these differences are attributable to the test's structure, its high demands on reading, and test-taking flexibility afforded under the paper testing mode. Item-level evaluation allows us to identify item characteristics that are prone to mode effects, including targeted cognitive skill, response type, and the amount of adaptation between modes. Implications for test design are discussed, and actionable design suggestions are offered with the goal of minimizing mode effect.

Critical thinking is a learning outcome highly valued by higher education institutions and the workforce. The Educational Testing Service (ETS) has designed a next generation assessment, the HEIghten™ critical thinking assessment, to measure students' critical thinking skills in analytical and synthetic dimensions. This paper introduces the theoretical framework that guided the assessment design, and also reports on the preliminary validity evidence of the pilot data from over 3000 students from 35 two and four-year institutions. The critical thinking scores demonstrated satisfactory total and subscale reliabilities, were reasonably correlated with SAT scores, high school grade point average (GPA), and college GPA, and were able to detect cross-sectional performance difference between freshmen and seniors. In addition, most examinees reported having tried their best when taking the test. Results show that test-taking motivation has a significant impact on performance. We encourage institutions to pay attention to motivational issues in implementing low-stakes learning outcomes assessment such as the HEIghten™ critical thinking assessment.

The Commission on the Future of Higher Education in the USA emphasises accountability in higher education as one of its key areas of interest. A programme, called the Voluntary System of Accountability, was developed to evaluate the effectiveness of general public college education. This study examines how students progress in college, indicated by the performance difference between freshmen and seniors after controlling for admission scores, can be measured using the Measure of Academic Proficiency and Progress[TM] (MAPP[TM]) test. A total of 6196 students from 23 institutions was included in this study. Results indicated that MAPP was able to differentiate between the performance of freshmen and seniors after controlling for SAT[R]/ACT scores. The institutions were classified into 10 groups on the basis of the difference in the actual versus expected MAPP performance. The assumptions and implications of linking student learning growth to institutional effectiveness are discussed. Methodological issues on value-added calculation and student motivation in taking standardised tests are also noted. (Contains 3 figures, 5 tables and 1 note.)