Search

The replacement of existing technology or the introduction of novel technology into the day-to-day routines of higher education institutions is not a trivial task. Currently, many higher education institutions are faced with the challenge of replacing existing procedures for administering written exams with e-exams. To guide this process, this paper proposes the novel technology-based exams acceptance model (TEAM) and empirically evaluates its model structure and usefulness from the perspective of higher education teachers. The model can be used to guide the transition from paper-based exams to e-exams and the implementation of innovative (e.g., adaptive) e-exam formats. The model includes perceived usefulness, computer self-efficacy, computer anxiety, prior experience, facilitating conditions, and subjective norm as predictors of the behavioral intention to use e-exams. To test the model empirically, the responses of 992 teachers at 63 German universities to a standardized online questionnaire were analyzed using structural equation modeling. The model fit was acceptable. With 77% (conventional e-exams) and 82% (adaptive e-exams), a large proportion of the variance of the intention to use these types of exams was explained. With TEAM, a highly predictive model for explaining the behavioral intention to use e-exams is now available. It offers a theoretical basis that can be used for the successful implementation of e-exams in higher education.

COVID-19 led to school closures and the necessity to use remote learning in 2020 and 2021 around the globe. This article provides results for a three-level random-effects meta-analysis examining the average effect of the COVID-19-related school closures with respect to several moderator variables. The results showed a robust average effect of d=-0.175(SE=0.063, p=0.013, 95% CI[-0.308,-0.041]). The moderator analysis was largely insignificant; however, the results tentatively point out that younger students in schools were more negatively affected compared to older students, and that the negative effect reduced with subsequent lockdowns in autumn and winter 2020/2021. The results are discussed with respect to potential explanations.

Large-scale assessments such as the Programme for International Student Assessment (PISA) have field trials where new survey features are tested for utility in the main survey. Because of resource constraints, there is a trade-off between how much of the sample can be used to test new survey features and how much can be used for the initial item response theory (IRT) scaling. Utilizing real assessment data of the PISA 2015 Science assessment, this article demonstrates that using fixed item parameter calibration (FIPC) in the field trial yields stable item parameter estimates in the initial IRT scaling for samples as small as n = 250 per country. Moreover, the results indicate that for the recovery of the county-specific latent trait distributions, the estimates of the trend items (i.e., the information introduced into the calibration) are crucial. Thus, concerning the country-level sample size of n = 1,950 currently used in the PISA field trial, FIPC is useful for increasing the number of survey features that can be examined during the field trial without the need to increase the total sample size. This enables international large-scale assessments such as PISA to keep up with state-of-the-art developments regarding assessment frameworks, psychometric models, and delivery platform capabilities.

This methods paper describes the methodological and statistical underpinnings of the highly adaptive testing design (HAT), which was developed for the Programme for International Student Assessment (PISA). The aim of HAT is to allow for a maximum of adaptivity in selecting items while taking the constraints of PISA into account with appropriate computer algorithms. HAT combines established methods from the area of computerized adaptive testing (a) to improve item selection when items are nested in units, (b) to make use of the correlation between the dimensions measured, (c) to efficiently accomplish constraint management, (d) to control for item position effects, and (e) to foster students' test-taking experience. The algorithm is implemented using the programming language R and readers are provided with the necessary code. This should facilitate future implementations of the HAT design and inspire other adaptive testing designs that aim to maximize adaptivity while meeting constraints. [ABSTRACT FROM AUTHOR]

This paper focuses on the operation and results of the gas treatment pilot plant for the purification and conditioning of steel mill gases, a key element of the Carbon2Chem® Technical Center in Duisburg, Germany. Using highly sensitive analytics, it is demonstrated that steel mill gas impurities which are detrimental to downstream catalytic conversion are reduced to single‐digit ppb levels. Strategies for adapting the gas purification process to take account of the transformation of steelmaking towards direct reduction are also discussed. [ABSTRACT FROM AUTHOR]

Position effects (PE) cause decreasing probabilities of correct item responses towards the end of a test. We analysed PEs in science, mathematics and reading tests administered in the German extension to the PISA 2006 study with respect to their variability at the student- and school-level. PEs were strongest in reading and weakest in mathematics. Variability in PEs was found at both levels of analysis. PEs were stronger for male students, for students with a migration background (science and mathematics), and for students with a less favourable socio-economic background (reading). At the school level, PEs were stronger in lower school tracks and in schools with a high proportion of students with a migration background. The relationships of the test scores with the covariates partly reflected the covariates' relationships with PEs. Our findings suggest that PEs should be taken seriously in large-scale assessments as they have an undesirable impact on the results.

In the existing multidimensional extensions of the log-normal response time (LNRT) model, the log response times are decomposed into a linear combination of several latent traits. These models are fully compensatory as low levels on traits can be counterbalanced by high levels on other traits. We propose an alternative multidimensional extension of the LNRT model by assuming that the response times can be decomposed into two response time components. Each response time component is generated by a one-dimensional LNRT model with a different latent trait. As the response time components—but not the traits—are related additively, the model is partially compensatory. In a simulation study, we investigate the recovery of the model's parameters. We also investigate whether the fully and the partially compensatory LNRT model can be distinguished empirically. Findings suggest that parameter recovery is good and that the two models can be distinctly identified under certain conditions. The utility of the model in practice is demonstrated with an empirical application. In the empirical application, the partially compensatory model fits better than the fully compensatory model. [ABSTRACT FROM AUTHOR]

Although multidimensional adaptive testing (MAT) has been proven to be highly advantageous with regard to measurement efficiency when several highly correlated dimensions are measured, there are few operational assessments that use MAT. This may be due to issues of constraint management, which is more complex in MAT than it is in unidimensional adaptive testing. Very few studies have examined the performance of existing constraint management methods (CMMs) in MAT. The present article focuses on the effectiveness of two promising heuristic CMMs in MAT for varying levels of imposed constraints and for various correlations between the measured dimensions. Through a simulation study, the multidimensional maximum priority index (MMPI) and multidimensional weighted penalty model (MWPM), as an extension of the weighted penalty model, are examined with regard to measurement precision and constraint violations. The results show that both CMMs are capable of addressing complex constraints in MAT. However, measurement precision losses were found to differ between the MMPI and MWPM. While the MMPI appears to be more suitable for use in assessment situations involving few to a moderate number of constraints, the MWPM should be used when numerous constraints are involved.

The highly adaptive testing (HAT) design is introduced as an alternative test design for the Programme for International Student Assessment (PISA). The principle of HAT is to be as adaptive as possible when selecting items while accounting for PISA's nonstatistical constraints and addressing issues concerning PISA such as item position effects. HAT combines established methods from the field of computerized adaptive testing. It is implemented in R and code is provided. HAT was compared to the PISA 2018 multistage design (MST) in a simulation study based on a factorial design with the independent variables response probability (RP; .50, .62), item pool optimality (PISA 2018, optimal), and ability level (low, medium, high). PISA‐specific conditions regarding sample size, missing responses, and nonstatistical constraints were implemented. HAT clearly outperformed MST regarding test information, RMSE, and constraint management across ability groups but it showed slightly weaker item exposure. Raising RP to .62 did not decrease test information much and is therefore a viable option to foster students’ test‐taking experience with HAT. Test information for HAT was up to three times higher than for MST when using a hypothetical optimal item pool. Summarizing, HAT proved to be a promising and applicable test design for PISA. [ABSTRACT FROM AUTHOR]

The selection of an appropriate booklet design is an important element of large-scale assessments of student achievement. Two design properties that are typically optimized are the "balance" with respect to the positions the items are presented and with respect to the mutual occurrence of pairs of items in the same booklet. The purpose of this study is to investigate the effects of these two design properties on bias and root mean square error of item parameter estimates from the Rasch model. First, position effects are estimated using data from a large-scale assessment study measuring the competencies of 19,107 ninth graders in science. These results were then used for a simulation study with 1,540 booklet designs with systematically varied position balance and cluster pair balance. The simulation results showed a small effect of position balancing on bias and root mean square error of the item parameter estimates while the cluster pair balance was ignorable. This null effect is actually good news for test designers since it allows for deliberately reducing the degree of cluster pair balance without negative effects on item parameter estimates. However, it is recommended to try to achieve a high position balance when designing large-scale assessment studies.

Multiple matrix designs are commonly used in large-scale assessments to distribute test items to students. These designs comprise several booklets, each containing a subset of the complete item pool. Besides reducing the test burden of individual students, using various booklets allows aligning the difficulty of the presented items to the assumed performance level of examined subgroups. While this may improve measurement precision and students' test-taking motivation, using several booklets might influence response behavior and thus constitute a potential source of unwanted variation. To provide guidance to identify and model booklet effects, this study presents statistical models accounting for booklet effects and applies these models in a large-scale assessment setting. Three models are derived from the Rasch model employing the generalized linear mixed models framework. The models were applied to data from a national educational standards assessment study for scientific competence. A total of 1,021 items were compiled to 74 booklets distributed to a sample of 9,044 students of Grades 9 and 10. The results revealed a small but nonnegligible booklet effect. For further large-scale assessment studies, it is recommended to examine whether booklet effects occur and to adequately account for them in the subsequent analyses where necessary.

This study examined students' competencies in engineering education at the university level. First, we developed a competency model in one specific field of engineering: process dynamics and control. Then, the theoretical model was used as a frame to construct test items to measure students' competencies comprehensively. In the empirical application, the reliability of the test was examined. The results confirmed the applicability of the theory-based test. Also, associations between students' competencies and individual characteristics were explored. Students who showed higher competencies could be identified based on their individual characteristics, such as participation in self-assessments. Finally, we demonstrated how to apply the standards of theory-based testing to everyday routines at the university and also highlighted some challenging issues to be considered when implementing theory-based testing in practice. This approach could be valuable: Tests based on theoretically sound models allow for appropriate measurement of students' competencies in higher education.