Search

Admission tests are among the most widespread and effective criteria for student selection in medicine in Germany. As such, the Test for Medical Studies (TMS) and the Hamburg Assessment Test for Medicine, Natural Sciences (HAM-Nat) are two major selection instruments assessing applicants’ discipline-specific knowledge and cognitive abilities. Both are currently administered in a paper-based format and taken by a majority of approximately 40,000 medicine applicants under high-stakes conditions yearly. Computer-based formats have not yet been used in the high-stakes setting, although this format may optimize student selection processes substantially. For an effective transition to computer-based testing, the test formats’ equivalence (i.e., measurement invariance) is an essential prerequisite. The present study examines measurement invariance across test formats for both the TMS and HAM-Nat. Results are derived from a large, representative sample of university applicants in Germany. Measurement invariance was examined via multiple-group confirmatory factor analysis. Analyses demonstrated partial scalar invariance for both admission tests indicating initial evidence of equivalence across test formats. Generalizability of the results is examined, and implications for the transition to computer-based testing are discussed.

Student Evaluation of Teaching (SET) instruments can be administered online or with paper and pencil. These procedures involve different interactions between respondents and instruments which could cause errors when administrators and researchers want to compare or relate online results with paper-based results. This study aimed to analyze if the Romanian version of the Student Evaluations of Educational Quality instrument presents similar psychometric characteristics regardless if it is completed online or by paper and pencil. Data was collected from two groups of students (N = 809). One group completed the SEEQ on paper-pencil support (312 responders, 38.6%). The other group completed the scale for the same teachers and courses, but online and one year later. Psychometrical checks and measurement invariance analysis suggested that the Romanian version of SEEQ has passed the internal validity and reliability test and presented configural, metric, and scalar invariance across the results obtained from both administration procedures. • Nowadays, SET instruments are administered online or by paper-pencil procedures. • The procedures involve different interactions between respondents and instruments. • Checking the measure invariance is necessary when using both SET procedures. • The SEEQ has good invariance indices across data collected with both procedures. • The SEEQ has a good fit for the Romanian population. [ABSTRACT FROM AUTHOR]

This study aimed to use the Alexa vocal assistant as an administerer of psychometric tests, assessing the efficiency and validity of this measurement. A total of 300 participants were administered the Interpersonal Reactivity Index (IRI). After a week, the administration was repeated, but the participants were randomly divided into groups of 100 participants each. In the first, the test was administered by means of a paper version; in the second, the questionnaire was read to the participants in person, and the operator contemporaneously recorded the answers declared by the participants; in the third group, the questionnaire was directly administered by the Alexa voice device, after specific reprogramming. The third group was also administered, as a post-session survey, the Engagement and Perceptions of the Bot Scale (EPVS), a short version of the Communication Styles Inventory (CSI), the Marlowe–Crowne Social Desirability Scale (MCSDS), and an additional six items to measure degrees of concentration, ease, and perceived pressure at the beginning and at the end of the administration. The results confirmed that the IRI did keep measurement invariance within the three conditions. The administration through vocal assistant showed an empathic activation effect significantly superior to the conditions of pencil–paper and operator-in-presence. The results indicated an engagement and positive evaluation of the interactive experience, with reported perceptions of closeness, warmth, competence, and human-likeness associated with higher values of empathetic activation and lower values of personal discomfort. [ABSTRACT FROM AUTHOR]

Measurement invariance holds when a latent construct is measured in the same way across different levels of background variables (continuous or categorical) while controlling for the true value of that construct. Using Monte Carlo simulation, this paper compares the multiple indicators, multiple causes (MIMIC) model and MIMIC-interaction to a novel use of alignment optimization (AO) for detecting measurement noninvariance when the violator is a continuous variable. Results showed that MIMIC and MIMIC-interaction in sequential likelihood ratio tests and Wald tests with a Bonferroni correction provided a good balance between identifying invariant and noninvariant (linear violations) items when n ≥ 500 in terms of classification accuracy (CA). AO (CA ≥ .86) was as competitive as MIMIC and MIMIC-interaction to linear invariance violations but was far better under nonlinear quadratic violations when n ≥ 1,000 (i.e., 100 per group for 10 groups). [ABSTRACT FROM AUTHOR]

Promoting the emotional and social development of students with and without special needs is a central goal of implementing inclusive education in the school system. The entry into school, and thus into the formal education system, is accompanied by emotions and changes in self-image and social relationships. For assessing emotional inclusion, social inclusion, and academic self-concept, the Perceptions of Inclusion Questionnaire (PIQ) is a widely used instrument. To date, the paper-pencil questionnaire has been used from third through ninth grades but has not yet been used with younger ages. This paper presents an adapted version of the PIQ for first-and second-grade students, which was used on two measurement time occasions (T1, N = 407, MAge = 7.2; T2, N = 613, MAge = 7.6). Information on students’ reading and listening comprehension was collected from the class teachers to verify whether the adapted questionnaire can be used for all students with different levels of language competencies. Measurement invariance was demonstrated to be at least scalar for all groups considered in the analyses. Students with higher rankings of reading and listening comprehension skills reported significant higher levels of emotional inclusion and academic self-concept while there were no significant differences in social inclusion. The findings suggest that the PIQ-EARLY is a suitable instrument for assessing self-perceived inclusion in first- and second-grade students. The results also highlight the importance of students’ language competencies for adjustment to school in early school years. [ABSTRACT FROM AUTHOR]

The onset of the coronavirus pandemic forced schools and universities across the nation and world to close and move to distance learning rather immediately. Almost two years later, colleges and universities have reopened, and most students have returned to campuses, but distance learning still occurs at a much higher rate than before the beginning of the pandemic. A few preliminary studies have shown that online assessment results are similar to in‐person assessment results. However, as colleges and universities have expanded their online options, new issues have arisen related to best practices, equity, fairness, test security, and test integrity. Paper‐and‐pencil assessments should not merely be moved to online assessments without careful consideration to how this administration change affects all students. With the switch to more online assessments, educators and researchers should thoroughly evaluate how to provide these assessments in the most secure, fair, and valid manner while also maintaining the test or assessment's integrity and interpretation. [ABSTRACT FROM AUTHOR]

Longitudinal measurement invariance—the consistency of measurement in data collected over time—is a prerequisite for any meaningful inferences of growth patterns. When one or more items measuring the construct of interest show noninvariant measurement properties over time, it leads to biased parameter estimates and inferences on the growth parameters. In this paper, I extend the recently developed alignment-within-confirmatory factor analysis (AwC) technique to adjust for measurement biases for growth models. The proposed AwC method does not require a priori knowledge of noninvariant items and the iterative searching of noninvariant items in typical longitudinal measurement invariance research. Results of a Monte Carlo simulation study comparing AwC with the partial invariance modeling method show that AwC largely reduces biases in growth parameter estimates and gives good control of Type I error rates, especially when the sample size is at least 1,000. It also outperforms the partial invariance method in conditions when all items are noninvariant. However, all methods give biased growth parameter estimates when the proportion of noninvariant parameters is over 25%. Based on the simulation results, I conclude that AO is a viable alternative to the partial invariance method in growth modeling when it is not clear whether longitudinal measurement invariance holds. The current paper also demonstrates AwC in an example modeling neuroticism over three time points using a public data set, which shows how researchers can compute effect size indices for noninvariance in AwC to assess to what degree invariance holds and whether AwC results are trustworthy. [ABSTRACT FROM AUTHOR]

In a recent paper, the first version of the contemporary psychometrics (ConPsy) checklist for assessing measurement tool quality has been published. This checklist aims to provide guidelines and references to researchers to assess measurement properties for newly developed measurement instruments. The ConPsy checklist recommends appropriate statistical methods for measurement instrument evaluation to guide researchers in instrument development and to support peer review. In this opinion article, I critically review some aspects of the checklist and question the usefulness of certain psychometric analyses in research practice. [ABSTRACT FROM AUTHOR]

This paper demonstrates the process of invariance testing in diagnostic classification models in the presence of attribute hierarchies via an extension of the log-linear cognitive diagnosis model (LCDM). This extension allows researchers to test for measurement (item) invariance as well as attribute (structural) invariance simultaneously in a single analysis. The structural model of the LCDM was parameterized as a Bayesian network, which allows attribute hierarchies to be modeled and tested for attribute invariance via a series of latent regression models. We illustrate the steps for carrying out the invariance analyses through an in-depth case study with an empirical dataset and provide JAGS code for carrying out the analysis within the Bayesian framework. The analysis revealed that a subset of the items exhibit partial invariance, and evidence of full invariance was found at the structural level. [ABSTRACT FROM AUTHOR]

This paper aims to examine the psychometric properties of social capital indicators, comparing Black and White respondents to identify the extent of measurement invariance in social capital by race. We used data from the longitudinal study Midlife in the United States (MIDUS), waves 1 through 3 (1995-2016). Data were from 6513 respondents (5604 White and 909 Black respondents). Social capital indicators were social cohesion, contributions to community, and community involvement. We used Structural Equation Modeling and Item Response Theory methods to test for measurement invariance in social capital by race. We observed violations of longitudinal and multi-group measurement invariance (MI) at configural and metric levels on two scales. Factor structures and indicator loadings were inconsistent over time. In IRT analysis, 'Many people come for advice' exhibited Differential Item Functioning (DIF), indicating a consistent advantage for White respondents on the contributions to community scale. Despite similar social capital levels (P(χ2,2) = 0.00), DIF was found in all contributions to community items and some community involvement items when examining race and education interaction. Invariance issues in social capital items suggest potential biases in comparing Black and White respondents. Recognizing these biases is essential. Future social capital research should assess existing data assumptions and involve stakeholders from diverse communities in creating new items. [ABSTRACT FROM AUTHOR]

Copyright of Revista do Serviço Público (Civil Service Review) is the property of Revista do Servico Publico and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)