Search

Assessment literacy is a crucial aspect of teachers' professional knowledge and relevant to fostering students' learning. Concerning experimentation, teachers have to be able to assess student achievement when students form hypotheses, design experiments, and analyze data. Therefore, teachers need to be familiar with criteria for experimentation as well as student conceptions of experimentation. The present study modeled and measured 495 German pre-service teachers' knowledge of what to assess regarding experimentation competences in biology. We applied an open-answer format for the measurement instrument. For modeling we used item response theory (IRT). We argue that "knowledge of what to assess" regarding experimentation competences is a one-dimensional construct and we provide evidence for the validity of the measurement. Furthermore, we describe qualitative findings of pre-service teachers' "knowledge of what to assess," in particular difficulties concerning the assessment of student conceptions as well as the use of scientific terms in the assessments. We discuss the findings in terms of implications for science teacher education and further research perspectives.

The investigation of learning processes by assessing students' experience along with objective characteristics within a classroom context has a long tradition in empirical learning process research (e.g. Sembill, 1984 et passim; Wild & Krapp, 1996). However, most of the existing studies confine themselves to psychological variables that seem to be too narrowly considered, as there is theoretical and empirical evidence proving the involvement of somatic and psychological processes in learning and in stress reactions. Furthermore there is a lack of studies that investigate situation-related experience (states) as an outcome of interactions between relatively stable characteristics (traits) and continuously changing "objective" context conditions. Against this background, we will present an approach for cross-classified multilevel longitudinal modelling of person-situation interactions in naturalistic educational settings. We illustrated our model with the example of students' stress experience referring to empirical data that we measured within a multidisciplinary research project (pedagogy, psychology, adolescent medicine). 53 students at a public German vocational school were investigated during 9 lessons. There are up to 38 state measurements per person, resulting in 2,014 measurements in total. Taking into account that states are nested within persons and within situations we applied a cross-classified multilevel model to analyse effects on students' stress experience. The analysis shows significant person-situation interactions between academic self-concept and classroom demands and between baseline cortisol concentration and classroom demands: the relation between classroom demands and stress experience depends on relatively stable person-related characteristics. A deeper knowledge about the complex interrelations between traits, states, and continuously changing context conditions seems to be essential for a more holistic understanding of learning at school and for the identification of crucial aspects for an evidence-based design and implementation of teaching and learning arrangements.

This study applied a three-step latent class analysis (LCA) approach to explore latent classes of learning strategy use and their moderation effects on the relationships between motivation and mathematics performance. The data of 15-year-old students from five East Asian educational systems related to Chinese culture in the Programme of International Student Assessment (PISA) in 2012 were analyzed. The findings indicated that Shanghai, Singapore, Taiwan, and Macau showed three latent classes of learning strategies, whereas Hong Kong had two latent classes. Most students in the five educational systems reported to use the control strategy, some students reported the use of combined learning strategies, and few students reported the use of memorization except for students in Shanghai. Furthermore, we found the moderation effects of learning strategy use on mathematics performance depended on the types of motivation and educational systems. This study provides insights into the advantages of a three-step LCA approach in educational research.

Background: After elementary school, students in Germany are separated into different school tracks (i.e., school types) with the aim of creating homogeneous student groups in secondary school. Consequently, the development of students' reading achievement diverges across school types. Findings on this achievement gap have been criticized as depending on the quality of the administered measure. Therefore, the present study examined to what degree differential item functioning affects estimates of the achievement gap in reading competence. Methods: Using data from the German National Educational Panel Study, reading competence was investigated across three timepoints during secondary school: in grades 5, 7, and 9 (N = 7276). First, using the invariance alignment method, measurement invariance across school types was tested. Then, multilevel structural equation models were used to examine whether a lack of measurement invariance between school types affected the results regarding reading development. Results: Our analyses revealed some measurement non-invariant items that did not alter the patterns of competence development found among school types in the longitudinal modeling approach. However, misleading conclusions about the development of reading competence in different school types emerged when the hierarchical data structure (i.e., students being nested in schools) was not taken into account. Conclusions: We assessed the relevance of measurement invariance and accounting for clustering in the context of longitudinal competence measurement. Even though differential item functioning between school types was found for each measurement occasion, taking these differences in item estimates into account did not alter the parallel pattern of reading competence development across German secondary school types. However, ignoring the clustered data structure of students being nested within schools led to an overestimation of the statistical significance of school type effects.

This study examined learning strategy use in mathematics among East Asian students in East Asian educational systems. By employing latent class analysis on the Programme for International Student Assessment (PISA) 2012 data, we found four classes of learning strategy types, namely memorization with metacognitive strategies (17.49%), metacognitive strategies with memorization (50.70%), elaboration only (10.33%), and metacognitive strategies with elaboration (16.47%). The results showed that the majority of the students in all seven East Asian educational systems belonged to the 'metacognitive strategies with memorization' class, and most students adopted more than one type of learning strategy when learning mathematics. Additionally, students who reported the use of metacognitive strategies along with either memorization or elaboration showed higher mathematics achievement. We conclude that the cognitive processes employed by students of East Asian backgrounds are more complex and nuanced than the previous perception that they relied heavily on memorization.

Standard unidimensional Rasch models assume that persons with the same ability parameters are comparable. That is, the same interpretation applies to persons with identical ability estimates as regards the underlying mental processes triggered by the test. However, research in cognitive psychology shows that persons at the same trait level may employ different strategies to arrive at the solutions. This is a major threat to the construct validity of a test since the construct representation of the test changes for different classes of respondents. In this study a reading comprehension test composed of 20 multiple-choice items is analysed with mixed Rasch model. Findings show that a two-class model fits the data best. After investigating class specific item profiles the implications of the study for test validation along with the contribution of the research to our understanding of reading processes are discussed. (Contains 2 tables and 2 figures.)

Measuring and linking competencies require special instruments, special data collection designs, and special statistical models. The measurement instruments are tests or tests forms, which can be used in the following situations: The same test can be given repeatedly; two or more parallel tests forms (i.e., forms intended to be similar in difficulty and content and that still need to equated) can be given at different time points; or two or more test forms that may be less parallel to various degrees can be given at different points in time. In some circumstances, the goal of the analysis is to make the scores of parallel test forms comparable across different time points and different samples of relatively similar ability (horizontal equating). In other situations, we aim at the comparability of scores of tests forms that are not parallel, and although they are intended to measure the same competencies, they are of different difficulties and are taken by samples that show large differences in ability (vertical linking). In other cases, we want to evaluate the change in competencies over time (with or without covariates) for the same individuals measured by the same instrument or by different instruments (longitudinal linking). This paper will briefly discuss a variety of techniques for relating scale scores from different data collections points and will then discuss models for measuring growth. Each of these areas is a large field in itself, and each has a potential strong impact on educational policies such as the No Child Left Behind Act (for example, the vertical linking of state or national assessments and longitudinal studies are a potential basis for informative analyses for the policy makers), on the life of students and parents (equating of achievement tests), or on the life of professionals (equating of licensure tests). In these times when more and more standardized testing is used for assessing competencies in different domains nationally and internationally, we are also discovering more challenges in ensuring that the process and the results are fair and accurate. In turn, these challenges and these new social implications open the door toward more research in support of fair assessments, both in improving upon the test construction process and in advancing the statistical methods involved.

In Germany, the respondents who had participated in the 2012 survey of the Programme for the International Assessment of Adult Competencies (PIAAC) were re-approached for the panel study PIAAC-L. PIAAC-L aims at investigating the longitudinal effects of skill outcomes over the life course and the development of the key skills assessed in PIAAC. Moreover, additional and alternative background information was collected and analyzed within PIAAC-L. PIAAC-L consists of three follow-up waves to the initial PIAAC 2012 survey. The present paper describes the rationale for PIAAC-L and the benefits of conducting a longitudinal PIAAC follow-up study in Germany. In addition, we outline the general design of PIAAC-L and the specific design of the three waves of data collection. Finally, we address the analytic potential of PIAAC-L data set and its availability to the scientific community.

Disengaged responding poses a severe threat to the validity of educational large-scale assessments, because item responses from unmotivated test-takers do not reflect their actual ability. Existing identification approaches rely primarily on item response times, which bears the risk of misclassifying fast engaged or slow disengaged responses. Process data with its rich pool of additional information on the test-taking process could thus be used to improve existing identification approaches. In this study, three process data variables—text reread, item revisit, and answer change—were introduced as potential indicators of response engagement for multiple-choice items in a reading comprehension test. An extended latent class item response model for disengaged responding was developed by including the three new indicators as additional predictors of response engagement. In a sample of 1,932 German university students, the extended model indicated a better model fit than the baseline model, which included item response time as only indicator of response engagement. In the extended model, both item response time and text reread were significant predictors of response engagement. However, graphical analyses revealed no systematic differences in the item and person parameter estimation or item response classification between the models. These results suggest only a marginal improvement of the identification of disengaged responding by the new indicators. Implications of these results for future research on disengaged responding with process data are discussed. [ABSTRACT FROM AUTHOR]

When competence tests are administered, subjects frequently omit items. These missing responses pose a threat to correctly estimating the proficiency level. Newer model-based approaches aim to take nonignorable missing data processes into account by incorporating a latent missing propensity into the measurement model. Two assumptions are typically made when using these models: (1) The missing propensity is unidimensional and (2) the missing propensity and the ability are bivariate normally distributed. These assumptions may, however, be violated in real data sets and could, thus, pose a threat to the validity of this approach. The present study focuses on modeling competencies in various domains, using data from a school sample (N = 15,396) and an adult sample (N = 7,256) from the National Educational Panel Study. Our interest was to investigate whether violations of unidimensionality and the normal distribution assumption severely affect the performance of the model-based approach in terms of differences in ability estimates. We propose a model with a competence dimension, a unidimensional missing propensity and a distributional assumption more flexible than a multivariate normal. Using this model for ability estimation results in different ability estimates compared with a model ignoring missing responses. Implications for ability estimation in large-scale assessments are discussed.

In this paper, we use data from the German PISA 2003 sample to study the effects of central exit examinations on student performance and student attitudes. Unlike earlier studies we use (i) a value-added measure to pin down the effect of central exit exams on learning in the last year before the exam and (ii) separate test scores for mathematical literacy and curriculum-based knowledge. The findings indicate that central exit exams improve curriculum-based knowledge but do not affect mathematical literacy. Students, although showing a better performance, are less intrinsically motivated in school. (Contains 5 tables.)

The aim of the research presented here is the use of extensions of longitudinal item response theory (IRT) models in the analysis and comparison of group-specific growth in large-scale assessments of educational outcomes. A general discrete latent variable model was used to specify and compare two types of multidimensional item-response-theory (MIRT) models for longitudinal data: (a) a model that handles repeated measurements as multiple, correlated variables over time and (b) a model that assumes one common variable over time and additional variables that quantify the change. Using extensions of these MIRT models, we approach the issue of modeling and comparing group-specific growth in observed and unobserved subpopulations. The analyses presented in this paper aim at answering the question whether academic growth is homogeneous across types of schools defined by academic demands and curricular differences. In order to facilitate answering this research question, (a) a model with a single two-dimensional ability distribution was compared to (b) a model assuming multiple populations with potentially different two-dimensional ability distributions based on type of school and to (c) a model that assumes that the observations are sampled from a discrete mixture of (unobserved) populations, allowing for differences across schools with respect to mixing proportions. For this purpose, we specified a hierarchical-mixture distribution variant of the two MIRT models. The latter model, (c), is a growth-mixture MIRT model that allows for variation of the mixing proportions across clusters in a hierarchically organized sample. We applied the proposed models to the PISA-I-Plus data for assessing learning and change across multiple subpopulations. The results of this study support the hypothesis of differential growth.

This paper focuses "interest in science" as one of the attitudinal aspects of scientific literacy. Large-scale data from the Programme for International Student Assessment (PISA) 2006 are analysed in order to describe student interest more precisely. So far the analyses have provided a general indicator of interest, aggregated over all contexts and contents in the science test. With its innovative approach PISA embeds interest items within the cognitive test unit and its contents and contexts. The main difference from conventional interest measures is that in most questionnaires, a relatively small number of interest items cover broad fields of contents and contexts. The science units represent a number of systematically differentiated scientific contexts and contents. The units' stimulus texts allow for concrete descriptions of relevant content aspects, applications, and contexts. In the analyses, multidimensional item response models are applied in order to disentangle student interest. The results indicate that multidimensional models fit the data. A two-dimensional model separating interest into two different "knowledge of science dimensions" described in the PISA science framework is further analysed with respect to gender, performance differences, and country. The findings give a comprehensive description of students' interest in science. The paper deals with methodological problems and describes requirements of the test construction for further assessments. The results are discussed with regard to their significance for science education. (Contains 5 tables and 6 figures.)

The study analyzes the effects of grade repetition on science and mathematics achievement and on self-concept in mathematics using longitudinal data from a representative sample of 9th graders in Germany. Same-age comparisons were applied between three groups: (a) the retained students, (b) a matched group of promoted students, and (c) the entire group of promoted students. The results show that the retained students have a higher self-concept in mathematics after one year of retention than before. Summarizing these empirical findings concerning achievement, we conclude that retention in the 9th grade does not reduce the gap in academic achievement to the regular promoted students. (Contains 5 tables and 3 figures.)

Correct responses to the unitized items of PISA 2006 rely to differing extents on the contextual stimulus supplied. This difference is referred to in this study as the degree of contextualization. A selection of science items from PISA 2006 has been assigned to two categories, not by competencies as in the framework for the PISA survey, but by the degree to which the item requires the ability to extract and apply information from the contextual stimulus provided. The article explores how students in Germany and in other selected OECD countries perform in solving these two types of items. The results show that German students' performance is satisfactory when solving items which require knowledge to be recalled and applied but that they are also quite capable of extracting and using information from the item stimuli. Somewhat different distributions are observed in other selected OECD countries. (Contains 9 tables and 1 figure.)

On a general level, the objective of diagnostic classifications models (DCMs) lies in a classification of individuals regarding multiple latent skills. In this article, the authors show that this objective can be achieved by multidimensional adaptive testing (MAT) as well. The authors discuss whether or not the restricted applicability of DCMs can be overcome by combining DCMs with MAT. The main concepts of MAT are first briefly described, and then the classification of individuals within a MAT-framework is discussed.