Search

In this dissertation, we examine how teachers and school leaders make sense and make use of school performance feedback (SPF) from external standardized assessments, and we explore factors that promote or hinder these processes. The conceptual framework is rooted in research on data-based decision making in education, SPF systems, and score reporting. We also incorporated perspectives from social and cognitive psychology, such as attribution theory, the theory of planned behavior, and, particularly, sensemaking. Theoretical insights are presented, as well as findings from original empirical research that was conducted within the context of the Flemish national assessments and parallel tests. The different studies highlight that making sense and making use of educational data in general, and SPF in particular, is not a rational, linear, predictable endeavor. Findings confirm that the mere availability of data such as SPF does not necessarily drive data-driven or data-based decision making in schools, and illustrate that SPF use is no one-size-fits-all phenomenon. Implications for policy and practice include that more emphasis should be placed on data cultures and collective sensemaking, on (the balance between) ownership and clear expectations regarding data use, and on the provision of sufficient cues in SPF to aid sensemaking. By involving users and taking into account actual user interpretations to a greater extent, (the delivery of) SPF can be improved. Such measures are necessary to ensure that SPF lives up to its potential: to effectively inform educational decisions and truly contribute to school improvement. By examining how teachers and school leaders engage with SPF from external standardized assessments, this dissertation is at the nexus of educational effectiveness and school improvement research.

nrpages: 452 status: published

Methodology and content ideally go hand in hand in research. However, often there is a lack of communication between the worlds of ‘methodologists’ and ‘substantive researchers’, which is impoverishing both worlds. This is no different in educational research. In the current project an attempt is made to integrate developments in the field of psychometric research into ‘real-life’ modeling issues in educational research, hence combining methodology and substantive research. The use of multilevel models in educational research is fairly widespread. This is predominantly so in educational effectiveness research. Multilevel models are crucial to account for the nested structure of educational data. They help educational researchers in looking for explanations for differences in educational outcomes, both cognitive and affective. However, in this explanatory approach the quality of measures often seems to be of secondary importance. In the domain of psychometrics on the other hand providing accurate measures is the central topic. However, this descriptive measurement approach often comes to a halt when the explaining commences. Throughout the studies in the current PhD project it will be illustrated that an explanatory and measurement approach intertwine and share a common foundation. In the present project the commonly applied multilevel models and item response theory models are introduced with reference to one common framework. This framework is that of generalized linear mixed models. By referring to this framework, it becomes clear that models in explanatory educational research and psychometrics have considerable commonalities. The framework also makes it more straightforward to extend the models and to accommodate specific research questions more easily. The PhD project consists of four studies. In a first study a more explanatory approach will be adopted in an investigation of the labor market outcomes of graduates from Italian universities. This study will adopt a strategy that is common in school effectiveness research and that is based on multilevel modeling. A second study addresses a crucial topic in national assessments of educational outcomes: the standard setting. An alternative empirical criterion to evaluate the performance of different standard-setting methods is introduced. The distinction between quantitative and qualitative differences in test performance is the central theme of the third study. A modeling approach that combines elements of latent class analysis and elements of item response theory is shown to be able to reveal this kind of differences in data from a spelling test in primary education. Finally, a fourth study concerns the modeling of data stemming from multiple choice items. These items might elicit guessing behavior from the test takers and this needs to be taken into account when modeling the data. A model is presented that can accommodate guessing behavior and simultaneously can be used to explain the occurrence of guessing behavior. This model will be applied to data from a Flemish national assessment. status: published

nrpages: 86 p. status: published