Search

Formal achievement data such as test scores and school performance feedback from standardised assessments can be a powerful tool for data-based decision making and school improvement. However, teachers' and school leaders' usage of these data is not necessarily straightforward or predictable. In order to illuminate how educational professionals engage with data in their daily practice, from their own subjective backgrounds and within their own contexts, data use researchers increasingly adopt a sensemaking perspective. Sensemaking, a theoretical construct grounded in psychological and organisational scholarship, offers a framework and a vocabulary to explain how cues such as educational output data are processed in real-life educational settings. As such, sensemaking research sheds light on reasons why educational professionals' use of these formal achievement data may deviate from normative expectations. The present study is a conceptual review of how sensemaking is conceived and applied in literature on educational professionals' use of formal achievement data. In total, 25 empirical and theoretical studies were selected and subjected to thematic analysis. Findings include that sensemaking is used as a lens to study data use, as well as a label for interpretive micro-processes of data analysis and interpretation, and that formal achievement data can be regarded as sensemaking resources. An integrated conceptual framework on educational professionals' sensemaking of formal achievement data is presented, including a discussion of critical insights that may inspire future research on data-based decision making in education.

The present study explores predictors of school performance feedback (SPF) use. In total, 470 Flemish educational professionals were surveyed about their use of SPF from school-external, low-stakes standardized assessments. A path analysis was conducted in order to investigate how individual user beliefs impact SPF use at the school level and how those beliefs mediate the effects of school-level features pertaining to school organization, performance, and voluntariness. Findings include that users' cognitive attitude and perceived expectations of others have a small effect on engagement with SPF in schools, and that these predictors mediate the effects of certain organizational characteristics. Whereas performance levels do not impact school-level feedback use, voluntariness in feedback pursuit and particularly an SPF-oriented school culture emerge as drivers. Implications for practice include the need for stimulating ownership in data-based decision making. Suggestions for further research are also discussed.

In contrast with the assumptions made in standard measurement models used in large-scale assessments, students' performance may change during the test administration. This change can be modeled as a function of item position in case of a test booklet design with item-order manipulations. The present study used an explanatory item response theory (IRT) framework to analyze item position effects in the 2012 European Survey on Language Competences. Consistent item position effects were found for listening but not for reading. More specifically, for a large subset of items, item difficulty decreased along with item position, which is known as a "practice effect." The effect was found across all tested languages, although the effect sizes varied across items, test levels, and countries.

The acquisition of research skills is an important goal of secondary education. Still, insights on what learner characteristics stimulate and hamper research skills are lacking. Therefore, this study investigates the relationship between students' research skills (assessed by means of the Leuven Research Skills Test (LRST) and students' gender, educational track, grade, autonomous and controlled motivation. The LRST was administered to 405 11th and 12th grade students, and to 156 first-year university students in Flanders. Multiple regression analyses reveal a significant relationship of grade and track with students' research skills in upper secondary education. Also in higher education, a significant relationship was found between students' former educational track and their research skills. In both groups, no significant differences were found in research skills based on gender and on students' controlled and autonomous motivation. This study contributes to future research by stressing the importance of curricular characteristics for fostering students' research skills.

According to (inter)national policy and curriculum documents, the acquisition of research skills is an important objective of secondary education. However, the conceptualization and hence the operationalization of this concept seems ambiguous. Furthermore, no test exists to assess students' proficiency in (a broad range of) research skills in a 11th- and 12th-grade behavioral sciences classroom context. This article first elaborates on what constitutes research skills in this educational context. Second, the development and testing process of the Leuven Research Skills Test (LRST) is described. Third, the psychometric properties and the dimensional structure of the LRST are presented, based on a large-scale sample (n = 405) of Belgian students in 11th and 12th grade. The results revealed that (a) the LRST is an internal consistent instrument and that (b) a hierarchical model with eight subordinate factors and one single uniting upper level factor appears to be the best fit to the data (in comparison with a unidimensional model and an eight-factor multidimensional model). It is argued that the LRST can be used to assess (individual differences in) overall research skills proficiency and to investigate the effect of particular interventions to foster research skills in future studies.

Although the development of critical thinking (CT) is a major goal of science education, adequate emphasis has not been given to the measurement of CT skills in specific science domains such as physics. Recognizing that adequately assessing CT implies the assessment of both domain-specific and domain-general CT skills, this study reports on the development and validation of a test designed to measure students' acquisition of CT skills in electricity and magnetism (CTEM). The CTEM items were designed to mirror the structural components of items identified in an existing standardized domain-general CT test, and targeted content from an introductory Electricity and Magnetism (E&M) course. A preliminary version of the CTEM test was initially piloted on three groups of samples: interviews with physics experts (N = 3), student cognitive interviews (N = 6), and small-scale paper and pencil administration (N = 19). Modifications were made afterwards and the test was administered to a different group of second-year students whose major was mechanical engineering (N = 45). The results showed that the internal consistency (Cronbach's a = 0.72) and inter-rater reliability (Cohen's kappa = 0.83) of the CTEM test are acceptable. The findings overall suggest that the CTEM test can be used to measure the acquisition of domain-specific CT skills in E&M, and a good basis for future empirical research that focuses on the integration of CT skills within specific subject matter instruction. A broader CT assessment framework is proposed and possible research questions that can be addressed through the CTEM test are discussed.

This study discusses the self-assessments of over 22,500 students on a set of 12 can-do statements that are taken from the Common European Framework of Reference (CEFR). The students' endorsement or negation of the statements reflects their sense of self-efficacy in English reading, writing, and listening. Their self-efficacy beliefs are compared to their proficiency level as measured in the European Survey on Language Competences. As expected, four-level mixed-effects logistic regression analyses show positive correlations between proficiency and the probability of endorsing the can-do statements. However, patterns in this relationship are influenced by the students' gender and country. As such, the study shows how far the understanding of 15-year-olds' own skills can vary across countries and between boys and girls, even when the skill descriptions that the students are offered rely on the highly concrete "common language" that is provided by the CEFR.

While understanding rational numbers forms an essential part of mathematical literacy, research has repeatedly shown that they form a stumbling block in education. A big source of difficulty is the inappropriate application of natural number knowledge. The research literature points at three main aspects where natural number knowledge is inappropriately applied: density, operations, and size. While there are numerous test instruments to assess each of these aspects, none of them combines all aspects in an integrated manner. By creating such a comprehensive test instrument and applying it to fourth graders, we found that the strength of the natural number bias was largest in density items, but still clearly present in items about size and operations. We further found that fourth graders' rational number reasoning--more specifically their ability to inhibit the inappropriate use of natural number knowledge--can be described by a one-dimensional construct.

According to the big-fish-little-pond (BFLP) model, the self-concept is not only influenced in a positive way by one's own achievement, but also in a negative way by one's relative achievement in comparison with one's immediate peers. This study investigates whether the BFLP effect also holds for second language acquisition. A random sample of 9167 Dutch-speaking students from Grade 8 took a reading comprehension test in French followed by a short background questionnaire in which they were asked for their self-concept of reading French. Multilevel regressions of the latter on individual and group achievement confirmed the BFLP effect but did not confirm an interaction of the effect with gender, individual achievement or class size. Moreover, in line with the local dominance hypothesis, it was shown that the negative effect of the average performance of peers is stronger for classmates than for schoolmates.

In this study it is investigated to what extent contextualized and non-contextualized mathematics test items have a differential impact on examinee effort. Mixture item response theory (IRT) models are applied to two subsets of items from a national assessment on mathematics in the second grade of the pre-vocational track in secondary education in Flanders. One subset focused on elementary arithmetic and consisted of non-contextualized items. Another subset of contextualized items focused on the application of arithmetic in authentic problem-solving situations. Results indicate that differential performance on the subsets is to a large extent due to test effort. The non-contextualized items appear to be much more susceptible to low examinee effort in low-stakes testing situations. However, subgroups of students can be found with regard to the extent to which they show low effort. One can distinguish a compliant, an underachieving, and a dropout group. Group membership is also linked to relevant background characteristics.

This study investigates willingness to communicate (WTC) and its determinants through structural equation modelling (SEM). Building on models by MacIntyre and Charos (1996) and Yashima (2002), it addresses 3 apparent gaps in the current knowledge base: It is the first SEM-based WTC study in a Western European context, investigating French as a second language in a representative sample of over 1,000 grade 12 students in Flanders, the Dutch-speaking part of Belgium; it is among the few studies that compare classroom WTC with WTC outside the classroom; and it investigates the underexplored relationship between WTC and language proficiency through extensive standardized listening tests. Among the study's findings are that classroom WTC is a strong predictor of WTC outside the classroom; however, in naturalistic settings the role of integrativeness decreases while anxiety levels play a larger role. The study provides new support for several existing suggestions for L2 pedagogy.

In this article, the change in examinee effort during an assessment, which we will refer to as persistence, is modeled as an effect of item position. A multilevel extension is proposed to analyze hierarchically structured data and decompose the individual differences in persistence. Data from the 2009 Program of International Student Achievement (PISA) reading assessment from N = 467,819 students from 65 countries are analyzed with the proposed model, and the results are compared across countries. A decrease in examinee effort during the PISA reading assessment was found consistently across countries, with individual differences within and between schools. Both the decrease and the individual differences are more pronounced in lower performing countries. Within schools, persistence is slightly negatively correlated with reading ability; but at the school level, this correlation is positive in most countries. The results of our analyses indicate that it is important to model and control examinee effort in low-stakes assessments.

Objective: The study examined the concordance among rating sources on attention problems of elementary school-aged children. Method: A randomly selected sample ("N" = 841) of children was rated by the physical educators, the teachers, and the parents, using the Attention Scales of the Motor Behavior Checklist (MBC), the Teacher Report Form, the Child Behavior Checklist, and the ADHD-Rating Scale-IV (ADHD-RS-IV). Results: Convergent validity of the Lack of Attention Scale of the MBC with the corresponding subscales was supported. Correlations were higher between teachers' ratings and between physical educators' and teachers' ratings than between physical educators' and parents' ratings or between teachers' and parents' ratings. Conclusion: Findings underscore the importance of taking the child's settings and observer influences into account and suggest that MBC is a new promising instrument for screening attention problems in school settings. (Contains 2 tables.)

Changing the order of items between alternate test forms to prevent copying and to enhance test security is a common practice in achievement testing. However, these changes in item order may affect item and test characteristics. Several procedures have been proposed for studying these item-order effects. The present study explores the use of descriptive and explanatory models from item response theory for detecting and modeling these effects in a one-step procedure. The framework also allows for consideration of the impact of individual differences in position effect on item difficulty. A simulation was conducted to investigate the impact of a position effect on parameter recovery in a Rasch model. As an illustration, the framework was applied to a listening comprehension test for French as a foreign language and to data from the PISA 2006 assessment. (Contains 6 tables and 3 figures.)

The study examines the agreement among raters on children's problematic behaviors. A multitrait-multimethod (MTMM) matrix was applied to a normative sample of elementary school-aged children (N = 841). The participants were rated by their physical educators, using the Motor Behavior Checklist for children (MBC; Efstratopoulou, Janssen, & Simons, 2012). Teachers and parents rated the same students using the Teacher Report Form (TRF; Achenbach, 1991b), the Child Behavior Checklist (CBCL; Achenbach, 1991a) and the ADHD Rating Scale-IV (DuPaul, Power, Anastopoulos, & Reid, 1998). The resulting matrix revealed significant correlations for the Rules Breaking, Lack of Attention, Hyperactivity/Impulsivity, Lack of Social interaction problem scale and for the Internalizing, Externalizing and Total scores. Convergent validity of the specific MBC subscales was supported by significant correlations with the corresponding subscales of TRF, CBCL and ADHD Rating Scale-IV. Findings underscore the importance of taking child's settings and observer influences into account and suggest that MBC is a new promising instrument that can provide valid ratings on externalizing behavior and social problems in children when used by physical educators in school settings. (Contains 3 tables.)

The study was designed to investigate the discriminant validity of the Motor Behavior Checklist (MBC) for distinguishing four group of children independently classified with Attention-Deficit/Hyperactivity Disorder, (ADHD; N = 22), Conduct Disorder (CD; N = 17), Learning Disabilities (LD; N = 24) and Autistic Spectrum Disorders (ASD; N = 20). Physical education teachers used the MBC for children to rate their pupils based on their motor related behaviors. A multivariate analysis revealed significant differences among the groups on different problem scales. The results indicated that the MBC for children may be effective in discriminating children with similar disruptive behaviors (e.g., ADHD, CD) and autistic disorders, based on their motor behavior characteristics, but not children with Learning Disabilities (LD), when used by physical education teachers in school settings. (Contains 2 tables.)

The distinction between quantitative and qualitative differences in mastery is essential when monitoring student progress and is crucial for instructional interventions to deal with learning difficulties. Mixture item response theory (IRT) models can provide a convenient way to make the distinction between quantitative and qualitative differences in mastery. The use of latent groups, rather than focusing on manifest groupings like gender or grade, in these models is very informative to give a substantive interpretation to the qualitative differences. In the current study, mixture IRT modeling is applied to the mastery of two crucial rules in vowel duration spelling in Dutch by pupils in the four final grades of primary school. Results indicate that differences in mastery of the spelling rules are not strictly quantitative. Three latent groups of pupils can be distinguished that show qualitative differences in the mastery of one of the crucial spelling rules involved. (Contains 3 tables and 1 figure.)

A large-scale paper-and-pencil assessment of the attainment targets of environmental studies with a focus on the subject area nature was held in primary education in Flanders (Belgium). The tests on different subfields of nature, i.e. the human body, healthcare, organisms, ecosystems, environmental care and non-living nature, were administered to a representative sample of 4556 pupils of the sixth grade from 145 different schools. The percentage of students mastering the attainment targets differed clearly across the tested subfields. Moreover, gender differences were found. Multilevel analyses revealed that 13% of the total variance of the performance on an overall scale could be attributed to differences between schools. For 82%, these school differences were accounted for by factors mainly referring to the language spoken at home and to the socio-economic background of the pupils. Differences among pupils within schools were explained only to a small extent. (Contains 1 figure and 5 tables.)

This longitudinal study examined the relationship between working memory and individual differences in mathematics. Working memory measures, comprising the phonological loop, the visuospatial sketchpad, and the central executive, were administered at the start of first grade. Mathematics achievement was assessed 4 months later (at the middle of first grade) and 1 year later (at the start of second grade). Working memory was significantly related to mathematics achievement in both grades, showing that working memory clearly "predicts" later mathematics achievement. The central executive was a unique predictor of both first- and second-grade mathematics achievement. There were age-related differences with regard to the contribution of the slave systems to mathematics performance; the visuospatial sketchpad was a unique predictor of first-grade, but not second-grade, mathematics achievement, whereas the phonological loop emerged as a unique predictor of second-grade, but not first-grade, mathematics achievement. (Contains 6 tables.)

This study introduces an approach for modeling multidimensional response data with construct-relevant group and domain factors. The item level parameter estimation process is extended to incorporate the refined effects of test dimension and group factors. Differences in item performances over groups are evaluated, distinguishing two levels of differential item functioning (DIF): a domain level and an item level. An illustration is presented using a Dutch spelling proficiency scale administered to two subgroups. DIF is modeled by the interaction between group and item domain (domain level DIF) and by the interaction between groups and items within each domain (item level DIF). A set of item response theory models was estimated using an adaptation of the logistic regression approach. The model with domain specific item-by-group interactions or DIF performed better than the other models neglecting domain or group differences. The method appears to be promising in that explicit domain factors can be implemented into model estimation procedure to better understand why items favor a specific language group over another. (Contains 3 tables and 1 figure.)

Essential for the validity of the judgments in a standard-setting study is that they follow the implicit task assumptions. In the Angoff method, judgments are assumed to be inversely related to the difficulty of the items; contrasting-groups judgments are assumed to be positively related to the ability of the students. In the present study, judgments from both procedures were modeled with a random-effects probit regression model. The Angoff judgments showed a weaker link with the position of the items on the latent scale than the contrasting-groups judgments with the position of the students. Hence, in the specific context of the study, the contrasting-groups judgments were more aligned with the underlying assumptions of the method than the Angoff judgments.

Performed a baseline assessment of the new curriculum standards of mathematics for elementary education in Flanders, studying the achievement of 5,763 sixth graders. Overall, the more innovative curriculum standards were less likely to be well mastered. Few gender differences in performance were observed. (SLD)

Speeded performance on simple mental addition problems of 6- and 7-year-olds with and without mild mental retardation was modeled from a person perspective and an item perspective, both inferred from Siegler's work. Models from item response theory were used to test hypotheses. Found that all children follow same developmental path in acquiring automated performance on simple mental addition problems. (EV)

Studied the componential structure of synonym tasks using confirmatory multidimensional two-parameter item-response theory models. Reanalysis of data from 218 and 258 Dutch secondary school students from a previous study (R. Janssen and P. De Boeck, 1997) supports the componential hypothesis. Discusses issues related to one of the subtasks studied. (SLD)