Search

To ensure that all children are on positive developmental trajectories, early education professionals need scientifically-based assessments of children's growth and development. Such assessments meet established standards for measurement and therefore support valid and reliable inferences about children's functioning. This study used the Mental Measurement Yearbook to provide a comprehensive examination of the degree to which early childhood assessment developers report evidence of validity and reliability in line with the Standards for Educational and Psychological Testing. The Mental Measurement Yearbook provides the most comprehensive and continuous examination of assessment developers' development and validation practices. Findings from this study suggest that only a small percentage of the 404 early childhood assessment reviews in the MMY provide evidence on all of the critical sources of validity outlined in the Standards. Response processes and consequences of testing are the most neglected sources of validity evidence. Implications of the findings from this study for early childhood assessment development, validation, and use are discussed.

This study provided validity evidence of the Questionnaire of English Writing Self-Efficacy (QEWSE) based on the five sources of validity evidence described in the Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014). Items of this measure demonstrated close alignment with the linguistic domain and the writing self-efficacy and self-regulation theories, providing validity evidence based on test content. Confirmatory factor analysis revealed a satisfactory global fit of the four-factor (ideation, skills, use, and self-regulation) model to the data, providing internal structure-based evidence. Significant correlations between self-efficacy ratings, writing proficiency, and writing self-regulated learning strategies supported the validity evidence based on relations to other variables. The reliability evidence was evaluated with consistencies of participants’ responses to items and further supported using Item Response Theory (IRT). IRT analyses suggested that measurement precision was greatest at the lower range of the latent trait scale, which provided justification for the primary use of the instrument to identify students with low writing self-efficacy.

Learners’ difficulties with proof have been ascribed to their lack of understanding of functions that proof performs in mathematics, namely, verification, explanation, communication, discovery, and systematization. However, the extant mathematics education literature on validation of instruments designed to measure learners’ beliefs about the functions of proof is scant. The purpose of this mixed methods study was to use the Standards for Educational and Psychological Testing (as discussed by the American Educational Research Association/American Psychological Association/National Council on Measurement in Education (AERA/APA/NCME), 2014) as a theoretical and methodological platform to support arguments for the validity of the learners’ beliefs about the functions of proof (BAFP) instrument. Scale items were generated from de Villiers’ model and a review of the literature, panel of experts, and key informants. The resulting instrument, comprising 28 Likert-scale items and 5 open questions that assessed the five functions of proof, was administered to 87 grade 11 learners in one high school in the Pinetown Education District, KwaZulu-Natal, South Africa. The results put the validity of the scores on the BAFP instrument into question. The study was an exploration of validity arguments based on the evidence from various data sources. Confirmatory factor analysis with a larger sample in a different context needs to be conducted to warrant the use of the BAFP instrument. The key contribution of this study to the field is that it sheds light on the complexities of instrument validation; the scope of the effort may explain why comprehensive validation efforts have not been documented extensively in literature.

The past 100 years have witnessed an evolution of the meaning of validity and validation within the fields of education and psychology. Validity was once viewed as a property of tests and scales, but is now viewed as the extent to which theory and evidence support proposed interpretations and uses of test scores. Uncertainty about what types of validity evidence were needed motivated the current “argument-based” approach, as reflected in the 2014 Standards for Educational and Psychological Testing. According to this approach, investigators should delineate the assumptions required in order for a proposed interpretation or use to be plausible and then seek evidence that supports or refutes those assumptions. Though validation practices within the field of patient-reported outcome measurement have implicitly included many elements of the argument-based approach, the approach has yet to be explicitly adopted. To facilitate adoption, this article proposes an initial set of assumptions that might be included in most arguments for research-related interpretations and uses of scores from patient-reported outcome measures. The article also includes brief descriptions of the types of evidence that would be best suited for evaluating each assumption. It is hoped that these generic assumptions will stimulate further discussion and debate among quality of life researchers regarding how best to adopt modern validity theory to patient-reported outcome measures.

For this study, researchers critically reviewed documents pertaining to the highest profile of the 15 teacher evaluation lawsuits that occurred throughout the U.S. as pertaining to the use of student test scores to evaluate teachers. In New Mexico, teacher plaintiffs contested how they were being evaluated and held accountable using a homegrown value-added model (VAM) to hold them accountable for their students’ test scores. Researchers examined court documents using six key measurement concepts (i.e., reliability, validity [i.e., convergent-related evidence], potential for bias, fairness, transparency, and consequential validity) defined by the Standards for Educational and Psychological Testing and found evidence of issues within both the court documents as well as the statistical analyses researchers conducted on the first three measurement concepts (i.e., reliability, validity [i.e., convergent-related evidence], and potential for bias).

The purpose of this study was to develop and evaluate the Family Caregiver Identity Scale (FCIS), an instrument designed to measure the extent to which an individual identifies with the family caregiver role. The process of instrument development outlined in the Standards for Educational and Psychological Testing was combined with Dillman’s four stages of pretesting. This was a multistage, iterative process, including several revisions based on feedback from experts, interviews, and pilot testing. Factor analyses were performed to test the hypothesized model of caregiver identity. A version of the FCIS consisting of 18 items was created and demonstrated initial evidence of validity. The FCIS will enable gerontological professionals to assess caregiver identity. The absence of caregiver identity is a factor in caregivers not accessing support services. This study contributes to the growing body of research connecting caregiver identity and support service utilization by caregivers.

Antecedentes: La validez es uno de los conceptos más fundamentales en el contexto de pruebas educativas y psicológicas y se refiere al grado en el que la evidencia teórica y empírica respaldan las interpretaciones de las puntaciones obtenidas a partir de una prueba utilizada para un fin determinado. En este trabajo, trazamos la historia de la teoría de la validez, centrándonos en su evolución y explicamos cómo validar el uso de una prueba para un propósito determinado. Método: Nos basamos en gran parte en los Estándares para Pruebas Educativas y Psicológicas, propuestos por la American Educational Research Association (AERA), la American Psychological Association (APA) y el National Council on Measurement in Education (NCME), los cuales proporcionan un marco conceptual para la validación de pruebas. También proporcionamos una breve descripción de la validación basada en argumentos y sus componentes, esbozando las dificultades asociadas a la operacionalización del proceso de validación desde una perspectiva de argumentación. Resultados: Se proponen cinco fuentes de evidencias de validez de las puntuaciones de una prueba: contenido, procesos de respuesta, estructura interna, relaciones con otras variables y consecuencias. Conclusión: El uso de los Estándares permite que la evidencia de validez pueda ser acumulada de forma sistemática para respaldar la interpretación y el uso de las puntuaciones de una prueba para un propósito específico, promoviendo así prácticas solidas en cuanto al uso de un instrumento de medida lo cual que puede contribuir a reducir las consecuencias negativas provenientes de la utilización de pruebas de alto riesgo.

Fairness is one of the most important quality criteria of assessments and a necessary condition for valid test score interpretations. In this paper, we describe findings related to an assessment of N = 7664 beginning business and economics students at 46 universities across Germany using a domain-specific higher education entry test. From the perspective of test fairness as defined by the internationally established validation standards outlined by AERA et al. (Standards for Educational and Psychological Testing, AERA, Washington, DC, 2014), we identify which students had particular difficulty completing the test, taking into account gender- and language-related influence factors in the assessment of their test performance. Our results highlight the particular challenges in admissions testing in higher education. One of these challenges is finding a suitable way to address the disadvantages experienced by various groups of students to guarantee fairness and ethical integrity when developing and administering tests. Current migration developments and the overall internationalization of higher education have led to an increasing heterogeneity of student bodies. These challenges are therefore of particular relevance for education policy and practice when it comes to ensuring fairness when assessing learning outcomes for all student groups. This also applies in the context of commonly used standardized assessments and established examination practices in higher education.

The Standards for Educational and Psychological Testing recommend an argument-based approach to validation that involves a clear statement of the intended interpretation and use of test scores, the identification of the underlying assumptions and inferences in that statement—termed the interpretation/use argument, and gathering of evidence to support or refute the assumptions and inferences. We present two approaches to articulating the interpretation/use argument. One approach uses the five sources of validity evidence in the Standards for Educational and Psychological Testing as a framework and the other approach uses Kane’s chain of assumptions/inferences as a framework. Through this process we identified aspects of these approaches that need to be further clarified for instrument developers to consistently implement either approach, identified important differences in the perspective each approach takes on validation, and highlight important questions for the measurement and mathematics educat...

The assessment of physical literacy is vital and challenging for researchers and practitioners. Passport for Life is a Canadian global practitioner-based, formative, criterion-referenced assessment of physical literacy that is recorded in an online program. Students complete eight assessments divided across four broad categories of physical literacy (active participation, fitness, movement, and living skills). This study is an endeavor to uncover initial validation and feasibility evidence for the proposed use of Passport for Life with Grade 10–12 physical education students from various regions of Canada. The Standards for Educational and Psychological Testing were used as the theoretical framework. The results with 1,003 secondary school students provide score validity evidence on the content, format, and administrative guidelines of the items and scales; response process evidence about how the program was valued and how the assessments were implemented; internal structure evidence about how participants performed and the extent of associations between items within each scale; predictive evidence as to how scales related temporally to a second measurement of each assessment scale; and concurrent evidence about how the scales related to those of other components. While drawing attention to certain limitations and adding some recommendations, this article shows that the results generally support the Grade 10–12 Passport for Life relative to its intended use. Subscribe to TPE

Objective: To evaluate the current usage of simulation in urological education in the USA and the barriers to incorporating a simulation-based educational curriculum, as the shift towards competency-based medical education has necessitated the introduction of simulation for training and assessing both non-technical and technical skills. Materials and methods: Residency programme directors at Accreditation Council for Graduate Medical Education (ACGME)-accredited urology training programmes in the USA were invited to respond to an anonymous electronic survey. The study evaluated the programme directors’ experiences and opinions for the current usage of existing urology simulators. The survey also elicited receptiveness and the barriers for incorporating simulation-based training curricula within urology training programmes. Results: In all, 43 completed surveys were received (35% response rate). Amongst responders, 97% (42/43) reported having access to a simulation education centre, and 60% (25/42) have incorporated simulation into their curriculum. A total of 87% (37/43) agreed that there is a role for a standardised simulator training curriculum, and 75% (30/40) agreed that simulators would improve operating room performance. A total of 64% (27/42) agreed that cost was a limiting factor, 12% (5/42) agreed on the cost-effectiveness of simulators, 35% (17/41) agreed there was an increased need for simulator education within work-hour limitations, and 38% (16/42) agreed a simulation programme would reduce patient risks and complications. Conclusions: The majority of urology programme directors consider that there is a role for incorporating a simulation-based curriculum into urology training. Barriers to implementation include cost burden, need for constant technology updates, need for advanced planning, and willingness of faculty to participate in administration. Keywords: Computer simulation, Education, Residency, Curriculum

This chapter discusses how the collection of validity evidence is integral to the development process of the portfolio used in Chile’s National Teacher Evaluation System (NTES). Following the validity framework of the Standards for Educational and Psychological Testing (AERA, APA, and NCME, 2014), the chapter begins by clarifying the use of the portfolio in the context of this national evaluation system. It explains how different phases of the portfolio construction process lend themselves to the collection of various types of evidence considered in the Standards, and how these inquiries are part of a logic of continuous improvement of this instrument. The relationship between instrument development and evidence collection is approached from the conceptual framework of Construct Modeling (Wilson, 2005). We discuss how various stages of the instrument development process considered in this framework can contribute to the collection of evidence of content, response process, and internal structure that will inform the construction of a validity argument (Kane, 1992). We describe how during each stage of portfolio development and review data are collected from multiple sources to improve the quality of the instrument and to document the evidence needed to support a validity argument regarding its use. The chapter concludes by emphasizing the value of understanding the validation process as a simultaneous aspect of developing an assessment instrument, not just for documenting in retrospect the evidence supporting the intended uses, but also to examine the quality of the instrument in a framework of continuous improvement.

This chapter addresses validity of educational assessments from a particular case: the Learning Progress Assessment System. This assessment program, developed by the Measurement Center at Pontificia Universidad Catolica de Chile, offers standardized tests based on the national curriculum, to schools and educational institutions that aim to measure students’ performance in language and mathematics throughout their schooling. First, this assessment program is described, through the presentation of its main objectives and the types of information it produces. Subsequently, we thoroughly present the procedures developed within the framework of the validation that, from the Standards for Educational and Psychological Testing, is conceived as a continuous process of evidence collection that supports the proposed uses and interpretations of tests. The routine validation agenda is presented first, which includes: evidence based on test content, analysis of its internal structure, and reliability and fairness corroborations. Secondly, we present studies separated in time, aimed at supporting arguments about relations to other variables (convergent and discriminant evidence of validity) and about the uses and consequences of the tests. In every case, we describe the procedures developed and the obtained results. Finally, the conclusion summarizes the scope of the presented topics and the challenges for the future.

Educational assessment and testing systems have become increasingly important in Latin America and their results are used for various purposes. However, are different interpretations and uses of these results appropriate? The content presented in this chapter introduces the current definition of validity, based primarily on the Standards for Educational and Psychological Testing (American Educational Research Association et al., 2014), and argues that validation should be a fundamental process in educational assessment and measurement systems to ensure legitimate interpretations and uses of their results. At the beginning of the chapter, the current concept of validity and validation is defined. The historical evolution of these concepts is then briefly reviewed, taking into account the contributions of Messick, Cronbach, and Kane. Subsequently, the types of validity evidence that can be part of a validation process are described: (a) evidence based on test content, (b) on response processes, (c) on internal structure, (d) on relations to other variables, and (e) on intended and unintended consequences. The chapter also explains how these different pieces of evidence should be integrated into a validity argument. Finally, the political dimension of validation is explained, pointing out the main difficulties encountered when research, policy, and practice converge as a validation agenda is created.

Background The current state of evidence regarding measures that assess evidence-informed decision-making (EIDM) competence attributes (i.e., knowledge, skills, attitudes/beliefs, behaviours) among nurses is unknown. This systematic review provides a narrative synthesis of the psychometric properties and general characteristics of EIDM competence attribute measures in nursing. Methods The search strategy included online databases, hand searches, grey literature, and content experts. To align with the Cochrane Handbook of Systematic Reviews, psychometric outcome data (i.e., acceptability, reliability, validity) were extracted in duplicate, while all remaining data (i.e., study and measure characteristics) were extracted by one team member and checked by a second member for accuracy. Acceptability data was defined as measure completion time and overall rate of missing data. The Standards for Educational and Psychological Testing was used as the guiding framework to define reliability, and validity evidence, identified as a unified concept comprised of four validity sources: content, response process, internal structure and relationships to other variables. A narrative synthesis of measure and study characteristics, and psychometric outcomes is presented across measures and settings. Results A total of 5883 citations were screened with 103 studies and 35 unique measures included in the review. Measures were used or tested in acute care (n = 31 measures), public health (n = 4 measures), home health (n = 4 measures), and long-term care (n = 1 measure). Half of the measures assessed a single competence attribute (n = 19; 54.3%). Three measures (9%) assessed four competence attributes of knowledge, skills, attitudes/beliefs and behaviours. Regarding acceptability, overall missing data ranged from 1.6–25.6% across 11 measures and completion times ranged from 5 to 25 min (n = 4 measures). Internal consistency reliability was commonly reported (21 measures), with Cronbach’s alphas ranging from 0.45–0.98. Two measures reported four sources of validity evidence, and over half (n = 19; 54%) reported one source of validity evidence. Conclusions This review highlights a gap in the testing and use of competence attribute measures related to evidence-informed decision making in community-based and long-term care settings. Further development of measures is needed conceptually and psychometrically, as most measures assess only a single competence attribute, and lack assessment and evidence of reliability and sources of established validity evidence. Registration PROSPERO #CRD42018088754.

IntroductionContemporary validity testing theory holds that validity lies in the extent to which a proposed interpretation and use of test scores is justified, the evidence for which is dependent on both quantitative and qualitative research methods. Despite this, we hypothesise that development and validation studies for assessments in the field of health primarily report a limited range of statistical properties, and that a systematic theoretical framework for validity testing is rarely applied. Using health literacy assessments as an exemplar, this paper outlines a protocol for a systematic descriptive literature review about types of validity evidence being reported and if the evidence is reported within a theoretical framework.Methods and analysisA systematic descriptive literature review of qualitative and quantitative research will be used to investigate the scope of validation practice in the rapidly growing field of health literacy assessment. This review method employs a frequency analysis to reveal potentially interpretable patterns of phenomena in a research area; in this study, patterns in types of validity evidence reported, as assessed against the criteria of the 2014Standards for Educational and Psychological Testing, and in the number of studies using a theoretical validity testing framework. The search process will be consistent with the Preferred Reporting Items for Systematic Reviews and Meta-analyses statement. Outcomes of the review will describe patterns in reported validity evidence, methods used to generate the evidence and theoretical frameworks underpinning validation practice and claims. This review will inform a theoretical basis for future development and validity testing of health assessments in general.Ethics and disseminationEthics approval is not required for this systematic review because only published research will be examined. Dissemination of the review findings will be through publication in a peer-reviewed journal, at conference presentations and in the lead author’s doctoral thesis.

The key-features (KFs) approach to assessment was initially proposed during the First Cambridge Conference on Medical Education in 1984 as a more efficient and effective means of assessing clinical decision-making skills. Over three decades later, we conducted a comprehensive, systematic review of the validity evidence gathered since then. The evidence was compiled according to the Standards for Educational and Psychological Testing's five sources of validity evidence, namely, Content, Response process, Internal structure, Relations to other variables, and Consequences, to which we added two other types related to Cost-feasibility and Acceptability. Of the 457 publications that referred to the KFs approach between 1984 and October 2017, 164 are cited here; the remaining 293 were either redundant or the authors simply mentioned the KFs concept in relation to their work. While one set of articles reported meeting the validity standards, another set examined KFs test development choices and score interpretation. The accumulated validity evidence for the KFs approach since its inception supports the decision-making construct measured and its use to assess clinical decision-making skills at all levels of training and practice and with various types of exam formats. Recognizing that gathering validity evidence is an ongoing process, areas with limited evidence, such as item factor analyses or consequences of testing, are identified as well as new topics needing further clarification, such as the use of the KFs approach for formative assessment and its place within a program of assessment.

The paper focuses on the role of teachers’ psychological culture in supporting the implementation of the Federal State Educational Standard for Basic Education.The authors stress that formation of personal educational outcomes in students is a task closely associated with the level of psychological culture of teachers.Using a questionnaire designed to explore the teachers’ need to improve their psychological competence, the authors carried out a survey of 69 teachers and outlined the priority areas of psychological culture improvement for the teachers basing on the outcomes of the survey.The paper analyses the relationship between the professionally significant qualities of the teachers (such as communicative and organizational skills) and their views on the formation of personal educational outcomes in students.Finally, the paper outlines the possibilities of developing psychological culture in teachers as a system of means allowing them to fulfill the requirements of modern educational standards, in particular, to resolve the task of forming personal educational outcomes in students.

Since the 1970s, successive editions of the authoritative American ‘Standards For Educational and Psychological Testing’ (hereafter referred to as ‘the Standards’) have included references to fairn...

The paper deals with the problems of the new approach to the planning of the physical education. The research aim was to determine the influence of aerobic activity on the dynamics of physical training and health promotion among students by involving cyclical sports in the physical culture classes at higher educational institutions. The teachers are advised to give consideration not to the sport results but to the students’ state of health at the physical culture classes. Traditionally formed classes directed at the fulfillment of control demands are not always effective in the context of health promotion. Special attention must be paid to the solution of health life and educational problems of students, formation of their methodical knowledge, raising in them a stable need in systematic exercising. Teachers should help students control the development of their physical qualities and teach them to make conclusions about their own health state.The research results confirmed the supposition that physical exercises of cyclical character have a more positive influence on health dynamics, and the 100 % scale grade improves students’ physical activity.

Objectives The aim of this study was to examine the validity of a simulator test designed to evaluate focused assessment with sonography for trauma (FAST) skills. Methods Participants included a group of ultrasound novices (n = 25) and ultrasound experts (n = 10). All participants had their FAST skills assessed using a virtual reality ultrasound simulator. Procedural performance on the 4 FAST windows was assessed by automated simulator metrics, which received a passing or failing score. The validity evidence for these simulator metrics was examined by a stepwise approach according to the Standards for Educational and Psychological Testing. Metrics with validity evidence were included in a simulator test, and the reliability of test scores was determined. Finally, a pass/fail level for procedural performance was established. Results Of the initial 55 metrics, 34 (61.8%) had validity evidence (P

What makes for a high-quality educational or psychological test? The standard answer is that it must have good reliability and validity (e.g. Clausen-May, 2001). Reliability, the degree to which a ...

Physical and Health Education Canada has developed and implemented a formative, criterion-referenced, and practitioner-based national (Canadian) online educational assessment and support resource called Passport for Life (PFL). It was developed to support the awareness and advancement of physical literacy among PE students and teachers. PFL consists of three assessments for each of the four components (active participation, fitness, movement, and living skills). The aim of this study was to uncover initial validation evidence for its current uses using four of the five broad guidelines (content, response processes, internal structure, relations with other variables) for establishing satisfactory score validity as established by the Standards for Educational and Psychological Testing. We conducted a pilot test with 860 students in Grades 4 and 5 in 2013–2014. We analyzed these data, along with data collected across 2 years for Grades 3 to 6 (n = 1,036 in 2013–2014 and 1,254 in 2014–2015) and Grades 7 to 9 (n = 1,793 in 2013–2014 and 1,151 in 2014–2015). A portion (15 to 25%) of these students completed some of the assessments a second time in each of these years. Validation evidence included the development of PFL by a number of domain experts, the nature and format of the components and scales relative to existing literature and evidence, the administrative procedures to guide teachers to implement the assessments, the alignment of items with each component construct and scale, teacher feedback, and positive and significant relations and temporal (predictive) consistency over the 2 years within and across scales and components. These results provide general support for the PFL and its intended use and highlight several cautions and recommendations. Subscribe to TPE

The Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014) were written primarily for psychometricians who work on large-scale educational assessments, licensure and certification tests, and workplace assessments. Some psychometricians suggest that the Standards could be useful to guide classroom assessment. We test that claim by selecting 12 psychometric standards that are likely to be relevant to classroom assessment and translating them into formulations that might make sense to teachers, for use with four types of classroom assessments. We conclude that it is possible to make translations that are reasonably relevant to classroom assessment practices, that translation of the psychometric standards should be combined with the Classroom Assessment Standards (Joint Committee on Standards for Educational Evaluation, 2015) to ensure full coverage of all considerations in classroom assessment, and that any translation of psychometric standards for classroom assessment must be formulated into language and concepts that are relevant to and informed by teachers.

IntroductionAs adults age, their senses tend to decline and a large portion of those most affected by sensory decline reside in long-term care. At present, the creation of a sense-sensitive environment in long-term care is a difficult task as there is minimal evidence or tools available to guide this process. The 5Senses screening tool was developed to measure the sense-sensitivity of a particular environment, with a focus on long-term care. The purpose of this paper is to describe a study protocol to assess the psychometric properties of the newly developed 5Senses screening tool.Methods and analysisWe will conduct a psychometric evaluation of the 5Senses screening tool in long-term care based on the Standards for Educational and Psychological Testing Framework. In phase I, we will seek input from international content experts (n=20) to assess the content validity of all sections of the tool. In phase II, we will invite auditors (n=3-9), residents (n=3-9) and staff (n=3-9) to partake in think-aloud sessions to assess response process validity. In phase III, we will conduct field testing of the revised 5Senses screening tool with auditors (n=100), residents (n=100) and staff (n=100) to evaluate additional measures including acceptability, inter-rater reliability, internal structure validity and internal consistency reliability, where possible.Ethics and disseminationEthical approval was obtained from the University of Ottawa Research Ethics Board. Findings will be disseminated through a peer-reviewed manuscript, through a dedicated website, through presentations in long-term care communities and through presentations at research conferences.

This study involved empirical investigation of a moves-based conceptualization of teacher practices of planning, enacting, and reflecting on formative assessment (FA) in mathematics classrooms in a high-needs school district in California. A qualitative case study of six middle school mathematics teachers’ practices of posing questions, pausing to foster equity of participation and quality of response, and probing student thinking, the study provides empirical evidence of qualitatively distinct levels of teacher posing, pausing, and probing moves. The study utilized a National Research Council-based educational assessment design framework that employed construct maps, multi-faceted items design, and scoring guides to examine teacher practice and to provide feedback protocols for teachers engaged in FA. Guided by the 2014 Standards for Educational and Psychological Testing, the study provides evidence for content validity and tools for future rater reliability investigations. The study found levels of teacher questioning practice, operationalized as posing, pausing, and proving moves, could be represented along generalized continua in the context of middle school mathematics instruction. The study’s work toward the development of a teacher learning progression framework in the formative assessment domain has implications for establishing an empirically-based, common grammar of practice in mathematics instruction and preparation.

The need for quality in score reporting practices is represented in the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 2014). The purpose of this study was to introduce a ratings-based instrument to assess the quality of score reports and examine the reliability of scores obtained. Quality criteria were derived from best-practices published within the literature (Hambleton & Zenisky, 2013). The rating scale was used to assess a sample of 40 English-language individual student score reports for elementary-level accountability tests representing 42 states and 5 provinces in the United States and Canada. A two-facet generalizability study (i.e., sr x d x r) was completed with an overall reliability coefficient of G=0.78. Application of the rating scale may provide a means to support empirical study of relationships between score report quality and stakeholder outcomes including interpretation, use, and impact.

Background Simulation is being increasingly used worldwide in healthcare education. However, it is costly both in terms of finances and human resources. As a consequence, several institutions have designed programs offering several short immersive simulation sessions, each followed by short debriefings. Although debriefing is recommended, no tool exists to assess appropriateness of short debriefings after such simulation sessions. We have developed the Simulation in Healthcare retrOaction Rating Tool (SHORT) to assess short debriefings, and provide some validity evidence for its use. Methods We designed this scale based on our experience and previously published instruments, and tested it by assessing short debriefings of simulation sessions offered to emergency medicine residents at Laval University (Canada) from 2015 to 2016. Analysis of its reliability and validity was done using Standards for educational and psychological testing. Generalizability theory was used for testing internal structure evidence for validity. Results Two raters independently assessed 22 filmed short debriefings. Mean debriefing length was 10:35 (min 7:21; max 14:32). Calculated generalizability (reliability) coefficients are φ = 0.80 and φ-λ3 = 0.82. The generalizability coefficient for a single rater assessing three debriefings is φ = 0.84. Conclusions The G study shows a high generalizability coefficient (φ ≥ 0.80), which demonstrates a high reliability. The response process evidence for validity provides evidence that no errors were associated with using the instrument. Further studies should be done to demonstrate validity of the English version of the instrument and to validate its use by novice raters trained in the use of the SHORT. Electronic supplementary material The online version of this article (10.1186/s12909-019-1503-4) contains supplementary material, which is available to authorized users.

Latent class models are powerful tools in psychological and educational measurement. These models classify individuals into subgroups based on a set of manifest variables, assisting decision making in a diagnostic system. In this article, based on information theory, the authors propose a mutual information reliability (MIR) coefficient that summaries the measurement quality of latent class models, where the latent variables being measured are categorical. The proposed coefficient is analogous to a version of reliability coefficient for item response theory models and meets the general concept of measurement reliability in the Standards for Educational and Psychological Testing. The proposed coefficient can also be viewed as an extension of the McFadden’s pseudo R-square coefficient, which evaluates the goodness-of-fit of logistic regression model, to latent class models. Thanks to several information-theoretic inequalities, the MIR coefficient is unitless, lies between 0 and 1, and receives good interpretation from a measurement point of view. The coefficient can be applied to both fixed and computerized adaptive testing designs. The performance of the MIR coefficient is demonstrated by simulated examples.

The Standards for Educational and Psychological Testing states that there are five resources for evidence of validity which need to be examined in the evaluation of a measurement implement. The aim of this study is to evaluate the validity of the Academic Motivation Scale (AMS) - Short Indonesian Language Version. The three stages of this study were conducted over differing time periods, as well as differing resources of evidence of validity. In the first stage, the only resource of evidence of validity used was in the form of testing for reliability and correlation between the items, which aimed at determining just which items would be utilized in the short version. Meanwhile, with the second and third stages, after the items had been reduced, evaluation was conducted based on two resources of evidence of validity, the internal structure, covering reliability testing and analysis of confirmatory factors, and resources of evidence of connections to other variables. Results of the reliability testing showed an alpha Cronbach value of more than 0.7, with corrected item total correlation of all items above 0.3 Results of factor analysis also indicated that all items grouped within its dimension, as well as the results of correlation analysis also indicating results in accord with earlier pieces of research. It may be concluded that the Academic Motivation Scale-Short Indonesian Version may be considered to be a reliable measurement instrument to measure academic motivation.

The focus in this article is on Brazilian education policy, specifically quality assurance and evaluation. The starting point is that quality, measured by means of large-scale assessments, is one of the key discursive justifications for educational change. The article addresses the questions of how quality evaluation became a significant feature of Brazilian education, and how international organisations are related to this. The notion of measurement emerged at the beginning of the 20th century among an active educational community, strongly inspired by US educational thinking and supported by instrumental educational cooperation with US agencies and universities. Large-scale assessment became more and more popular in the 1960s–1980s, mainly within this setting. Educational indicators have been under development since the 1990s, alongside the education-related programmes of international organisations. Drawing on earlier research and documents, the article analyses historical and political influences from both international and national actors on the emergence of quality assessment in Brazilian basic education. The authors also point out the need for further research from a more complex perspective on the interplay between the various actors responsible for policies focusing on educational quality.

The Standards for Educational and Psychological Testing indicate that validation of assessments should include analyses of participants’ response processes. However, such analyses typically are conducted only to supplement quantitative field studies with qualitative data, and seldom are such data connected to quantitative data on student or item performance. This paper presents an example of how data from an analysis of mental operations collected using a sociocognitive approach can be quantitatively integrated with other data on student and item performance to validate in part an assessment of higher education students’ competency in business and economics. Evidence of forward reasoning and paraphrasing as mental operations is obtained using the think-aloud method. As part of the validity argument and to enhance credibility of the findings, the generalized linear models are expressed as multilevel models in which the analyses of response processes are aligned with quantitative findings from large-scale field studies.

The article analyzes the models and strategies, as well as the specifics of the implementation of Federal state educational standards in the practice of school psychological services. Paying particular attention to changes in the situation in the whole educational school, the authors define the place of the forms and types of application of psychological knowledge in the content and organization of the educational environment of the school, making it obligatory, specific and measurable activities educational psychologist as a full member of the educational process.

This brief presents the most significant recommendations based on a review of key findings from research presented in this special issue. The authors offer what they believe to be the most important considerations of what works for improving Black male school achievement in the domains of research, practice, and policy.

Standards provide a foundation for thinking about leadership development and practice. This article explores three sets of standards that impact educational leadership preparation: (a) the recently revised Interstate School Leaders Licensure Consortium (ISLLC) standards, which are now known as the Practice Standards for Educational Leaders (PSEL); (b) the Educational Leadership Constituent Council (ELCC) standards, which are currently under revision; and (c) the University Council for Educational Administration (UCEA) standards. In addition to reviewing each set, how they are used, and how they differ, we share findings from a survey of preparation program directors focused on how influential and beneficial these standards are in promoting leadership preparation program improvement.

This document analysis provides a summary of the research from high-impact journals published between 2008 and 2013 with the explicit purpose of determining the extent to which the current empirical evidence supports the individual 2011 Educational Leadership Constituent Council Program Standards and their elements. We found that the standards are unevenly represented and supported in the research. In general, the research was stronger for Standards 2 and 3 and building-level leadership standards. Without a more substantial research foundation for leadership preparation, program faculty are limited in their ability to design evidence-supported learning content and experiences for the next generation of educational leaders.

This review critically evaluates reporting and use of standardized measures to assess community based treatments for young children with Autism Spectrum Disorder (ASD). The Standards for Educational and Psychological Testing (AERPA, APA & NCME, 1999), a best practice framework for reporting standardized test results, guides the evaluation. Fifty three different outcome measures are identified across 45 studies representing twelve countries. Adaptive behavior, specifically the Vineland Adaptive Behavior Scales and cognitive measures continue to be primary outcome tools, despite a lack of clear fit to core ASD diagnostic constructs. Behavioral, ASD specific, language, social communication, and family wellness tools are under represented. Reporting strengths are use of multiple measures, clear sample descriptions, and use of specialized tools for ASD. Reporting weaknesses are assessment bias, test substitution, and under reporting of test modifications. Clinical and research implications are discussed.

In the modern world, the environment of any educational institution represents a spectrum of ethnic groups and subcultures: a multicultural educational environment. Pupils who are aware of their national identity often demonstrate intolerance toward students of other nationalities, which threatens pupils’ psychological safety. In this article, we present the results of pilot research examining the level of a pupil’s psychological safety in the multicultural educational environment and identifying the criteria that influence a pupil’s psychological safety. The research sample comprised 127 pupils aged 13–14 years from different schools living in various places that differed by the type of settlement, industrial development and level of science and culture. We isolated the following criteria for a pupil’s psychological safety in the multicultural educational environment: satisfaction with the educational environment, protection from psychological abuse and self-confidence. According to pupils, the essential characteristics of safety in the educational environment, regardless of school category and type, are being able to ask for help, protection of personal dignity, interactions with other students, and self-respect. Empirical data reveal the current status of the psychological safety of the entire sample group (n = 127) and compare indices of psychological safety in the educational institutions under study. Analysis of the results of our research indicates that protecting a pupil’s personality in the multicultural educational environment has the greatest influence on his/her psychological safety. In addition, a comfortable psychological atmosphere, mutual aid and support of pupils and low levels of classmates’ and coevals’ aggression positively influence the protection experience.

The problem of sharing psychological tests with non-psychologists concerns the sharing of chosen psychological tools with people without full psychological education. An analysis of the principles of sharing psychological tests was attempted. The Workgroup of Psychological Tests of The Polish Psychological Association (Pracownia Testow Psychologicznych Polskiego Towarzystwa Psychologicznego) makes psychological tools available in 4 categories, each of which includes various psychological tests. The issues of concern are the platform “Epsilon,” which has been recently introduced by The Workgroup of Psychological Tests, a training program, which is supposed to endorse the basics of psychometrics, or the possibility to access psychological tests from the category “for psychologists only,” given to people with a “doctor of psychology” degree. It is necessary to discuss the subject of the use of psychological tests by people without adequate psychological education. Easy access to the tests by non-psychologists can be harmful to the prestige of the psychological profession.

This article describes the development of a problem-solving instrument intended for classroom use that addresses the Common Core State Standards for Mathematics. In this study, 137 students completed the assessment, and their responses were analyzed. Evidence for validity was collected and examined using the current standards for educational and psychological testing. Instrument validation findings regarding internal consistency reliability were high, and multiple forms of validity (i.e., content, response processes, internal structure, relationship to other variables, and consequences of testing) were found to be appropriate. The resulting instrument provides teachers and researchers with a sound tool to gather data about sixth-grade students' problem solving in the Common Core era.

ObjectiveValidity refers to the extent to which evidence and theory support the adequacy and appropriateness of inferences based on score interpretations. The health sector is lacking a theoretically-driven framework for the development, testing and use of health assessments. This study used the Standards for Educational and Psychological Testing framework of five sources of validity evidence to assess the types of evidence reported for health literacy assessments, and to identify studies that referred to a theoretical validity testing framework.MethodsA systematic descriptive literature review investigated methods and results in health literacy assessment development, application and validity testing studies. Electronic searches were conducted in EBSCOhost, Embase, Open Access Theses and Dissertations and ProQuest Dissertations. Data were coded to the Standards’ five sources of validity evidence, and for reference to a validity testing framework.ResultsCoding on 46 studies resulted in 195 instances of validity evidence across the five sources. Only nine studies directly or indirectly referenced a validity testing framework. Evidence based on relations to other variables is most frequently reported.ConclusionsThe health and health equity of individuals and populations are increasingly dependent on decisions based on data collected through health assessments. An evidence-based theoretical framework provides structure and coherence to existing evidence and stipulates where further evidence is required to evaluate the extent to which data are valid for an intended purpose. This review demonstrates the use of the Standards’ theoretical validity testing framework to evaluate sources of evidence reported for health literacy assessments. Findings indicate that theoretical validity testing frameworks are rarely used to collate and evaluate evidence in validation practice for health literacy assessments. Use of the Standards’ theoretical validity testing framework would improve evaluation of the evidence for inferences derived from health assessment data on which public health and health equity decisions are based.

Examining science motivation is essential for science learning in the classroom, even more, science motivation is often related to students’ achievement and performance. The test of science motivation has broadly employed and some of the instrument has been translated into a local language where the instrument used. The current study explores the validity evidence of Science Motivation Questionnaire II (SMQ-II) by IRT-Rasch based analysis using Test Analysis Models ‘TAM’ Package v. 3.1-45 in R software from Indonesian context. This study reveals construct validity of SMQ-II instrument based on The Standards for Educational and Psychological Testing using Messick’s (1989) validity theories and construct validity. The data collected from 984 Indonesian middle and high school students. As result, the current study had SMQ-II validity evidence which includes (1) Dimensionality of the instrument using Exploratory Factor Analysis and Rasch-analysis likelihood ratio approach, (2) Expected A Posteriori (EAP) reliability and Weighted Maximum Likelihood Estimation (WLE) reliability, (3) Item fit and item properties, (4) Wright Map using Rasch modeling. The result of construct analysis shows that the instrument was appropriate with a theory where the construct was categorized into five dimensions, all the items of SMQ-II fit the rasch modelling, and the internal reliability of each dimension is categorized as good. In general, the current study suggests that SMQ-II instrument of Indonesian version is adequate to be implemented in Indonesia.