Search

The assessment of physical literacy is vital and challenging for researchers and practitioners. Passport for Life is a Canadian global practitioner-based, formative, criterion-referenced assessment of physical literacy that is recorded in an online program. Students complete eight assessments divided across four broad categories of physical literacy (active participation, fitness, movement, and living skills). This study is an endeavor to uncover initial validation and feasibility evidence for the proposed use of Passport for Life with Grade 10–12 physical education students from various regions of Canada. The Standards for Educational and Psychological Testing were used as the theoretical framework. The results with 1,003 secondary school students provide score validity evidence on the content, format, and administrative guidelines of the items and scales; response process evidence about how the program was valued and how the assessments were implemented; internal structure evidence about how participants performed and the extent of associations between items within each scale; predictive evidence as to how scales related temporally to a second measurement of each assessment scale; and concurrent evidence about how the scales related to those of other components. While drawing attention to certain limitations and adding some recommendations, this article shows that the results generally support the Grade 10–12 Passport for Life relative to its intended use. Subscribe to TPE

This chapter addresses validity of educational assessments from a particular case: the Learning Progress Assessment System. This assessment program, developed by the Measurement Center at Pontificia Universidad Catolica de Chile, offers standardized tests based on the national curriculum, to schools and educational institutions that aim to measure students’ performance in language and mathematics throughout their schooling. First, this assessment program is described, through the presentation of its main objectives and the types of information it produces. Subsequently, we thoroughly present the procedures developed within the framework of the validation that, from the Standards for Educational and Psychological Testing, is conceived as a continuous process of evidence collection that supports the proposed uses and interpretations of tests. The routine validation agenda is presented first, which includes: evidence based on test content, analysis of its internal structure, and reliability and fairness corroborations. Secondly, we present studies separated in time, aimed at supporting arguments about relations to other variables (convergent and discriminant evidence of validity) and about the uses and consequences of the tests. In every case, we describe the procedures developed and the obtained results. Finally, the conclusion summarizes the scope of the presented topics and the challenges for the future.

Educational assessment and testing systems have become increasingly important in Latin America and their results are used for various purposes. However, are different interpretations and uses of these results appropriate? The content presented in this chapter introduces the current definition of validity, based primarily on the Standards for Educational and Psychological Testing (American Educational Research Association et al., 2014), and argues that validation should be a fundamental process in educational assessment and measurement systems to ensure legitimate interpretations and uses of their results. At the beginning of the chapter, the current concept of validity and validation is defined. The historical evolution of these concepts is then briefly reviewed, taking into account the contributions of Messick, Cronbach, and Kane. Subsequently, the types of validity evidence that can be part of a validation process are described: (a) evidence based on test content, (b) on response processes, (c) on internal structure, (d) on relations to other variables, and (e) on intended and unintended consequences. The chapter also explains how these different pieces of evidence should be integrated into a validity argument. Finally, the political dimension of validation is explained, pointing out the main difficulties encountered when research, policy, and practice converge as a validation agenda is created.

What makes for a high-quality educational or psychological test? The standard answer is that it must have good reliability and validity (e.g. Clausen-May, 2001). Reliability, the degree to which a ...

Physical and Health Education Canada has developed and implemented a formative, criterion-referenced, and practitioner-based national (Canadian) online educational assessment and support resource called Passport for Life (PFL). It was developed to support the awareness and advancement of physical literacy among PE students and teachers. PFL consists of three assessments for each of the four components (active participation, fitness, movement, and living skills). The aim of this study was to uncover initial validation evidence for its current uses using four of the five broad guidelines (content, response processes, internal structure, relations with other variables) for establishing satisfactory score validity as established by the Standards for Educational and Psychological Testing. We conducted a pilot test with 860 students in Grades 4 and 5 in 2013–2014. We analyzed these data, along with data collected across 2 years for Grades 3 to 6 (n = 1,036 in 2013–2014 and 1,254 in 2014–2015) and Grades 7 to 9 (n = 1,793 in 2013–2014 and 1,151 in 2014–2015). A portion (15 to 25%) of these students completed some of the assessments a second time in each of these years. Validation evidence included the development of PFL by a number of domain experts, the nature and format of the components and scales relative to existing literature and evidence, the administrative procedures to guide teachers to implement the assessments, the alignment of items with each component construct and scale, teacher feedback, and positive and significant relations and temporal (predictive) consistency over the 2 years within and across scales and components. These results provide general support for the PFL and its intended use and highlight several cautions and recommendations. Subscribe to TPE

The Standards for Educational and Psychological Testing (American Educational Research Association [AERA], American Psychological Association [APA], & National Council on Measurement in Education [NCME], 2014) were written primarily for psychometricians who work on large-scale educational assessments, licensure and certification tests, and workplace assessments. Some psychometricians suggest that the Standards could be useful to guide classroom assessment. We test that claim by selecting 12 psychometric standards that are likely to be relevant to classroom assessment and translating them into formulations that might make sense to teachers, for use with four types of classroom assessments. We conclude that it is possible to make translations that are reasonably relevant to classroom assessment practices, that translation of the psychometric standards should be combined with the Classroom Assessment Standards (Joint Committee on Standards for Educational Evaluation, 2015) to ensure full coverage of all considerations in classroom assessment, and that any translation of psychometric standards for classroom assessment must be formulated into language and concepts that are relevant to and informed by teachers.

This study involved empirical investigation of a moves-based conceptualization of teacher practices of planning, enacting, and reflecting on formative assessment (FA) in mathematics classrooms in a high-needs school district in California. A qualitative case study of six middle school mathematics teachers’ practices of posing questions, pausing to foster equity of participation and quality of response, and probing student thinking, the study provides empirical evidence of qualitatively distinct levels of teacher posing, pausing, and probing moves. The study utilized a National Research Council-based educational assessment design framework that employed construct maps, multi-faceted items design, and scoring guides to examine teacher practice and to provide feedback protocols for teachers engaged in FA. Guided by the 2014 Standards for Educational and Psychological Testing, the study provides evidence for content validity and tools for future rater reliability investigations. The study found levels of teacher questioning practice, operationalized as posing, pausing, and proving moves, could be represented along generalized continua in the context of middle school mathematics instruction. The study’s work toward the development of a teacher learning progression framework in the formative assessment domain has implications for establishing an empirically-based, common grammar of practice in mathematics instruction and preparation.

Latent class models are powerful tools in psychological and educational measurement. These models classify individuals into subgroups based on a set of manifest variables, assisting decision making in a diagnostic system. In this article, based on information theory, the authors propose a mutual information reliability (MIR) coefficient that summaries the measurement quality of latent class models, where the latent variables being measured are categorical. The proposed coefficient is analogous to a version of reliability coefficient for item response theory models and meets the general concept of measurement reliability in the Standards for Educational and Psychological Testing. The proposed coefficient can also be viewed as an extension of the McFadden’s pseudo R-square coefficient, which evaluates the goodness-of-fit of logistic regression model, to latent class models. Thanks to several information-theoretic inequalities, the MIR coefficient is unitless, lies between 0 and 1, and receives good interpretation from a measurement point of view. The coefficient can be applied to both fixed and computerized adaptive testing designs. The performance of the MIR coefficient is demonstrated by simulated examples.

The focus in this article is on Brazilian education policy, specifically quality assurance and evaluation. The starting point is that quality, measured by means of large-scale assessments, is one of the key discursive justifications for educational change. The article addresses the questions of how quality evaluation became a significant feature of Brazilian education, and how international organisations are related to this. The notion of measurement emerged at the beginning of the 20th century among an active educational community, strongly inspired by US educational thinking and supported by instrumental educational cooperation with US agencies and universities. Large-scale assessment became more and more popular in the 1960s–1980s, mainly within this setting. Educational indicators have been under development since the 1990s, alongside the education-related programmes of international organisations. Drawing on earlier research and documents, the article analyses historical and political influences from both international and national actors on the emergence of quality assessment in Brazilian basic education. The authors also point out the need for further research from a more complex perspective on the interplay between the various actors responsible for policies focusing on educational quality.

The development of any assessment should be an iterative and careful process. Ideally, this process is guided by a well-defined framework (see for example Downing in: Downing and Haladyna (eds) Handbook of test development, Lawrence Erlbaum Associates, Mahwah, 2006; Mislevy et al. in On the roles of task model variables in assessment design (CSE Technical Report 500), Educational Testing Service, Princeton, 1999; AERA et al. in Standards for educational and psychological testing, AERA, Washington, DC, 2004), but such a framework is not always available when the instrument to be developed is new or innovative. Frameworks for the development of traditional computer-based tests have been published and experimented with since the late 1990s, by which time CBT had already existed for more than a decade. In an earlier empirical pilot study, we described a new type of assessment for Dutch vocational education, called multimedia-based performance assessment (MBPA) (self-revealing reference 2014). This CBT uses multiple media formats and interactive tasks to measure skills that are currently measured by performance-based assessment. In conducting that pilot study, deficits in the existing literature made it difficult to ground all developmental steps in sound scientific theory. To remedy those deficits, this article presents and validates a framework for the design and development of MBPA, combining a search of the relevant literature from several subfields of educational assessment and consultation with assessment experts. The framework unites assessment development and multimedia development theory, focus solely on vocational education, and answers the call for a framework from the scientific community. The first step in validating the prototype framework involved five semi-structured interviews with Dutch assessment and multimedia experts to produce a final version of the framework. Second, the pilot MBPA was reconstructed in accordance with this finalized framework, resulting in an improved MBPA and demonstrating that the proposed framework is a useful and applicable tool for the design and development of MBPA in vocational education.

Educational agencies are going through a difficult transition, with important consequences in terms of de-legitimacy of the social mandate historically assigned to the school. The reason for this widespread unease can be traced to the fact that today we live in a complex system. This paper arises within this problematic framework with the intent to outline the most important lines of change affecting education agencies during the last fifteen years, paying particular attention to the new institutional and normative framework that redefined the whole educational structure in Italy. The objective of this paper is to contribute to the development of a new vision of the education system, more in line with the challenges posed by the global and complex society that emphasizes both the importance of evaluation of education agencies, on the basis of Quality Assurance more attentive to results, and the community’s responsibility to education.

Low-stakes assessment is supposed to improve educational practice by providing feedback to different actors in educational systems. However, the process of assessment from design to the point of a final impact on student learning outcomes is complex and diverse. It is hard to identify reasons for substandard achievement on assessments, let alone the means that should be undertaken to improve the educational setting. Furthermore, it is difficult to show a causal link between educational reforms and change in test achievement over time. This paper examines the potential impact that low-stakes testing initiatives have in engineering higher education on educational improvement by examining two case studies. It discusses how the design of constructs and particular assessment conditions may foster or limit educational improvement and outlines conditions to ensure that low-stakes assessment designs achieve educational improvement.

Background This brief reflection on the work of the Gordon Commission calls out significant themes and implications found in the various papers authored by the commissioners and other scholars, especially those included in this special issue of Teachers College Record. Purpose The forward-looking vision of the Gordon Commission is contrasted with contemporary teaching and testing practices to highlight implications for new assessment purposes and methods. It is argued that a new vision of assessment is inseparable from a new vision of teaching and learning. To realize this new vision, some current practices, especially uses of testing to sort and select students and to rank teachers and schools, will need to be greatly attenuated or even abandoned. Research Design This is a narrative review expressing the author's own point of view. No empirical findings are cited. Conclusions A conservative reading of the Gordon Commission's work might suggest that educational assessment tomorrow should function much as it does today, only better. A closer reading, however, suggests a more radical view. Assessment FOR education must break free from the constraints of standardization and consequential comparison.

This chapter provides an overview of the use of professional standards of practice in assessment and of the Council for the Advancement of Standards in Higher Education (CAS). It outlines a model for conducting program self-studies and discusses the importance of implementing change based on assessment results.

This third edition of the Standards for Academic and Professional Instruction in Foundations of Education, Educational Studies, and Educational Policy Studies is presented to the educational commun...

Germany has relied on its well-established tradition of preparing the workforce through the so-called dual system of practical in-firm training (apprenticeships) and theoretical foundations conveyed by vocational schools. Believing in the high quality of academic school programmes that prepare a small elite for university studies, the German public remained convinced until well into the 1990s that there was little need for innovation and improvement in German education. Although there had been earlier signs of concern, this conviction remained unchanged until the publication of the outcomes of the International Association for the Evaluation of Educational Achievement (IEA) Third International Mathematics and Science Study (TIMSS) in 1996. The ‘TIMSS shock’, reinforced by later studies, has resulted in a host of changes in educational policy, perhaps best characterised by the late Hermann Lange as the ‘turn towards evidence-based educational policy making’. Hallmarks of this trend were the decision by all 16 federal states in Germany to participate regularly in international large-scale assessments (ILSAs), including interstate comparisons (taken in 1997); the decision to define standards of educational performance (corresponding to the federally recognised school leaving certificates); the initiation of academic study programmes intended to foster the development of the required advanced technical skills; and the founding and/or expansion of agencies for quality management (including systems of reports on the state of education, both at the federal and the state levels). In this article, the role of ILSAs (including both IEA and others) in these policy changes are described and examined from a critical perspective.

The year 2015 is the deadline for most of the Education for All (EFA) goals. As we get closer to this date, reviews about what has been done and reflection about future agendas will multiply. This ...

Rationale and Objectives The American Institute of Ultrasound in Medicine (AIUM) recommends that all providers performing ultrasound-guided invasive procedures be competent in a core set of guidance skills common to all ultrasound-guided procedures, including in-plane and out-of-plane needle guidance and needle imaging optimization techniques such as probe translation, rotation, and heel-toe standoff. To allow for the practice and assessment of these core skills, we have created a novel task trainer and set of targeting tasks, and sought to obtain validity evidence in the content and response process domains for this training and assessment system according to the Standards for Educational and Psychological Testing. Materials and Methods We have constructed an ultrasound-guided invasive procedure training system and five targeting tasks that focused on the needle guidance skills outlined by the AIUM. All tasks were performed by obstetrics and gynecology or maternal-fetal medicine physicians with and without experience in ultrasound-guided invasive procedures during a series of simulation workshops. All participants completed a survey regarding the trainer's and the tasks' usefulness in the training of inexperienced physicians. Results The physicians who completed the tasks had favorable views of task trainer and curriculum. The targeting curriculum was felt to allow practice of all of the core guidance skills outlined by the AIUM. The average response provided for all of the tasks was 4.0 or greater, with half of the items having an average response of 4.5 or higher. Conclusions We have constructed a task trainer that incorporates all of the core skills outlined by the AIUM. All five tasks received very favorable reviews from both experienced and inexperienced providers. Taken together, our findings suggest they have strong content and response process validity evidence.

Borrowing educational institutions and ideas from the past and from other contemporary education systems is a significant part of the educational history of many nations. Many of the current nations continue to probe the past for pleasure or insights, while some nations seek to erase much of their educational past. Further, an increasing number of contemporary governments and educational leaders actively seek to improve or redesign their educational programmes by borrowing current educational ideas, programmes, and institutions from other countries. This Introduction recognizes the legitimacy of multiple definitions of the term “world-class” but limits discussion to basic education. Thus, the focus here is on pre-university schooling. Attention will be given to the measurement of quality as an integral component of world-class education, approaches to its planning, and discussions of relevant contemporary issues and trends.

Considering consequences in the evaluation of validity is not new although it is still debated by Paul E. Newton (this issue) and others (Mehrens, 1997; Popham, 1997). The argument-based approach t...

Academic program review has become a consistent and integral part of the life of a Communication Department. Nationally, program review began as part of the assessment movement, and has taken its place as a regular feature in higher education. Program review generally has two purposes—to demonstrate program effectiveness and to serve as data for department planning. The National Communication Association initiated the development of program review standards for communication departments. After the National Communication Association approved an initial draft, a task force was formed to finalize the standards, to develop supporting material, and to gather departmental data to be used for comparison purposes. This article describes the background for the standards, discusses the application of the standards, presents the data gathered, and suggests application of both the standards and the data.

Influenced by governmental policy, school psychological services in the Chinese mainland are a recent phenomenon with a history of only about 20 years, which can be divided into two stages: exploration and development. To give a comprehensive account of the development of school psychological services in the Chinese mainland, this article reviews their developmental history, analyses their status quo and features, identifies some contemporary problems, and suggests possible directions for the future development.

This study followed up on previous work that examined the incidence of reporting evidence based on test consequences in Mental Measurements Yearbook. In the present study, additional possible outlets for what has been called ‘‘consequential validity’’ evidence were investigated, including all articles published in the past 10 years in several applied journals devoted to educational assessment and educational policy, and all presentations at recent annual meetings for the three organizations that sponsor the Standards for Educational and Psychological Testing (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education). Consistent with previous findings, consequences of testing as a source of validity evidence is essentially nonexistent in the professional literature and applied measurement and policy work. The article concludes with implications of these findings for extending and refining current validity theory and validation practice.

Contemporary educational evaluation policy and practice can be challenged by applying critical theory and hermeneutics perspectives in evaluation and by asking whose interests are being served through evaluation. Using educational evaluation in Sweden as the context, the author argues for an explanation-oriented evaluation approach that assumes education is complex, multilayered, and infused with conflicting interests and intentions. The author concludes that evaluations that attend to this complexity have the potential to influence educational policy significantly. © Wiley Periodicals, Inc., and the American Evaluation Association.

Student assessment is central to the educational process and can be used for multiple purposes including, to promote student learning, to grade student performance and to evaluate the educational quality of qualifications. It is, therefore, of utmost importance that assessment instruments are of a high quality. In this article, we present various tools that instructors could use, both to improve instrument design and validity before presentation to students and, to evaluate the reliability and quality of the assessment after students have answered the questions. In keeping with our goals of the Bridging-the-Gap series, we also present various ideas from the educational literature regarding the evaluation of assessment quality and end with a list of criteria that instructors could use to guide their evaluation process.

Public-opinion surveys on recent changes in education and current policy priorities show that levels of satisfaction are dependent on social status and the experiences of different educational institutions.

This paper proposes a framework for assessing the quality of education, based on the outcomes defined in educational standards. The author takes the view that educational standards reflect the mission that schools must fulfill. He explains that the classroom curriculum, derived from educational standards, must customize the learning process to respond to the teaching–learning environment. Defining quality as the extent to which the delivery of the school curriculum realises the learning outcomes defined in the educational standards, the author proposes that quality in education should be evaluated using two approaches: relative achievement assessment and absolute achievement assessment. In elaborating on the two dimensions of assessment on which the model is based, the author highlights the need for greater attention to be paid to values and attitudes in assessing quality of education.

Issues related to the national assessment of children with special educational needs, including standard assessment tasks, are discussed by Rea Reason. The author is tutor for professional development in educational psychology, Centre for Educational Guidance and Special Needs, Manchester University, and senior educational psychologist in Oldham.

In order to better understand the relationship between educational psychology research and educational reform, this essay reviews the development of educational psychology studies in the last 20 years in China. The study shows that: (1) Rapid development has been made in the areas of establishing discipline systems; (2) Research fields have been expanded and research directions have been localized; (3) The theoretical basis for quality education has been derived from education psychology studies; (4) The psychological foundation for curriculum reform and instruction has been implemented, and (5) China has committed to serve and expand current educational practices.

Some of the challenges that test publishers face in constructing educational assessments that meet high technical quality as prescribed in the Standards for Educational and Psychological Testing (AERA, APA, NCME, 1999) are examined. Federal educational initiatives are used to illustrate demands on technical quality that challenge the efforts of test publishers to adhere to the Standards. Possible pitfalls to avoid when updating the Standards are also discussed.

The Standards for Educational and Psychological Testing have evolved in the breadth and depth of coverage of issues in educational testing and measurement since their first publication in 1954. There were a number of substantive changes in the 1999 revision that addressed validity, fairness, accommodations, and compliance with the Standards. In addition, there was nearly a 50% increase in the number of standards contained in the last revision. The next revision of the Standards may be initiated in 2007 and there are remaining concerns about access and awareness by non-measurement professionals, compliance by test publishers and users, relevance in addressing mandates for accountability, and substantive areas of educational assessment. This review of major changes to the Standards and discussion of future topics is designed to inform the next revision.