Search

Although scholars have discussed in depth about the evaluation content and the construction of evaluation index system of emergency management and crisis management, a unified and standardized interpretation of system construction concepts and empirical research on the performance evaluation of urban emergency management is still lacking. In view of this, this paper is based on the theory of the crisis life cycle, with the four phases of urban emergency management and the content of the task as a clue, constructed a performance evaluation index system containing 12 primary indicators and 44 secondary indicators, and centered on the emergency management work situation of S city. The study reveals that emergency management underperforms in S city, with a score of 5.948, qualifying as the "defined level". The "prevention and preparedness stage" receives poor evaluation, indicating a significant bias for crisis response instead of prevention. The report suggests the improvement of emergency material stockpile planning, the formation of grassroots emergency response teams, the emergency management publicity, and special emergency drill plans, etc., with a view to providing reference for city managers to comprehensively review and evaluate the emergency management system, and to promote the construction of a more systematic, complete and scientific urban emergency management system. [ABSTRACT FROM AUTHOR]

This study investigates whether family owners consider the stages of CEO tenure in their use of firm performance to assess CEOs. Integrating insights on family owners' long-term orientation and stages of CEO tenure, we theorize that family owners rely more on firm performance to make CEO replacement decisions during the mid-stage of CEO tenure than during the early and late stages. Moreover, we predict that they hold nonfamily CEOs more accountable for firm performance than family CEOs only during the mid-stage of CEO tenure. Using data from family-controlled firms in Taiwan, we found empirical evidence supportive of our theoretical predictions. [ABSTRACT FROM AUTHOR]

India's healthcare system is, for a large part, organized around a vast network of Primary Healthcare Centres (PHCs) that form the pillar on which the public healthcare sector functions. The World Health Organization (WHO) has emphasized the important role that PHCs play in strengthening community health and the provision of healthcare. Although a few studies have assessed specific elements of services offered by PHCs, only a few have studied the patients' perspectives on the functioning and performance of PHCs in the Indian context. A qualitative research methodology was employed to explore the opinions of 188 patients attending one of three PHCs in Bengaluru (India), using in-depth interviews and thematic analysis. Results showed that patients assessed PHC based on the nine themes that broadly can be classified into components of the center, and that of the personnel. The patients valued the behavioural aspects of the personnel during service delivery and should be configured into the PHC performance. [ABSTRACT FROM AUTHOR]

Teacher preparation programs are increasingly expected to use data on preservice teacher (PST) skills to drive program improvement and provide targeted supports. Observational ratings are especially vital, but also prone to measurement issues. Scores may be influenced by factors unrelated to PSTs' instructional skills, including rater standards. Yet we know little about how these measurement challenges play out in the preservice context specifically. Here, we investigate the reliability and sensitivity of two observational measures. We find measures collected during student teaching are especially prone to measurement issues; only 3% to 4% of variation in scores reflects consistent differences between PSTs, while 9% to 17% of variation can be attributed to the mentors with whom they work. When high scores stem not from strong instructional skills, but instead from external circumstances, we cannot use them to make consequential decisions about PSTs' individual needs or readiness for independent teaching. [ABSTRACT FROM AUTHOR]

Preservice teacher performance assessments, such as the edTPA, are one of the accountability policies from states and local authorities designed to ensure the quality of beginning teachers and standardize teacher education. We studied experiences of 65 preservice teachers regarding the effect of the edTPA on their learning in field-placement classrooms. These cases revealed that the edTPA created "protected teaching spaces" for participants to experiment with student-centered instructional practices supported in university courses and codified in edTPA rubrics. This was especially impactful for novices who previously had limited opportunities to try out equitable reform-oriented instruction in their placements. In these cases, the edTPA also helped mitigate inequities in learning to teach, an unintended outcome that is important for policymakers to consider when deciding on credentialing requirements. [ABSTRACT FROM AUTHOR]

The duration, accuracy, and consistency of responses to various types of stimuli are widely accepted as indirect indicators of the efficiency of brain information processing, but current clinical tests appear to lack sufficient sensitivity to detect subtle impairments. Immersive virtual reality (VR) offers a new means to acquire measures of perceptual-motor responses to moving visual stimuli that require rapid conflict resolution, but their test-retest reliability has not yet been demonstrated. Repeated measures. We analyzed data from 19 healthy young adults who performed a 40-trial VR test on three consecutive days. We focused on response time (RT) and perceptual latency (PL) for eye, neck, arm, and whole-body step displacements involved in executing a reaching/lunging movement in a right or left direction toward a peripherally located virtual target. Measures of RT and PL included a 40-trial mean, an intra-individual variability (IIV) value, and a rate correct score (RCS) that incorporated both response duration and accuracy. Most mean and IIV values for PL and RT demonstrated a positive distributional skew that was substantially reduced by natural logarithm transformation. While a learning effect was evident between sessions 1 and 2 for 7 of 8 mean PL and RT measures, 3-session intraclass correlation coefficient (ICC) values were moderate to excellent for 15 of 16 transformed PL and RT measures (range:.618 to.922). The composite RCS metric did not require transformation for either PL or RT, whose respective 3-session ICC values were.877 and.851. This moderate to excellent test-retest reliability for various VR measures of perceptual-motor function, combined with evidence of their validity from both past and future research, suggest that these measures can advance clinical detection of impaired brain processing and longitudinal assessments of potentially modifiable performance deficiencies. [ABSTRACT FROM AUTHOR]

Rating scale analysis techniques provide researchers with practical tools for examining the degree to which ordinal rating scales (e.g., Likert-type scales or performance assessment rating scales) function in psychometrically useful ways. When rating scales function as expected, researchers can interpret ratings in the intended direction (i.e., lower ratings mean "less" of a construct than higher ratings), distinguish between categories in the scale (i.e., each category reflects a unique level of the construct), and compare ratings across elements of the measurement instrument, such as individual items. Although researchers have used these techniques in a variety of contexts, studies are limited that systematically explore their sensitivity to problematic rating scale characteristics (i.e., "rating scale malfunctioning"). I used a real data analysis and a simulation study to systematically explore the sensitivity of rating scale analysis techniques based on two popular polytomous item response theory (IRT) models: the partial credit model (PCM) and the generalized partial credit model (GPCM). Overall, results indicated that both models provide valuable information about rating scale threshold ordering and precision that can help researchers understand how their rating scales are functioning and identify areas for further investigation or revision. However, there were some differences between models in their sensitivity to rating scale malfunctioning in certain conditions. Implications for research and practice are discussed. [ABSTRACT FROM AUTHOR]

Sparse rating designs, where each examinee's performance is scored by a small proportion of raters, are prevalent in practical performance assessments. However, relatively little research has focused on the degree to which different analytic techniques alert researchers to rater effects in such designs. We used a simulation study to compare the information provided by two popular approaches: Generalizability theory (G theory) and Many-Facet Rasch (MFR) measurement. In previous comparisons, researchers used complete data that were not simulated—thus limiting their ability to manipulate characteristics such as rater effects, and to understand the impact of incomplete data on the results. Both approaches provided information about rating quality in sparse designs, but the MFR approach highlighted rater effects related to centrality and bias more readily than G theory. [ABSTRACT FROM AUTHOR]

We study a decade of achievement gaps for fifth-, eighth-, and 10th-grade students in Norway using administrative population data. Norway is a wealthy and egalitarian country with a homogeneous educational system, yet achievement gaps between students at the 90th and 10th percentiles of parental income and between students whose parents have at least a master and at most a high school degree are found to be large (0.55–0.93 and 0.70–0.99 SD), equivalent to about 2 to 2.5 years of schooling, and increasing by grade level. Achievement gaps by parental income, but not by parental education, increased over the time period, underscoring the different ways these two socioeconomic status components relate to achievement and the potential for policy to alter gaps. [ABSTRACT FROM AUTHOR]

The National Building Code of Canada (NBC) employs seismic force modification factors (i.e. ductility-related factor, Rd, and an overstrength-related factor, Ro) for different Seismic Force Resisting Systems (SFRSs), which, since 2005, are largely based on engineering judgment and qualitative comparisons of seismic response characteristics of different SFRSs. Currently, there is no unified procedure in Canada for systematically quantifying these values, which has potentially led to perceived inconsistent levels of performance between buildings built with different SFRSs. While some research has been carried out in Canada to quantitatively evaluate the seismic response of different SFRSs using the FEMA P695 methodology with some modifications, inconsistencies exist among different research works. Moreover, the number of archetypes developed in these studies was limited, due to the computationally laborious incremental dynamic analysis (IDA) step in FEMA P695. Therefore, there is a need for a consistent and efficient methodology for structural performance assessment of different SFRSs in Canada. The proposed performance-based unified (PBU) procedure is to systematically quantify Rd and Ro factors for different SFRSs in the NBC. The PBU procedure is designed to be a performance-based approach (vs the risk-based approach of FEMA P695). In addition, our procedure is devised to optimize the number of archetypes through a two-level screening procedure using a nonlinear pushover and time history analyses before running the IDA. To validate the PBU procedure, we developed 21 different concrete moment frame archetypes covering various key parameters. The results verify the efficiency of the screening feature of the PBU procedure by a significant reduction in the number of archetypes requiring IDA. [ABSTRACT FROM AUTHOR]

In the present study, we examine whether characteristics of student teaching sites and cooperating teachers predict teacher candidates' edTPA scores. Using data from North Carolina, we find that candidates earn higher edTPA scores if they student-taught in a high value-added school and with a cooperating teacher earning higher evaluation ratings. Evidence also suggests that candidates earn higher scores when they are mentored by a cooperating teacher who is more familiar with performance assessments. Finally, we find that the relationships between student teaching environments and edTPA scores vary by characteristics of the candidate. This study reinforces prior work on the importance of learning environments to teacher development and the importance of preparation programs securing placements in high-quality learning environments. Our results may also have implications for the use of performance assessments by states and preparation programs. [ABSTRACT FROM AUTHOR]

Adoption of new instructional standards in science demands high-quality information about classroom practice. Teacher portfolios can be used to assess instructional practice and support teacher self-reflection anchored in authentic evidence from classrooms. This study investigated a new type of electronic portfolio tool that allows efficient capture of classroom artifacts in multimedia formats using mobile devices. We assess the psychometric properties of measures of quality instruction in middle school science classrooms derived from the contents of portfolios collected using this novel tool—with instruction operationalized through dimensions aligned to the Next Generation Science Standards. Results reflect low rater error and adequate reliability for several dimensions, a dominant underlying factor, and significant relations to some relevant concurrent indicators. Although no relation was found to student standardized test scores or course grades, portfolio ratings did relate to student self-efficacy perceptions and enjoyment of science. We examine factors influencing measurement error, and consider the broader implications of the results for assessing the validity of portfolio score interpretations, and the feasibility and potential value of this type of tool for summative and formative uses, in the context of large-scale instructional improvement efforts. [ABSTRACT FROM AUTHOR]

Structural faults like damage and degradations will cause changes in structure response data. Performance assessment can be conducted by investigating such changes. In real implementations however, structural responses are affected by environmental and operational variations (EOVs) as well. Such variation should be well captured by the assessment model when detecting structural changes. It should be noted that not all EOVs can be measured by the monitoring system. When both observed and latent EOVs have significant effects on the monitored structural responses, these two effects should be considered properly. Furthermore, uncertainties can be significant for the monitoring data since loads and EOVs cannot be directly controlled under working conditions. To address these problems, this work proposes a performance assessment method considering both observed and latent EOVs. A Gaussian process is used to model the functional behaviour between structural response and observed EOVs whilst principal component analysis is used to eliminate the effect of latent EOVs. These two methods are combined using a Bayesian formulation and the effect of both observed and latent EOVs are modelled. The associated model parameters are inferred through probability density functions to account for the uncertainties. A synthetic data example is presented to validate the proposed method. It is also applied to the monitoring data of a long-span cable-stayed bridge with different damage scenarios considered, illustrating its capability of real implementations in structural health monitoring. [ABSTRACT FROM AUTHOR]

The development of full-body exoskeletons has been limited due to design complexities, mechanical integration intricacies, and heavier weight, among others. Consequently, very few full-body powered exoskeletons were developed to address these challenges, in spite of increasing demand for physical assistance at full-body level. This article presents an overall design and development of a powered full-body exoskeleton called "FB-AXO." Primarily, FB-AXO consists of two main subsystems, a lower-body and an upper-body subsystem connected together through waist and spine modules. FB-AXO is developed for the support of weaker ageing adults so that they can continue functioning their daily activities. At the onset of the project, a set of functional and design requirements has been formulated with an extensive end-user involvement and then used in realizing the FB-AXO. The final FB-AXO design comprises of 27 degrees of freedom, of which 10 are active and 17 are passive, having a total system weight of 25 kg. Overall, the article elaborates comprehensively the design, construction, and preliminary testing of FB-AXO. The work effectively addresses design challenges including kinematic compatibility and modularity with innovative solutions. The details of the mechanics, sensors, and electronics of the two subsystems along with specifics of human-exoskeleton interfaces and ranges of motion are also provided. The FB-AXO exoskeleton effectively demonstrated to assist full-body motions such as normal walking, standing, bending as well as executing lifting and carrying tasks to meet the daily living demands of older users. [ABSTRACT FROM AUTHOR]

Practical constraints in rater-mediated assessments limit the availability of complete data. Instead, most scoring procedures include one or two ratings for each performance, with overlapping performances across raters or linking sets of multiple-choice items to facilitate model estimation. These incomplete scoring designs present challenges for detecting rater biases, or differential rater functioning (DRF). The purpose of this study is to illustrate and explore the sensitivity of DRF indices in realistic sparse rating designs that have been documented in the literature that include different types and levels of connectivity among raters and students. The results indicated that it is possible to detect DRF in sparse rating designs, but the sensitivity of DRF indices varies across designs. We consider the implications of our findings for practice related to monitoring raters in performance assessments. [ABSTRACT FROM AUTHOR]

This study aimed to test the reliability, sensitivity, construct and logical validity of an adapted Special Judo Fitness Test (SJFT) for judo athletes with visual impairment. Twenty judo athletes with visual impairments performed both the adapted SJFT with tactile and sonorous stimuli (experimental conditions) and the typically administered SJFT (standard condition). We used analyses of variance (ANOVAs) with repeated-measures to compare the groups' SJFT performances, and one-way ANOVAs to compare different visual ability classes of athletes (B1, B2 and B3). We used t-tests to compare SJFT variables between elite and sub-elite groups. We set statistical significance for all tests at p < 0.05. The standard SJFT showed excellent test-retest reliability for number of throws and overall index (ICC = 0.91–0.95), and both sonorous and tactile sensitivity adaptations of the SJFT showed medium sensitivity for detecting performance changes. The number of throws and SJFT index were higher with the sonorous adaptation of the test, compared to the tactile and standard versions (p < 0.001). Athletes who were blind (B1) presented similar performances to athletes who were partially sighted (B2 and B3) only on the SJFT with the sonorous stimulus. Moreover, only the sonorous SJFT adaptation discriminated between the performances of elite and sub-elite athletes (p < 0.001). In conclusion, both SJFT adaptations showed excellent reliability and medium sensitivity on test-retest, but, only the SJFT with the sonorous stimulus seemed valid for assessing judo athletes with varying degrees of visual impairment, and only the sonorous stimulus SJFT discriminated elite from sub-elite athletes. [ABSTRACT FROM AUTHOR]

The estimation of high-dimensional latent regression item response theory (IRT) models is difficult because of the need to approximate integrals in the likelihood function. Proposed solutions in the literature include using stochastic approximations, adaptive quadrature, and Laplace approximations. We propose using a second-order Laplace approximation of the likelihood to estimate IRT latent regression models with categorical observed variables and fixed covariates where all parameters are estimated simultaneously. The method applies when the IRT model has a simple structure, meaning that each observed variable loads on only one latent variable. Through simulations using a latent regression model with binary and ordinal observed variables, we show that the proposed method is a substantial improvement over the first-order Laplace approximation with respect to the bias. In addition, the approach is equally or more precise to alternative methods for estimation of multidimensional IRT models when the number of items per dimension is moderately high. Simultaneously, the method is highly computationally efficient in the high-dimensional settings investigated. The results imply that estimation of simple-structure IRT models with very high dimensions is feasible in practice and that the direct estimation of high-dimensional latent regression IRT models is tractable even with large sample sizes and large numbers of items. [ABSTRACT FROM AUTHOR]

The Personal Background Preparation Survey (PBPS) identifies students at risk for academic nonadvancement. Uniquely, the PBPS produces individualized reports making evidence-based risk-specific recommendations prescribing interventions targeting students' empirically identified risk indicators. At a large southwestern health sciences community college, after baseline PBPS administration among 409 diverse first-semester-fall 2010 students, fall 2011 PBPS administration helped target PBPS-individualized interventions among 618 first-semester-fall 2011 students. Group-oriented Advanced Academic Training (AAT) workshops augmented PBPS-targeted individualized interventions among 1,183 additional first-semester students during fall 2012 and fall 2013. AAT participants practiced a daily self-testing retrieval regimen to reduce PBPS-identified cognitive processing, information-, and time-management risk indicators. Controlling PBPS risk level, underrepresented minority status, and gender as covariates, first-semester student nonadvancement rate decreased from baseline's 42.3% and PBPS-individualized interventions' 41.4% to 16.2% and 11.6% postAAT (p <.001), respectively. AAT was designed to reduce primarily higher risk student nonadvancement; yet, retention gains did not differ significantly across risk levels, underrepresented minority students status, and gender. [ABSTRACT FROM AUTHOR]

This study introduces a novel differential item functioning (DIF) method based on propensity score matching that tackles two challenges in analyzing performance assessment data, that is, continuous task scores and lack of a reliable internal variable as a proxy for ability or aptitude. The proposed DIF method consists of two main stages. First, propensity score matching is used to eliminate preexisting group differences before the test, ideally creating equivalent groups as in a randomized experimental study. Then, linear mixed effects models are adopted to perform DIF analysis based on the matched data set. We demonstrate this propensity DIF method using a high-stakes functional English language proficiency test. DIF due to education was investigated in the writing component, which consists of two continuously scored performance-based tasks. Although the proposed method is demonstrated in the context of language testing, it can be applied to other types of performance assessments. [ABSTRACT FROM AUTHOR]

Random-vibration-based statistical time series structural health monitoring methods utilize small-scale, compact, and data-based, time series stochastic representations of the structural dynamics for damage diagnosis. In this study, a comprehensive and critical assessment of the diagnostic performance of five prominent response-only methods is presented based on incipient, 'minor' to 'mild', damages on a lab-scale wind turbine jacket structure. Statistically reliable damage detection and identification results are obtained via a 'rotation' procedure resulting into thousands of test cases, with the performance analysed in terms of receiver operating characteristic curves and confusion matrices. The results indicate not only challenging of the methods' capabilities, but also the achievement of good to excellent performance for the 'minor' to 'mild' damages, respectively, with the model parameter–based method offering the best performance. In addition, the use of a vibration signal measured via a laser vibrometer leads to slightly improved detection performance over that obtained via a classical accelerometer. [ABSTRACT FROM AUTHOR]

Background. The English National Health Service publishes hospital performance indicators based on average postoperative EQ-5D index scores after hip replacement surgery to inform prospective patients' choices of hospital. Unidimensional index scores are derived from multidimensional health-related quality-of-life data using preference weights estimated from a sample of the UK general population. This raises normative concerns if general population preferences differ from those of the patients who are to be informed. This study explores how the source of valuation affects hospital performance estimates. Methods. Four different value sets reflecting source of valuation (general population v. patients), valuation technique (visual analog scale [VAS] v. time tradeoff [TTO]), and experience with health states (currently experienced vs. experimentally estimated) were used to derive and compare performance estimates for 243 hospitals. Two value sets were newly estimated from EQ-5D-3L data on 122,921 hip replacement patients and 3381 members of the UK general public. Changes in hospital ranking (nationally) and performance outlier status (nationally; among patients' 5 closest hospitals) were compared across valuations. Results. National rankings were stable under different valuations (rank correlations >0.92). Twenty-three (9.5%) hospitals changed outlier status when using patient VAS valuations instead of general population TTO valuations, the current approach. Outlier status also changed substantially at the local level. This was explained mostly by the valuation technique, not the source of valuations or experience with the health states. Limitations. No patient TTO valuations were available. The effect of value set characteristics could be established only through indirect comparisons. Conclusion. Different value sets may lead to prospective patients choosing different hospitals. Normative concerns about the use of general population valuations are not supported by empirical evidence based on VAS valuations. [ABSTRACT FROM AUTHOR]

Longitudinally measured biomarkers are useful to predict the risk of clinical endpoints, since subject-specific marker trajectory contains additional information on pathology and critical windows. The work is motivated by the Scandinavian Fetal Growth Study, aiming at predicting pregnancy outcomes with repeated ultrasound measurements during pregnancy. While the observation time of markers often varies across individuals, it is not well understood how the variations affect risk prediction. Existing methods of longitudinal risk prediction, such as shared random effects model and pattern mixture model, construct a prediction implicitly as a function of the biomarkers and their observation time. Methods that ignore the longitudinal structure, such as sufficient dimension reduction and logistic regression, have better interpretability regarding how a biomarker measured at specific time window contributes to the disease risk, but often have reduced accuracy because of ignoring the observation time information. We propose a novel imputation approach to handle the random observation time, while preserving the direct interpretation. Through extensive simulation studies and analyses of the Scandinavian Fetal Growth Study data, we systematically compared the discrimination and calibration performance of different risk prediction methods, and found that the imputation method has comparable performance to longitudinal methods with an advantage of better interpretability. [ABSTRACT FROM AUTHOR]

edTPA is designed to strengthen teacher professionalization and provide a framework for program redesign. However, using a national assessment to shift the content of local programs is challenging because of their inherent organizational complexity. In this article, we focus on this complexity, using a systems lens to analyze edTPA implementation at a large, public university. Employing a mixed-methods case study design, we survey 250 teacher educators and candidates to understand how they interpret the demands of edTPA and how their varied perspectives impact each other. We interview a stratified, purposive subset of participants to explore mechanisms underlying quantitative findings. We find substantial internal variation in edTPA implementation that translates into differential support for candidates. This variation could not be explained by duration of implementation of edTPA. Varied perspectives may stem from distinct perceptions of teacher educators' professional roles and the role they see edTPA playing in teacher professionalization. [ABSTRACT FROM AUTHOR]

Contemporary teacher evaluation systems use multiple measures of performance to construct ratings of teacher quality. While the properties of constituent measures have been studied, little is known about whether composite ratings themselves are sufficiently reliable to support high-stakes decision making. We address this gap by estimating the consistency of composite ratings of teacher quality from New Mexico's teacher evaluation system from 2015 to 2016. We estimate that roughly 40% of teachers would receive a different composite rating if reevaluated in the same year; 97% of teachers would receive ratings within ±1 level of their original rating. We discuss mechanisms by which policymakers can improve rating consistency, and the implications of those changes to other properties of teacher evaluation systems. [ABSTRACT FROM AUTHOR]

This article is an overview of the National Assessment of Education Quality (NAEQ) of China in reading, mathematics, sciences, arts, physical education, and moral education at Grades 4 and 8. After a review of the background and history of NAEQ, we present the assessment framework with students' holistic development at the core and the design for each subject used in the 2015–2017 assessment cycle. Technical details including item response modeling and the standard setting procedure are presented. We conclude with a discussion of the social impact, current issues, and future directions of national educational assessment. [ABSTRACT FROM AUTHOR]

This exploratory study examines the relationship between preservice teachers' performance on a teaching performance assessment for licensure in elementary mathematics and a measure of knowledge that, in studies of practicing teachers, was found to predict effective mathematics teaching. A sample of 89 preservice teachers completed the Performance Assessment for California Teachers (PACT) and the classroom video analysis (CVA) instrument. Correlation analyses found overall weak associations between measures for the whole group. For groups of high, medium, and low performers on PACT, average scores on the CVA measure and its subscales varied in the predicted direction. However, individuals within the PACT performing groups had a relatively broad range of scores on the CVA, and several preservice teachers who performed poorly on PACT had average or better than average performance on the CVA. Similarly, several of the PACT high performers scored below the whole group CVA average. In addition to suggesting areas for future research, the findings raise questions about assessing preservice teachers' readiness to teach mathematics and the use of a single measure to make licensing decisions. [ABSTRACT FROM AUTHOR]

Rater effects, or raters' tendencies to assign ratings to performances that are different from the ratings that the performances warranted, are well documented in rater-mediated assessments across a variety of disciplines. In many real-data studies of rater effects, researchers have reported that raters exhibit more than one effect, such as a combination of misfit and systematic biases related to student subgroups (i.e., differential rater functioning [DRF]). However, researchers who conduct simulation studies of rater effects usually focus on the effects in isolation. The purpose of this study was to explore the degree to which rater effect indicators are sensitive to rater effects when raters exhibit more than one type of effect, and to explore the degree to which this sensitivity changes under different data collection designs. We used a simulation study to explore combinations of DRF and rater misfit. Overall, our findings suggested that it is possible to use common numeric and graphical indicators of DRF and rater misfit when raters exhibit both these effects, but that these effects may be difficult to distinguish using only numeric indicators. We also observed that combinations of rater effects are easier to identify when complete rating designs are used. We discuss implications of our findings as they result to research and practice. [ABSTRACT FROM AUTHOR]

Social-emotional learning (SEL) is gaining increasing attention in education policy and practice due to growing evidence that related constructs are strongly predictive of long-term academic achievement and attainment. However, the work of educators to support SEL is hampered by a lack of available, unbiased measures of related competencies. In this study we conducted a literature review to investigate whether assessment metadata (typically data relevant to how students behave on a test or survey) can provide information on SEL constructs. Implications of this new source of SEL data for practice, policy, and research are discussed. [ABSTRACT FROM AUTHOR]

Flipped classroom is an emerging course delivery method, where the traditional in-class and out-of-class activities flip places. The lectures are given out-of-class with electronic resources and in-class time is reserved for group activities and discussion with the instructor. In this study, we present our implementation of a flipped classroom in a senior level undergraduate engineering course. An extensive perception analysis was carried out with anonymous surveys and face-to-face interviews with students. Performance analysis of the flipped class was carried out by comparing the achievement level in the course learning outcomes of previous years' traditional classes and recent year's flipped class. The out-of-class lectures were implemented as machine-voice-narrated presentations. The effect of machine voice on the flipped classroom is also examined. Students' perception and reaction to various aspects of the flipped class are documented. Performance analysis showed a significant improvement in one category of course learning outcomes. Student perception was positive overall. [ABSTRACT FROM AUTHOR]

This study examines how secondary school teachers have implemented educational reforms in the Philippines. Data sources were surveys for sample teachers and students in 20 schools on how often a teaching or evaluation strategy was used in various courses. The study concluded that most teachers were able to adopt most of the reforms. Even though teacher-directed lessons still dominated, there were sufficient student-centered lessons. Tests, exams, and performance tasks were used for evaluations. However, classroom management needs increased attention. More resources and professional development are needed. Policymakers and educators in all jurisdictions can learn from the reform efforts. [ABSTRACT FROM AUTHOR]

Large-scale assessments still focus on those aspects of students' competence that can be evaluated using paper-and-pencil tests (or computer-administered versions thereof). Performance tests are considered costly due to administration and scoring, and, more importantly, they are limited in reliability and validity. In this article, we demonstrate how a sociocognitive perspective provides an understanding of these issues and how, based on this understanding, an argument-based approach to assessment design, interpretation, and use can help to develop comprehensive, yet reliable and valid, performance-based assessments of student competence. More specifically, we describe the development of a computer-administered, simulation-based assessment that can reliably and validly assess students' competence to plan, perform, and analyze physics experiments at a large scale. Data from multiple validation studies support the potential of adopting a sociocognitive perspective and assessments based on an argument-based approach to design, interpretation, and use. We conclude by discussing the potential of simulations and automated scoring methods for reliable and valid performance-based assessments of student competence. [ABSTRACT FROM AUTHOR]

Rater effects such as severity, centrality, and misfit are recurrent concerns in performance assessments. Despite their persistence in operational assessment settings and frequent discussion in research, researchers have not fully explored the impacts of rater effects as they relate to estimates of student achievement. The purpose of this study is to explore the impacts of rater severity, centrality, and misfit on student achievement estimates and on classification decisions. The results suggest that these three types of rater effects have substantial impacts on estimates of student achievement and on classification decisions that impact the fairness of rater-mediated assessments. Accordingly, it is essential that researchers and practitioners evaluate ratings across all stages of rater-mediated assessment procedures, including rater training and operational scoring. [ABSTRACT FROM AUTHOR]

Preparing, recruiting, and retaining high-quality teachers into the profession has been a concern of policy makers and practitioners for some time. Teacher attrition is problematic and costly for schools and districts. However, relatively few studies have investigated the relationship between preservice teacher quality and teacher attrition. In this study, we analyze data from an apprenticeship-style teacher preparation program to understand the relationship between a measure of preservice teacher quality-student teachers' observational scores-and their decisions to (a) enter into the profession, and (b) stay in the profession within the first 2 years after graduation. We find that more qualified student teachers are more likely to enter into the profession and stay in the profession, even after controlling for student teachers' demographic characteristics and their academic achievement. [ABSTRACT FROM AUTHOR]

In an era of increased teacher accountability, teacher preparation programs across the country are faced with increasing pressure to adopt rigorous and high stakes performance-based assessments, such as edTPA, that provide data-based evidence that their candidates are ready to teach upon program completion. Furthermore, in response to new accreditation standards for teacher training programs established by the Council for Accreditation of Educator Preparation, colleges of education are now required to demonstrate enduring and mutually beneficial partnerships with K-12 partners. Given the influence of the K-12 cooperating teacher (CT) on candidate classroom practice, providing professional development (PD) to CTs on edTPA represents a possible means of accomplishing both goals. This study examined the impact of an edTPA PD workshop designed specifically for CTs on CT practice and candidate edTPA scores. Results suggest that candidates placed with CTs who have received edTPA PD can benefit from increased CT knowledge about the edTPA assessment. [ABSTRACT FROM AUTHOR]

Anchor-based, end-of-shift ratings are commonly used to conduct performance assessments of resident physicians. These performance evaluations often include narrative assessments, such as solicited or "free-text" commentary. Although narrative commentary can help to create a more detailed and specific assessment of performance, there are limited data describing the effects of narrative commentary on the global assessment process. This single-group, observational study examined the effect of narrative comments on global performance assessments. A subgroup of the clinical competency committee, blinded to resident identity, assigned a single, consensus-based performance score (1-6) to each resident based solely on end-of-shift milestone scores. De-identified narrative comments from end-of-shift evaluations were then included and the process was repeated. We compared milestone-only scores to milestone plus narrative commentary scores using a nonparametric sign test. During the study period, 953 end-of-shift evaluations were submitted on 41 residents. Of these, 535 evaluations included free-text narrative comments. In 17 of the 41 observations, performance scores changed after the addition of narrative comments. In two cases, scores decreased with the addition of free-text commentary. In 15 cases, scores increased. The frequency of net positive change was significant (p = .0023). The addition of narrative commentary to anchor-based ratings significantly influenced the global performance assessment of Emergency Medicine residents by a committee of educators. Descriptive commentary collected at the end of shift may inform more meaningful appraisal of a resident's progress in a milestone-based paradigm. The authors recommend clinical training programs collect unstructured narrative impressions of residents' performance from supervising faculty. [ABSTRACT FROM AUTHOR]

Teacher candidate performance assessments represent a promising source of data for evidence-based program improvement. However, teacher preparation programs (TPPs) interested in reform face a crucial question: how to identify actionable evidence in performance-assessment data. To address this concern, we propose a two-pronged empirical framework that TPPs can use to analyze performance-assessment data. The first approach, latent class analysis, creates profiles of instructional practice by grouping candidates together based on similarities in their performance-assessment scores. This can help TPPs provide targeted supports to candidates. The second approach, predictive validity analyses, estimates relationships between candidates’ performance-assessment scores and their performance as teachers-of-record. This can help TPPs identify programmatic elements significantly related to teacher outcomes. We illustrate this framework with Educative Teacher Performance Assessment (edTPA) data from a Partner University and contend that the impact of performance assessments can be amplified by these common strategies for analyzing performance-assessment data. [ABSTRACT FROM AUTHOR]

The interpretation of ratings from educational performance assessments assumes that rating scale categories are ordered as expected (i.e., higher ratings correspond to higher levels of judged student achievement). However, this assumption must be verified empirically using measurement models that do not impose ordering constraints on the rating scale category thresholds, such as item response theory models based on adjacent-categories probabilities. This study considers the application of an adjacent-categories formulation of polytomous Mokken scale analysis (ac-MSA) models as a method for evaluating the degree to which rating scale categories are ordered as expected for individual raters in performance assessments. Using simulated data, this study builds on the preliminary application of ac-MSA models to rater-mediated performance assessments, in which a real data analysis suggested that these models can be used to identify disordered rating scale categories. The results suggested that ac-MSA models are sensitive to disordered categories within individual raters. Implications are discussed as they relate to research, theory, and practice for rater-mediated educational performance assessments. [ABSTRACT FROM AUTHOR]

Model-data fit indices for raters provide insight into the degree to which raters demonstrate psychometric properties defined as useful within a measurement framework. Fit statistics for raters are particularly relevant within frameworks based on invariant measurement, such as Rasch measurement theory and Mokken scale analysis. A simple approach to examining invariance is to examine assessment data for evidence of Guttman errors. I used real and simulated data to illustrate and explore a nonparametric procedure for evaluating rater errors based on Guttman errors and to examine the alignment between Guttman errors and other indices of rater fit. The results suggested that researchers and practitioners can use summaries of Guttman errors to identify raters who exhibit misfit. Furthermore, results from the comparisons between summaries of Guttman errors and parametric fit statistics suggested that both approaches detect similar problematic measurement characteristics. Specifically, raters who exhibit many Guttman errors tended to have higher-than-expected Outfit MSE statistics and lower-than-expected estimated slope statistics. I discuss implications of these results as they relate to research and practice for rater-mediated assessments. [ABSTRACT FROM AUTHOR]

This article presents pilot study results of two leadership performance assessments, designed for a California principal preparation program and embedded in preparation using two learning approaches. The pilot study had two purposes: to evaluate the assessments' content validity and to evaluate the candidates' leadership skills as demonstrated through their assessment products and an independent self-assessment tool. The evidence showed the tasks to be valid and useful tools for formative leadership development for different candidates and school settings. Participant feedback on the assessments' benefits and independent self-assessment ratings provided construct validation. We concluded that these are promising assessment tools for programs' use in candidate assessment. [ABSTRACT FROM AUTHOR]

Recent research has explored the use of models adapted from Mokken scale analysis as a nonparametric approach to evaluating rating quality in educational performance assessments. A potential limiting factor to the widespread use of these techniques is the requirement for complete data, as practical constraints in operational assessment systems often limit the use of complete rating designs. In order to address this challenge, this study explores the use of missing data imputation techniques and their impact on Mokken-based rating quality indicators related to rater monotonicity, rater scalability, and invariant rater ordering. Simulated data and real data from a rater-mediated writing assessment were modified to reflect varying levels of missingness, and four imputation techniques were used to impute missing ratings. Overall, the results indicated that simple imputation techniques based on rater and student means result in generally accurate recovery of rater monotonicity indices and rater scalability coefficients. However, discrepancies between violations of invariant rater ordering in the original and imputed data are somewhat unpredictable across imputation methods. Implications for research and practice are discussed. [ABSTRACT FROM AUTHOR]

Although teacher education is the formal means by which novices are prepared for teaching, they come having already had significant experience in schools. Preservice teachers have formed habits of “teaching” which influence their learning to teach. This article reports a study of the specific knowledge of and skills with teaching practice that novices bring to teacher education with respect to one teaching practice, eliciting student thinking in elementary mathematics, and describes the use of a standardized teaching simulation to learn about novices’ skills. The findings reveal details about preservice teachers’ skills and habits of practice at the point that they enter formal teacher preparation. Preservice teachers’ ways of carrying out this particular practice are categorized into three distinct categories: (a) skills that need to be learned, (b) skills that can be built on, and (c) approaches that need to be unlearned. [ABSTRACT FROM AUTHOR]

The trends in students’ performance in science and mathematics at secondary school level were examined within a decade-long interval in Ghana. Using West African Examinations Council (WAEC) data from approximately 20% of schools countrywide, it was observed that the results did not provide a definitive pattern. Factors such as teachers’ inability to complete between 50% and 75% of the curricula, insufficient-time to enact and complete the curriculum, lack of enough qualified teachers, and use of poor teaching approaches accounted for the trends. This study in Ghana has implications in teacher recruitment deployment in general as well as duration of secondary education globally to examine their efficacy. [ABSTRACT FROM AUTHOR]

This work proposes a method for assessment of evaluation processes, applicable to systems in which both measurable and policy-based performance criteria coexist. It intends to provide a mapping procedure, based solely on measurable criteria, aiming at visual inspection of the outcome, which allows the introduction of two types of indicators at different phases of the process. The definition of performance-based zones is suggested: two of them clearly correspond to high/low performance, in which the overall evaluation result is essentially established on the basis of measurable indicators, i.e. on the basis of objective criteria, and an in-between area, more prone to a less deterministic outcome. The method of evaluation is applied in three successive steps: (i) build the representation and identify the performance zones, (ii) introduce additional policy criteria to complete a first draft of the outcome, and, finally (iii) assess relative positioning, correct if necessary, and finalize. Performance maps are presented and inspected for a set of Research and Development institutions, to illustrate examples of application of the methodology proposed. [ABSTRACT FROM AUTHOR]

The acquisition of teacher digital competence is a key aspect in the initial training of teachers. However, most existing evaluation instruments do not provide sufficient evidence of this teaching competence. In this study, we describe the design and development process of a three-dimensional (3D) virtual environment for evaluating the teacher digital competence of future teachers, through a performance-based, collaborative and contextual evaluation. This environment, named ETeach3D, has been constructed using the educational design research approach. It is based on successive iterative cycles and is in accordance with the criteria of usefulness, validity, and effectiveness. In addition to the research team responsible for the project, participating in this study were 187 Spanish undergraduate students of Education and 22 experts in the field of educational technology. Results show that these environments, in addition to other characteristics, should (a) function smoothly and have simple interfaces, realistic scenes, and interactive activities and (b) follow a systematic evaluation procedure that integrates several strategies and levels of complexity. This research helps to improve the initial training of preservice teachers and contributes to the growing number of educational design research studies that focus in the field of evaluation of the curriculum domain. [ABSTRACT FROM AUTHOR]

In this article, the author describes a different theory of multiple abilities, the Prism Model, developed with Usanee Anuruthwong from Thailand, and further evolved with Belle Wallace from the UK and Leonie Kronborg from Australia. Using this model, insights gained will be shared that have been gained from working with children and adults from different cultures, especially the Diné people in northern Arizona, USA, and my journey toward development of my spiritual awareness and abilities. These insights will help readers find their own ways to recognize and develop these abilities in their students. Our current work with the Real Engagement in Active Problem Solving model will illustrate practical ways educators can develop spiritual abilities in classroom contexts while teaching the required academic content. [ABSTRACT FROM AUTHOR]

Prevalence of academic risk (PAR) group profiles provide data enabling empirically based group-specialized prescriptions for targeted academic success interventions to increase student retention, completion, and graduation rates, while improving allocation of institutional resources. Postsecondary student attrition engenders student debt, shortages of highly trained professionals, and inadequate workforce diversity. Diagnostic PAR profiles, employing a proactive Diagnosis ⇒ Prescription ⇒ Intervention ⇒ Evaluation Model adapted from clinical medicine, enable empirically targeted, group-specialized, student-success interventions. Two-step cluster analysis of survey responses from 512 students entering first-semester urban health science community college differentiated three PAR-profile groups: (a) traditional, (b) language/education/financially challenged, and (c) older/supporting families/financially challenged. The latter two groups’ end-of-first-semester odds of adverse academic status events were, respectively, 1.73 (p = .016) and 2.08 (p = .002) times the traditional group’s. PAR profiles enable empirically based group-tailored interventions for increasing student retention, completion, and graduation rates, while improving allocation of institutional resources. [ABSTRACT FROM AUTHOR]

The No Child Left Behind (NCLB) Act required states to adopt accountability systems measuring student proficiency on state-administered exams. The federal legislation contained several strict requirements for NCLB implementation, such as escalating student proficiency targets that reach 100% proficiency by 2014. But it also gave states considerable flexibility to interpret and implement components of NCLB. Using a data set we constructed, this paper is the first national study examining which schools failed during the early years of NCLB and which performance targets they failed to meet. We explore how states’ NCLB implementation decisions were related to their schools’ failure rates, which ranged from less than 1% to more than 80% across states. Wide cross-state variation in failure rates resulted from how states’ decisions interacted with each other and with school characteristics, like enrollment size, grade span, and ethnic diversity. Subtle differences in policy implementation may cause dramatic differences in measured outcomes. [ABSTRACT FROM AUTHOR]

The purpose of this study was to propose extensions of reliability estimation methods that could be used to determine the conditions under which single scoring for constructed-response (CR) items is as effective as double scoring in mixed-format licensure tests. Multivariate generalizability theory methods traditionally used to estimate overall composite score reliability were extended with simulations so that classification consistency and classification accuracy estimates could also be obtained. Composite score reliabilities, classification consistencies, and accuracies were estimated based on the double and single scoring of the CR items of three licensure tests. Composite score reliabilities, classification consistencies, and accuracies were also estimated in decision studies considering varied testing situations such as different numbers of CR items and different CR section weights. [ABSTRACT FROM AUTHOR]

This commentary on the article titled “Examining the Internal Structure Evidence for the Performance Assessment for California Teachers: A Validation Study of the Elementary Literacy Teaching Event for Tier 1 Teacher Licensure” provides an overview of Performance Assessment for California Teachers (PACT), its relationship to edTPA and the construct (domains) measured, a summary of the major findings of the article commented on, a discussion of PACT validity issues and rubrics, and the need for instructionally sensitive and formative assessments. It concludes with recommendations to the State of California that include attention to construct validity, improved rubrics, formative use of the data, and attention to utility as well as validity and reliability. [ABSTRACT FROM PUBLISHER]

The edTPA, a nationally available performance assessment for teacher candidates, has recently been developed and implemented in teacher education programs across the United States. Advocates make arguments for the need for such an assessment while critics of standardized performance assessments point out the dangers of standardization. This article takes a step back from the arguments in support of or in opposition to the assessment and asks fundamentally what the underlying conception of teaching of the edTPA is. After examining conceptions of teaching articulated by scholars such as Nathanial Gage, Larry Cuban, Lee Shulman, Paulo Freire, and bell hooks, this article argues that the underlying conception of teaching of the edTPA is one of professional practice, not only at the individual level but also at the level of teaching as a collective enterprise. The conception of teaching argument is also connected to discussions of the validity arguments for the edTPA with specific attention to face validity, content validity, and construct validity. [ABSTRACT FROM PUBLISHER]