Search

Background: Given pressures to control costs and improve quality of care, one of the most prevalent transformational performance improvement approaches in health care is Lean management. However, the roles of support functions such as human resource (HR), finance, and information technology (IT) in Lean management and the relationships of these support functions with performance are unknown. Purpose: The aim of this studywas to examine the relationships between the HR, finance, and IT functions, overall Lean implementation, and self-reported performance improvement in hospitals that have implemented Lean. Methodology/Approach: Data from a national survey of Lean in U.S. hospitals (N = 1,222; 847 reported using Lean) were analyzed using multivariable regression and bootstrapped mediation analysis. The extent to which HR, finance, and IT functions support Lean management was measured using indices including six, three, and six items respectively. Lean implementation was measured by the number of units doing Lean (up to 29) and by a four-level self-reported maturity scale. Performance improvement was measured using an index of self-reported achievements (ranging from 0 to 16). Results: There were significant positive associations between Lean HR, finance, and IT functions and self-reported performance impact (controlling for organizational and market variables). Tests of mediation indicated that the associations of HR, finance, and IT functions with self-reported performance were significantly mediated by the number of Lean units (mediated proportion ranging from 40% to 73%), and HR function was also mediated by self-reported maturity (61% mediated). There were no moderating effects. Conclusion: HR, finance, and IT functions are positively associated with self-reported Lean impact on performance and primarily explained by the overall degree of Lean implementation. Practice Implications: Efforts to align HR, finance, and IT functions with overall Lean implementation can help to ensure that frontline caregivers and managers have the data and skills required to meet transformational improvement goals. [ABSTRACT FROM AUTHOR]

Educator preparation programs across the United States are increasingly adopting performance-based assessments, including edTPA, as a mechanism to measure candidates' development of the knowledge and skills necessary to enter the classroom upon program completion. Within this context, this investigation examined associations between elementary candidates' edTPA scores, background characteristics, preparation program features, and performance on practice tasks. EdTPA scores were collected for six cohorts of elementary teaching candidates, including 392 undergraduate and 115 graduate certificate candidates. Multiple imputation and ordinary least squares regression were used to assess associations. Better Grade Point Averages and performance on practice tasks were consistently associated with higher final edTPA scores for undergraduates. Perceived preparation from coursework predicted higher final edTPA scores in some cases. The results are discussed relative to their implications for identifying candidates who may require additional supports and for targeting program resources to supports that have the strongest associations with final scores. [ABSTRACT FROM AUTHOR]

The Performance Assessment for Leaders (PAL) is a four-task evaluation system designed to improve the quality of initial school leaders in Massachusetts by evaluating candidate readiness using a rigorous, standard assessment system. By design, it is signalling leadership quality expectations to preparation programmes. Through two years of survey research with leadership preparation faculty (n = 12) and candidates (n = 53) who completed PAL, we explored faculty understanding the PAL assessment, assessment alignment task to programme content, candidate preparedness, and impact on programmes. Results show that faculty used PAL to improve their preparation programmes and provide high-quality learning experiences for candidates. [ABSTRACT FROM AUTHOR]

This article presents pilot study results of two leadership performance assessments, designed for a California principal preparation program and embedded in preparation using two learning approaches. The pilot study had two purposes: to evaluate the assessments' content validity and to evaluate the candidates' leadership skills as demonstrated through their assessment products and an independent self-assessment tool. The evidence showed the tasks to be valid and useful tools for formative leadership development for different candidates and school settings. Participant feedback on the assessments' benefits and independent self-assessment ratings provided construct validation. We concluded that these are promising assessment tools for programs' use in candidate assessment. [ABSTRACT FROM AUTHOR]

Packaged air-conditioning systems, such as rooftop units (RTUs), are widely used in small commercial buildings in the United States. Although new regulations and standards require new air-conditioning equipment to have multi-speed supply fan operation, most of the installed units employ fixed-speed supply fans. The electrical energy usage associated with constant volumetric air delivery fan systems is on the order of half the total RTU electrical energy usage because supply fans operate continuously during the occupied periods in commercial buildings for ventilation. In addition, compressors within RTUs typically feature single- or two-stage capacity controls leading to frequent unit cycling and thus, deteriorated performance in energy efficiency. This paper presents relatively comprehensive assessment results for three RTU variable-speed retrofit options: 1) retrofits of fixed-speed supply fans with variable-speed fans for single-compressor-speed units, 2) replacement of existing fixed-speed RTUs with RTUs having two-speed compressors with variable-speed supply fans and 3) retrofits of RTUs with new RTUs that incorporate variable-speed compressors and fans (indoor and outdoor). The assessment study involves parametric simulations across different building types and locations in the U.S. to understand the performance variation with respect to location and building use. Performance improvements in terms of energy efficiency, operation cost and indoor comfort delivery for the various retrofit options are demonstrated compared to conventional single-stage RTUs with constant speed supply fans. The assessment results show that all three retrofit options offer significant fan energy savings, but much smaller savings potential for compressor and condenser fan electricity consumption. The variable-speed-unit retrofit provides the highest cost savings potentials, ranging from 40 to 250 $ per rated cooling ton per year depending on the building type and location. In addition to efficiency and economic benefits, the two-stage-unit and variable-speed-unit retrofits also lead to improved indoor humidity regulation performance due to reduced unit cycling. [ABSTRACT FROM AUTHOR]

This case study was conducted within the context of a place-based education project that was implemented with primary school students in the USA. The authors and participating teachers created a performance assessment of standards-aligned tasks to examine 6–10-year-old students’ graph interpretation skills as part of an exploratory research project. Fifty-five students participated in a performance assessment interview at the beginning and end of a place-based investigation. Two forms of the assessment were created and counterbalanced within class at pre and post.In situscoring was conducted such that responses were scored as correct versus incorrect during the assessment's administration. Criterion validity analysis demonstrated an age-level progression in student scores. Tests of discriminant validity showed that the instrument detected variability in interpretation skills across each of three graph types (line, bar, dot plot). Convergent validity was established by correlatingin situscores with those from the Graph Interpretation Scoring Rubric. Students’ proficiency with interpreting different types of graphs matched expectations based on age and the standards-based progression of graphs across primary school grades. The assessment tasks were also effective at detecting pre–post gains in students’ interpretation of line graphs and dot plots after the place-based project. The results of the case study are discussed in relation to the common challenges associated with performance assessment. Implications are presented in relation to the need for authentic and performance-based instructional and assessment tasks to respond to the Common Core State Standards and the Next Generation Science Standards. [ABSTRACT FROM PUBLISHER]

This exploratory study examines the relationship between personality and classroom performance. The personality measure is True Colors, the testing is measured by results on a paper-and-pencil test in a senior-level human resource management course at a large southeastern U.S. university, and the performance assessment is grades on written assignments. The link between personality and testing is contentious, with some researchers stating a definitive link between the two and other authors stating no link between the two components. Results of the data analysis suggest that the Blue, Gold, and Orange colors of the True Colors personality traits influence student learning in the classroom. By understanding the relationship between a student’s personality and his or her success in the classroom, an instructor can enhance student learning. [ABSTRACT FROM AUTHOR]

The edTPA, a nationally available performance assessment for teacher candidates, has recently been developed and implemented in teacher education programs across the United States. Advocates make arguments for the need for such an assessment while critics of standardized performance assessments point out the dangers of standardization. This article takes a step back from the arguments in support of or in opposition to the assessment and asks fundamentally what the underlying conception of teaching of the edTPA is. After examining conceptions of teaching articulated by scholars such as Nathanial Gage, Larry Cuban, Lee Shulman, Paulo Freire, and bell hooks, this article argues that the underlying conception of teaching of the edTPA is one of professional practice, not only at the individual level but also at the level of teaching as a collective enterprise. The conception of teaching argument is also connected to discussions of the validity arguments for the edTPA with specific attention to face validity, content validity, and construct validity. [ABSTRACT FROM PUBLISHER]

As combined sewer systems and centralized wastewater treatment facilities age, many communities in the world are challenged by management of combined sewer overflow (CSO). Constructed wetlands are considered to be one of the green infrastructure solutions to CSOs in the US. Despite the wide application of constructed wetlands to different types of wastewaters, the stochastic and intermittent nature of CSO presents challenges for design and performance assessment of constructed wetlands. This paper reviews the application status of CSO constructed wetlands in the US, assesses the benefits of CSO constructed wetlands, identifies challenges to designing CSO constructed wetlands, and proposes design considerations. This review finds that constructed wetlands are effective in CSO treatment and relatively less expensive to build than comparable grey infrastructure. Constructed wetlands not only remove pollutants, but also mitigate the event-associated flow regime. The design challenges include incorporating considerations of green infrastructure into permit requirements, determining design capacity for highly variable flows, requiring pretreatment, and needing adaptive design and intensive monitoring. Simultaneous monitoring of flow rate and water quality at both the inflow and outflow of CSO constructed wetlands is required for performance assessment and needed to support design, but is rarely available. [ABSTRACT FROM AUTHOR]

The goal of the present study was to explore the training effectiveness of a feedback method, which utilises eye tracking technology to support both expert demonstration and trainee assessment by measuring trainee scan data, diagnosing deficiencies, and providing feedback allowing a trainee to compare their scan to an expert scan. A training effectiveness evaluation (TEE) was performed to examine the ability of this method to improve individual search strategies and anomaly detection performance, both during training and in the field. The effects of the training were explored during virtual and practical application exercises within a Combat Hunter training course at the United States Marine Corps (USMC) School of Infantry (SOI). Of approximately 50 USMC participants, those who trained utilising eye tracking-based feedback showed significant improvements in search strategies compared to those who received traditional instructor-based feedback. This paper presents the TEE methodology and results and discusses implications of the findings. [ABSTRACT FROM AUTHOR]

Abstract: Extensive work has been carried out by the U.S. Department of Energy (DOE) in the development of a proposed geologic repository at Yucca Mountain (YM), Nevada, for the disposal of high-level radioactive waste. As part of this development, a detailed performance assessment (PA) for the YM repository was completed in 2008 and supported a license application by the DOE to the U.S. Nuclear Regulatory Commission (NRC) for the construction of the YM repository. The following aspects of the 2008 YM PA are described in this presentation: (i) conceptual structure and computational organization, (ii) uncertainty and sensitivity analysis techniques in use, (iii) uncertainty and sensitivity analysis for physical processes, and (iv) uncertainty and sensitivity analysis for expected dose to the reasonably maximally exposed individual (RMEI) specified the NRC’s regulations for the YM repository. [Copyright &y& Elsevier]

This article examines the logic underlying different models for assessing the college readiness of high school students. It focuses on benchmark scores that purportedly identify students who are college ready and presents the challenges of using threshold scores from a single assessment instrument to represent readiness. As well as providing logical and methodological reasons to be wary of threshold scores, analyses of Minnesota and national data illustrate their shortcomings. The article argues that assessments of college readiness should (a) use benchmarks with meaning and consequences for students, (b) employ multiple measures to provide readiness information more precise than from a threshold score derived from any single assessment, and (c) present readiness in terms of probabilities or likelihoods rather than as ready or not. [ABSTRACT FROM PUBLISHER]

Examinees who initially fail and later repeat an SP-based clinical skills exam typically exhibit large score gains on their second attempt, suggesting the possibility that examinees were not well measured on one of those attempts. This study evaluates score precision for examinees who repeated an SP-based clinical skills test administered as part of the US Medical Licensing Examination sequence. Generalizability theory was used as the basis for computing conditional standard errors of measurement ( SEM) for individual examinees. Conditional SEMs were computed for approximately 60,000 single-take examinees and 5,000 repeat examinees who completed the Step 2 Clinical Skills Examination between 2007 and 2009. The study focused exclusively on ratings of communication and interpersonal skills. Conditional SEMs for single-take and repeat examinees were nearly indistinguishable across most of the score scale. US graduates and IMGs were measured with equal levels of precision at all score levels, as were examinees with differing levels of skill speaking English. There was no evidence that examinees with the largest score changes were measured poorly on either their first or second attempt. The large score increases for repeat examinees on this SP-based exam probably cannot be attributed to unexpectedly large errors of measurement. [ABSTRACT FROM AUTHOR]

The American Recovery and Reinvestment Act (ARRA) channeled billions of dollars through the states in an effort to stimulate the economy and shore up faltering state services. It presented unprecedented accountability expectations in the form of required state transparency with regard to ARRA expenditures. This study examines the variations in state responses to ARRA accountability expectations and analyzes state characteristics associated with greater accountability efforts. There is considerable variability in state ARRA accountability reporting but no apparent underlying pattern to the variability, which seems to be largely unaffected by political or other characteristics of the states. The study reveals that most states could have done a much better job of informing their citizens about how ARRA funds were spent and what was accomplished with those funds. [ABSTRACT FROM AUTHOR]

Abstract: In 2001, the National Nuclear Security Administration of the U.S. Department of Energy in conjunction with the national security laboratories (i.e., Los Alamos National Laboratory, Lawrence Livermore National Laboratory and Sandia National Laboratories) initiated development of a process designated Quantification of Margins and Uncertainties (QMU) for the use of risk assessment methodologies in the certification of the reliability and safety of the nation''s nuclear weapons stockpile. This presentation discusses and illustrates the conceptual and computational basis of QMU in analyses that use computational models to predict the behavior of complex systems. The following topics are considered: (i) the role of aleatory and epistemic uncertainty in QMU, (ii) the representation of uncertainty with probability, (iii) the probabilistic representation of uncertainty in QMU analyses involving only epistemic uncertainty, and (iv) the probabilistic representation of uncertainty in QMU analyses involving aleatory and epistemic uncertainty. [Copyright &y& Elsevier]

Abstract: In 2001, the National Nuclear Security Administration (NNSA) of the U.S. Department of Energy (DOE) in conjunction with the national security laboratories (i.e., Los Alamos National Laboratory, Lawrence Livermore National Laboratory, and Sandia National Laboratories) initiated development of a process designated quantification of margins and uncertainties (QMU) for the use of risk assessment methodologies in the certification of the reliability and safety of the nation''s nuclear weapons stockpile. A previous presentation, “Quantification of Margins and Uncertainties: Conceptual and Computational Basis,” describes the basic ideas that underlie QMU and illustrates these ideas with two notional examples. The basic ideas and challenges that underlie NNSA''s mandate for QMU are present, and have been successfully addressed, in a number of past analyses for complex systems. To provide perspective on the implementation of a requirement for QMU in the analysis of a complex system, three past analyses are presented as examples: (i) the probabilistic risk assessment carried out for the Surry Nuclear Power Station as part of the U.S. Nuclear Regulatory Commission''s (NRC''s) reassessment of the risk from commercial nuclear power in the United States (i.e., the NUREG-1150 study), (ii) the performance assessment for the Waste Isolation Pilot Plant carried out by the DOE in support of a successful compliance certification application to the U.S. Environmental Agency, and (iii) the performance assessment for the proposed high-level radioactive waste repository at Yucca Mountain, Nevada, carried out by the DOE in support of a license application to the NRC. Each of the preceding analyses involved a detailed treatment of uncertainty and produced results used to establish compliance with specific numerical requirements on the performance of the system under study. As a result, these studies illustrate the determination of both margins and the uncertainty in margins in real analyses. [Copyright &y& Elsevier]

In 2005, the largest field study to-date concluded that automated dimming controls in side-lighted spaces consistently underperformed relative to expectation, and were still to be considered an 'emerging' technology. Motivated by a desire to accelerate the adoption and acceptance of such controls in, this paper details a field-measured assessment of a digital daylighting system recently introduced to the US market. Two intentionally challenging offices were retrofit with controls, and automated data acquisition hardware, and system performance was evaluated across several parameters throughout a twelve-month period. The field data and user interviews indicate that the system was relatively straightforward to commission compared to other dimming controls, and overall was well received by the occupants. Energy savings in the private and open-plan installations were between 17 percent and 20 percent. In the private office, the pre-existence of occupancy sensing, and automatic-ON in the retrofit system precluded even deeper energy savings, whereas false triggering of the occupancy sensor limited the achieved savings in the open plan space. The controller was largely able to maintain target illuminance on the worksurface, although increased daylight availability led to increased variability and a 100-lx increase between noon and 2PM. Key recommendations to increase the acceptance of similar systems include the provision of shielding options for occupancy sensors, manual-ON options for private offices, remote-access commissioning tools, and improved clarity of written materials describing system function, operation, and installation. [ABSTRACT FROM AUTHOR]

This article describes the results of a proof of concept project, Data Sets for Inquiry in Geoscience (DIGS), funded by the National Science Foundation. The three goals of the project were to: 1 study the impacts on student learning of web-based supplementary curriculum modules in which secondary students used real geoscience datasets, visualisations, and software tools to conduct investigations within two fundamental topics of study, climate change and plate boundaries 2 develop designs and prototypes of technology-based performance assessments to provide evidence of students' geoscientific knowledge and inquiry skills 3 develop extension scenarios to overview examples of curriculum modules and performance assessments that could be developed for other Earth Science standards and curriculum topics. The article describes the features of the climate module curricular activities and performance assessments that can serve as models for additional topics. The project promoted and tested knowledge and inquiry strategies not typically addressed in geoscience curricula and assessments. [ABSTRACT FROM AUTHOR]

This article provides an overview of the development and validation of the Student Tool for Technology Literacy (ST²L). Developing valid and reliable objective performance measures for monitoring technology literacy is important to all organizations charged with equipping students with the technology skills needed to successfully participate in and contribute to a digital and global society. The purpose of the ST²L is to measure student technology literacy for low-stakes purposes of reporting aggregated school results, curricular planning, and students' self-assessment of technology skills. This article reports the development procedures and results of the pilot test conducted with eighth grade students (N = 1,561) to validate the functioning of this online, interactive tool. Analyses focused on item difficulty and discrimination by ability groups, completion time analysis, internal consistency reliability, and construct validity. ST²L was found to be a sound assessment tool for the intended purpose of low-stakes assessment of technology literacy. [ABSTRACT FROM AUTHOR]

Intracorporeal suturing is one of the most difficult laparoscopic tasks. The purpose of this study was to assess the impact of robotic assistance on novice suturing performance, safety, and workload in the operating room. Medical students ( n = 34), without prior laparoscopic suturing experience, were enrolled in an Institutional Review Board-approved, randomized protocol. After viewing an instructional video, subjects were tested in intracorporeal suturing on two identical, live, porcine Nissen fundoplication models; they placed three gastro-gastric sutures using conventional laparoscopic instruments in one model and using robotic assistance (da Vinci®) in the other, in random order. Each knot was objectively scored based on time, accuracy, and security. Injuries to surrounding structures were recorded. Workload was assessed using the validated National Aeronautics and Space Administration (NASA) task load index (TLX) questionnaire, which measures the subjects’ self-reported performance, effort, frustration, and mental, physical, and temporal demands of the task. Analysis was by paired t-test; p < 0.05 was considered significant. Compared with laparoscopy, robotic assistance enabled subjects to suture faster (595 ± 22 s versus 459 ± 137 s, respectively; p < 0.001), achieve higher overall scores (0 ± 1 versus 95 ± 128, respectively; p < 0.001), and commit fewer errors per knot (1.15 ± 1.35 versus 0.05 ± 0.26, respectively; p < 0.001). Subjects’ overall score did not improve between the first and third attempt for laparoscopic suturing (0 ± 0 versus 0 ± 0; p = NS) but improved significantly for robotic suturing (49 ± 100 versus 141 ± 152; p < 0.001). Moreover, subjects indicated on the NASA-TLX scale that the task was more difficult to perform with laparoscopic instruments compared with robotic assistance (99 ± 15 versus 57 ± 23; p < 0.001). Compared with standard laparoscopy, robotic assistance significantly improved intracorporeal suturing performance and safety of novices in the operating room while decreasing their workload. Moreover, the robot significantly shortened the learning curve of this difficult task. Further study is needed to assess the value of robotic assistance for experienced surgeons, and validated robotic training curricula need to be developed. [ABSTRACT FROM AUTHOR]

The article offers the author's insights on the profession of medicine and the obligation of medical practitioners to their patients and the public. He discusses the importance of The American Board of Physical Medicine and Rehabilitation (ABPMR) which was created to certify physicians based on the standards of excellence. The author emphasizes the significance of board certifications to provide assurance to the public of a high-quality health care given to patients.

Whenever a change of administration occurs in Washington, classic issues of public administration are brought out of the academic closet. The dichotomy between politics and administration moves from discussion in classrooms and textbooks to the pages of the Washington press. The role of the career bureaucracy is acknowledged as a way the new administration will translate its policy and political agenda to the machinery of government. During this stage, after election results are clear, the term "permanent government" is used with the assumption that everyone knows what it means. But the concept of "the permanent government" is an idea borrowed from the British system, one that cannot be automatically applied to the American case, where a system of shared powers plus the structure and culture of the US career bureaucracy are quite different from the British Westminster model. This article focuses on two management efforts within one of the great overhead agencies of American Government, the Office of Management and Budget (OMB). These involve the Office of Information and Regulatory Affairs (OIRA) and the staff itself within the OMB Management office that has responsibility for performance assessment. They provide evidence of an inherent tension between aspirations of the Obama administration and the orientation of existing staff. Discussion of these two management efforts provides a glimpse of a set of decision processes that is usually absent from discussion of "the permanent government." It indicates that "permanent government" is more than just career-political interactions, and it is evidence that career staff operate differently in different contexts. To understand this role, it is important to look inside the detailed processes of government to determine how these actually operate within the Executive Office of the President. [ABSTRACT FROM AUTHOR]

In 2000, the Accreditation Council for Medical Education introduced a new initiative that substantively changed the method by which residency programs are evaluated. In this new competency-based approach to residency education, assessment of performance became a main area of interest, and direct observation was offered as a tool to assess knowledge and skills. Despite being an inherent part of medical education as faculty and learners work together in clinical experiences, direct observation has traditionally been an informal and underused assessment method across all specialties. Residents and students report rarely being observed during their educational process, even though they value the experience. Reasons for this include a lack of faculty time, a lack of faculty skills, a potential stressful effect on the learner, and a perceived lack of validation of the assessment. This article examines the literature regarding the use of direct observation in medical education with a focus on validity evidence. We performed a PubMed search of articles pertaining to direct observation, using key words such as direct observation, performance observation, clinical observation, students, find residents. A subsequent search was conducted in known articles, focusing on variations of the term observation in the titles of articles and introducing the concept of clinical competence. In conclusion, direct observation is a unique and useful tool in the assessment of medical students and residents. Assessing learners in natural settings offers the opportunity to see beyond what they know find into what they actually do, which is fundamentally essential to training qualified physicians. Although the literature identifies several threats to its validity as an assessment, it also demonstrates methods to minimize those threats. Based on the current recommendations and need for performance assessment in education and with attention paid to the development and design, direct observation can and should be included in medical education curricula. [ABSTRACT FROM AUTHOR]

This article addresses the application of the assessment triangle developed by the National Research Council (Pellegrino, Chudowsky, & Glaser, 2001), most specifically the cognition vertex of that triangle, to the unique learning characteristics of students with significant cognitive disabilities in developing and demonstrating academic competence. Given the inclusion of all students, including students with significant cognitive disabilities, in measures of large-scale educational assessment and accountability under the No Child Left Behind Act, it is essential to examine how the primary elements of knowledge representation and competence identified by Pellegrino et al. for all students have special ramifications for students with significant cognitive disabilities. It is only in the development of such a model of competence that it is possible to construct alternate assessments for these students that validly represent what these students know and can do. [ABSTRACT FROM AUTHOR]

Hurricanes are one of the major causes of damage and loss to residential wooden buildings in the United States. The study of hurricane-imposed risks to residential wooden buildings is, therefore, a fundamental step to mitigate these damages and losses. Within this context, a probabilistic methodology is provided for the fragility analysis of residential wooden buildings. Two damage mechanisms are considered in this methodology, excessive dynamic wind pressure and impact from windborne debris. Unlike existing frameworks, the methodology defines the geometric configuration as well as the required extension of the neighboring buildings to estimate the damage from both mechanisms. A case study illustrates the methodology on residential communities composed of archetype gable-roof buildings. Ten construction cases are defined to cover a range of resistances of building components for both considered damage mechanisms. Three floor-area ratios (F A R) are also considered. The resulting fragility curves highlight: (1) the effect of the F A R on enhancing the performance of residential buildings due to the effect of shielding; and (2) the significance of windborne debris in increasing the estimated hurricane risk. The estimated fragility curves can be used to directly model residential buildings in community resilience frameworks. • A methodology for the risk and fragility analysis of wooden buildings is presented. • Damage from windborne debris is considered in the developed methodology. • Fragility curves for archetype gable-roof buildings are presented. • The significance of windborne debris and floor-area ratio are highlighted. • The estimated fragility curves can be used in community resilience frameworks. [ABSTRACT FROM AUTHOR]

In the United States, the performance of public transit systems is often characterized, in both the popular press and academic literature, as being mediocre at best and growing steadily worse. Drawing on insights from the extant literature and a census of all U.S. transit systems, this research argues that "multiple definitions" of transit performance exist and are composed of three macro-constructs which are weighted differently by different stakeholder groups and thereby provide unique definitions for each stakeholder group. Statistical analysis of the data suggests that there are differences in both the absolute importance and relative importance placed on the three macro-constructs by U.S. transit stakeholder groups. Therefore, the examination of public transit performance from a multiple stakeholder points-of-view perspective appears warranted in order to better evaluate the performance of public transit systems. [ABSTRACT FROM AUTHOR]

BSTRACT: This article reports on Beyond the OPI: Integrated Performance Assessment (IPA) Design Project, a three-year (1997-2000) research initiative sponsored by the U.S. Department of Education International Research and Studies Program. The primary goal of the project was to develop an integrated skills assessment prototype that would measure students' progress towards the Standards for Foreign Language Learning in the 21st Century (National Standards, 1999, 2006). A second goal of the project was to use the assessment prototype as a catalyst for curricular and pedagogical reform. This paper presents the Integrated Performance Assessment (IPA) prototype, illustrates a sample IPA, and discusses how classroom-based research on the IPA demonstrated the washback effect of integrated performance-based assessment on teachers' perceptions regarding their instructional practices. [ABSTRACT FROM AUTHOR]

As part of its preparation to review a potential license application from the U.S. Department of Energy (DOE), the U.S. Nuclear Regulatory Commission (NRC) is examining the performance of the proposed Yucca Mountain nuclear waste repository. In this regard, we evaluated postclosure repository performance using Monte Carlo analyses with an NRC-developed system model that has 950 input parameters, of which 330 are sampled to represent system uncertainties. The quantitative compliance criterion for dose was established by NRC to protect inhabitants who might be exposed to any releases from the repository. The NRC criterion limits the peak-of-the-mean dose, which in our analysis is estimated by averaging the potential exposure at any instant in time for all Monte Carlo realizations, and then determining the maximum value of the mean curve within 10,000 years, the compliance period. This procedure contrasts in important ways with a more common measure of risk based on the mean of the ensemble of peaks from each Monte Carlo realization. The NRC chose the former ( peak-of-the-mean) because it more correctly represents the risk to an exposed individual. Procedures for calculating risk in the expected case of slow repository degradation differ from those for low-probability cases of disruption by external forces such as volcanism. We also explored the possibility of risk dilution (i.e., lower calculated risk) that could result from arbitrarily defining wide probability distributions for certain parameters. Finally, our sensitivity analyses to identify influential parameters used two approaches: (1) the ensemble of doses from each Monte Carlo realization at the time of the peak risk (i.e., peak-of-the-mean) and (2) the ensemble of peak doses calculated from each realization within 10,000 years. The latter measure appears to have more discriminatory power than the former for many parameters (based on the greater magnitude of the sensitivity coefficient), but can yield different rankings, especially for parameters that influence the timing of releases. [ABSTRACT FROM AUTHOR]

The use of experts to judge performance assessments is desirable because ratings of performances, carried out by experts in the content domain of the examination, are often considered to be the "gold standard." However, one drawback of using experts to rate performances is the high cost involved. A more economic alternative for scoring performance assessments entails using analytic scoring, which typically involves assigning points to individual traits present in the performance, and summing to arrive at a single score. This strategy is less costly, but may lack the richness of holistic scoring. This study investigates the use of regression-based techniques to predict expert judgments on a written performance task from a combination of analytic scores. Potentially, this will result in scores that approximate the richness of holistic ratings while maintaining the cost-effectiveness of analytic scoring. Results show that a substantial proportion of variance in expert judgments can be explained by the analytic scores, but that decisions based on actual expert judgments and the predicted expert judgments were not sufficiently consistent to warrant the substitution of one score for the other. [ABSTRACT FROM AUTHOR]

Performance assessments have been touted for their multidimensional and ‘realworld’, or authentic, appearance, yet this complexity is at the heart of the most serious problems in the use of performance assessments. If performance assessments are more multidimensional and situational, then perhaps performance assessment scores represent other things besides the construct of interest. This study empirically explored that possibility, namely that scores on a performance assessment reflect a motivational variable (perception of control), in addition to the construct intended, while an objective test does not. Data from high school Spanish students who took an objective test, a performance assessment and a measure of perceptions of control suggest that perceptions of control indeed predict performance assessment scores but not objective test scores. [ABSTRACT FROM AUTHOR]

Much has been written about the development and application of quantitative methods for estimating under uncertainty the long-term radiological performance of underground disposal of radioactive wastes. Until recently, interest has been focused almost entirely on the technical challenges regardless of the role of the organization responsible for these analyses. Now the dialogue between regulators, the repository developer or operator, and other interested parties in the decision-making process receives increasing attention, especially in view ofsome current difficulties in obtaining approvals to construct or operate deep facilities for intermediate or high-level wastes. Consequently, it is timely to consider the options for regulators' review and evaluation of safety submissions, at the various stages in the site selection to repository closure process, and to consider, especially, the role for performance assessment (PA) within the programs of a regulator both before and after delivery of such a submission. The origins and broad character of present regulations in the European Union (EU) and in the OECD countries are outlined and some regulatory PA reviewed. The issues raised are discussed, especially in regard to the interpretation of regulations, the dangers from the desire for simplicity in argument, the use of regulatory PA to review and challenge thePA in the safety case, and the effects of the relationship between proponent and regulator. Finally, a very limited analysis of the role of PA in public hearings is outlined and recommendations are made, together with proposals for improving the mechanisms for international collaboration on technical issues of regulatory concern. [ABSTRACT FROM AUTHOR]

The Waste Isolation Pilot Plant (WIPP) is a geological repository for disposal of U.S. defense transuranic radioactive waste. Built and operated by the U.S. Department of Energy (DOE), it is located in the Permian age salt beds in southeastern New Mexico at a depth of 655 m.Performance assessment for the repository's compliance with the 10,000-year containment standards was completed in 1996 and the U.S. Environmental Protection Agency (EPA) certified in 1998 that the repository meets compliance with the EPA standards 40 CFR 191 and 40 CFR 194.The Environmental Evaluation Group (EEG) review of the DOE's application for certification identified a number of issues. These related to the scenarios, conceptual models, and values of the input parameters used in the calculations. It is expected that these issues will be addressed and resolved during the first 5-year recertification process that began with the first receipt of waste at WIPP on March 26, 1999, and scheduled to be completed in March 2004. [ABSTRACT FROM AUTHOR]

Performance Assessment (PA) is the use of mathematical models to simulate the long-term behavior of engineered and geologic barriers in anuclear waste repository; methods of uncertainty analysis are used to assess effects of parametric and conceptual uncertainties associated with the model system upon the uncertainty in outcomes of the simulation. PA is required by the U.S. Environmental Protection Agency as part of its certification process for geologic repositories for nuclear waste. This paper is a dialogue to explore the value and limitations of PA. Two 'skeptics' acknowledge the utility of PA in organizing the scientific investigations that are necessary for confident sitingand licensing of a repository; however, they maintain that the PA process, at least as it is currently implemented, is an essentially unscientific process with shortcomings that may provide results of limited use in evaluating actual effects on public health and safety. Conceptual uncertainties in a PA analysis can be so great that results can be confidently applied only over short time ranges, the antithesis of the purpose behind long-term, geologic disposal. Two 'proponents' of PA agree that performance assessment is unscientific, but only in the sense that PA is an engineering analysis that uses existing scientific knowledge to support public policy decisions, rather than an investigation intended to increase fundamental knowledge of nature; PA has different goals and constraints than a typical scientific study. The 'proponents' describe an ideal, six-step process for conducting generalized PA, here called probabilistic systems analysis (PSA); theynote that virtually all scientific content of a PA is introduced during the model-building steps of a PSA; they contend that a PA based on simple but scientifically acceptable mathematical models can provide useful and objective input to regulatory decision makers. The valueof the results of any PA must lie between these two views and will depen [ABSTRACT FROM AUTHOR]

The conceptual and computational structure of a performance assessment (PA) for the Waste Isolation Pilot Plant (WIPP) is described. Important parts of this structure are (1) maintenance of a separation between stochastic (i.e., aleatory) and subjective (i.e., epistemic) uncertainty, with stochastic uncertainty arising from the many possible disruptions that could occur over the 10,000-year regulatory period that applies to the WIPP, and subjective uncertainty arising from the imprecision with which many of the quantities required in the analysis are known, (2) use of Latin hypercube sampling to incorporate the effects of subjective uncertainty, (3) use of Monte Carlo (i.e., random) sampling to incorporate the effects of stochastic uncertainty, and(4) efficient use of the necessarily limited number of mechanistic calculations that can be performed to support the analysis. The WIPP is under development by the U.S. Department of Energy (DOE) for the geologic (i.e., deep underground) disposal of transuranic (TRU) waste, with the indicated PA supporting a Compliance Certification Application (CCA) by the DOE to the U.S. Environmental Protection Agency (EPA)in October 1996 for the necessary certifications for the WIPP to begin operation. The EPA certified the WIPP for the disposal of TRU waste in May 1998, with the result that the WIPP will be the first operational facility in the United States for the geologic disposal of radioactive waste. [ABSTRACT FROM AUTHOR]

The U.S. Nuclear Regulatory Commission (NRC) staff has developed a performance assessment capability to address three programmatic areas in nuclear waste management: high-level waste, low-level waste, and decommissioning of licensed facilities (license termination). The NRC capability consists of: (1) methodologies for performance assessment;(2) models and computer codes for estimating system performance; (3)regulatory guidance in various forms, such as regulations, Branch Technical Positions, and Standard Review Plans; and (4) a technical staff experienced in executing and evaluating performance assessments for a variety of waste systems. Although the tools and techniques are refined for each programmatic area, general approaches and similar issues are encountered in all areas. [ABSTRACT FROM AUTHOR]

Objective: To determine whether changes in the demographic/educational mix of those entering internal medicine from 1986 to 1989 were associated with differences among them at the time of certification.Participants: Included in the study were all candidates for the 1989 to 1992 American Board of Internal Medicine certifying examinations in internal medicine.Measurements: Demographic information and medical school, residency training, and examination experience were available for each candidate. Data defining quality, size, and number of subspecialties were available for internal medicine training programs.Results: From 1990 to 1992, the total number of men and women candidates increased as did the numbers of foreign-citizen non-U.S. medical school graduates and osteopathic medical school graduates; the number of U.S. medical school graduates remained nearly constant and the number of U.S.-citizen graduates of non-U.S. medical schools declined. The pass rates for all groups of first-time examination takers decreased, while the ratings of program directors remained relatively constant. Program quality, size, and number of subspecialty programs had modest positive relationships with examination performance.Conclusions: Changes in the characteristics of those entering internal medicine from 1986 to 1989 were associated with declines in performance at the time of certification. These declines occurred in all content areas of the test and were apparent regardless of program quality. These data identify some of the challenges internal medicine faces in the years ahead. [ABSTRACT FROM AUTHOR]

Objectives: 1) To determine whether nurse evaluations of humanistic behavior discriminate between houseofficers in an internal medicine training program, and 2) to compare nurse and attending physician evaluations.Design: Prospective, six-month comparison of nurse and attending ratings of houseofficer humanistic behavior.Procedure: Using a six-item, Likert-scale humanistic behavior rating form, nurses and ward attendings evaluated 76 PGY-1, PGY-2, and PGY-3 houseofficers over a six-month period. Nurses and attendings voluntarily evaluated houseofficers on all inpatient units in both university and Veterans Administration teaching hospitals.Measurements and Main Results: Nurse ratings discriminated residents from one another throughout the six months of the study and over all units in both hospitals. Attending physician ratings were only moderately correlated with nurses' and were significantly more lenient. Exploratory analyses of the nursing evaluations revealed that female houseofficers received significantly more favorable evaluations than did men and that ward nurses were significantly more lenient than were critical care nurses. Nurse ratings did not differ by hospital, training year, or month of evaluation.Conclusions: Nurses can provide information about humanistic behavior that will allow program directors to discriminate among different levels of houseofficer behavior. Information from nurses differs from that provided by attending physicians. Nurse ratings are affected by gender and by the type of unit from which they are obtained. [ABSTRACT FROM AUTHOR]

Over the past 30 years, there have been numerous, noteworthy successes in the development, validation, and implementation of clinical skills assessments. Despite this progress, the medical profession has barely scratched the surface towards developing assessments that capture the true complexity of hands-on skills in procedural medicine. This paper highlights the development implementation and new discoveries in performance metrics when using sensor technology to assess cognitive and technical aspects of hands-on skills., (© 2021 Wiley Periodicals LLC.)

Competing Interests: Disclosure Dr. Gerald Fried has served as a consultant for Theator, an AI-powered video education platform for surgery. Rest of the authors have nothing to disclose.

Purpose: To assess radiologists' performance in the Merit-Based Incentive Payment System (MIPS), with attention to the impact of the novel MIPS-Alternative Payment Models (APMs) participation option created under the Medicare Access and CHIP Reauthorization Act., Methods: Data regarding radiologists' 2017 MIPS performance was extracted from the Physician Compare 2017 Individual EC Public Reporting-overall MIPS Performance data set, and additional physician characteristics were extracted from multiple CMS data sets., Results: Among 20,956 MIPS-participating radiologists, 16.6% participated using individual reporting, 68.9% group reporting, and 13.4% APM reporting. Average Quality scores were 59.7 84.0, and 92.5, respectively. The fractions of radiologists scored in Advancing Care Information were 4.1%, 27.0%, and 100.0%. When scored, average scores in this category were 61.9, 94.6, and 80.9. A total of 27.7% and 42.7% of interventional radiologists were scored in this category using individual and group reporting, respectively. However, general radiologists and subspecialists other than interventional radiologists were rarely (<5%) scored. Average scores in Improvement Activities were 37.5, 92.5, and 100.0 for individual reporting, group reporting, and APM reporting, respectively. Average Final Scores were 56.5, 85.6, and 90.6. The better performance of APM versus group reporting was most apparent for smaller practices (ie, for practice sizes ≤15, average Final Score of 84.1 for APM versus 75.0 for group reporting)., Conclusion: Although radiologists perform much better in MIPS using group versus individual reporting, performance improves even further through APM reporting, particularly for smaller practices. Radiologists seeking better performance under MIPS should carefully explore APM opportunities., (Copyright © 2019 American College of Radiology. Published by Elsevier Inc. All rights reserved.)

For 223 residents from eight teaching hospitals, the results of the second-year in-training examination and the first-sitting certifying examination of the American Board of Internal Medicine were highly correlated. The results of the in-training examination can serve residents as an important measure of their preparedness for certification and can be useful in identifying the need for more intensive self-study strategies during the subsequent one and a half years. [ABSTRACT FROM AUTHOR]

Purpose: Radiologists have historically participated as individuals in CMS pay-for-performance programs, but little is known about how radiologists perform under increasingly available group participation. We aimed to assess radiologists' relative national performance on CMS quality metrics using group versus individual participation., Methods: Radiologists' group- and individual-level 2016 performance on Physician Quality Reporting System (PQRS) and non-PQRS Qualified Clinical Data Registry (QCDR) measures were obtained from the CMS national Physician Compare database and compared., Results: Radiology groups reported an average 4.6 ± 2.0 quality measures; individual radiologists reported 2.3 ± 1.2 (P < .001). At least six measures were reported by 31.5% of groups versus 1.0% of individuals. Only one measure was reported by 5.4% of groups versus 33.0% of individuals. Groups reported 21 unique measures (20 via registries and one via QCDR). For 8 of the 11 measures reported by 20 or more groups, the average group performance rate was 3% or better than the average performance rate among radiologists participating as individuals (maximum 14% improvement with group participation versus individual participation for any individual measure). Group and individual performance were similar for the remaining three such measures. For measures reported by 20 or more groups in which a higher score indicates better performance, average group performance rates ranged from 86.2% to 98.9%., Conclusion: Compared with individual participation in CMS quality performance programs, radiologists participating as a group reported larger numbers of quality measures and achieved higher performance rates on those measures. Radiology practices seeking success under Medicare's new Quality Payment Program should carefully explore group participation., (Copyright © 2018 American College of Radiology. Published by Elsevier Inc. All rights reserved.)

Background: The effects of underlying noncodified risks are unclear on the prognosis of patients with end-stage renal disease (ESRD). We aimed to evaluate the association of residential area life expectancy with outcomes and processes of care for patients with ESRD in the United States., Study Design: Retrospective cohort study., Setting & Participants: Adult patients with incident ESRD between 2006 and 2013 recorded in the US Renal Data System (n=606,046)., Predictor: The primary exposure was life expectancy in the patient's residential county estimated by the Institute for Health Metrics and Evaluation., Outcomes: Death, placement on the kidney transplant wait list, living and deceased donor kidney transplantation, and posttransplantation graft loss., Results: Median life expectancies of patients' residences were 75.6 (males) and 80.4 years (females). Compared to the highest life expectancy quintile and adjusted for demographic factors, disease cause, and multiple comorbid conditions, the lowest quintile had adjusted HRs for mortality of 1.20 (95% CI, 1.18-1.22); placement onto the waiting list, 0.68 (95% CI, 0.67-0.70); living donor transplantation, 0.53 (95% CI, 0.51-0.56); posttransplantation graft loss, 1.35 (95% CI, 1.27-1.43); and posttransplantation mortality, 1.29 (95% CI, 1.19-1.39). Patients living in areas with lower life expectancy were less likely to be informed about transplantation, be under the care of a nephrologist, or receive an arteriovenous fistula as the initial dialysis access. Results remained consistent with additional adjustment for zip code-level median income, population size, and urban-rural locality., Limitations: Potential residual confounding and attribution of effects to individuals based on residential area-level data., Conclusions: Residential area life expectancy, a proxy for socioeconomic, environmental, genetic, and behavioral factors, was independently associated with mortality and process-of-care measures for patients with ESRD. These results emphasize the underlying effect on health outcomes of the environment in which patients live, independent of patient-level factors. These findings may have implications for provider assessments., (Copyright © 2018 National Kidney Foundation, Inc. Published by Elsevier Inc. All rights reserved.)

Purpose: The Medicare Access and CHIP Reauthorization Act (MACRA) Quality performance category is the successor to the Physician Quality and Reporting System (PQRS) program and now contributes to physicians' income adjustments based upon performance rates calculated for a minimum of six measures. We assess radiologists' frequency of reporting PQRS measures as a marker of preparedness for MACRA., Methods: Medicare-participating radiologists were randomly searched through the Physician Compare website until identifying 1,000 radiologists who reported at least one PQRS measure. Associations were explored between the number of reported measures and radiologist characteristics., Results: For PQRS-reporting radiologists, the number of reported PQRS measures was 1 (25.2%), 2 (27.3%), 3 (18.2%), 4 (19.3%), 5 (8.3%), and 6 (1.7%). The most commonly reported measures were "documenting radiation exposure time for procedures using fluoroscopy" (64.3%) and "accurate measurement of carotid artery narrowing" (56.8%). Reporting at least two measures was significantly (P < .001) more likely for nonacademic (77.3%) versus academic (44.9%) radiologists, generalists (82.7%) versus subspecialists (59.1%), and radiologists in smaller (≤9 members) (84.7%) versus larger (≥100 members) (39.7%) practices. Reporting six measures was significantly (P < .05) more likely for generalists (2.6%) versus subspecialists (0.4%)., Conclusion: Most PQRS-reporting radiologists reported only one or two measures, well below MACRA's requirement of six. Radiologists continuing such reporting levels will likely be disadvantaged in terms of potential payment adjustments under MACRA. Lower reporting rates for academic and subspecialized radiologists, as well as those in larger practices, may relate to such radiologists' reliance on their hospitals or networks for PQRS reporting. Qualified clinical data registries should be embraced to facilitate more robust measure reporting., (Copyright © 2017 American College of Radiology. Published by Elsevier Inc. All rights reserved.)

This article comments on "performance assessment in support of the 1996 compliance certification application for the waste isolation pilot plant." The probability methodology described in the paper is textbook statistics and is an appropriate tool to support Waste Isolation Pilot Plant (WIPP) compliance certification under the U.S. Environmental Protection Agency (EPA) standards. The effort described in the paper is the first fully probabilistic performance assessment to be implemented by the U.S. Department of Energy that meets performance objectives set by an external agency (i.e., EPA) and therefore stands as a significant milestone. The superiority of this methodology to a deterministic approach is found in the sheer amount of information that a probabilistic assessment provides over a deterministic one. This work represents ground- breaking application of a probabilistic method to a large-scale national problem: transuranic radioactive waste (TRU) disposal. The importance of this performance assessment lies not only in answering regulatory issues, but also in serving as a template for certifications of future nuclear waste repositories.

Background: Our aim was to report the longitudinal assessment of technical performance of general surgery residents on select tasks from multiple programs over a 2-year period., Methods: An institutional review board-approved, multi-institutional collaborative study was undertaken with yearly resident performance assessments over a 3-year period. General surgery residents (postgraduate year [PGY] 1 to 5) were tested on 3 laparoscopic and 5 open simulated surgical tasks. Resident performance was compared individually over time and among interns and more senior residents., Results: Forty-one residents from 4 residency programs were evaluated. Scores increased in all tasks with each assessment, plateauing at a lower PGY level for open tasks compared with laparoscopic tasks. Change in performance scores between assessments were higher for interns compared with more senior residents (P < .003)., Conclusions: Resident performance on basic open and laparoscopic tasks assessed over time improved the most between the PGY 1 and 2 levels and was dependent on task difficulty. This documented skill evolution may allow tailoring of skills curricula to both meet existing needs and minimize performance variability., (Copyright © 2015 Elsevier Inc. All rights reserved.)

This article comments on "Performance Assessment in Support of the 1996 Compliance Certification Application for the Waste Isolation Pilot Plant." The probability methodology described in the paper is textbook statistics and is an appropriate tool to support Waste Isolation Pilot Plant (WIPP) compliance certification under the U.S. Environmental Protection Agency (EPA) standards of 40 CFR 191 and 194. The most difficult aspect of the approach is assigning probability distributions to address stochastic and subjective uncertainties. The effort described in the paper is the first fully probabilistic performance assessment to be implemented by the U.S. Department of Energy that meets performance objectives set by an external agency (i.e., EPA) and therefore stands as a significant milestone. This work represents ground- breaking application of a probabilistic method to a large-scale national problem: transuranic radioactive waste (TRU) disposal. The importance of this performance assessment (and the probabilistic methodology that it employs) lies not only in answering regulatory issues, but also in serving as a template for certifications of future nuclear waste repositories.

Objectives: Insufficient attention has been given to how information from computer-based clinical case simulations is presented, collected, and scored. Research is needed on how best to design such simulations to acquire valid performance assessment data that can act as useful feedback for educational applications. This report describes a study of a new simulation format with design features aimed at improving both its formative assessment feedback and educational function., Methods: Case simulation software (LabCAPS) was developed to target a highly focused and well-defined measurement goal with a response format that allowed objective scoring. Data from an eight-case computer-based performance assessment administered in a pilot study to 13 second-year medical students was analyzed using classical test theory and generalizability analysis. In addition, a similar analysis was conducted on an administration in a less controlled setting, but to a much large sample (n = 143), within a clinical course that utilized two random case subsets from a library of 18 cases., Results: Classical test theory case-level item analysis of the pilot assessment yielded an average case discrimination of 0.37, and all eight cases were positively discriminating (range = 0.11-0.56). Classical test theory coefficient alpha and the decision study showed the eight-case performance assessment to have an observed reliability of σ = G = 0.70. The decision study further demonstrated that a G = 0.80 could be attained with approximately 3 h and 15 min of testing. The less-controlled educational application within a large medical class produced a somewhat lower reliability for eight cases (G = 0.53). Students gave high ratings to the logic of the simulation interface, its educational value, and to the fidelity of the tasks., Conclusions: LabCAPS software shows the potential to provide formative assessment of medical students' skill at diagnostic test ordering and to provide valid feedback to learners. The perceived fidelity of the performance tasks and the statistical reliability findings support the validity of using the automated scores for formative assessment and learning. LabCAPS cases appear well designed for use as a scored assignment, for stimulating discussions in small group educational settings, for self-assessment, and for independent learning. Extension of the more highly controlled pilot assessment study with a larger sample will be needed to confirm its reliability in other assessment applications.