Search

The evidence-based policy movement promotes the use of empirical evidence to inform policy decision-making. While several social science disciplines are undergoing a ‘credibility revolution’ focused on openness and replication, policy analysis has yet to systematically embrace transparency and reproducibility. We argue that policy analysis should adopt the open research practices increasingly espoused in related disciplines to advance the credibility of evidence-based policy making. We first discuss the importance of evidence-based policy in an era of increasing disagreement about facts, analysis, and expertise. We present a novel framework for ‘open’ policy analysis (OPA) and how to achieve it, focusing on examples of recent policy analyses that have incorporated open research practices such as transparent reporting, open data, and code sharing. We conclude with recommendations on how key stakeholders in evidence-based policy can make OPA the norm and thus safeguard trust in using empirical evidence to inform important public policy decisions.

Contains fulltext : 214941.pdf (Publisher’s version ) (Open Access) We present a consensus-based checklist to improve and document the transparency of research reports in social and behavioural research. An accompanying online application allows users to complete the form and generate a report that they can submit with their manuscript or post to a public repository. 3 p.

Author(s): Aczel, Balazs; Szaszi, Barnabas; Sarafoglou, Alexandra; Kekecs, Zoltan; Kucharský, Simon; Benjamin, Daniel; Chambers, Christopher D; Fisher, Agneta; Gelman, Andrew; Gernsbacher, Morton A; Ioannidis, John P; Johnson, Eric; Jonas, Kai; Kousta, Stavroula; Lilienfeld, Scott O; Lindsay, D Stephen; Morey, Candice C; Munafo, Marcus; Newell, Benjamin R; Pashler, Harold; Shanks, David R; Simons, Daniel J; Wicherts, Jelte M; Albarracin, Dolores; Anderson, Nicole D; Antonakis, John; Arkes, Hal R; Back, Mitja D; Banks, George C; Beevers, Christopher; Bennett, Andrew A; Bleidorn, Wiebke; Boyer, Ty W; Cacciari, Cristina; Carter, Alice S; Cesario, Joseph; Clifton, Charles; Conroy, Ronan M; Cortese, Mike; Cosci, Fiammetta; Cowan, Nelson; Crawford, Jarret; Crone, Eveline A; Curtin, John; Engle, Randall; Farrell, Simon; Fearon, Pasco; Fichman, Mark; Frankenhuis, Willem; Freund, Alexandra M; Gaskell, M Gareth; Giner-Sorolla, Roger; Green, Don P; Greene, Robert L; Harlow, Lisa L; de la Guardia, Fernando Hoces; Isaacowitz, Derek; Kolodner, Janet; Lieberman, Debra; Logan, Gordon D; Mendes, Wendy B; Moersdorf, Lea; Nyhan, Brendan; Pollack, Jeffrey; Sullivan, Christopher; Vazire, Simine; Wagenmakers, Eric-Jan | Abstract: An amendment to this paper has been published and can be accessed via a link at the top of the paper.

Report cards classifying physicians into performance tiers are central to health care quality improvement initiatives. Misclassification is a concern since physicians often have small patient panels for standard performance measures. Given that report cards are used for different purposes by different stakeholders, we specify loss functions and evaluate the potential cost of misclassification for physician report card designs. Monte Carlo simulation to explore misclassification risk and cost for four illustrative physician report card designs and three loss functions representing overall misclassification, patient, and pay-for-performance program perspectives. True physician performance is simulated under a beta-binomial model with parameters yielding simulated true scores resembling previously reported estimates. Misclassification risk varied across report card designs. Overall misclassification risk increased with the number of performance tiers for our simulated scenarios. However, the relationship between misclassification cost and number of tiers was inconsistent across the loss functions. The report card with the lowest misclassification cost varied across stakeholders. Within stakeholder, the costs of a two-tier report card with a high or low hurdle (25th and 75th percentile, respectively) varied. Loss functions and report card designs are illustrative and not intended to exhaustively catalog all possibilities. Little guidance exists on misclassification costs from the patient perspective. Misclassification cost depends on how performance information will be used and by whom. Selecting the lowest-cost design for a given stakeholder could maximize the usefulness of physician performance data. Misclassification cost could guide report card design, improving the usefulness of a report card for one stakeholder without disadvantaging others .

There are many recommendations of "best practices" for those doing data science, data-intensive research, and research in general. These documents usually present a particular vision of how people should work with data and computing, recommending specific tools, activities, mechanisms, and sensibilities. However, implementation of best (or better) practices in any setting is often met with resistance from individuals and groups, who perceive some drawbacks to the proposed changes to everyday practice. We offer some definitions of resistance, identify the sources of researchers' hesitancy to adopt new ways of working, and describe some of the ways resistance is manifested in data science teams. We then offer strategies for overcoming resistance based on our group members' experiences working alongside resistors or resisting change themselves. Our discussion concluded with many remaining questions left to tackle, some of which are listed at the end of this piece.

Background Public report card designers aim to provide comprehensible provider performance information to consumers. Report cards often display classifications of providers into performance tiers that reflect whether performance is statistically significantly above or below average or not statistically significantly different from average. To further enhance the salience of public reporting to consumers, report card websites often allow a user to compare a subset of selected providers on tiered performance rather than direct statistical comparisons of the providers in a consumer's personal choice set. Objective We illustrate the differences in conclusions drawn about relative provider performance using tiers versus conducing statistical tests to assess performance differences. Methods Using publicly available cross-sectional data from Medicare Hospital Compare on three mortality and three readmission outcome measures, we compared each provider in the top or bottom performance tier with those in the middle tier and assessed the proportion of such comparisons that exhibited no statistically significant differences. Results Across the six outcomes, 1.3-6.1% of hospitals were classified in the top tier. Each top-tier hospital did not statistically significantly differ in performance from at least one mid-tier hospital. The percentages of mid-tier hospitals that were not statistically significantly different from a given top-tier hospital were 74.3-81.1%. The percentages of hospitals classified as bottom tier were 0.6-4.0%. Each bottom-tier hospital showed no statistically significant difference from at least one mid-tier hospital. The percentage of mid-tier hospitals that were not significantly different from a bottom-tier hospital ranged from 60.4% to 74.8%. Conclusions Our analyses illustrate the need for further innovations in the design of public report cards to enhance their salience for consumers.

As part of an effort to assist Chile in developing a strategic program to foster the development of the health information technology (health IT) sector over the next five to ten years, this study assesses the current state of health IT adoption and implementation in Chile, as well as the challenges and opportunities facing the sector over the coming years. The authors conducted an environmental scan and ten key informant interviews and found that there are a number of successful health IT projects and strategies for further development currently underway in Chile, but that the successful projects are generally localized within specific health care providers and lack integration. These and other challenges suggest significant potential for the Ministry of Economy and other stakeholders to take specific actions designed to encourage further development of the health IT sector in Chile over the coming years. The next phase of this effort will use the results from this study to develop a roadmap for the Ministry of Economy to encourage health IT development in Chile over the short, medium, and long terms.

This dissertation translates to policy analysis solutions developed to address the lack of credibility in the reproducibility crisis in science.

As part of an effort to assist Chile in developing a strategic program to foster the development of the health information technology (health IT) sector over the next five to ten years, this study assesses the current state of health IT adoption and implementation in Chile, as well as the challenges and opportunities facing the sector over the coming years. The authors conducted an environmental scan and ten key informant interviews and found that there are a number of successful health IT projects and strategies for further development currently underway in Chile, but that the successful projects are generally localized within specific health care providers and lack integration. These and other challenges suggest significant potential for the Ministry of Economy and other stakeholders to take specific actions designed to encourage further development of the health IT sector in Chile over the coming years. The next phase of this effort will use the results from this study to develop a roadmap for the Ministry of Economy to encourage health IT development in Chile over the short, medium, and long terms.

As part of its Health and Development Strategic Program, Chile's Ministry of Economy, Development and Tourism needs a roadmap to foster the development of health information technology in the public health system. This report describes the roadmap, a guide for developing more-specific and detailed activities and timelines, and presents some recommendations for the ministry to consider when implementing the roadmap.

Describes a roadmap with five objectives for the Chilean government to expand its health information technology (IT) capabilities over the next ten years.