Search

A “two Americas” narrative emerged in the summer of 2021: one with high demand for COVID-19 vaccines, and a second with widespread vaccine hesitancy and opposition to mask mandates. But our analysis of excess mortality shows that the U.S. has been a divided nation at least since the start of the pandemic. Through April, 2022, there were 1,335,292 excess deaths associated with COVID-19, 37% more than reported as such. After the first wave, death rates in the South were more than double those in the Northeast; 45% of deaths were in the South, with 38% of the population.While some regard vaccination and other measures as matters of personal choice, the population impact is striking. If every region had the same mortality rate as the lowest regional rate in each period, more than 418,763 COVID-19 deaths were “avoidable,” more than half (58%) in the South and almost half before vaccines were available. These results show that population-based COVID-19 policies can still play an important role in protecting those most vulnerable to severe disease and death and reducing the spread of the virus.This example illustrates the importance of excess mortality measures as part of a comprehensive surveillance system. Official mortality counts rely on complete recording of COVID-19 as a cause of death, but COVID-19 deaths are under reported for many reasons. Indeed, the proportion of COVID-19 deaths reported as such varied markedly over time, and from 67% in the West to 87% the Northeast. In 2022, some regions cut back on testing making it harder to see a re-emergence of COVID-19 in those places. More extensive surveillance based on wastewater testing and other means that do not depend on testing are needed to get a more accurate picture. Excess mortality estimates are more tenuous years beyond the pre-pandemic period.

Population-based seroprevalence surveys can provide useful estimates of the number of individuals previously infected with serious acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and still susceptible, as well as contribute to better estimates of the case-fatality rate and other measures of coronavirus disease 2019 (COVID-19) severity. No serological test is 100% accurate, however, and the standard correction that epidemiologists use to adjust estimates relies on estimates of the test sensitivity and specificity often based on small validation studies. We have developed a fully Bayesian approach to adjust observed prevalence estimates for sensitivity and specificity. Application to a seroprevalence survey conducted in New York State in 2020 demonstrates that this approach results in more realistic—and narrower—credible intervals than the standard sensitivity analysis using confidence interval endpoints. In addition, the model permits incorporating data on the geographical distribution of reported case counts to create informative priors on the cumulative incidence to produce estimates and credible intervals for smaller geographic areas than often can be precisely estimated with seroprevalence surveys.

Timely and accurate data on COVID-19 cases and COVID-19‒related deaths are essential for making decisions with significant health, economic, and policy implications. A new report from the National Academies of Sciences, Engineering, and Medicine proposes a uniform national framework for data collection to more accurately quantify disaster-related deaths, injuries, and illnesses. This article describes how following the report’s recommendations could help improve the quality and timeliness of public health surveillance data during pandemics, with special attention to addressing gaps in the data necessary to understand pandemic-related health disparities.

Tests for Coronavirus disease 2019 (COVID-19) are intended for a disparate and shifting range of purposes: (1) diagnosing patients who present with symptoms to inform individual treatment decisions; (2) organizational uses such as "cohorting" potentially infected patients and staff to protect others; and (3) contact tracing, surveillance, and other public health purposes. Often lost when testing is encouraged is that testing does not by itself confer health benefits. Rather, testing is useful to the extent it forms a critical link to subsequent medical or public health interventions. Such interventions might be individual level, like better diagnosis, treatment, isolation, or quarantine of contacts. They might aid surveillance to understand levels and trends of disease within a defined population that enables informed decisions to implement or relax social distancing measures. In this article, we describe the range of available COVID-19 tests; their accuracy and timing considerations; and the specific clinical, organizational, and public health considerations that warrant different testing strategies. Three representative clinical scenarios illustrate the importance of appropriate test use and interpretation. The reason a patient seeks testing is often a strong indicator of the pretest probability of infection, and thus how to interpret test results. In addition, the level of population spread of the virus and the timing of testing play critical roles in the positive or negative predictive value of the test. We conclude with practical recommendations regarding the need for testing in various contexts, appropriate tests and testing methods, and the interpretation of test results.

Background: Over the course of the COVID-19 pandemic in Italy, different response measures were taken to contain the spread of the virus. These include a variety of non-pharmaceutical interventions and a mass vaccination campaign. While not definitive, epidemiological measures provide some indication of the impact of such measures on the dynamics of the pandemic and lessons to better prepare for future emergencies. Objective: To describe the impact of vaccine rollout and health policies on the evolution of the COVID-19 pandemic in Italy from March 2020 to October 2021 using a set of epidemiological indicators. Methods: We performed a time-trend analysis of new confirmed COVID-19 cases, patients in hospital, and deaths. Using line charts, we informally assessed the relationship of these indicators with the immunization campaign and other health policies. Daily aggregate data were gathered from GitHub repositories of certified data from Italy's Government and Civil Protection. Results: The immunization coverage increased starting in March 2021, with a parallel decrease in COVID-19 infections, hospitalizations, and deaths. Despite different implementation approaches, the vaccine coverage growth rate had a similar pattern across regions. A comprehensive approach including measures such as requiring face masks and a “Green Pass” to enter indoor places also helped contain the pandemic. Conclusions: The vaccine rollout had a major effect on COVID-19 in Italy, especially on hospitalizations and deaths. Before the vaccine was available, however, other non-pharmaceutical interventions also helped to contain the spread of the virus and mitigate its effect on the population.

In 2019, Clark County experienced a measles outbreak that challenged local and state public health authorities. Clark County Public Health (CCPH) led the response in collaboration with public and private entities. The first public health response included active surveillance and monitoring and to inform the public about where exposure might be taking place. Eventually, 71 confirmed cases were identified, 4,138 contacts traced, and 53 public exposure sites identified. As a social distancing measure unvaccinated students, teachers, and staff were excluded from schools in which a case had been reported. Accurately ascertaining vaccine status was challenging, especially for faculty and staff. The national anti-vaccination sentiment created challenges for emergency risk communication. CCPH monitored social media and developed talking points to address emerging issues. CCPH’s response was very resource intensive. Employing the Incident Command System (ICS) was critical, yet there were issues with role management, training, and communication outside ICS channels.

Issue Clark County experienced a measles outbreak that challenged public health authorities. Description of the practice: We conducted a formal After Action Review with state and local health officials, school officials, and others to identify lessons for public health practitioners facing future outbreaks. Results Following the early identification of measles in a child who had recently arrived from Ukraine, active surveillance identified 71 confirmed cases, most in unvaccinated persons under 18 years of age. 4,138 contacts were traced and public health personnel made daily monitoring calls to 816. 53 potential exposure sites in healthcare facilities, schools and other public places were identified and communicated to the public. As a social distancing measure, unvaccinated students, teachers, and staff were excluded from schools in which exposure had occurred. Ascertaining susceptibility status was challenging. The national anti-vaccination sentiment and a parallel outbreak in a New York religious community created challenges in representing community risk while avoiding stigmatization of a community in which the first reported case was identified. Rather than respond to every false claim on social media, the health department developed talking points about emerging issues and engaged the community in dialogue. Lessons Responding to the measles outbreak required innovative approaches to surveillance and contact tracing, social distancing (school exclusions), and emergency risk communication. The response required extensive coordinated efforts of the county and state health departments, school systems, and many other organizations. Mutual aid enabled an influx of resources but managing the surge of responders proved challenging. Key messages Public health emergencies require effective emergency management practices. Carefully conducted After Action Reviews of health emergencies can help public health practitioners identify challenges and innovative practices.

To demonstrate how public health emergency systems can use health systems tools to analyze and learn from critical incidents, we employed a facilitated look-back approach to review the public response to a chemical spill in Charleston, West Virginia. We reviewed official reports, news articles, and other documents; conducted in-person interviews with key public health and emergency response officials and local community stakeholders; and organized a facilitated look-back meeting to identify root causes of the problems that were encountered. The primary response challenges were (1) public distrust stemming from scientific uncertainty about potential harms of chemicals involved in the spill and how this uncertainty was communicated and (2) communication within the public health system, broadly defined. We found that to address inherent uncertainty, health officials should acknowledge uncertainty and tell the public what is known and unknown, and what they are doing to get more information.

Zika virus provides an example for which public health surveillance is based primarily on health care provider notifications to health departments of potential cases. This case-based surveillance is commonly used to understand the spread of disease in a population. However, case-based surveillance is often biased—whether testing is done and which tests are used and the accuracy of the results depend on a variety of factors including test availability, patient demand, perceptions of transmission, and patient and provider awareness, leading to surveillance artifacts that can provide misleading information on the spread of a disease in a population and have significant public health practice implications. To better understand this challenge, we first summarize the process that health departments use to generate surveillance reports, then describe factors influencing testing and reporting patterns at the patient, provider, and contextual level. We then describe public health activities, including active surveillance, that influence both patient and provider behavior as well as surveillance reports, and conclude with a discussion about the interpretation of surveillance data and approaches that could improve the validity of surveillance reports.

Quality measurement is at the heart of efforts to achieve high-quality surgical and medical care at a lower cost. Without accurate quality measures, it is not possible to appropriately align incentives with quality. The aim of these National Quality Forum (NQF) guidelines is to provide measure developers and other stakeholders with guidance on the standards used by the NQF to evaluate the scientific acceptability of performance measures. Using a methodologically rigorous and transparent process for evaluating health care quality measures is the best insurance that alternative payment plans will truly reward and promote higher quality care. Performance measures need to be credible in order for physicians and hospitals to willingly partner with payers in efforts to improve population outcomes. Our goal in creating this position paper is to promote the transparency of NQF evaluations, improve the quality of performance measurements, and engage surgeons and all other stakeholders to work together to advance the science of performance measurement.

Background In 2017, under the supervision of the European Centre for Disease Prevention and Control (ECDC), a list of competencies and knowledge and skill statements for EU professionals working in Public Health Emergency Preparedness (PHEP) systems were identified and integrated into a Competency Model. Building on these efforts, a PHEP competency-based training toolkit was developed, pilot tested and finalized with the goal of providing a standardized training resource for experts involved in response to cross-border threats to health. Objectives Led by the ECDC, an international research team drafted a preliminary version of a curriculum consisting of five modules, with a list of targeted competencies, learning objectives, reading and training methods including case studies and simulation exercises. In May 2018, the curriculum was pilot tested in a 3.5-day training course at the University of Bologna, Italy. The participants included 20 representatives from 8 EU Member States. An evaluation strategy, combining both quantitative and qualitative methods, was designed to assess the impact of the training on participants’ applied knowledge and competence. Results Overall, the participants’ feedback on the pilot course was positive, with scores from 4.0 to 4.8 out of 5 for all the items considered. Several suggestions were made both by the trainers and the trainees to enhance the effectiveness of the learning activities, and the curriculum was integrated into a training toolkit with more exercises, teaching materials and a guide for facilitators. Conclusions This European Public Health Emergency Preparedness Training Toolkit is the first EU competency-based toolkit intended to help Member States improve their response to public health emergencies through standardized training. Designed for individuals working in strategic positions at the national level, its general approach, modular structure and content are relevant and can be tailored to a wider range of professionals. Key messages To improve PHEP in the EU, competency-based curricula and standardized training programs and resources for health professionals are needed. By enhancing the overall public health professionals’ experience and competency, the European PHEP Training Toolkit represents a step towards strengthening preparedness and response systems in the EU.

Background After Action Reviews (AARs) provide a means to observe how well preparedness systems perform in real world conditions and can help to identify – and address – gaps in national and global public health emergency preparedness (PHEP) systems. WHO has recently published guidance for voluntary AARs. This analysis builds on this guidance by reviewing evidence on the effectiveness of AARs as tools for system improvement and by summarizing some key lessons about ensuring that AARs result in meaningful learning from experience. Results Empirical evidence from a variety of fields suggests that AARs hold considerable promise as tools of system improvement for PHEP. Our review of the literature and practical experience demonstrates that AARs are most likely to result in meaningful learning if they focus on incidents that are selected for their learning value, involve an appropriately broad range of perspectives, are conducted with appropriate time for reflection, employ systems frameworks and rigorous tools such as facilitated lookbacks and root cause analysis, and strike a balance between attention to incident specifics vs. generalizable capacities and capabilities. Conclusions Employing these practices requires a PHEP system that facilitates the preparation of insightful AARs, and more generally rewards learning. The barriers to AARs fall into two categories: concerns about the cultural sensitivity and context, liability, the political response, and national security; and constraints on staff time and the lack of experience and the requisite analytical skills. Ensuring that AARs fulfill their promise as tools of system improvement will require ongoing investment and a change in mindset. The first step should be to clarify that the goal of AARs is organizational learning, not placing blame or punishing poor performance. Based on experience in other fields, the buy-in of agency and political leadership is critical in this regard. National public health systems also need support in the form of toolkits, guides, and training, as well as research on AAR methods. An AAR registry could support organizational improvement through careful post-event analysis of systems’ own events, facilitate identification and sharing of best practices across jurisdictions, and enable cross-case analyses.

Background Guangdong, which stands at the frontier of emerging infections, has developed an innovative system approach to this threat. The province of 110 million inhabitants, a major hub for domestic and international trade, includes mega-cities side-by-side with small-scale agricultural areas in a region from which recent strains of influenza and other respiratory viruses have emerged. Guangdong residents consume chicken and purchase it at live poultry markets, which can accelerate the emergence of influenza strains and transmission to humans. We describe Guangdong's approach to surveillance and public health emergency response. # Methods We use H7N9 as a case study to illustrate how Guangdong's system functions. # Results Guangdong's approach includes five core elements -- real-time surveillance, risk assessment, rapid response, active and accurate intervention, and multilateral communication. This requires partnerships among governmental agencies at all levels as well as healthcare professionals and residents. Risk management, for instance, involves not only provincial and local Centers for Disease Control, other government agencies, hospitals and clinics, and wholesale and live poultry markets. The implementation of Guangdong's risk management strategy depends, in part, on available resources and existing capacities. Guangzhou and a majority of Guangdong Province implement "control risk" level strategies. However, the city of Shenzhen (as well as neighboring Hong Kong), implement "eliminate risk." level strategies. # Conclusions Effective disease surveillance requires vertical and horizontal coordination among public health agencies. Similarly, disease prevention and control efforts also require partnerships with other governmental agencies at all levels, as well as with healthcare professionals and civil society. Guangdong Province has developed an approach for surveillance and control of H7N9 that may represent a robust and potentially scalable model for China and other nations.

In 2017, the European Centre for Disease Prevention and Control (ECDC) developed a competency model for individuals who work in public health emergency preparedness (PHEP) in European Union (EU) countries. The model serves as the basis for developing competency-based training programmes to support professionals in PHEP efforts at the country level. The competency model describes the knowledge and skills professionals need when working in national-level PHEP, such as preparedness committee members or their equivalents. In order to develop the model, existing competency statements were reviewed, as well as case studies and reports. Fifty-three professionals from the EU and other countries provided feedback to the model by participating in a three-stage consultation process. The model includes 102 competency, 100 knowledge and 158 skill statements. In addition to specifying the appropriate content for training programmes, the proposed common competency model can help to standardise terminology and approaches to PHEP training.

Learning health systems use routinely collected electronic health data (EHD) to advance knowledge and support continuous learning. Even without randomization, observational studies can play a central role as the nation’s health care system embraces comparative effectiveness research and patient-centered outcomes research. However, neither the breadth, timeliness, volume of the available information, nor sophisticated analytics, allow analysts to confidently infer causal relationships from observational data. However, depending on the research question, careful study design and appropriate analytical methods can improve the utility of EHD.The introduction to a series of four papers, this review begins with a discussion of the kind of research questions that EHD can help address, noting how different evidence and assumptions are needed for each. We argue that when the question involves describing the current (and likely future) state of affairs, causal inference is not relevant, so randomized clinical trials (RCTs) are not necessary. When the question is whether an intervention improves outcomes of interest, causal inference is critical, but appropriately designed and analyzed observational studies can yield valid results that better balance internal and external validity than typical RCTs. When the question is one of translation and spread of innovations, a different set of questions comes into play: How and why does the intervention work? How can a model be amended or adapted to work in new settings? In these “delivery system science” settings, causal inference is not the main issue, so a range of quantitative, qualitative, and mixed research designs are needed.We then describe why RCTs are regarded as the gold standard for assessing cause and effect, how alternative approaches relying on observational data can be used to the same end, and how observational studies of EHD can be effective complements to RCTs. We also describe how RCTs can be a model for designing rigorous observational studies, building an evidence base through iterative studies that build upon each other (i.e., confirmation across multiple investigations).

To demonstrate how public health systems can use root-cause analysis (RCA) to improve learning from critical incidents, the research team utilized a facilitated look-back meeting to examine the public health systems' response to a Salmonella outbreak in the water supply in Alamosa, Colorado. We worked with public health, emergency management agencies, and other stakeholders to identify response challenges related to public health emergency preparedness capabilities, root causes, and lessons learned. The results demonstrate that RCA can help identify systems issues that, if addressed, can improve future responses. Furthermore, RCA can identify more basic issues that go beyond a specific incident or setting, such as the need for effective communication and coordination throughout the public health system, and the social capital needed to support it.

Zika virus disease provides the latest example of a critical nexus between public health and clinical practice. Interpreting Zika virus test results is complicated by the absence of a single testing approach with superior validity across contexts and populations. Molecular tests are highly specific, variably sensitive, and have a short window period. Serologic tests identify antibodies against Zika virus and are more likely than molecular tests to cross-react with other related viruses, reducing specificity. The type of test performed and timing relative to possible Zika virus exposure depend on public health guidance, testing algorithms, test availability, and capacity. Guidance from the Centers for Disease Control and Prevention and local health departments have changed throughout the course of the US epidemic based on prevalence, geography, and clinical concerns. Women with a low pretest probability of infection should be counseled against testing. Women with a high pretest probability of Zika virus infection should still receive enhanced prenatal monitoring and newborn evaluation, regardless of the test result. An appropriate interpretation of results depends on what tests are used, patient characteristics, and reasons for testing. Clinicians should take these factors into account in shared decision making discussions with pregnant women about Zika virus testing.

Among the many provisions of the Affordable Care Act (ACA) that aim to improve population health, one stands out as having the potential for bridging the efforts of the healthcare delivery sector, public health agencies, and other community organizations to improve population health outcomes. Under IRS §501(r) regulations, non-profit hospitals are required to work with health departments and other community organizations to conduct a Community Health Needs Assessment (CHNA) at least every three years and adopt an implementation strategy (IS) describing how identified needs will be addressed. These regulations have the potential to refocus some of the “community benefits” that hospitals are required to spend (estimated at $62 billion in 2011) towards improving population health. Taken together with other provisions in the ACA intended to encourage the healthcare delivery sector to be accountable for health outcomes in defined populations, as well as the increasing prominence of population health and performance management in the healthcare delivery sector, the new CHNA requirements – and especially their implementation strategies – have the potential to refocus healthcare providers’ attention “upstream” to address the social and behavioral determinants of health. Indeed, the CHNA requirements of the ACA and related initiatives offer an extraordinary promise for advancing a culture of health through multi-sector collaboration to build health partnerships. However, although some hospitals and communities have adopted promising approaches to meeting the IRS requirements, CHNA and other community health improvement processes vary markedly in how they define the community served, in how they receive input from community members, in the methods they use to set priorities, in the degree of collaboration among hospitals and other organizations, and in they measure performance. Community stakeholders often do not understand how to engage hospitals as partners. And hospitals often lack the staffing and competencies to engage in and sustain partnerships. Consequently, the IRS CHNA regulations so far do not seem to have transformed the healthcare delivery sector to the degree that they might have. To understand why the IRS regulations have not yet realized their promise, and to identify models that can help them do so, this environmental scan was designed to summarize the forces of change in the U.S. healthcare delivery system and other factors that affect population health and the context in which hospitals operate. This scan is conducted as part of a project supported by the Robert Wood Johnson Foundation to assess the impact of the CHNA requirements on improving population health in communities. To understand the context in which CHNAs are being conducted, we begin by examining changes in the healthcare delivery system, showing how hospitals, and healthcare more generally, are transitioning from a system in which they were reimbursed based on the volume of services they provided, to one based on the value of the results they produce. As a result, healthcare delivery systems are increasingly being held accountable for improving health outcomes, which they realize they cannot do without collaboration with other entities in the communities they serve. Section 3 discusses a series of parallel developments all based on the growing understanding that improving population health outcomes requires the engagement and collaboration of the healthcare delivery system and public health agencies, and indeed many other entities in the community. These developments create incentives for hospitals and other healthcare organizations to collaborate with multiple public and private organizations in the communities they serve. From this perspective, a community health improvement process provides the potential to leverage and coordinate the strengths and resources of both the healthcare and public health systems as well as many public and private sector agencies and organizations to create healthier communities (Kuehnert, 2012). Indeed, if implemented as intended, a coordinated community health improvement process can help communities achieve the five conditions that Kania and Kramer (2011) in their groundbreaking work on “collective impact” conclude are necessary for large-scale social change: a common agenda, shared measurement systems, mutually reinforcing activities, continuous communication, and backbone support organizations. Section 4 describes the IRS community benefit and CHNA regulations and the plethora of guidance, CHNA models, data, and other resources that have been developed to help hospitals implement the IRS regulations. These include models and other guidance for conducting community health assessments and developing implementation strategies for hospitals and local health departments. Also included are small-area data resources and web-portals providing access to evidence about effective community-based interventions. Section 5 describes activities undertaken by individual states – in some cases pre-dating the IRS regulations – that facilitate hospitals’ community health improvement processes in the ACA era. These include delivery system reforms, prevention strategies, community benefit regulations, CHNA mandates, data resources, and other activities intended to foster collaboration between healthcare, public health, and other community organizations. There have been no comprehensive assessments of the impact of either CHNAs per se or the ACA-mandated IRS regulations, but Section 6 summarizes the studies, mostly state-based, of community health improvement processes that have been published to date. Consistent with the results from the Public Health Institute analysis described above, the results of these studies demonstrate considerable variability in how CHNAs are done, and how seriously they are taken by hospital and other decision-makers. The paper concludes with a discussion of some open questions regarding approaches to conducting CHNAs and developing implementation strategies. In particular, we address: the definition of the community served by hospitals, methods for priority setting, collaboration among hospitals and with other organizations, expectations about collaboration among hospitals and with other organizations in implementation, as well as evaluation and performance measurement.

Public health authorities have adopted entry screening and subsequent restrictions on travelers from Ebola-affected West African countries as a strategy to prevent importation of Ebola virus disease (EVD) cases. We analyzed international, federal, and state policies-principally based on the policy documents themselves and media reports-to evaluate policy variability. We employed means-ends fit analysis to elucidate policy objectives. We found substantial variation in the specific approaches favored by WHO, CDC, and various American states. Several US states impose compulsory quarantine on a broader range of travelers or require more extensive monitoring than recommended by CDC or WHO. Observed differences likely partially resulted from different actors having different policy goals-particularly the federal government having to balance foreign policy objectives less salient to states. Further, some state-level variation appears to be motivated by short-term political goals. We propose recommendations to improve future policies, which include the following: (1) actors should explicitly clarify their objectives, (2) legal authority should be modernized and clarified, and (3) the federal government should consider preempting state approaches that imperil its goals.

The third paper in a series on how learning health systems can use routinely collected electronic health data (EHD) to advance knowledge and support continuous learning, this review describes how analytical methods for individual-level electronic health data EHD, including regression approaches, interrupted time series (ITS) analyses, instrumental variables, and propensity score methods, can also be used to address the question of whether the intervention “works.”The two major potential sources of bias in non-experimental studies of health care interventions are that the treatment groups compared do not have the same probability of treatment or exposure and the potential for confounding by unmeasured covariates. Although very different, the approaches presented in this chapter are all based on assumptions about data, causal relationships, and biases. For instance, regression approaches assume that the relationship between the treatment, outcome, and other variables is properly specified, all of the variables are available for analysis (i.e., no unobserved confounders) and measured without error, and that the error term is independent and identically distributed. The instrumental variables approach requires identifying an instrument that is related to the assignment of treatment but otherwise has no direct on the outcome. Propensity score methods approaches, on the other hand, assume that there are no unobserved confounders. The epidemiological designs discussed also make assumptions, for instance that individuals can serve as their own control.To properly address these assumptions, analysts should conduct sensitivity analyses within the assumptions of each method to assess the potential impact of what cannot be observed. Researchers also should analyze the same data with different analytical approaches that make alternative assumptions, and to apply the same methods to different data sets. Finally, different analytical methods, each subject to different biases, should be used in combination and together with different designs, to limit the potential for bias in the final results.

The last in a series of four papers on how learning health systems can use routinely collected electronic health data (EHD) to advance knowledge and support continuous learning, this review describes how delivery system science provides a systematic means to answer questions that arise in translating complex interventions to other practice settings. When the focus is on translation and spread of innovations, the questions are different than in evaluative research. Causal inference is not the main issue, but rather one must ask: How and why does the intervention work? What works for whom and in what contexts? How can a model be amended to work in new settings? In these settings, organizational factors and design, infrastructure, policies, and payment mechanisms all influence an intervention’s success, so a theory-driven formative evaluation approach that considers the full path of the intervention from activities to engage participants and change how they act to the expected changes in clinical processes and outcomes is needed. This requires a scientific approach to quality improvement that is characterized by a basis in theory; iterative testing; clear, measurable process and outcomes goals; appropriate analytic methods; and documented results.To better answer the questions that arise in delivery system science, this paper introduces a number of standard qualitative research approaches that can be applied in a learning health system: Pawson and Tilley’s “realist evaluation,” theory-based evaluation approaches, mixed-methods and case study research approaches, and the “positive deviance” approach.

Improving preparedness in the European region requires a clear understanding of what European Union (EU) member states should be able to do, whether acting internally or in cooperation with each other or the EU and other multilateral organizations. We have developed a preparedness logic model that specifies the aims and objectives of public health preparedness, as well as the response capabilities and preparedness capacities needed to achieve them. The capabilities, which describe the ability to effectively use capacities to identify, characterize, and respond to emergencies, are organized into 5 categories. The first 3 categories—(1) assessment; (2) policy development, adaptation, and implementation; and (3) prevention and treatment services in the health sector—represent what the public health system must accomplish to respond effectively. The fourth and fifth categories represent a series of interrelated functions needed to ensure that the system fulfills its assessment, policy development, and prevention and treatment roles: (4) coordination and communication regards information sharing within the public health system, incident management, and leadership, and (5) emergency risk communication focuses on communication with the public. This model provides a framework for identifying what to measure in capacity inventories, exercises, critical incident analyses, and other approaches to assessing public health emergency preparedness, not how to measure them. Focusing on a common set of capacities and capabilities to measure allows for comparisons both over time and between member states, which can enhance learning and sharing results and help identify both strengths and areas for improvement of public health emergency preparedness in the EU., Improving preparedness in Europe requires a clear understanding of what EU member states should be able to do, whether acting internally or in cooperation with each other. The authors developed a preparedness logic model that specifies the aims and objectives of public health preparedness, as well as the response capabilities and preparedness capacities needed to achieve them.

As an alternative to standard quality improvement approaches and to commonly used after action report/improvement plans, we developed and tested a peer assessment approach for learning from singular public health emergencies. In this approach, health departments engage peers to analyze critical incidents, with the goal of aiding organizational learning within and across public health emergency preparedness systems. We systematically reviewed the literature in this area, formed a practitioner advisory panel to help translate these methods into a protocol, applied it retrospectively to case studies, and later field-tested the protocol in two locations. These field tests and the views of the health professionals who participated in them suggest that this peer-assessment approach is feasible and leads to a more in-depth analysis than standard methods. Engaging people involved in operating emergency health systems capitalizes on their professional expertise and provides an opportunity to identify transferable best practices.

Research Objective: Non-profit hospitals are required to work with community organizations to prepare a Community Health Needs Assessment (CHNA) and implementation strategy (IS). In concert with the health care delivery system’s transformation from volume to value and efforts to enhance multi-sector collaboration, such community health improvement (CHI) processes have the potential to bridge efforts of the health care delivery sector, public health agencies, and community organizations to improve population health. Having a shared measurement system is critical to achieving collective impact, yet despite the availability of community-level data from a variety of sources, many CHI processes lack clear, measurable objectives and evaluation plans. Through an in-depth analysis of ten exemplary CHI processes, we sought to identify best practices for population health measurement with a focus on measures for needs assessments and priority setting.Study Design: Based on a review of the scientific literature, professional publications and presentations, and nominations from a national advisory panel, we identified 10 exemplary CHI processes. Criteria of choice were whether (1) the CHIs articulate a clear definition of intended outcomes; (2) clear, focused, measurable objectives and expected outcomes, including health equity; (3) expected outcomes are realistic and addressed with specific action plans; and (4) whether the plans and their associated performance measures become fully integrated into agencies and become a way of being for the agencies. We then conducted an in-depth analysis of CHNA, IS, and related documents created by health departments and leading hospitals in each process.Population Studied: U.S. hospitals.Principal Findings: Census, American Community Survey, and similar data are available for smaller areas are used to describe the populations covered, and, to a lesser extent, to identify health issues where there are disparities and inequities.Common data sources for population health profiles, including risk factors and population health outcomes, are vital statistics, survey data including BRFSS, infectious disease surveillance data, hospital & ED data, and registries. These data are typically available only at the county level, and only occasionally are broken down by race, ethnicity, age, poverty.There is more variability in format and content of ISs than CHNAs; the most developed models include population-level goals/objectives and strategies with clear accountability and metrics. Other hospital IS’s are less developed.Conclusions: The county is the unit of choice because most population health profile data are not available for sub-county areas, but when a hospital serves a population more broadly or narrowly defined, appropriate data are not available to set priorities or monitor progress.Measure definitions are taken from the original data sources, so comparisons across measures is difficult. Thus, although CHNAs cover many of the same topics, the measures used vary markedly. Using the same community health profile, e.g. County Health Rankings, would simplify benchmarking and trend analysis.Implications for Policy or Practice: It is important to develop population health data that can be disaggregated to the appropriate geographical level and to groups defined by race and ethnicity, socioeconomic status, and other factors associated with health outcomes.

Research Objective: Non-profit hospitals are required to work with community organizations to prepare Community Health Needs Assessment (CHNA) and implementation strategy (IS). In concert with the health care delivery system’s transformation from volume to value and efforts to enhance multi-sector collaboration, such community health improvement (CHI) processes have the potential to bridge efforts of the health care delivery sector, public health agencies, and community organizations to improve population health. Having a shared measurement system is critical to achieving collective impact, yet despite the availability of community-level data from a variety of sources, many CHI processes lack clear, measurable objectives and evaluation plans. Through an in-depth analysis of ten exemplary CHI processes, we sought to identify best practices for population health measurement with a focus on monitoring collaborative implementation strategies.Study Design: Based on a review of the scientific literature, professional publications and presentations, and nominations from a national advisory panel, we identified 10 exemplary CHI processes. Criteria of choice were whether (1) the CHIs articulate a clear definition of intended outcomes; (2) clear, focused, measurable objectives and expected outcomes, including health equity; (3) expected outcomes are realistic and addressed with specific action plans; and (4) whether the plans and their associated performance measures become fully integrated into agencies and become a way of being for the agencies. We then conducted an in-depth analysis of CHNA, IS, and related documents created by health departments and leading hospitals in each process.Population Studied: U.S. hospitals.Principal Findings: Community health improvement processes benefit from a shared measurement system that indicate accountability for specific activities. Despite the importance of measurement and evaluation, existing community health improvement efforts often fall short in these areas. There is more variability in format and content of ISs than CHNAs; the most developed models include population-level goals/objectives and strategies with clear accountability and metrics. Other hospital IS’s are less developed.Although all U.S. hospitals are familiar with performance measurement in their management, this familiarity does not seem to carry over to Community Benefit and CHNA efforts. Indeed, 5 of the 10 CHI processes we examined have some Accountable Care Organization (ACO) involvement, where population-health performance measures are commonplace. Yet this involvement is not mentioned in the CHNAs and ISs, nor are ACO data cited.Conclusions: Strengthening the CHNA regulations to require that hospitals report the evaluation measures they intend to monitor based on an established community health improvement model could help communities demonstrate impact. As in other areas of health care, performance measures should be tailored to implementation strategy, with clear indication of accountability, and move from outputs to process and outcome measures with established validity and reliability.Implications for Policy or Practice: Although performance measurement is now commonplace throughout the health care system, the individuals who manage CHI processes may not be that familiar with this approach. This suggests that it is important to develop practitioners’ knowledge and skills needed to use it population health data effectively.

To the Editor: Thank you to Drs Yasril and Wiwanitkit for raising these additional points regarding our article. We agree that clinicians should be aware of all possible sources of error in testing for Zika virus

Objective Identify lessons about the public health emergency preparedness system from after action report/improvement plans (AAR/IPs) authored by state and local health departments following the 2009 H1N1 influenza pandemic. Design Potentially generalizable findings were collected during a workshop attended by representatives from the Centers for Disease Control and Prevention (CDC), state and local public health departments, and other organizations that prepared 2009 H1N1AAR/IPs. Participants Workshop participants included state and local health department personnel who had submitted AAR/IPs to the CDC for review. Measures Workshop participants were asked to consider the question: What did you hear from other jurisdictions that resonated with your own experience and could be a generalized finding? Results Workshop discussions revealed potential lessons concerning: (1) situational awareness during the initial response; (2) resource mobilization and legal authority; (3) the complexity of vaccine distribution and administration; (4) balancing emergency response and routine operations; (5) communication and coordination among the many independent actors in the public health system; and (6) incident management in a long-duration incident. Conclusions The response to the 2009 H1N1 influenza pandemic provides an opportunity to learn about the public health system's emergency response capabilities and to identify ways to improve preparedness for future events. Perhaps the most important lessons from the 2009 H1N1 response reveal the complexity of coordinating actions among the many different actors, institutions, sectors, and disciplines involved in the public health system. While the response to the pandemic engendered creative "on the spot" solutions, continued effort is needed to better understand and manage the identified challenges.

Objective To analyze key variations in the after action report/improvement plan (AAR/IP) process used by state and local health departments following the 2009 H1N1 pandemic and identify ideas for improving that process. Design Workshop participants discussed their AAR findings and the methods used to prepare their reports and implications for improving the AAR/IP process in future events. Participants Workshop participants included state and local health department personnel who had submitted AAR/IPs to the Centers for Disease Control and Prevention (CDC) for review. Measures Workshop participants were asked to consider the question: On the basis of what you heard in this workshop, what would you do differently if you could redo your 2009 H1N1 AAR/IP? Results Workshop discussions revealed wide differences in the participants' understanding of the intended uses and users of the AAR/IPs, their scope, timing, and format, and the use of external consultants in their preparation, and on the strengths and weaknesses of various approaches. The AAR/IPs also varied in the extent to which they sought to identify root causes and the methods they used to do so. Conclusions The AAR/IPs can be useful for both accountability and quality improvement, but these objectives require different foci and methodological approaches. Notably, the AAR/IPs can also be used as an opportunity to hold health departments accountable for conducting root cause analyses and making the improvements that follow from them. Federal agencies requiring the AAR/IPs should clarify the purpose and issues of scope and timing; develop training materials and exemplary cases of effective AAR/IPs, particularly of root cause analysis applied to public health emergency preparedness, professional guidelines, and standards for consultants; and consider developing a peer model for preparing AAR/IPs.

The National Health Security Strategy calls for a systematic quality improvement efforts to improving health security. In addition, since 2001, billions of dollars have been spent on public health emergency preparedness at the federal, state, and local levels. Policy makers must now ask has this investment been effective, and in what way to measure this investment. Whether the goal is to ensure accountability to policy makers or to facilitate quality improvement, valid and reliable measures of preparedness are needed. Measuring and assessing the state of the nation's preparedness, however, are challenging. This article explores the current measurement climate and potential for improvement.

The Harvard School of Public Health Center for Public Health Preparedness exercise program has two aims: ( 1) educating the public health workforce on key public health system emergency preparedness issues, and ( 2) identifying specific systems-level challenges in the public health response to large-scale events. Rigorous evaluation of 38 public health emergency preparedness (PHEP) exercises employing realistic scenarios and reliable and accurate outcome measures has demonstrated the usefulness of PHEP exercises in clarifying public health workers' roles and responsibilities, facilitating knowledge transfer among these individuals and organizations, and identifying specific public health systems-level challenges.

Objectives: The Medical Reserve Corps (MRC) is a national network of community-based units created to promote the local identification, recruitment, training, and activation of volunteers to assist local health departments in public health activities. This study aimed to develop a toolkit for MRC coordinators to assess and monitor volunteer units' performance and identify barriers limiting volunteerism.Methods: In 2008 and 2009, MRC volunteers asked to participate in influenza clinics were surveyed in 7 different locations throughout the United States. Two survey instruments were used to assess the performance of the volunteers who were able to participate, the specific barriers that prevented some volunteers from participating, and the overall attitudes of those who participated and those who did not. Validity and reliability of the instruments were assessed through the use of factor analysis and Cronbach's alpha.Results: Two survey instruments were developed: the Volunteer Self-Assessment Questionnaire and the Barriers to Volunteering Questionnaire. Data were collected from a total of 1059 subjects, 758 participated in the influenza clinics and 301 were unable to attend. Data from the 2 instruments were determined to be suitable for factor analysis. Factor solutions and inter-item correlations supported the hypothesized domain structure for both survey questionnaires. Results on volunteers' performance were consistent with observations of both local health departments' staff and external observers.Conclusions: The survey instruments developed for this study appear to be valid and reliable means to assess the performance and attitudes of MRC volunteers and barriers to their participation. This study found these instruments to have face and content validity and practicality. MRC coordinators can use these questionnaires to monitor their ability to engage volunteers in public health activities.(Disaster Med Public Health Preparedness. 2010;4:213-219)

Purpose Because health insurance claims lack clinical information, comparative effectiveness research studies that rely on these data may be challenging to interpret and may result in biased inference. We conducted an exploratory study to determine if medical information contained in patient charts could offer clinical details that would assist in interpreting the results of a claims-based comparative effectiveness study of selective serotonin reuptake inhibitors (SSRIs). Methods Retrospective review of 457 charts of patients initiating SSRI treatment. Descriptive data elements included patient diagnosis, symptoms of depressive and anxiety disorders, provider's assessment, and medication treatment and side effects. Results Most subjects were excluded from the study because their charts were not accessible (58.7%), they did not have a follow-up visit (55.6%), providers could not be contacted (58.0%), or providers refused participation in the study (36.5%). Among those included in the study, most patients were noted to have depression, but most charts lacked information on the majority of depression symptoms at baseline and follow-up. Few concomitant symptoms, side effects, and other important clinical and treatment characteristics were recorded. Conclusions Inability to obtain charts due to plan or provider refusal, lack of available information in charts at key times in the course of illness, and missing data elements posed considerable challenges and prevented firm conclusions beyond those drawn from the parent, claims-based study. Copyright © 2010 John Wiley & Sons, Ltd.

No standard case definition exists for interstitial cystitis/painful bladder syndrome for patient screening or epidemiological studies. As part of the RAND Interstitial Cystitis Epidemiology study, we developed a case definition for interstitial cystitis/painful bladder syndrome with known sensitivity and specificity. We compared this definition with others used in interstitial cystitis/painful bladder syndrome epidemiological studies.We reviewed the literature and performed a structured, expert panel process to arrive at an interstitial cystitis/painful bladder syndrome case definition. We developed a questionnaire to assess interstitial cystitis/painful bladder syndrome symptoms using this case definition and others used in the literature. We administered the questionnaire to 599 women with interstitial cystitis/painful bladder syndrome, overactive bladder, endometriosis or vulvodynia. The sensitivity and specificity of each definition was calculated using physician assigned diagnoses as the reference standard.No single epidemiological definition had high sensitivity and high specificity. Thus, 2 definitions were developed. One had high sensitivity (81%) and low specificity (54%), and the other had the converse (48% sensitivity and 83% specificity). These values were comparable or superior to those of other epidemiological definitions used in interstitial cystitis/painful bladder syndrome prevalence studies.No single case definition of interstitial cystitis/painful bladder syndrome provides high sensitivity and high specificity to identify the condition. For prevalence studies of interstitial cystitis/painful bladder syndrome the best approach may be to use 2 definitions that would yield a prevalence range. The RAND Interstitial Cystitis Epidemiology interstitial cystitis/painful bladder syndrome case definitions, developed through structured consensus and validation, can be used for this purpose.

Background Despite the acknowledged promise of developing a public health systems research (PHSR) agenda for emergency preparedness, there has been no systematic review of the literature in this area. The purpose of this study was to conduct a systematic literature review in order to identify and characterize the PHSR literature produced in the U.S. in the past 11 years in the field of public health emergency preparedness. Evidence acquisition Articles were searched in MEDLINE and EMBASE, as well as in the gray literature. Two independent reviewers classified the articles according to study design and IOM public health emergency preparedness (PHEP) research goal areas. Evidence synthesis From January 1, 1997, through December 31, 2008, there were 547 articles that met the inclusion criteria that were published. It was possible to classify 314 (57%) articles into at least one of the four IOM PHEP research goal areas. Of these, 61 (11%) addressed Research Area 1 (usefulness of training); 39 (7%) addressed Research Area 2 (communications in preparedness and response); 193 (35%) addressed Research Area 3 (sustainable preparedness and response systems); and 39 (7%) addressed Research Area 4 (criteria and metrics to measure effectiveness and efficiency). Twenty-one studies (4%) could be classified into more than one category. The majority of the articles (81%), including commentaries/reviews and case studies, were based on qualitative analysis. Commentaries/review articles were the most common study types (62%). Conclusions Since 2001, the PHSR literature on PHEP issues has grown at about 33% per year. However, most studies lack a rigorous design, raising questions about the validity of the results.

Although previous studies have found no differences in response to antidepressant pharmacotherapy between selective serotonin reuptake inhibitors (SSRIs), some recent trials suggest benefits associated with more rapid onset of action.The aim of this work was to compare the likelihood that patients initiating treatment with branded escitalopram, rather than with any of 3 SSRIs (ie, citalopram, fluoxetine, and paroxetine) that are available in generic or branded formulations, would continue therapy with the initial medication after 2 and 6 months.We used propensity score-weighted logistic regression to assess the effect of antidepressant choice on the likelihood of continuing treatment, based on data from a large administrative claims database with information about US patients. We modeled the propensity to initiate treatment with escitalopram based on demographic, diagnostic, insurance, and service-use characteristics in the 6 months before treatment initiation and used the results to calculate weights for analysis of treatment continuation. The primary outcome measures were receipt of 2 prescriptions of the index drug in the first 2 months and, among those continuing at 2 months, 4 prescriptions in the first 6 months. Antidepressant choice, cost, and service-use characteristics during the treatment period were included as covariates. Patients who initiated therapy between July 2002 and April 2005 were eligible for inclusion.Based on data for 43,921 patients, at 2 months, escitalopram initiators were more likely to have continued initial medication than those receiving the other SSRIs (66.1% vs 61.9%, respectively; P0.01) and less likely to have switched or augmented treatment (4.8% vs 7.6%; P0.01). At 6 months, escitalopramtreated patients were also more likely to have continued initial medication (47.1% vs 41.0%; P0.01) and less likely to have switched or augmented treatment (9.4% vs 14.4%; P0.01).Patients initiating treatment with escitalopram were more likely to continue and less likely to switch or augment treatment at 2 and 6 months of therapy compared with those who initiated with alternative SSRIs.

The Washington metropolitan area was closely examined to understand how these regional preparedness structures have been organized, implemented, and governed, as well as to assess the likely impact of such regional structures on public health preparedness and public health systems more generally. It was found that no single formal regional structure for the public health system exists in the Washington metropolitan area, although the region is designated by the Department of Homeland Security as the National Capital Region (NCR). In fact, the vast majority of preparedness planning and response activities in this area are the result of voluntary self-organization through both governmental and nongovernmental organizations. Some interviewed felt that this was an optimal arrangement, as personal relationships prove crucial in responding to a public health emergency and an informal response is often more timely than a formal response. The biggest challenge for public health preparedness in the NCR is incorporating all federal government agencies in the area in NCR preparedness planning.