Search

Objective We used experiences in multiple countries to determine that owner engagement is critical for successful evaluations of surveillance system viability. Introduction Pilot projects help determine utility and feasibility of a system, but even if considered successful, cost could prevent further scale-up. When evaluating a surveillance system pilot, cost and benefits are key factors to examine. In Cote d’Ivoire and Tanzania, Ministry of Health (MoH) and non-governmental partners receive funding under the Global Health Security Agenda to strengthen disease surveillance for earlier detection and improved response to potential infectious disease outbreaks. To this end, Community based surveillance (CBS) projects were implemented in 2016 as a means for early warning of potential events to facilitate a more rapid response. Currently, these CBS projects are being evaluated collaboratively with the primary stakeholder, the host country government (HCG), as lead, and partners such as CDC providing technical assistance. In other instances, partners may conduct an evaluation and share the results and recommendations with the HCG; however, if the HCG is not actively engaged as the primary executor, outcomes may not be endorsed or implemented. Therefore, these evaluations were approached from an owner’s (HCG) perspective. In this way, the governmental agencies develop capabilities to conduct similar activities in other areas, reduce dependencies on outside entities, and promote enactment of resulting recommendations. Methods Once the determination was made that an evaluation was necessary to decide the usefulness of the projects for future planning, key stakeholders worked together to design and execute the evaluation. For Cote d’Ivoire, the evaluation team consisted of representatives from the MoH’s National Institute of Public Health, Directorate of Informatics and Health Information, and Directorate for the Coordination of the Expanded Immunization Program, along with delegates from CDC and implementing partners. In Tanzania, evaluation team members came from the MoH, the Ministry of Agriculture, Forestries and Livestock, WHO, CDC and implementing partners. Team members participated in either planning, conducting or analyzing the evaluation, while some contributed to a combination or to all aspects. MoH members led the effort with CDC and other partners providing technical assistance, while implementing partners contributed only to planning and logistics to reduce the potential for bias. For the initial step, representatives came together to fully document the system to be evaluated. This system description details the purpose, relevant stakeholders and current operation of the pilot system. As the evaluation question should remain within the scope of the system’s purpose, it was necessary to definitively understand and confirm the goal and objectives set out for the system. Next, the sites, participants and roles, and data flows were described, noting that verification of the actual processes would occur during the site visit portion of the evaluation. Total cost of ownership was calculated by considering solution costs, implementation costs and ongoing support, and then broken down by district. The CBS pilots implemented early warning notification systems in two districts in Cote d’Ivoire and in five districts in Tanzania using a combination of paper-based and electronic reporting formats. Evaluation teams visited pilot sites and routine surveillance sites for comparison and conducted in-person interviews using questionnaires specific to the individual’s role. Data were either collected in the field on paper forms or electronically on tablets for subsequent upload to a centralized database for later analysis. Data from project and routine reporting databases were comparatively analyzed to calculate timeliness, validity, usefulness, acceptability and value of the early warning system pilots. Results Although final interpretations of the evaluation results are pending, the evaluations were successfully led by the HCG and jointly conducted with other stakeholder engagement. Leadership by the owners of the systems has already resulted in the recognition that certain aspects of the pilot surveillance systems demonstrate a successful and affordable approach, while others will need to consider more cost-effective strategies. Though further analysis will likely continue to show the utility of CBS strategies, the ownership approach is resulting in an outcome of broad stakeholder input with approval from the host country government. Conclusions Community based surveillance can help to detect events of public health importance and effect earlier introduction into the health system for more timely situational awareness and response. However, it is difficult to determine the costs associated with different strategies of implementation and operation in order to ascertain the value for public health action. Additionally, pilot implementations of these systems are often funded at a level that cannot be replicated nationally and not for a prolonged period of time. While it is believed that CBS can be a cost effective early notification system, continual monitoring and routine evaluation is required. By routinely monitoring cost and quality, sustainability of the system can be continually assessed and system adaptations made accordingly. Key to remember is that evaluation must occur from an owner’s perspective and must engage the people who are going to govern, operate and provide the ongoing resources for system operation. In this way, effectiveness and efficiency can be continually monitored within the parameter of cost so that viability of the system can be ascertained.

Objective To describe the results of the evaluation of the PAILD active surveillance system and lessons learned for similar surveillance efforts in a resource-limited setting. Introduction A liver disease of unknown etiology, called unknown liver disease (ULD) by the community, was first identified in 2002 in Tigray; a rugged, semi-arid, mountainous region that is considered one of the most drought-prone and food insecure regions of Ethiopia. ULD is a chronic condition characterized by epigastric pain, abdominal distention, ascites, emaciation, and hepato/splenomegaly. In 2005, the Ethiopian Health and Nutritional Research Institute was assigned by the Ethiopia Ministry of Health to assist the Tigray Regional Health Bureau and oversee the disease investigation. In 2008, Centers for Disease Control and Prevention (CDC) assisted the Ethiopian team and jointly developed the surveillance tools. The surveillance system was implemented in 2009 with the objectives to determine the magnitude and distribution of the disease; identify disease trends; detect cases to provide them with clinical care; and inform health officials and funding bodies for resource allocation. After several investigations, a local plant containing a particular type of pyrrolizidine alkaloid (PA) toxin that contaminated local foodstuffs was identified as the etiologic agent, and ULD was renamed PAILD in 2011. Methods From 20 September to 1 October 2011, we conducted site visits, held semi-structured interviews with 20 staff members, reviewed reporting materials, and summarized the information flow including data collection, reporting, analysis, and dissemination. Results This surveillance system was implemented in 13 rural, resource-limited districts in the NW, Central and Western Zones. The system identified a total of 1033 cases, including 314 deaths, as of September 2011; guided medication distribution to the health facilities; served as a registry for patient follow up; and provided decision-makers with information needed to allocate resources. A large-scale training was conducted in 2010; however, high staff turnover and a lack of backup surveillance staff at each site suggested that additional training may be needed. Due to the absence of a diagnostic test, the case definition was very simple to enable frontline staff in the communities and at the health posts/centers to identify disease cases. These individuals travelled long distances by foot to deliver paper surveillance forms to the district health offices. A surveillance team placed in the NW Zonal office collected missing reports from the health facilities given limited transportation; however they have left since this evaluation. Information from the surveillance system was shared with partner agencies at the national level every 3 to 6 months; however, this information was not shared with frontline staff. Conclusions The PAILD Active Surveillance System met its objectives as originally defined. Evaluation of this unique surveillance system for a chronic disease with unknown cause in a resource-limited setting provides several lessons that can inform similar surveillance efforts. Ongoing logistical challenges (e.g., shortage of paper forms, lack of transportation, and long distances between locations) complicated data collection and reporting. While electronic reporting may have helped overcome some of these difficulties, it was not feasible in this setting. Frontline staff identified cases in the community so that they could receive treatment; these key staff can be further incentivized by receiving regular training and surveillance reports. Ongoing support will be critical to overcome these unique challenges to ensure continual disease monitoring as interventions to disrupt PA exposure are implemented in the community.

ObjectiveTo assess evidence for public health impact of syndromic surveillance.IntroductionSystematic syndromic surveillance is undergoing a transition. Building on traditional roots in bioterrorism and situational awareness, proponents are demonstrating the timeliness and informative power of syndromic surveillance data to supplement other surveillance data.MethodsWe used PubMed and Google Scholar to identify articles published since 2007 using key words of interest (e.g., syndromic surveillance in combinations with emergency, evaluation, quality assurance, alerting). The following guiding questions were used to abstract impact measures of syndromic surveillance: 1) what was the public health impact; what decisions or actions occurred because of use of syndromic surveillance data?, 2) were there specific interventions or performance measures for this impact?, and 3) how, and by whom, was this information used?ResultsThirty-five papers were included. Almost all articles (n=33) remarked on the ability of syndromic surveillance to improve public health because of timeliness and/or accuracy of data. Thirty-four articles mentioned that syndromic surveillance data was used or could be useful. However, evidence of health impact directly attributable to syndromic surveillance efforts were lacking. Two articles described how syndromic data were used for decision-making. One article measured the effect of data utilization.ConclusionsWithin the syndromic surveillance literature instances of a conceptual shift from detection to practical response are plentiful. As the field of syndromic surveillance continues to evolve and is used by public health institutions, further evaluation of data utility and impact is needed.ReferencesAyala, A., Berisha, V., Goodin, K., Pogreba-Brown, K., Levy, C., McKinney, B., Koski, L., & Imholte, S. (2016). Public health surveillance strategies for mass gatherings: Super Bowl XLIX and related events, Maricopa County, Arizona, 2015. Health Security, 14(3), 173-84. doi: 10.1089/hs.2016.0029.Bermis, K., Frias, M., Patel, M.T., & Christiansen, D. (2017). Using an Emergency Department Syndromic Surveillance System to Evaluate Reporting of Potential Rabies Exposures, Illinois, 2013-2015. Public Health Reports 132(Supplement 1) 59S-64S."Borroto, R., Williamson, B., Pitcher, P., Ballester, L., Smith, W., Soetebier, K., & Drenzek, C. (2016). Using Syndromic Surveillance Alert Protocols for Epidemiologic Response in Georgia. Online Journal of Public Health Informatics 9(1):e123. doi:10.5210/ojphi.v9i1.7707."Daly, E.R., Dufault, K., Swenson, D.J., Lakevicius, P., Metcalf, E., & Chan, B.P. (2017). Use of emergency department data to monitor and respond to an increase in opioid overdoses in New Hampshire 2011-2015. Public Health Reports 132(Supplement 1) 73S-79S. doi: 10.1177/0033354917707934Deyneka, L., Hakenewerth, A., Faigen, Z., Ising, A., & Barnett, C. (2017). Using syndromic surveillance data to monitor endocarditis and sepsis among drug users. Online Journal of Public Health Informatics, (9)1. doi: http://dx.doi.org/10.5210/ojphi.v9i1.7708DeYoung, K., Chen, Y., Beum, R., Askenazi, M., Zimmerman, C., & Davidson, A. J. (2017). Validation of a syndromic case definition for detecting emergency department visits potentially related to marijuana. Public Health Reports, epublication.doi: 10.1177/0033354917708987"Dinh, M.M., Kastelein, C., Bein, K.J., Bautovich, T., & Ivers, R. (2015). Use of a syndromic surveillance system to describe the trend in cycling-related presentations to emergency departments in Sydney. Emergency Medicine Australasia, 27(4), 343-7. doi: 10.1111/1742-6723.12422Gevitz, K., Madera, R., Newbern, C., Lojo, J., & Johnson, C. Risk of Fall-Related Injury due to Adverse Weather Events, Philadelphia, Pennsylvania, 2006-2011. Public Health Reports (132) 53S-58S. doi: 10.1177/0033354917706968"Gonzales-Colon, F.J., Lake, I., Barker, G., Smith, G.E., Elliot, A.J., & Morbey, R. (2016). Using Bayesian Networks to assist decision-making in syndromic surveillance. Online Journal of Public Health Informatics, 8(1), e15. doi:10.5210/ojphi.v8i1.6415"Harmon, KJ., Proescholdbell, S., Marshall, S., & Waller, A. (2014). Utilization of emergency department data for drug overdose surveillance in North Carolina. Online Journal of Public Health Informatics 6(1), e174. doi: 10.5210/ojphi.v6i1.5200Harris, J.K., Mansour, R., Choucair, B., Olson, J., Nissen, C., & Bhatt, J. (2014). Health department use of social media to identify foodborne illness—Chicago, Illinois, 2013-2014. MMWR Morbidity and Mortality Weekly Report 63(32), 681-685. Retrieved from: https://www.cdc.gov/mmwr/preview/mmwrhtml/mm6332a1.htm"Harrison, C., Jorder, M., Stern, H., Stavinksy, F., Reddy, V., Hanson, H., Waechter, H., Lowe, L., Gravano, L., & Balter, S. (2014). Using online reviews by restaurant patrons to identify unreported cases of foodborne illness — New York City, 2012–2013. MMWR Morbidity and Mortality Weekly Report 63(20), 441-445. Retrieved from:https://www.cdc.gov/MMWr/preview/mmwrhtml/mm6320a1.htm"Hawkins, J.B., Tuli, G., Kluberg, S., Harris, J., Brownstein, J.S., & Nsoesie, E. (2016). A digital platform for local foodborne illness and outbreak surveillance. Online Journal of Public Health Informatics 8(1), e60. http://dx.doi.org/10.5210/ojphi.v8i1.6474Hines, J.Z., Bancroft, J., Powell, M., & Hedberg, K. (2017). Case finding using syndromic surveillance data during an outbreak of Shiga Toxin–Producing Escherichia coli O26 infections, Oregon, 2015. Public Health Reports, epublication. https://doi.org/10.1177/0033354917708994Hudson, L. T., Klekamp, B.G., & Matthews, S.D. (2017). Local Public Health Surveillance of Heroin-Related Morbidity and Mortality, Orange County, Florida, 2010-2014. Public Health Reports (132), 80S-87SHughes, H.E., Morbey, R., Hughes, T.C., Locker, T.E., Pebody, R., Green, H.K., Ellis, J., Smith, G.E., & Elliot, A.J. (2016). Emergency department syndromic surveillance providing early warning of seasonal respiratory activity in England. Epidemiology and Infection, 144(5), 1052-64. doi: 10.1017/S0950268815002125Hughes, H.E., Morbey, R., Hughes, T.C., Locker, T.E., Shannon, T., Carmichael, C., Murray, V., Ibbotson, S., Catchpole, M., McCloskey, B., Smith, G., & Elliot, A.J. (2014). Using an emergency department syndromic surveillance system to investigate the impact of extreme cold weather events. Public Health, 128(7), 628-635. doi: 10.1016/j.puhe.2014.05.007Ising, A., Proescholdbell, S., Harmon, K.J., Sachdeva, N., Marshall, S.W., & Waller, A.E. (2016). Use of syndromic surveillance data to monitor poisonings and drug overdoses in state and local public health agencies. Injury Prevention 22:i43-i49.http://dx.doi.org/10.1136/injuryprev-2015-041821"Johnson, J. I., & Brown, K. (2015). Validation of emergency department and outpatient data using ILI syndrome classifiers. Online Journal of Public Health Informatics, 7(1), e83. http://doi.org/10.5210/ojphi.v7i1.5749Lall, R., Abdelnabi , J., Ngai, S., Parton, H.B., Saunders, K., Sell, J., Wahnich, A., Weiss, D., Marthes, R.W. (2017). Advancing the Use of Emergency Department Syndromic Surveillance Data, New York City, 2012-2016. Public Health Reports (132), 23S-30SLiljeqvist, H. T., Muscatello, D., Sara, G., Dinh, M., & Lawrence, G. L. (2014). Accuracy of automatic syndromic classification of coded emergency department diagnoses in identifying mental health-related presentations for public health surveillance. BMC Medical Informatics and Decision Making, 14(84). http://doi.org/10.1186/1472-6947-14-84Lober, W. B., Reeder, B., Painter, I., Revere, D., Goldov, K., Bugni, P. F., & Olson, D. R. (2014). Technical description of the Distribute Project: a community-basedsyndromic surveillance system implementation. Online Journal of Public Health Informatics, 5(3), 224. http://doi.org/10.5210/ojphi.v5i3.4938Mathes, R. W., Ito, K., & Matte, T. (2011). Assessing syndromic surveillance of cardiovascular outcomes from emergency department chief complaint data in New York City. Public Library of Science ONE, 6(2), e14677. http://doi.org/10.1371/journal.pone.0014677O’Connell, E. K., Zhang, G., Leguen, F., Llau, A., & Rico, E. (2010). Innovative uses for syndromic surveillance. Emerging Infectious Diseases, 16(4), 669–671. http://doi.org/10.3201/eid1604.090688Rumoro, D.P., Hallock, M.M., Silva, J., Shah, S.C., Gibbs, G., Trenholme G.M., & Waddell, M.J. (2013). Why does Influenza-Like Illness surveillance miss true influenza cases in the emergency department?: Implications for health care providers. Annals of Emergency Medicine, 62(4), S75. https://doi.org/10.1016/j.annemergmed.2013.07.024Samoff E, Waller A, Fleischauer A, et al. Integration of Syndromic Surveillance Data into Public Health Practice at State and Local Levels in North Carolina. Public Health Reports. 2012;127(3):310-317.Savard, N., Bédard, L., Allard, R., & Buckeridge, D.L. (2015). Using age, triage score, and disposition data from emergency department electronic records to improve Influenza-Like Illness surveillance. Journal of the American Medical Informatics Association, 22(3): 688-696. doi: 10.1093/jamia/ocu002Seil, K., Marcum, J., Lall, R., & Stayton, C. (2015). Utility of a near real-time emergency department syndromic surveillance system to track injuries in New York City. Injury Epidemiology, 2(1), 11. http://doi.org/10.1186/s40621-015-0044-5Smith, S., Elliot, A. J., Hajat, S., Bone, A., Smith, G. E., & Kovats, S. (2016). Estimating the burden of heat illness in England during the 2013 summer heatwave using syndromic surveillance. Journal of Epidemiology and Community Health, 70(5), 459–465. http://doi.org/10.1136/jech-2015-206079Stephens, E. (2017). Development of syndrome definitions for acute unintentional drug and heroin overdose. Online Journal of Public Health Informatics, (9)1. http://dx.doi.org/10.5210/ojphi.v9i1.7593.Stigi, K., Baer, A., Duchin, J., & Lofy, K. (2014). Evaluation of electronic ambulatory care data for Influenza-Like Illness surveillance, Washington state. Journal of Public Health Management & Practice, 20(6)580-582.doi: 10.1097/PHH.0b013e3182aaa29bVilain, P., Larrieu, S., Mougin-Damour, K., Marianne Dit Cassou, P.J., Weber, M., Combes, X., & Filleul, L. (2017). Emergency department syndromic surveillance to investigate the health impact and factors associated with alcohol intoxication in Reunion Island. Emergency medicine journal 34(6), 386-390. doi: 10.1136/emermed-2015-204987Walsh, A. (2017). Going beyond chief complaints to identify opioid-related emergency department visits. Online Journal of Public Health Informatics, (9)1. http://dx.doi.org/10.5210/ojphi.v9i1.7617.White, J.R., Berisha, V., Lane, K., Menager, H., Gettel, A., & Braun, C.R. (2017). Evaluation of a Novel Syndromic Surveillance Query for Heat-Related Illness Using Hospital Data From Maricopa County, Arizona, 2015. Public Health Reports (132), 31S-39SYih WK, Deshpande S, Fuller C, et al. Evaluating Real-Time Syndromic Surveillance Signals from Ambulatory Care Data in Four States. Public Health Reports. 2010;125(1):111-120.

This paper examines the state of the art in mobile clinical and health-related apps. A 2012 estimate puts the number of health-related apps at no fewer than 40,000, as healthcare professionals and consumers continue to express concerns about the quality of many apps, calling for some form of app regulatory control or certification to be put in place. We describe the range of apps on offer as of 2013, and then present a brief survey of evaluation studies of medical and health-related apps that have been conducted to date, covering a range of clinical disciplines and topics. Our survey includes studies that highlighted risks, negative issues and worrying deficiencies in existing apps. We discuss the concept of ‘apps as a medical device’ and the relevant regulatory controls that apply in USA and Europe, offering examples of apps that have been formally approved using these mechanisms. We describe the online Health Apps Library run by the National Health Service in England and the calls for a vetted medical and health app store. We discuss the ingredients for successful apps beyond the rather narrow definition of ‘apps as a medical device’. These ingredients cover app content quality, usability, the need to match apps to consumers’ general and health literacy levels, device connectivity standards (for apps that connect to glucometers, blood pressure monitors, etc.), as well as app security and user privacy. ‘Happtique Health App Certification Program’ (HACP), a voluntary app certification scheme, successfully captures most of these desiderata, but is solely focused on apps targeting the US market. HACP, while very welcome, is in ways reminiscent of the early days of the Web, when many “similar” quality benchmarking tools and codes of conduct for information publishers were proposed to appraise and rate online medical and health information. It is probably impossible to rate and police every app on offer today, much like in those early days of the Web, when people quickly realised the same regarding informational Web pages. The best first line of defence was, is, and will always be to educate consumers regarding the potentially harmful content of (some) apps.