Search

BackgroundDespite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase. ObjectiveThe objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase. MethodsThree consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an “eHealth evaluation cycle” and subsequently compose the online “eHealth methodology guide.” ResultsThe scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an “eHealth evaluation cycle” was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the “eHealth methodology guide” was composed by assigning the 75 evaluation approaches to the specific study phases of the “eHealth evaluation cycle.” ConclusionsSeventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online “eHealth methodology guide.” By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the “eHealth evaluation cycle,” the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Abstract Background Previously we described parents' and professionals' experiences with a web-based communication system in a 6-month pilot in three Dutch cerebral palsy care settings. We found that half of the participating professionals had not used the system, and of those who had used the system one third had used it only once. The present study aimed to evaluate whether professionals' system use was associated with their a priori expectancies and background. Methods Professionals who had not used the system (n = 54) were compared with professionals who had used the system more than once (n = 46) on the basis of their questionnaire responses before the pilot, their affiliation and the number of patients which they represented in the study. The questionnaire items comprised professionals' expectancies regarding the system's performance and ease of use, as well as the expected time availability and integration into daily care practice. Results Overall, users had higher a priori expectancies than non-users. System use was associated with expected ease of use (p = .046) and time availability (p = .005): 50% of the users (vs. 31% of the non-users) expected that the system would be easy to use and 93% of the users (vs. 72% of the non-users) expected that they would be able to reserve a time slot each week for responding to submitted questions. With respect to professionals' affiliation, system use was associated with professionals' institution (p = .003) and discipline (p = .001), with more (para-) medical professionals among users (93% vs. 63% among non-users), and more education professionals among non-users (37% vs. 7% among users). In addition, users represented more patients (mean 2, range 1-8) than non-users (mean 1.1, range 1-2) (p = .000). Conclusions Professionals' system use was associated with expected ease of use and time availability, professionals' affiliation and the number of represented patients, while no association was found with expected performance of the system. To achieve higher adoption rates in the future, it is important to further develop the technology by optimizing the system's ease of use and interoperability and including advanced consultation options. In addition, better identified end users should be more extensively informed about the system's possibilities through tailored education.

Background: For eHealth technologies in general and audit and feedback (AF) systems specifically, integrating interdisciplinary theoretical underpinnings is essential, as it increases the likelihood of achieving desired outcomes by ensuring a fit among eHealth technology, stakeholders, and their context. In addition, reporting on the development and implementation process of AF systems, including substantiations of choices, enables the identification of best practices and accumulation of knowledge across studies but is often not elaborated on in publications., Objective: This scoping review aims to provide insights into the development and implementation strategies for AF systems for a real-world problem that threatens modern health care-antimicrobial resistance-and provide an interdisciplinary conceptual framework that can serve as a checklist and guidance for making informed choices in the development and implementation of future AF systems., Methods: A scoping review was conducted by querying PubMed, Scopus, Web of Science, IEEE Xplore Digital Library, and Embase (≥2010) for studies describing either the development or implementation process, or both, of an AF system for antimicrobial resistance or infections in hospitals. Studies reporting only on effectiveness or impact were excluded. A total of 3 independent reviewers performed the study selection, and 2 reviewers constructed the conceptual framework through the axial and selective coding of often-used theories, models, and frameworks (TMFs) from the literature on AF and eHealth development and implementation. Subsequently, the conceptual framework was used for the systematic extraction and interpretation of the studies' descriptions of AF systems and their development and implementation., Results: The search resulted in 2125 studies that were screened for eligibility, of which 12 (0.56%); 2012-2020) were included. These studies described the development and implementation processes heterogeneously in terms of study aims, study targets, target groups, methods, and theoretical underpinnings. Few studies have explicitly explained how choices for the development and implementation of AF systems were substantiated by the TMFs. The conceptual framework provided insights into what is reported on the development and implementation process and revealed underreported AF system constructs (eg, AF system design; engagement with the AF system; and comparison, goal setting, and action planning) and development and implementation (eg, champions) constructs., Conclusions: This scoping review showed the current heterogeneous reporting of AF systems and their development and implementation processes and exemplified how interdisciplinary TMFs can (and should) be balanced in a conceptual framework to capture relevant AF systems and development and implementation constructs. Thereby, it provides a concrete checklist and overall guidance that supports the professionalization and harmonization of AF system development and implementation. For the development and implementation of future AF systems and other eHealth technologies, researchers and health care workers should be supported in selecting and integrating TMFs into their development and implementation process and encouraged to explicitly report on theoretical underpinnings and the substantiation of choices., (©Julia Keizer, Britt E Bente, Nashwan Al Naiemi, Lisette JEWC Van Gemert-Pijnen, Nienke Beerlage-De Jong. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 11.03.2022.)