Search

BackgroundDespite the increase in use and high expectations of digital health solutions, scientific evidence about the effectiveness of electronic health (eHealth) and other aspects such as usability and accuracy is lagging behind. eHealth solutions are complex interventions, which require a wide array of evaluation approaches that are capable of answering the many different questions that arise during the consecutive study phases of eHealth development and implementation. However, evaluators seem to struggle in choosing suitable evaluation approaches in relation to a specific study phase. ObjectiveThe objective of this project was to provide a structured overview of the existing eHealth evaluation approaches, with the aim of assisting eHealth evaluators in selecting a suitable approach for evaluating their eHealth solution at a specific evaluation study phase. MethodsThree consecutive steps were followed. Step 1 was a systematic scoping review, summarizing existing eHealth evaluation approaches. Step 2 was a concept mapping study asking eHealth researchers about approaches for evaluating eHealth. In step 3, the results of step 1 and 2 were used to develop an “eHealth evaluation cycle” and subsequently compose the online “eHealth methodology guide.” ResultsThe scoping review yielded 57 articles describing 50 unique evaluation approaches. The concept mapping study questioned 43 eHealth researchers, resulting in 48 unique approaches. After removing duplicates, 75 unique evaluation approaches remained. Thereafter, an “eHealth evaluation cycle” was developed, consisting of six evaluation study phases: conceptual and planning, design, development and usability, pilot (feasibility), effectiveness (impact), uptake (implementation), and all phases. Finally, the “eHealth methodology guide” was composed by assigning the 75 evaluation approaches to the specific study phases of the “eHealth evaluation cycle.” ConclusionsSeventy-five unique evaluation approaches were found in the literature and suggested by eHealth researchers, which served as content for the online “eHealth methodology guide.” By assisting evaluators in selecting a suitable evaluation approach in relation to a specific study phase of the “eHealth evaluation cycle,” the guide aims to enhance the quality, safety, and successful long-term implementation of novel eHealth solutions.

Abstract Background Previously we described parents' and professionals' experiences with a web-based communication system in a 6-month pilot in three Dutch cerebral palsy care settings. We found that half of the participating professionals had not used the system, and of those who had used the system one third had used it only once. The present study aimed to evaluate whether professionals' system use was associated with their a priori expectancies and background. Methods Professionals who had not used the system (n = 54) were compared with professionals who had used the system more than once (n = 46) on the basis of their questionnaire responses before the pilot, their affiliation and the number of patients which they represented in the study. The questionnaire items comprised professionals' expectancies regarding the system's performance and ease of use, as well as the expected time availability and integration into daily care practice. Results Overall, users had higher a priori expectancies than non-users. System use was associated with expected ease of use (p = .046) and time availability (p = .005): 50% of the users (vs. 31% of the non-users) expected that the system would be easy to use and 93% of the users (vs. 72% of the non-users) expected that they would be able to reserve a time slot each week for responding to submitted questions. With respect to professionals' affiliation, system use was associated with professionals' institution (p = .003) and discipline (p = .001), with more (para-) medical professionals among users (93% vs. 63% among non-users), and more education professionals among non-users (37% vs. 7% among users). In addition, users represented more patients (mean 2, range 1-8) than non-users (mean 1.1, range 1-2) (p = .000). Conclusions Professionals' system use was associated with expected ease of use and time availability, professionals' affiliation and the number of represented patients, while no association was found with expected performance of the system. To achieve higher adoption rates in the future, it is important to further develop the technology by optimizing the system's ease of use and interoperability and including advanced consultation options. In addition, better identified end users should be more extensively informed about the system's possibilities through tailored education.