Search

BackgroundIncreasing adoption of sensor-based digital health technologies (sDHTs) in recent years has cast light on the many challenges in implementing these tools into clinical trials and patient care at scale across diverse patient populations; however, the methodological approaches taken toward sDHT usability evaluation have varied markedly. ObjectiveThis review aims to explore the current landscape of studies reporting data related to sDHT human factors, human-centered design, and usability, to inform our concurrent work on developing an evaluation framework for sDHT usability. MethodsWe conducted a scoping review of studies published between 2013 and 2023 and indexed in PubMed, in which data related to sDHT human factors, human-centered design, and usability were reported. Following a systematic screening process, we extracted the study design, participant sample, the sDHT or sDHTs used, the methods of data capture, and the types of usability-related data captured. ResultsOur literature search returned 442 papers, of which 85 papers were found to be eligible and 83 papers were available for data extraction and not under embargo. In total, 164 sDHTs were evaluated; 141 (86%) sDHTs were wearable tools while the remaining 23 (14%) sDHTs were ambient tools. The majority of studies (55/83, 66%) reported summative evaluations of final-design sDHTs. Almost all studies (82/83, 99%) captured data from targeted end users, but only 18 (22%) out of 83 studies captured data from additional users such as care partners or clinicians. User satisfaction and ease of use were evaluated for 83% (136/164) and 91% (150/164) of sDHTs, respectively; however, learnability, efficiency, and memorability were reported for only 11 (7%), 4 (2%), and 2 (1%) out of 164 sDHTs, respectively. A total of 14 (9%) out of 164 sDHTs were evaluated according to the extent to which users were able to understand the clinical data or other information presented to them (understandability) or the actions or tasks they should complete in response (actionability). Notable gaps in reporting included the absence of a sample size rationale (reported for 21/83, 25% of all studies and 17/55, 31% of summative studies) and incomplete sociodemographic descriptive data (complete age, sex/gender, and race/ethnicity reported for 14/83, 17% of studies). ConclusionsBased on our findings, we suggest four actionable recommendations for future studies that will help to advance the implementation of sDHTs: (1) consider an in-depth assessment of technology usability beyond user satisfaction and ease of use, (2) expand recruitment to include important user groups such as clinicians and care partners, (3) report the rationale for key study design considerations including the sample size, and (4) provide rich descriptive statistics regarding the study sample to allow a complete understanding of generalizability to other patient populations and contexts of use.

Background: Digital measures offer an unparalleled opportunity to create a more holistic picture of how people who are patients behave in their real-world environments, thereby establishing a better connection between patients, caregivers, and the clinical evidence used to drive drug development and disease management. Reaching this vision will require achieving a new level of co-creation between the stakeholders who design, develop, use, and make decisions using evidence from digital measures. Summary: In September 2022, the second in a series of meetings hosted by the Swiss Federal Institute of Technology in Zürich, the Foundation for the National Institutes of Health Biomarkers Consortium, and sponsored by Wellcome Trust, entitled “Reverse Engineering of Digital Measures,” was held in Zurich, Switzerland, with a broad range of stakeholders sharing their experience across four case studies to examine how patient centricity is essential in shaping development and validation of digital evidence generation tools. Key Messages: In this paper, we discuss progress and the remaining barriers to widespread use of digital measures for evidence generation in clinical development and care delivery. We also present key discussion points and takeaways in order to continue discourse and provide a basis for dissemination and outreach to the wider community and other stakeholders. The work presented here shows us a blueprint for how and why the patient voice can be thoughtfully integrated into digital measure development and that continued multistakeholder engagement is critical for further progress.

Background: Digital health technologies are attracting attention as novel tools for data collection in clinical research. They present alternative methods compared to in-clinic data collection, which often yields snapshots of the participants’ physiology, behavior, and function that may be prone to biases and artifacts, e.g., white coat hypertension, and not representative of the data in free-living conditions. Modern digital health technologies equipped with multi-modal sensors combine different data streams to derive comprehensive endpoints that are important to study participants and are clinically meaningful. Used for data collection in clinical trials, they can be deployed as provisioned products where technology is given at study start or in a bring your own “device” (BYOD) manner where participants use their technologies to generate study data. Summary: The BYOD option has the potential to be more user-friendly, allowing participants to use technologies that they are familiar with, ensuring better participant compliance, and potentially reducing the bias that comes with introducing new technologies. However, this approach presents different technical, operational, regulatory, and ethical challenges to study teams. For example, BYOD data can be more heterogeneous, and recruiting historically underrepresented populations with limited access to technology and the internet can be challenging. Despite the rapid increase in digital health technologies for clinical and healthcare research, BYOD use in clinical trials is limited, and regulatory guidance is still evolving. Key Messages: We offer considerations for academic researchers, drug developers, and patient advocacy organizations on the design and deployment of BYOD models in clinical research. These considerations address: (1) early identification and engagement with internal and external stakeholders; (2) study design including informed consent and recruitment strategies; (3) outcome, endpoint, and technology selection; (4) data management including compliance and data monitoring; (5) statistical considerations to meet regulatory requirements. We believe that this article acts as a primer, providing insights into study design and operational requirements to ensure the successful implementation of BYOD clinical studies.

The EVIDENCE (EValuatIng connecteD sENsor teChnologiEs) checklist was developed by a multidisciplinary group of content experts convened by the Digital Medicine Society, representing the clinical sciences, data management, technology development, and biostatistics. The aim of EVIDENCE is to promote high quality reporting in studies where the primary objective is an evaluation of a digital measurement product or its constituent parts. Here we use the terms digital measurement product and connected sensor technology interchangeably to refer to tools that process data captured by mobile sensors using algorithms to generate measures of behavioral and/or physiological function. EVIDENCE is applicable to 5 types of evaluations: (1) proof of concept; (2) verification, (3) analytical validation, and (4) clinical validation as defined by the V3 framework; and (5) utility and usability assessments. Using EVIDENCE, those preparing, reading, or reviewing studies evaluating digital measurement products will be better equipped to distinguish necessary reporting requirements to drive high-quality research. With broad adoption, the EVIDENCE checklist will serve as a much-needed guide to raise the bar for quality reporting in published literature evaluating digital measurements products.

Abstract Digital medicine is an interdisciplinary field, drawing together stakeholders with expertize in engineering, manufacturing, clinical science, data science, biostatistics, regulatory science, ethics, patient advocacy, and healthcare policy, to name a few. Although this diversity is undoubtedly valuable, it can lead to confusion regarding terminology and best practices. There are many instances, as we detail in this paper, where a single term is used by different groups to mean different things, as well as cases where multiple terms are used to describe essentially the same concept. Our intent is to clarify core terminology and best practices for the evaluation of Biometric Monitoring Technologies (BioMeTs), without unnecessarily introducing new terms. We focus on the evaluation of BioMeTs as fit-for-purpose for use in clinical trials. However, our intent is for this framework to be instructional to all users of digital measurement tools, regardless of setting or intended use. We propose and describe a three-component framework intended to provide a foundational evaluation framework for BioMeTs. This framework includes (1) verification, (2) analytical validation, and (3) clinical validation. We aim for this common vocabulary to enable more effective communication and collaboration, generate a common and meaningful evidence base for BioMeTs, and improve the accessibility of the digital medicine field.

As part of a biomarker discovery effort in peripheral blood, we acquired an immunological profile of cell-surface markers from healthy control and untreated subjects with relapsing–remitting MS (RRMS). Fresh blood from each subject was screened ex vivo using a panel of 50 fluorescently labelled monoclonal antibodies distributed amongst 56 pools of four antibodies each. From these 56 pools, we derived an immunological profile consisting of 1018 ‘features’ for each subject in our analysis using a systematic gating strategy. These profiles were interrogated in an analysis with a screening phase (23 patients) and an extension phase (15 patients) to identify cell populations in peripheral blood whose frequency is altered in untreated RRMS subjects. A population of CD8lowCD4− cells was identified as being reduced in frequency in untreated RRMS subjects (P = 0.0002), and this observation was confirmed in an independent sample of subjects from the Comprehensive Longitudinal Investigation of MS at the Brigham & Womens Hospital (P = 0.002). This reduction in the frequency of CD8lowCD4− cells is also observed in 38 untreated subjects with a clinically isolated demyelination syndrome (CIS) (P = 0.0006). We also show that these differences may be due to a reduction in the CD8lowCD56࠽−CD4− subset of CD8low cells, which have a natural killer cell profile. Similarities between untreated CIS and RRMS subjects extend to broader immunological profiles: consensus clustering of our data suggests that there are three distinct populations of untreated RRMS subjects and that these distinct phenotypic categories are already present in our sample of untreated CIS subjects. Thus, our large-scale immunophenotyping approach has yielded robust evidence for a reduction of CD8lowCD4− cells in both CIS and RRMS in the absence of treatment as well as suggestive evidence for the existence of immunologically distinct subsets of subjects with a demyelinating disease. [ABSTRACT FROM AUTHOR]