Search

Over 50 million people undergo surgeries each year in the United States, with over 70% of them filling opioid prescriptions within one week of the surgery. Due to the highly addictive nature of these opiates, a post-surgical window is a crucial time for pain management to ensure accurate prescription of opioids. Drug prescription nowadays relies primarily on self-reported pain levels to determine the frequency and dosage of pain drug. Patient pain self-reports are, however, influenced by subjective pain tolerance, memories of past painful episodes, current context, and the patient's integrity in reporting their pain level. Therefore, objective measures of pain are needed to better inform pain management. This paper explores a wearable system, named Painometry, which objectively quantifies users' pain perception based-on multiple physiological signals and facial expressions of pain. We propose a sensing technique, called sweep impedance profiling (SIP), to capture the movement of the facial muscle corrugator supercilii, one of the important physiological expressions of pain. We deploy SIP together with other biosignals, including electroencephalography (EEG), photoplethysmogram (PPG), and galvanic skin response (GSR) for pain quantification. From the anatomical and physiological correlations of pain with these signals, we designed Painometry, a multimodality sensing system, which can accurately quantify different levels of pain safely. We prototyped Painometry by building a custom hardware, firmware, and associated software. Our evaluations use the prototype on 23 subjects, which corresponds to 8832 data points from 276 minutes of an IRB-approved experimental pain-inducing protocol. Using leave-one-out cross-validation to estimate performance on unseen data shows 89.5% and 76.7% accuracy of quantification under 3 and 4 pain states, respectively.

Background Studies that use ecological momentary assessments (EMAs) or wearable sensors to track numerous attributes, such as physical activity, sleep, and heart rate, can benefit from reductions in missing data. Maximizing compliance is one method of reducing missing data to increase the return on the heavy investment of time and money into large-scale studies. Objective This paper aims to identify the extent to which compliance can be prospectively predicted from individual attributes and initial compliance. Methods We instrumented 757 information workers with fitness trackers for 1 year and conducted EMAs in the first 56 days of study participation as part of an observational study. Their compliance with the EMA and fitness tracker wearing protocols was analyzed. Overall, 31 individual characteristics (eg, demographics and personalities) and behavioral variables (eg, early compliance and study portal use) were considered, and 14 variables were selected to create beta regression models for predicting compliance with EMAs 56 days out and wearable compliance 1 year out. We surveyed study participation and correlated the results with compliance. Results Our modeling indicates that 16% and 25% of the variance in EMA compliance and wearable compliance, respectively, could be explained through a survey of demographics and personality in a held-out sample. The likelihood of higher EMA and wearable compliance was associated with being older (EMA: odds ratio [OR] 1.02, 95% CI 1.00-1.03; wearable: OR 1.02, 95% CI 1.01-1.04), speaking English as a first language (EMA: OR 1.38, 95% CI 1.05-1.80; wearable: OR 1.39, 95% CI 1.05-1.85), having had a wearable before joining the study (EMA: OR 1.25, 95% CI 1.04-1.51; wearable: OR 1.50, 95% CI 1.23-1.83), and exhibiting conscientiousness (EMA: OR 1.25, 95% CI 1.04-1.51; wearable: OR 1.34, 95% CI 1.14-1.58). Compliance was negatively associated with exhibiting extraversion (EMA: OR 0.74, 95% CI 0.64-0.85; wearable: OR 0.67, 95% CI 0.57-0.78) and having a supervisory role (EMA: OR 0.65, 95% CI 0.54-0.79; wearable: OR 0.66, 95% CI 0.54-0.81). Furthermore, higher wearable compliance was negatively associated with agreeableness (OR 0.68, 95% CI 0.56-0.83) and neuroticism (OR 0.85, 95% CI 0.73-0.98). Compliance in the second week of the study could help explain more variance; 62% and 66% of the variance in EMA compliance and wearable compliance, respectively, was explained. Finally, compliance correlated with participants’ self-reflection on the ease of participation, usefulness of our compliance portal, timely resolution of issues, and compensation adequacy, suggesting that these are avenues for improving compliance. Conclusions We recommend conducting an initial 2-week pilot to measure trait-like compliance and identify participants at risk of long-term noncompliance, performing oversampling based on participants’ individual characteristics to avoid introducing bias in the sample when excluding data based on noncompliance, using an issue tracking portal, and providing special care in troubleshooting to help participants maintain compliance.

Affect detection in the "real" wild - where people go about their daily routines in their homes and workplaces - is arguably a different problem than affect detection in the lab or in the "quasi" wild (e.g., YouTube videos). How will our affect detection systems hold up when put to the test in the real wild? Some in the Affective Computing community had an opportunity to address this question as part of the MOSAIC (Multimodal Objective Sensing to Assess Individuals with Context [1]) program which ran from 2017 to 2020. Results were sobering, but informative. I'll discuss those efforts with an emphasis on performance achieved, insights gleaned, challenges faced, and lessons learned.

We introduce the psychometric concepts of bias and fairness in a multimodal machine learning context assessing individuals’ hireability from prerecorded video interviews. We collected interviews from 733 participants and hireability ratings from a panel of trained annotators in a simulated hiring study, and then trained interpretable machine learning models on verbal, paraverbal, and visual features extracted from the videos to investigate unimodal versus multimodal bias and fairness. Our results demonstrate that, in the absence of any bias mitigation strategy, combining multiple modalities only marginally improves prediction accuracy at the cost of increasing bias and reducing fairness compared to the least biased and most fair unimodal predictor set (verbal). We further show that gender-norming predictors only reduces gender predictability for paraverbal and visual modalities, while removing gender-biased features can achieve gender blindness, minimal bias, and fairness (for all modalities except for visual) at the cost of some prediction accuracy. Overall, the reduced-feature approach using predictors from all modalities achieved the best balance between accuracy, bias, and fairness, with the verbal modality alone performing almost as well. Our analysis highlights how optimizing model prediction accuracy in isolation and in a multimodal context may cause bias, disparate impact, and potential social harm, while a more holistic optimization approach based on accuracy, bias, and fairness can avoid these pitfalls.

Several psychologists posit that performance is not only a function of personality but also of situational contexts, such as day-level activities. Yet in practice, since only personality assessments are used to infer job performance, they provide a limited perspective by ignoring activity. However, multi-modal sensing has the potential to characterize these daily activities. This paper illustrates how empirically measured activity data complements traditional effects of personality to explain a worker's performance. We leverage sensors in commodity devices to quantify the activity context of 603 information workers. By applying classical clustering methods on this multisensor data, we take a person-centered approach to describe workers in terms of both personality and activity. We encapsulate both these facets into an analytical framework that we call organizational personas. On interpreting these organizational personas we find empirical evidence to support that, independent of a worker's personality, their activity is associated with job performance. While the effects of personality are consistent with the literature, we find that the activity is equally effective in explaining organizational citizenship behavior and is less but significantly effective for task proficiency and deviant behaviors. Specifically, personas that exhibit a daily-activity pattern with fewer location visits, batched phone-use, shorter desk-sessions and longer sleep duration, tend to perform better on all three performance metrics. Organizational personas are a descriptive framework to identify the testable hypotheses that can disentangle the role of malleable aspects like activity in determining the performance of a worker population.

Collaborative problem solving (CPS) is a crucial 21st century skill; however, current technologies fall short of effectively supporting CPS processes, especially for remote, computer-enabled interactions. In order to develop next-generation computer-supported collaborative systems that enhance CPS processes and outcomes by monitoring and responding to the unfolding collaboration, we investigate automated detection of three critical CPS process ? construction of shared knowledge, negotiation/coordination, and maintaining team function ? derived from a validated CPS framework. Our data consists of 32 triads who were tasked with collaboratively solving a challenging visual computer programming task for 20 minutes using commercial videoconferencing software. We used automatic speech recognition to generate transcripts of 11,163 utterances, which trained humans coded for evidence of the above three CPS processes using a set of behavioral indicators. We aimed to automate the trained human-raters' codes in a team-independent fashion (current study) in order to provide automatic real-time or offline feedback (future work). We used Random Forest classifiers trained on the words themselves (bag of n-grams) or with word categories (e.g., emotions, thinking styles, social constructs) from the Linguistic Inquiry Word Count (LIWC) tool. Despite imperfect automatic speech recognition, the n-gram models achieved AUROC (area under the receiver operating characteristic curve) scores of .85, .77, and .77 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively; these reflect 70%, 54%, and 54% improvements over chance. The LIWC-category models achieved similar scores of .82, .74, and .73 (64%, 48%, and 46% improvement over chance). Further, the LIWC model-derived scores predicted CPS outcomes more similar to human codes, demonstrating predictive validity. We discuss embedding our models in collaborative interfaces for assessment and dynamic intervention aimed at improving CPS outcomes.

We conducted an exploratory study on affect regulation during game-based learning where 110 college-aged participants (Mage = 22.14, SDage = 1.24; 50.0% female; 70.0% White) played an easy, medium, and difficult level of an educational game (Physics Playground) while self-reporting their strongest affective state and regulation strategies associated with each level. Participants also self-reported their effort and completed a physics posttest after gameplay. We found that frustration, confusion, determination, and curiosity were the dominant affective states (81.4% of total reports) while cognitive reappraisal and acceptance were the major affective regulation strategies (others individually occurred less than 10.1% of the time). Engaging in cognitive reappraisal – an affective regulation strategy that involves changing the way one thinks about a situation – was beneficial for successfully solving a level when participants were frustrated or confused, but had no effect when participants were determined or curious. Engaging in cognitive reappraisal when experiencing high frustration/confusion positively predicted posttest scores, but only for those who put a high amount of effort into the game. For students who were low in effort or low in frustration/confusion, simply accepting one's emotions when experiencing high frustration/confusion was beneficial. We discuss theoretical implications and applications towards game-based learning supports to promote persistence and learning outcomes.

Virtual learning environments (VLEs) are increasingly used at-scale in educational contexts to facilitate teaching and promote learning, and the data they produce can be used for educationa...

Mental health issues, which can be difficult to diagnose, are a growing concern worldwide. For effective care and support, early detection of mood-related health concerns is of paramount importance. Typically, survey based instruments including Ecologically Momentary Assessments (EMA) and Day Reconstruction Method (DRM) are the method of choice for assessing mood related health. While effective, these methods require some effort and thus both compliance rates as well as quality of responses can be limited. As an alternative, We present a study that used passively sensed data from smartphones and wearables and machine learning techniques to predict mood instabilities, an important aspect of mental health. We explored the effectiveness of the proposed method on two large-scale datasets, finding that as little as three weeks of continuous, passive recordings were sufficient to reliably predict mood instabilities.

Achieving important goals is widely assumed to require confronting obstacles, failing repeatedly, and persisting in the face of frustration. Yet empirical evidence linking achievement and frustration tolerance is lacking. To facilitate work on this important topic, we developed and validated a novel behavioral measure of frustration tolerance: the Mirror Tracing Frustration Task (MTFT). In this 5-min task, participants allocate time between a difficult tracing task and entertaining games and videos. In two studies of young adults (Study 1: N = 148, Study 2: N = 283), we demonstrated that the MTFT increased frustration more than 18 other emotions, and that MTFT scores were related to self-reported frustration tolerance. Next, we assessed whether frustration tolerance correlated with similar constructs, including self-control and grit, as well as objective measures of real-world achievement. In a prospective longitudinal study of high-school seniors (N = 391), MTFT scores predicted grade-point average and standardized achievement test scores, and-more than 2 years after completing the MTFT-progress toward a college degree. Though small in size (i.e., rs ranging from .10 to .24), frustration tolerance predicted outcomes over and above a rich set of covariates, including IQ, sociodemographics, self-control, and grit. These findings demonstrate the validity of the MTFT and highlight the importance of frustration tolerance for achieving valued goals. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Assessing performance in the workplace typically relies on subjective evaluations, such as, peer ratings, supervisor ratings and self assessments, which are manual, burdensome and potentially biased. We use objective mobile sensing data from phones, wearables and beacons to study workplace performance and offer new insights into behavioral patterns that distinguish higher and lower performers when considering roles in companies (i.e., supervisors and non-supervisors) and different types of companies (i.e., high tech and consultancy). We present initial results from an ongoing year-long study of N=554 information workers collected over a period ranging from 2-8.5 months. We train a gradient boosting classifier that can classify workers as higher or lower performers with AUROC of 0.83. Our work opens the way to new forms of passive objective assessment and feedback to workers to potentially provide week by week or quarter by quarter guidance in the workplace.

Compared with admissions test scores, why are high school grades better at predicting college graduation? We argue that success in college requires not only cognitive ability but also self-regulatory competencies that are better indexed by high school grades. In a national sample of 47,303 students who applied to college for the 2009/2010 academic year, Study 1 affirmed that high school grades out-predicted test scores for 4-year college graduation. In a convenience sample of 1,622 high school seniors in the Class of 2013, Study 2 revealed that the incremental predictive validity of high school grades for college graduation was explained by composite measures of self-regulation, whereas the incremental predictive validity of test scores was explained by composite measures of cognitive ability.

We designed and tested an attention-aware learning technology (AALT) that detects and responds to mind wandering (MW), a shift in attention from task-related to task-unrelated thoughts, that is negatively associated with learning. We leveraged an existing gaze-based mind wandering detector that uses commercial off the shelf eye tracking to inform real-time interventions during learning with an Intelligent Tutoring System in real-world classrooms. The intervention strategies, co-designed with students and teachers, consisted of using student names, reiterating content, and asking questions, with the aim to reengage wandering minds and improve learning. After several rounds of iterative refinement, we tested our AALT in two classroom studies with 287 high-school students. We found that interventions successfully reoriented attention, and compared to two control conditions, reduced mind wandering, and improved retention (measured via a delayed assessment) for students with low prior-knowledge who occasionally (but not excessively) mind wandered. We discuss implications for developing gaze-based AALTs for real-world contexts.

Students using online learning environments need to effectively self-regulate their learning. However, with an absence of teacher-provided structure, students often resort to less effective, passive learning strategies versus constructive ones. We consider the potential benefits of interventions that promote retrieval practice – retrieving learned content from memory – which is an effective strategy for learning and retention. The goal is to nudge students towards completing short, formative quizzes when they are likely to succeed on those assessments. Towards this goal, we developed a machine-learning model using data from 32,685 students who used an online mathematics platform over an entire school year to prospectively predict scores on three-item assessments (N = 210,020) from interaction patterns up to 9 minutes before the assessment as well as Item Response Theory (IRT) estimates of student ability and quiz difficulty. These models achieved a student-independent correlation of 0.55 between predicted and actual scores on the assessments and outperformed IRT-only predictions (r = 0.34). Model performance was largely independent of the length of the analyzed window preceding a quiz. We discuss potential for future applications of the models to trigger dynamic interventions that aim to encourage students to engage with formative assessments rather than more passive learning strategies.

Teachers, like everyone else, need objective reliable feedback in order to improve their effectiveness. However, developing a system for automated teacher feedback entails many decisions regarding data collection procedures, automated analysis, and presentation of feedback for reflection. We address the latter two questions by comparing two different machine learning approaches to automatically model seven features of teacher discourse (e.g., use of questions, elaborated evaluations). We compared a traditional open-vocabulary approach using n-grams and Random Forest classifiers with a state-of-the-art deep transfer learning approach for natural language processing (BERT). We found a tradeoff between data quantity and accuracy, where deep models had an advantage on larger datasets, but not for smaller datasets, particularly for variables with low incidence rates. We also compared the models based on the level of feedback granularity: utterance-level (e.g., whether an utterance is a question or a statement), class session-level proportions by averaging across utterances (e.g., question incidence score of 48%), and session-level ordinal feedback based on pre-determined thresholds (e.g., question asking score is medium [vs. low or high]) and found that BERT generally provided more accurate feedback at all levels of granularity. Thus, BERT appears to be the most viable approach to providing automatic feedback on teacher discourse provided there is sufficient data to fine tune the model.

Emotional regularity is the degree to which a person maintains and returns to a set of emotional states over time. The present investigation examined associations between emotional regularity and extant emotion measures as well as psychologically relevant dimensions of personality, health, and real-world occupational outcomes. Participants included 598 U.S. adults who provided daily experience sampling reports on their emotional states for approximately two months. Results suggest that emotional regularity was related to, but distinct from, well-established measures of emotion including emotional intensity, variability, covariation, inertia, granularity, and emodiversity. Furthermore, emotional regularity significantly predicted measures of personality, psychological health, and occupational outcomes even when accounting for extant emotion measures and sociodemographic covariates. Finally, it explained modest (7.5%) improvement (in terms of cross-validated RSq.) over baseline models containing emotional intensity, variability, and sociodemographic covariates. These findings suggest that emotional regularity may provide an important indicator of healthy emotional functioning and may be a promising area for further scientific discovery.

Digital learning technologies that aim to measure and sustain student engagement typically use supervised machine learning approaches for engagement detection, which requires reliable “ground-truth” engagement annotations. The present study examined associations between student demographics (age [grade], gender, and ethnicity) and the reliability of engagement annotations based on visual behaviors. We collected videos of diverse students (N = 60) from grades 1–12 who engaged in one-hour online learning sessions with grade-appropriate content. Each student’s data was annotated by three trained coders for behavioral and emotional engagement. We found that inter-rater reliability (IRR) for behavioral engagement was higher for older students whereas IRRs for emotional engagement was higher for younger students. We also found that both rotational head movements and facial expressivity decreased with age, and critically, rotational head movements mediated the effects of grade on behavioral IRR; there was no mediation for emotional IRR. There were no effects of gender or ethnicity on IRR. We discuss the implications of our findings for annotating engagement in supervised learning models for diverse students and across grades.

Frequent monitoring of participant compliance is necessary when conducting large-scale, longitudinal studies to ensure that the collected data is of sufficiently high quality. While the need for achieving high compliance has been underscored and there are discussions on incentives and factors affecting compliance, little is shared about the actual processes and tools used for monitoring compliance in such studies. Monitoring participant compliance with respect to multi-modal data can be a tedious process, especially if there are only a few personnel involved. In this case study, we describe the iterative design of an interactive visualization system we developed for monitoring compliance and refined based on changing requirements in an ongoing study. We find that the visualization system, leveraging the digital medium, both facilitates the exploratory tasks of monitoring participant compliance and supports asynchronous collaboration among non-co-located researchers. Our documented requirements for checking participant compliance as well as the design of the visualization system can help inform the compliance-monitoring process in future studies., 8 pages, 4 figures

Human abnormal physical and psychological behaviors, such as high level of stress, may result in negative impacts on work and life, if not handled efficiently. However, the continuous collection of behavioral data from questionnaires is not feasible, as is often the case for the natural downside of survey data gathering. Thanks to the proliferation of mobile sensors, it brings compelling opportunities for us to more deeply analyze human behavior. In this work, we ask the question of detecting anomalies in human physical and psychological behaviors from multivariate temporal data from multi-modal sensors. In the past decades, many efforts have been made in developing anomaly detection methods, but there remain several challenges in this specific domain problem: 1) data contains missing values at random positions. 2) data from multiple sensors is of multi-resolution and multivariate. 3) human behaviors are correlated to each other, thus it poses a multi-label problem. 4) the available labeled instances are limited, which requires the semi-supervised learning setting. 5) the frequency of anomaly occurrence is much smaller than that of normal instances, leading to imbalance problems. We propose a novel framework MBead to resolve these concerns. MBead consists of three key components: reweighted autoencoder to capture the dependency across temporal domain and multiple modalities, relevance learning module to learn the pairwise relations among labeled instances, and temporal prediction module to detect the anomalies while trained in semi-supervised settings. Extensive experiments show our MBead outperforms seven state-of-art baselines on three tasks of behavior anomaly detection: stress, affect, and work performance.

Modeling team phenomena from multiparty interactions inherently requires combining signals from multiple teammates, often by weighting strategies. Here, we explored the hypothesis that strategic weighting signals from individual teammates would outperform an equal weighting baseline. Accordingly, we explored role-, trait-, and behavior-based weighting of behavioral signals across team members. We analyzed data from 101 triads engaged in computer-mediated collaborative problem solving (CPS) in an educational physics game. We investigated the accuracy of machine-learned models trained on facial expressions, acoustic-prosodics, eye gaze, and task context information, computed one-minute prior to the end of a game level, at predicting success at solving that level. AUROCs for unimodal models that equally weighted features from the three teammates ranged from .54 to .67, whereas a combination of gaze, face, and task context features, achieved an AUROC of .73. The various multiparty weighting strategies did not outperform an equal-weighting baseline. However, our best nonverbal model (AUROC = .73) outperformed a language-based model (AUROC = .67), and there were some advantages to combining the two (AUROC = .75). Finally, models aimed at prospectively predicting performance on a minute-by-minute basis from the start of the level achieved a lower, but still above-chance, AUROC of .60. We discuss implications for multiparty modeling of team performance and other team constructs.

Like anyone, teachers need feedback to improve. Due to the high cost of human classroom observation, teachers receive infrequent feedback which is often more focused on evaluating performance than on improving practice. To address this critical barrier to teacher learning, we aim to provide teachers with detailed and actionable automated feedback. Towards this end, we developed an approach that enables teachers to easily record high-quality audio from their classes. Using this approach, teachers recorded 142 classroom sessions, of which 127 (89%) were usable. Next, we used speech recognition and machine learning to develop teacher-generalizable computer-scored estimates of key dimensions of teacher discourse. We found that automated models were moderately accurate when compared to human coders and that speech recognition errors did not influence performance. We conclude that authentic teacher discourse can be recorded and analyzed for automatic feedback. Our next step is to incorporate the automatic models into an interactive visualization tool that will provide teachers with objective feedback on the quality of their discourse.

In an increasingly globalized and service-oriented economy, people need to engage in computer-mediated collaborative problem solving (CPS) with diverse teams. However, teams routinely fail to live up to expectations, showcasing the need for technologies that help develop effective collaboration skills. We take a step in this direction by investigating how different dimensions of team diversity (demographic, personality, attitudes towards teamwork, prior domain experience) predict objective (e.g. effective solutions) and subjective (e.g. positive perceptions) collaborative outcomes. We collected data from 96 triads who engaged in a 30-minute CPS task via videoconferencing. We found that demographic diversity and differing attitudes towards teamwork predicted impressions of positive engagement, while personality diversity predicted learning outcomes. Importantly, these relationships were maintained after accounting for team makeup. None of the diversity measures predicted task performance. We discuss how our findings can be incorporated into technologies that aim to help diverse teams develop CPS skills.

Collaborative problem solving (CPS) in virtual environments is an increasingly important context of 21st century learning. However, our understanding of this complex and dynamic phenomenon is still limited. Here, we examine unimodal primitives (activity on the screen, speech, and body movements), and their multimodal combinations during remote CPS. We analyze two datasets where 116 triads collaboratively engaged in a challenging visual programming task using video conferencing software. We investigate how UI-interactions, behavioral primitives, and multimodal patterns were associated with teams' subjective and objective performance outcomes. We found that idling with limited speech (i.e., silence or backchannel feedback only) and without movement was negatively correlated with task performance and with participants' subjective perceptions of the collaboration. However, being silent and focused during solution execution was positively correlated with task performance. Results illustrate that in some cases, multimodal patterns improved the predictions and improved explanatory power over the unimodal primitives. We discuss how the findings can inform the design of real-time interventions for remote CPS.

Assessment of job performance, personalized health and psychometric measures are domains where data-driven and ubiquitous computing exhibits the potential of a profound impact in the future. Existing techniques use data extracted from questionnaires, sensors (wearable, computer, etc.), or other traits, to assess well-being and cognitive attributes of individuals. However, these techniques can neither predict individual's well-being and psychological traits in a global manner nor consider the challenges associated to processing the data available, that is incomplete and noisy. In this paper, we create a benchmark for predictive analysis of individuals from a perspective that integrates: physical and physiological behavior, psychological states and traits, and job performance. We design data mining techniques as benchmark and uses real noisy and incomplete data derived from wearable sensors to predict 19 constructs based on 12 standardized well-validated tests. The study included 757 participants who were knowledge workers in organizations across the USA with varied work roles. We developed a data mining framework to extract the meaningful predictors for each of the 19 variables under consideration. Our model is the first benchmark that combines these various instrument-derived variables in a single framework to understand people's behavior by leveraging real uncurated data from wearable, mobile, and social media sources. We verify our approach experimentally using the data obtained from our longitudinal study. The results show that our framework is consistently reliable and capable of predicting the variables under study better than the baselines when prediction is restricted to the noisy, incomplete data.