Search

We provide a psychometric-grounded exposition of bias and fairness as applied to a typical machine learning pipeline for affective computing. We expand on an interpersonal communication framework to elucidate how to identify sources of bias that may arise in the process of inferring human emotions and other psychological constructs from observed behavior. Various methods and metrics for measuring fairness and bias are discussed along with pertinent implications within the United States legal context. We illustrate how to measure some types of bias and fairness in a case study involving automatic personality and hireability inference from multimodal data collected in video interviews for mock job applications. We encourage affective computing researchers and practitioners to encapsulate bias and fairness in their research processes and products and to consider their role, agency, and responsibility in promoting equitable and just systems., 21 pages, 4 figures

Collaborative problem-solving (CPS) is ubiquitous in everyday life, including work, family, leisure activities, etc. With collaborations increasingly occurring remotely, next-generation collaborative interfaces could enhance CPS processes and outcomes with dynamic interventions or by generating feedback for after-action reviews. Automatic modeling of CPS processes (called facets here) is a precursor to this goal. Accordingly, we build automated detectors of three critical CPS facets—construction of shared knowledge, negotiation and coordination, and maintaining team function—derived from a validated CPS framework. We used data of 32 triads who collaborated via a commercial videoconferencing software, to solve challenging problems in a visual programming task. We generated transcripts of 11,163 utterances using automatic speech recognition, which were then coded by trained humans for evidence of the three CPS facets. We used both standard and deep sequential learning classifiers to model the human-coded facets from linguistic, task context, facial expressions, and acoustic–prosodic features in a team-independent fashion. We found that models relying on nonverbal signals yielded above-chance accuracies (area under the receiver operating characteristic curve, AUROC) ranging from .53 to .83, with increases in model accuracy when language information was included (AUROCS from .72 to .86). There were no advantages of deep sequential learning methods over standard classifiers. Overall, Random Forest classifiers using language and task context features performed best, achieving AUROC scores of .86, .78, and .79 for construction of shared knowledge, negotiation/coordination, and maintaining team function, respectively. We discuss application of our work to real-time systems that assess CPS and intervene to improve CPS outcomes.

Whereas social visual attention has been examined in computer-mediated (e.g., shared screen) or video-mediated (e.g., FaceTime) interaction, it has yet to be studied in mixed-media interfaces that combine video of the conversant along with other UI elements. We analyzed eye gaze of 37 dyads (74 participants) who were tasked with negotiating the price of a new car (as a buyer and seller) using mixed-media video conferencing under competitive or cooperative negotiation instructions (experimental manipulation). We used multidimensional recurrence quantification analysis to extract spatio-temporal patterns corresponding to mutual gaze (individuals look at each other), joint attention (individuals focus on the same elements of the interface), and gaze aversion (an individual looks at their partner, who is looking elsewhere). Our results indicated that joint attention predicted the sum of points attained by the buyer and seller (i.e., the joint score). In contrast, gaze aversion was associated with faster time to complete the negotiation, but with a lower joint score. Unexpectedly, mutual gaze was highly infrequent and unrelated to the negotiation outcomes and none of the gaze patterns predicted subjective perceptions of the negotiation. There were also no effects of gender composition or negotiation condition on the gaze patterns or negotiation outcomes. Our results suggest that social visual attention may operate differently in mixed-media collaborative interfaces than in face-to-face interaction. As mixed-media collaborative interfaces gain prominence, our work can be leveraged to inform the design of gaze-sensitive user interfaces that support remote negotiations among other tasks.

The prevalence of the acronym tl;dr (“too long; didn’t read”) suggests that people intentionally disengage their attention from long sections of text. We studied this real-world phenomenon in an ed...

This study answered novel questions about the connection between high school extracurricular dosage (number of activities and participation duration) and the attainment of a bachelor's degree. Using data from the Common Application and the National Student Clearinghouse (N = 311,308), we found that greater extracurricular participation positively predicted bachelor's degree attainment. However, among students who ultimately earned a bachelor's degree, participating in more than a moderate number of high school activities (3 or 4) predicted decreasing odds of earning a bachelor's degree on time (within 4 years). This effect intensified as participation duration increased, such that students who participated in the greatest number of high school activities for the most years were the most likely to delay college graduation.

We zone out roughly 20-40% of the time during reading – a rate that is concerning given the negative relationship between mind-wandering and comprehension. We tested if Eye-Mind Reader – an intelli...

Background Studies that use ecological momentary assessments (EMAs) or wearable sensors to track numerous attributes, such as physical activity, sleep, and heart rate, can benefit from reductions in missing data. Maximizing compliance is one method of reducing missing data to increase the return on the heavy investment of time and money into large-scale studies. Objective This paper aims to identify the extent to which compliance can be prospectively predicted from individual attributes and initial compliance. Methods We instrumented 757 information workers with fitness trackers for 1 year and conducted EMAs in the first 56 days of study participation as part of an observational study. Their compliance with the EMA and fitness tracker wearing protocols was analyzed. Overall, 31 individual characteristics (eg, demographics and personalities) and behavioral variables (eg, early compliance and study portal use) were considered, and 14 variables were selected to create beta regression models for predicting compliance with EMAs 56 days out and wearable compliance 1 year out. We surveyed study participation and correlated the results with compliance. Results Our modeling indicates that 16% and 25% of the variance in EMA compliance and wearable compliance, respectively, could be explained through a survey of demographics and personality in a held-out sample. The likelihood of higher EMA and wearable compliance was associated with being older (EMA: odds ratio [OR] 1.02, 95% CI 1.00-1.03; wearable: OR 1.02, 95% CI 1.01-1.04), speaking English as a first language (EMA: OR 1.38, 95% CI 1.05-1.80; wearable: OR 1.39, 95% CI 1.05-1.85), having had a wearable before joining the study (EMA: OR 1.25, 95% CI 1.04-1.51; wearable: OR 1.50, 95% CI 1.23-1.83), and exhibiting conscientiousness (EMA: OR 1.25, 95% CI 1.04-1.51; wearable: OR 1.34, 95% CI 1.14-1.58). Compliance was negatively associated with exhibiting extraversion (EMA: OR 0.74, 95% CI 0.64-0.85; wearable: OR 0.67, 95% CI 0.57-0.78) and having a supervisory role (EMA: OR 0.65, 95% CI 0.54-0.79; wearable: OR 0.66, 95% CI 0.54-0.81). Furthermore, higher wearable compliance was negatively associated with agreeableness (OR 0.68, 95% CI 0.56-0.83) and neuroticism (OR 0.85, 95% CI 0.73-0.98). Compliance in the second week of the study could help explain more variance; 62% and 66% of the variance in EMA compliance and wearable compliance, respectively, was explained. Finally, compliance correlated with participants’ self-reflection on the ease of participation, usefulness of our compliance portal, timely resolution of issues, and compensation adequacy, suggesting that these are avenues for improving compliance. Conclusions We recommend conducting an initial 2-week pilot to measure trait-like compliance and identify participants at risk of long-term noncompliance, performing oversampling based on participants’ individual characteristics to avoid introducing bias in the sample when excluding data based on noncompliance, using an issue tracking portal, and providing special care in troubleshooting to help participants maintain compliance.

Several psychologists posit that performance is not only a function of personality but also of situational contexts, such as day-level activities. Yet in practice, since only personality assessments are used to infer job performance, they provide a limited perspective by ignoring activity. However, multi-modal sensing has the potential to characterize these daily activities. This paper illustrates how empirically measured activity data complements traditional effects of personality to explain a worker's performance. We leverage sensors in commodity devices to quantify the activity context of 603 information workers. By applying classical clustering methods on this multisensor data, we take a person-centered approach to describe workers in terms of both personality and activity. We encapsulate both these facets into an analytical framework that we call organizational personas. On interpreting these organizational personas we find empirical evidence to support that, independent of a worker's personality, their activity is associated with job performance. While the effects of personality are consistent with the literature, we find that the activity is equally effective in explaining organizational citizenship behavior and is less but significantly effective for task proficiency and deviant behaviors. Specifically, personas that exhibit a daily-activity pattern with fewer location visits, batched phone-use, shorter desk-sessions and longer sleep duration, tend to perform better on all three performance metrics. Organizational personas are a descriptive framework to identify the testable hypotheses that can disentangle the role of malleable aspects like activity in determining the performance of a worker population.

We conducted an exploratory study on affect regulation during game-based learning where 110 college-aged participants (Mage = 22.14, SDage = 1.24; 50.0% female; 70.0% White) played an easy, medium, and difficult level of an educational game (Physics Playground) while self-reporting their strongest affective state and regulation strategies associated with each level. Participants also self-reported their effort and completed a physics posttest after gameplay. We found that frustration, confusion, determination, and curiosity were the dominant affective states (81.4% of total reports) while cognitive reappraisal and acceptance were the major affective regulation strategies (others individually occurred less than 10.1% of the time). Engaging in cognitive reappraisal – an affective regulation strategy that involves changing the way one thinks about a situation – was beneficial for successfully solving a level when participants were frustrated or confused, but had no effect when participants were determined or curious. Engaging in cognitive reappraisal when experiencing high frustration/confusion positively predicted posttest scores, but only for those who put a high amount of effort into the game. For students who were low in effort or low in frustration/confusion, simply accepting one's emotions when experiencing high frustration/confusion was beneficial. We discuss theoretical implications and applications towards game-based learning supports to promote persistence and learning outcomes.

Mental health issues, which can be difficult to diagnose, are a growing concern worldwide. For effective care and support, early detection of mood-related health concerns is of paramount importance. Typically, survey based instruments including Ecologically Momentary Assessments (EMA) and Day Reconstruction Method (DRM) are the method of choice for assessing mood related health. While effective, these methods require some effort and thus both compliance rates as well as quality of responses can be limited. As an alternative, We present a study that used passively sensed data from smartphones and wearables and machine learning techniques to predict mood instabilities, an important aspect of mental health. We explored the effectiveness of the proposed method on two large-scale datasets, finding that as little as three weeks of continuous, passive recordings were sufficient to reliably predict mood instabilities.

Achieving important goals is widely assumed to require confronting obstacles, failing repeatedly, and persisting in the face of frustration. Yet empirical evidence linking achievement and frustration tolerance is lacking. To facilitate work on this important topic, we developed and validated a novel behavioral measure of frustration tolerance: the Mirror Tracing Frustration Task (MTFT). In this 5-min task, participants allocate time between a difficult tracing task and entertaining games and videos. In two studies of young adults (Study 1: N = 148, Study 2: N = 283), we demonstrated that the MTFT increased frustration more than 18 other emotions, and that MTFT scores were related to self-reported frustration tolerance. Next, we assessed whether frustration tolerance correlated with similar constructs, including self-control and grit, as well as objective measures of real-world achievement. In a prospective longitudinal study of high-school seniors (N = 391), MTFT scores predicted grade-point average and standardized achievement test scores, and-more than 2 years after completing the MTFT-progress toward a college degree. Though small in size (i.e., rs ranging from .10 to .24), frustration tolerance predicted outcomes over and above a rich set of covariates, including IQ, sociodemographics, self-control, and grit. These findings demonstrate the validity of the MTFT and highlight the importance of frustration tolerance for achieving valued goals. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Emotional regularity is the degree to which a person maintains and returns to a set of emotional states over time. The present investigation examined associations between emotional regularity and extant emotion measures as well as psychologically relevant dimensions of personality, health, and real-world occupational outcomes. Participants included 598 U.S. adults who provided daily experience sampling reports on their emotional states for approximately two months. Results suggest that emotional regularity was related to, but distinct from, well-established measures of emotion including emotional intensity, variability, covariation, inertia, granularity, and emodiversity. Furthermore, emotional regularity significantly predicted measures of personality, psychological health, and occupational outcomes even when accounting for extant emotion measures and sociodemographic covariates. Finally, it explained modest (7.5%) improvement (in terms of cross-validated RSq.) over baseline models containing emotional intensity, variability, and sociodemographic covariates. These findings suggest that emotional regularity may provide an important indicator of healthy emotional functioning and may be a promising area for further scientific discovery.

Frequent monitoring of participant compliance is necessary when conducting large-scale, longitudinal studies to ensure that the collected data is of sufficiently high quality. While the need for achieving high compliance has been underscored and there are discussions on incentives and factors affecting compliance, little is shared about the actual processes and tools used for monitoring compliance in such studies. Monitoring participant compliance with respect to multi-modal data can be a tedious process, especially if there are only a few personnel involved. In this case study, we describe the iterative design of an interactive visualization system we developed for monitoring compliance and refined based on changing requirements in an ongoing study. We find that the visualization system, leveraging the digital medium, both facilitates the exploratory tasks of monitoring participant compliance and supports asynchronous collaboration among non-co-located researchers. Our documented requirements for checking participant compliance as well as the design of the visualization system can help inform the compliance-monitoring process in future studies., 8 pages, 4 figures

Like anyone, teachers need feedback to improve. Due to the high cost of human classroom observation, teachers receive infrequent feedback which is often more focused on evaluating performance than on improving practice. To address this critical barrier to teacher learning, we aim to provide teachers with detailed and actionable automated feedback. Towards this end, we developed an approach that enables teachers to easily record high-quality audio from their classes. Using this approach, teachers recorded 142 classroom sessions, of which 127 (89%) were usable. Next, we used speech recognition and machine learning to develop teacher-generalizable computer-scored estimates of key dimensions of teacher discourse. We found that automated models were moderately accurate when compared to human coders and that speech recognition errors did not influence performance. We conclude that authentic teacher discourse can be recorded and analyzed for automatic feedback. Our next step is to incorporate the automatic models into an interactive visualization tool that will provide teachers with objective feedback on the quality of their discourse.

In an increasingly globalized and service-oriented economy, people need to engage in computer-mediated collaborative problem solving (CPS) with diverse teams. However, teams routinely fail to live up to expectations, showcasing the need for technologies that help develop effective collaboration skills. We take a step in this direction by investigating how different dimensions of team diversity (demographic, personality, attitudes towards teamwork, prior domain experience) predict objective (e.g. effective solutions) and subjective (e.g. positive perceptions) collaborative outcomes. We collected data from 96 triads who engaged in a 30-minute CPS task via videoconferencing. We found that demographic diversity and differing attitudes towards teamwork predicted impressions of positive engagement, while personality diversity predicted learning outcomes. Importantly, these relationships were maintained after accounting for team makeup. None of the diversity measures predicted task performance. We discuss how our findings can be incorporated into technologies that aim to help diverse teams develop CPS skills.

Assessment of job performance, personalized health and psychometric measures are domains where data-driven and ubiquitous computing exhibits the potential of a profound impact in the future. Existing techniques use data extracted from questionnaires, sensors (wearable, computer, etc.), or other traits, to assess well-being and cognitive attributes of individuals. However, these techniques can neither predict individual's well-being and psychological traits in a global manner nor consider the challenges associated to processing the data available, that is incomplete and noisy. In this paper, we create a benchmark for predictive analysis of individuals from a perspective that integrates: physical and physiological behavior, psychological states and traits, and job performance. We design data mining techniques as benchmark and uses real noisy and incomplete data derived from wearable sensors to predict 19 constructs based on 12 standardized well-validated tests. The study included 757 participants who were knowledge workers in organizations across the USA with varied work roles. We developed a data mining framework to extract the meaningful predictors for each of the 19 variables under consideration. Our model is the first benchmark that combines these various instrument-derived variables in a single framework to understand people's behavior by leveraging real uncurated data from wearable, mobile, and social media sources. We verify our approach experimentally using the data obtained from our longitudinal study. The results show that our framework is consistently reliable and capable of predicting the variables under study better than the baselines when prediction is restricted to the noisy, incomplete data.

Collaborative problem solving (CPS) is an essential skill for the 21st century workforce, but remains difficult to assess. Understanding how CPS skills affect CPS performance outcomes can inform CPS training, task design, feedback design, and automated assessment. We investigated CPS behaviors (individually and in co-occurring patterns) in 101 (N = 303) remote triads who collaboratively played an educational game called Physics Playground for 45-min. Team interactions consisting of open-ended speech occurring over video-conferencing with screen sharing. We coded participant's utterances relative to a CPS framework consisting of three facets (i.e., competencies such as constructing shared knowledge) manifested in 19 specific indicators (e.g., responds to other's questions/ideas). A matching technique was used to isolate the effect of CPS behaviors on CPS outcomes (quality of solution of a game level) controlling for pertinent covariates. Mixed-effects ordinal regression models indicated that proposing solution ideas and discussing results were the major predictors of CPS performance, and that team-member activities surrounding idea generation mattered. These findings highlighted the importance of both individual and collective contributions and social and cognitive skills in successful CPS outcomes.

We conducted a meta-analysis of 33 independent samples (N = 2924) to address whether adding anthropomorphic faces to multimedia graphics and/or adding pleasant colors are effective emotional design approaches. We found significant positive meta-analytic effects for retention (k = 18, d+ = 0.387), comprehension (k = 14, d+ = 0.317), and transfer (k = 27, d+ = 0.327) under a random-effects model. Effects for affective-motivational variables were mixed, with a robust effect for intrinsic motivation (k = 23, d+ = 0.255), a weaker effect for liking/enjoyment (k = 20, d+ = 0.109), and a marginal effect for positive affect (k = 15, d+ = 0.113). The manipulations did not significantly (ps > .227) influence perceptions of learning (k = 11, d+ = 0.097) or effort (k = 20, d+ = 0.051), but reduced perceptions of difficulty (k = 14, d+ = −0.208). Four of the outcome variables (retention, transfer, intrinsic motivation, and perceived effort) were sufficiently heterogeneous. There was no major issue with publication bias, influential cases, or outliers. With one exception, there was no evidence of moderation by experimental contrast, dynamicity of materials, age, language/culture, prior mood, time-on-task, and publication type after adjusting for multiple comparisons. There was provisional evidence that age moderated the effect of the manipulations on intrinsic motivation, such that larger effects were revealed for children compared to older learners. Altogether, anthropomorphisms/colors appear to be useful design principles.

Contains fulltext : 195646.pdf (Publisher’s version ) (Open Access) What do we think about when we mind wander and where do these thoughts come from? We tested the idea that semantically rich stimuli yield patterns of mind wandering that are closely coupled with the stimuli compared to being more internally triggered. We analyzed the content of 949 self-reported zone outs (1218 thoughts) and 519 of their triggers from 88 participants who read an instructional text and watched a film for 20 min each. We found that mind wandering associated with memory retrieval was more frequent than prospection and introspection across both stimuli. Over 70% of autobiographical and semantic memory retrievals were triggered by the content of the stimuli, compared to around 30% for prospective and introspective thoughts. Further, latent semantic analysis revealed that semantic and unspecific memories were more “semantically” similar to their triggers than prospective and introspective thoughts, suggesting that they arise from spontaneous associations with the stimulus. These findings suggest a re-evaluation of how internal concerns and the external world give rise to mind wandering and emphasize the importance of studying mind wandering in semantically rich contexts akin to much of the real world. 14 p.

There is increased interest in augmenting multimedia instructional materials to elevate learners’ positive affective-motivational states in order to improve learning. However, these efforts have only been partly successful and mediational effects of positive affective-motivational states have not always been established. In this study, university students (N = 65) from the Czech Republic, a country where beer brewing is a source of national pride, were informed that they would either study how to brew beer (high intrinsic motivation condition) or how to prepare a citrate substrate (low intrinsic motivation condition). The 90-min simulation environment used for learning was about beer brewing in both cases, with superficial changes to instructions and graphics to disguise the topic manipulation. Generalized positive affect, overall enjoyment, flow, and learning involvement were higher in the beer brewing condition (Cohen’s d = 0.44–0.87) as were learning gains when measured immediately (retention: d = 0.48; transfer: d = 0.46) and a month later (retention: d = 0.66; transfer: d = 0.62). However, only learning involvement and flow positively mediated the influence of the topic manipulation on immediate learning outcomes; there were no mediation effects on delayed learning outcomes after co-varying out immediate learning. The findings corroborate results from extant studies on the importance of topic interest in learning from instructional texts. They also indicate that affective-motivational mediation is one, but not the only, mechanism by which topic-based intrinsic motivation manipulations influence learning and that induced positive affective-motivational states can be differentially related or unrelated to learning.

We hypothesize that cognitively engaged readers dynamically adjust their reading times with respect to text complexity (i.e., reading times should increase for difficult sections and decrease for easier ones) and failure to do so should impair comprehension. This hypothesis is consistent with theories of text comprehension but has surprisingly been untested. We tested this hypothesis by analyzing 4 datasets in which participants (N = 484) read expository texts using a self-paced reading paradigm. Participants self-reported mind wandering in response to pseudorandom thought-probes during reading and completed comprehension assessments after reading. We computed two measures of cognitive coupling by regressing each participant's paragraph-level reading times on two measures of text complexity: Flesch-Kincaid Grade Level and Word Concreteness scores. The two coupling measures yielded convergent findings: coupling was a negative predictor of mind wandering and a positive predictor of both text- and inference-level comprehension. Goodness-of-fit, measured with Akaike information criterion, also improved after adding coupling to the reading-time only models. Furthermore, cognitive coupling mediated the relationship between mind wandering and comprehension, supporting the hypothesis that mind wandering engenders a decoupling of attention from external stimuli. (PsycINFO Database Record

Personality traits such as grit and self-control are important determinants of success in life outcomes. However, most measures of these traits, which rely on self-reports, might be biased when used for the purpose of evaluating education policies or interventions. Recent research has shown the potential of survey effort-in particular, item non-response and careless answering-as a proxy measure of these traits. The current investigation uses a dataset of high school seniors (N = 513) to investigate survey effort measures in relationship with teacher reports, performance task measures, high school academic outcomes, and college attendance. Our results show promise for use of survey effort as proxy measures of grit and self-control.

We hypothesize that effective collaboration is facilitated when individuals and environmental components form a synergy where they work together and regulate one another to produce stable patterns of behavior, or regularity, as well as adaptively reorganize to form new behaviors, or irregularity. We tested this hypothesis in a study with 32 triads who collaboratively solved a challenging visual computer programming task for 20 min following an introductory warm-up phase. Multidimensional recurrence quantification analysis was used to examine fine-grained (i.e., every 10 s) collective patterns of regularity across team members' speech rate, body movement, and team interaction with the shared user interface. We found that teams exhibited significant patterns of regularity as compared to shuffled baselines, but there were no systematic trends in regularity across time. We also found that periods of regularity were associated with a reduction in overall behavior. Notably, the production of irregular behavior predicted expert-coded metrics of collaborative activity, such as teams' ability to construct shared knowledge and effectively negotiate and coordinate execution of solutions, net of overall behavioral production and behavioral self-similarity. Our findings support the theory that groups can interact to form interpersonal synergies and indicate that information about system-level dynamics is a viable way to understand and predict effective collaborative processes.

We know that reading involves coordination between textual characteristics and visual attention, but research linking eye movements during reading and comprehension assessed after reading is surprisingly limited, especially for reading long connected texts. We tested two competing possibilities: (a) the weak association hypothesis: Links between eye movements and comprehension are weak and short-lived, versus (b) the strong association hypothesis: The two are robustly linked, even after a delay. Using a predictive modeling approach, we trained regression models to predict comprehension scores from global eye movement features, using participant-level cross-validation to ensure that the models generalize across participants. We used data from three studies in which readers (Ns = 104, 130, 147) answered multiple-choice comprehension questions ~30 min after reading a 6,500-word text, or after reading up to eight 1,000-word texts. The models generated accurate predictions of participants' text comprehension scores (correlations between observed and predicted comprehension: 0.384, 0.362, 0.372, ps

The Tesserae project investigates how a suite of sensors can measure workplace performance (e.g., organizational citizenship behavior), psychological traits (e.g., personality, affect), and physical characteristics (e.g., sleep, activity) over one year. We enrolled 757 information workers across the U.S. and measure heart rate, physical activity, sleep, social context, and other aspects through smartwatches, a phone agent, beacons, and social media. We report challenges that we faced with enrollment, privacy, and incentive structures while setting up such a long-term multimodal large-scale sensor study. We discuss the tradeoffs of remote versus in-person enrollment, and showed that directly paid, in-person enrolled participants are more compliant overall compared to remotely-enrolled participants. We find that providing detailed information regarding privacy concerns up-front is highly beneficial. We believe that our experiences can benefit other large sensor projects as this field grows.

Multiparty collaborative problem solving - an increasingly important context in the 21st century workforce - suffers from a degradation of social and behavioral signals when attempted remotely, resulting in suboptimal outcomes. We investigate teams' multidimensional patterns of visual attention during a collaborative problem-solving task with an eye for leveraging insights to improve collaborative interfaces. Fifty-seven novices (forming 19 triads) engaged in a challenging programming task (Minecraft Hour of Code) using videoconferencing software with screen sharing. To discover patterns of individual-level gaze-UI coupling(coordination of a teammate's attention with respect to changes in the user interface) and team-level gaze-UI regularity (dynamics of teams' collective attention in context with changes in the user interface), we applied cross- and multidimensional recurrence quantification analyses, respectively. Individuals' eye gaze was significantly coupled with the ongoing screen activity whereas teams displayed significant patterns of gaze regularity, suggesting repetitive patterns in teams' attention. These measures predicted expert-coded collaborative processes of constructing shared knowledge and negotiation and coordination (but not maintaining team function) and correlated with task score (r = .425). They also predicted individually assessed subjective perceptions of team performance and the collaboration process, but not individual's learning or team's task scores. We discuss implications of our findings for the design of intelligent collaborative interfaces.

Social media serves as a platform to share thoughts and connect with others. The ubiquitous use of social media also enables researchers to study human behavior as the data can be collected in an inexpensive and unobtrusive way. Not only does social media provide a passive means to collect historical data at scale, it also functions as a "verbal" sensor, providing rich signals about an individual's social ecological context. This case study introduces an infrastructural framework to illustrate the feasibility of passively collecting social media data at scale in the context of an ongoing multimodal sensing study of workplace performance (N=757). We study our dataset in its relationship with demographic, personality, and wellbeing attributes of individuals. Importantly, as a means to study selection bias, we examine what characterizes individuals who choose to consent to social media data sharing vs. those who do not. Our work provides practical experiences and implications for research in the HCI field who seek to conduct similar longitudinal studies that harness the potential of social media data.

Re-reading has been shown to have a minimal benefit on text comprehension, in comparison to reading only once or other types of study techniques (e.g., testing; self-explanation). In two experiments we examined the effect of re-reading on mind wandering. Participants read two texts, during which they responded to intermittent mind wandering probes. One text was read once and the other twice. Consistent with previous findings, there was no effect of re-reading on comprehension even though participants reported feeling more competent when they re-read the text. Critically, participants mind wandered more while re-reading. Furthermore, the effect of re-reading on mind wandering was specific to intentional forms of mind wandering rather than unintentional. The implications of these results for understanding mind wandering and the limited effectiveness of re-reading as a mnemonic are discussed.

Silent reading fluency has received limited attention in the school-based literatures across the past decade. We fill this gap by examining both oral and silent reading fluency and their relation to overall abilities in reading comprehension in fourth-grade students. Lower-level reading skills (word reading, rapid automatic naming) and vocabulary were included in structural equation models in order to determine their impact on reading fluency and comprehension. Results suggested that oral and silent reading fluency represent separate constructs, but only oral reading fluency contributed to reading comprehension. Vocabulary was found to contribute uniquely to comprehension even after controlling for reading fluency.

We tested the domain-specificity or domain-generality of academic diligence in middle-school students using the Academic Diligence Task (ADT), a performance task that assesses effort on tedious problems in the face of digital distractions. Students in 8th grade (N = 439) were randomly assigned to individually complete a math, verbal, or spatial ADT or to a combination of all three. Confirmatory factor analyses suggested domain-generality, as did the fact that ADT scores in a given domain did not differentially predict academic achievement in that domain. Results indicated that all three ADTs had adequate external and predictive validity, but convergent validity varied. Whereas both math and verbal ADT scores correlated with teacher-reports of grit and self-control, only math scores consistently correlated with self-reports of the same constructs; these measures did not correlate with spatial ADT scores. Thus, the math ADT is the best performance measure of diligence, followed by the verbal ADT.

We leverage a unique national dataset of 41,359 college applications to prospectively predict 4-year bachelor's graduation in a generalizable manner. Our features include sociodemographics, institutional graduation rates, academic achievement, standardized test scores, engagement in extracurricular activities, work experiences, and ratings by teachers and high-school guidance counselors. A random forest classifier successfully predicted 4-year graduation for 71.4% of the students (base rate = 44%) using all 166 of the aforementioned features and a split-half validation method. A stochastic hill-climbing feature selection procedure effectively maintained the same classification accuracy, but with a minimal set of 37 features, consisting of an approximately equal representation of sociodemographics, cognitive, and noncognitive factors. We advocate against using these results for admissions decisions, instead contemplating how they might be used to provide parents and educators with actionable information to guide students towards college success.

Item does not contain fulltext How does the dynamic structure of the external world direct attention? We examined the relationship between event structure and attention to test the hypothesis that narrative shifts (both theoretical and perceived) negatively predict attentional lapses. Self-caught instances of mind wandering were collected while 108 participants watched a 32.5 min film called The Red Balloon. We used theoretical codings of situational change and human perceptions of event boundaries to predict mind wandering in 5-s intervals. Our findings suggest a temporal alignment between the structural dynamics of the film and mind wandering reports. Specifically, the number of situational changes and likelihood of perceiving event boundaries in the prior 0–15 s interval negatively predicted mind wandering net of low-level audiovisual features. Thus, mind wandering is less likely to occur when there is more event change, suggesting that narrative shifts keep attention from drifting inwards.

Novice students (N = 99) participated in a lab study in which they learned the fundamentals of computer programming in Python using a self-paced computerized learning environment involving a 25-min scaffolded learning phase and a 10-min unscaffolded fadeout phase. Students provided affect judgments at approximately 100 points (every 15 s) over the course of viewing videos of their faces and computer screens recorded during the learning session. The results indicated that engagement, confusion, frustration, boredom, and curiosity were the most frequent affective states, while anxiety, happiness, anger, surprise, disgust, sadness, and fear were rare. Confusion + frustration and curiosity + engagement were identified as two frequently co-occurring pairs of affective states. An analysis of affect dynamics indicated that there were reciprocal transitions between engagement and confusion, confusion and frustration, and one way transitions between frustration and boredom and boredom and engagement. Considering interaction events in tandem with affect revealed that constructing code was the central activity that preceded and followed each affective state. Further, confusion and frustration followed errors and preceded hint usage, while curiosity and engagement followed reading or coding. An analysis of affect-learning relationships after partialling out control variables (e.g., scholastic aptitude, hint usage) indicated that boredom (r = −.149) and frustration (r = −.218) were negative correlated with learning while transitions between confusion → frustration (r = .103), frustration → confusion (r = .105), and boredom → engagement (r = .282) were positively correlated with learning. Implications of the results to theory on affect incidence and dynamics and on the design of affect-aware learning environments are discussed.

Mind wandering is a ubiquitous phenomenon where attention involuntarily shifts from task-related thoughts to internal task-unrelated thoughts. Mind wandering can have negative effects on performance; hence, intelligent interfaces that detect mind wandering can improve performance by intervening and restoring attention to the current task. We investigated the use of eye gaze and contextual cues to automatically detect mind wandering during reading with a computer interface. Participants were pseudorandomly probed to report mind wandering while an eye tracker recorded their gaze during the reading task. Supervised machine learning techniques detected positive responses to mind wandering probes from eye gaze and context features in a user-independent fashion. Mind wandering was detected with an accuracy of 72 % (expected accuracy by chance was 60 %) when probed at the end of a page and an accuracy of 67 % (chance was 59 %) when probed in the midst of reading a page. Global gaze features (gaze patterns independent of content, such as fixation durations) were more effective than content-specific local gaze features. An analysis of the features revealed diagnostic patterns of eye gaze behavior during mind wandering: (1) certain types of fixations were longer; (2) reading times were longer than expected; (3) more words were skipped; and (4) there was a larger variability in pupil diameter. Finally, the automatically detected mind wandering rate correlated negatively with measures of learning and transfer even after controlling for prior knowledge, thereby providing evidence of predictive validity. Possible improvements to the detector and applications that utilize the detector are discussed.

Summary The executive resource hypothesis assumes a positive relationship between resource availability and mind wandering. Under the assumption that different modalities of information delivery differentially tax resources, we compared mind wandering across different modalities during the presentation of The Red Headed League (Experiment 1 and 2) and Walden (Experiment 3). An Audio only condition produced the most mind wandering. Two conditions that presumably consumed more executive resources than the Audio Only condition (i.e. Audio + Text and Self-paced Reading) produced equivalent amounts of mind wandering during Experiments 1 and 2. In Experiment 3 the reading time of the Self-paced readers moderated the effect of mind wandering in that the fast readers mind wandered more than those in the Audio + Text condition. Results are discussed in the context of the demands of different modes of information delivery methods and mind wandering as well as the potential effects of material type. Copyright © 2015 John Wiley & Sons, Ltd.

This study investigated the relationships among incoming knowledge, persistence, affective states, in-game progress, and consequently learning outcomes for students using the game Physics Playground. We used structural equation modeling to examine these relations. We tested three models, obtaining a model with good fit to the data. We found evidence that both the pretest and the in-game measure of student performance significantly predicted learning outcome, while the in-game measure of performance was predicted by pretest data, frustration, and engaged concentration. Moreover, we found evidence for two indirect paths from engaged concentration and frustration to learning, via the in-game progress measure. We discuss the importance of these findings, and consider viable next steps concerning the design of effective learning supports within game environments. We model relations among various student variables and learning outcome in a game.Pretest and in-game performance significantly predict learning outcome.In-game performance is predicted by pretest data, frustration, and engagement.Two indirect paths involving frustration and engagement predict learning.

The current concerns hypothesis suggests that directing attention towards unfulfilled plans of the individual prior to a task would result in more off-task thoughts (or mind wandering). In this experiment, participants were asked to read a scientific text and self-report instances of mind wandering by indicating when they were experiencing task-unrelated thoughts (TUTs) or task-related interferences (TRIs). Prior to reading, participants in the individual plans experimental condition were asked to reflect upon their short-term plans by making a “to do” list while participants in a control condition were asked to make a list of the components of an automobile. In support of the current concerns hypothesis, directing attention towards the short-term plans resulted in significantly more TUTs, but not TRIs. Furthermore, participants in the individual plans condition had significantly lower scores on an assessment of reading comprehension, and this relationship was mediated by the frequency of TUTs.

Objective Can having too much self-control make people unhappy? Researchers have increasingly questioned the unilateral goodness of self-control and proposed that it is beneficial only up to a certain point, after which it becomes detrimental. The little empirical research on the issue shows mixed results. Hence, we tested whether a curvilinear relationship between self-control and subjective well-being exists. Method We used multiple metrics (questionnaires, behavioral ratings), sources (self-report, other-report), and methods (cross-sectional measurement, dayreconstruction method, experience sampling method) across six studies (Ntotal = 5,318). Results We found that self-control positively predicted subjective well-being (cognitive and affective), but there was little evidence for an inverted U-shaped curve. The results held after statistically controlling for demographics and other psychological confounds. Conclusion Our main finding is that self-control enhances subjective well-being with little to no apparent downside of too much self-control.

Mind wandering is a ubiquitous phenomenon in which attention shifts from task-related to task-unrelated thoughts. The last decade has witnessed an explosion of interest in mind wandering, but research has been stymied by a lack of objective measures, leading to a near-exclusive reliance on self-reports. We addressed this issue by developing an eye-gaze-based, machine-learned model of mind wandering during computerized reading. Data were collected in a study in which 132 participants reported self-caught mind wandering while reading excerpts from a book on a computer screen. A remote Tobii TX300 or T60 eyetracker recorded their gaze during reading. The data were used to train supervised classification models to discriminate between mind wandering and normal reading in a manner that would generalize to new participants. We found that at the point of maximal agreement between the model-based and self-reported mind-wandering means (smallest difference between the group-level means: M model = .310, M self = .319), the participant-level mind-wandering proportional distributions were similar and were significantly correlated (r = .400). The model-based estimates were internally consistent (r = .751) and predicted text comprehension more strongly than did self-reported mind wandering (r model = −.374, r self = −.208). Our results also indicate that a robust strategy of probabilistically predicting mind wandering in cases with poor or missing gaze data led to improved performance on all metrics, as compared to simply discarding these data. Our findings demonstrate that an automated objective measure might be available for laboratory studies of mind wandering during reading, providing an appealing alternative or complement to self-reports.

The last decade has witnessed considerable interest in the investigation of the affective dimensions of learning and in the development of advanced learning technologies that automatically detect and respond to student affect. Identifying the affective states that students experience in technology-enhanced learning contexts is a fundamental question in this area. This article provides an initial attempt to answer this question with a selective meta-analysis of 24 studies that utilized a mixture of methodologies (online self-reports, online observations, emote-aloud protocols, cued recall) and affect judges (students themselves, untrained peers, trained judges) for fine-grained monitoring of 14 discrete affective states of 1,740 middle school, high school, college, and adult students in 5 countries. Affective states occurred over the course of interactions with a range of learning technologies, including intelligent tutoring systems, serious games, simulation environments, and simple computer interfaces. Standardized effect sizes of relative frequency, computed by comparing the proportional occurrence of each affective state to the other states in each study, were modeled with random-effects models. Engagement/flow was consistently found to be relatively frequent (d+ = 2.5), and contempt, anger, disgust, sadness, anxiety, delight, fear, and surprise were consistently infrequent, with d+ ranging from −6.5 to −0.78. Effects for boredom (d+ = 0.19), confusion (d+ = 0.12), curiosity (d+ = −0.10), happiness (d+ = −0.13), and frustration (d+ = −2.5) varied substantially across studies. Mixed-effects models indicated that the source of the affect judgments (self vs. observers) and the authenticity of the learning contexts (classroom vs. laboratory) accounted for greater heterogeneity than the use of advanced learning technologies and training time. Theoretical and applied implications of the findings are discussed.

Assessing a person’s ability to multitask is a topic that is gaining increased attention. However, task constraints and difficulty rarely remain constant in real-world environments; when task constraints change, people must adapt to avoid diminished task performance or failure. But can we identify and predict differences in multitasking adaptability? This question was assessed in an experiment wherein participants multitasked in a flight simulator. Task difficulty was incrementally increased across three experimental manipulations. We measured participants' performance on tasks with baseline versus increased difficulty. Cluster analyses on performance identified three distinct adaptability groups in each condition, irrespective of performance at baseline. Furthermore, individual membership in each cluster was quite consistent across different difficulty conditions. Cluster membership in this task was predicted by spatial ability, which is a cognitive ability not related to general multitasking ability.

Objective: The aim of this study was to identify the cognitive factors that predictability and adaptability during multitasking with a flight simulator. Background: Multitasking has become increasingly prevalent as most professions require individuals to perform multiple tasks simultaneously. Considerable research has been undertaken to identify the characteristics of people (i.e., individual differences) that predict multitasking ability. Although working memory is a reliable predictor of general multitasking ability (i.e., performance in normal conditions), there is the question of whether different cognitive faculties are needed to rapidly respond to changing task demands ( adaptability). Method: Participants first completed a battery of cognitive individual differences tests followed by multitasking sessions with a flight simulator. After a baseline condition, difficulty of the flight simulator was incrementally increased via four experimental manipulations, and performance metrics were collected to assess multitasking ability and adaptability. Results: Scholastic aptitude and working memory predicted general multitasking ability (i.e., performance at baseline difficulty), but spatial manipulation (in conjunction with working memory) was a major predictor of adaptability (performance in difficult conditions after accounting for baseline performance). Conclusion: Multitasking ability and adaptability may be overlapping but separate constructs that draw on overlapping (but not identical) sets of cognitive abilities. Application: The results of this study are applicable to practitioners and researchers in human factors to assess multitasking performance in real-world contexts and with realistic task constraints. We also present a framework for conceptualizing multitasking adaptability on the basis of five adaptability profiles derived from performance on tasks with consistent versus increased difficulty.

Mind wandering (MW) is a ubiquitous phenomenon that has a negative influence on performance and productivity in many contexts. We propose that intelligent interfaces should have some mechanism to detect and respond to MW in real-time. Towards this end, we developed an interface that automatically detects MW from eye-gaze during computerized reading. When MW is detected, the interface intervenes by asking just-in-time questions and encouraging re-reading as needed. After multiple rounds of iterative refinement, we summatively compared the interface to a yoked control condition in a randomized control trial with 104 participants. Preliminary results suggest that the system was successful in correcting comprehension deficits attributed to MW, thereby highlighting the potential for intelligent interfaces that improve performance by "attending to attention."

The Fluid Events Model is aimed at predicting changes in the actions people take on a moment-by-moment basis. In contrast with other research on action selection, this work does not investigate why some course of action was selected, but rather the likelihood of discontinuing the current course of action and selecting another in the near future. This is done using both task-based and experience-based factors. Prior work evaluated this model in the context of trial-by-trial, independent, interactive events, such as choosing how to copy a figure of a line drawing. In this paper, we extend this model to more covert event experiences, such as reading narratives, as well as to continuous interactive events, such as playing a video game. To this end, the model was applied to existing data sets of reading time and event segmentation for written and picture stories. It was also applied to existing data sets of performance in a strategy board game, an aerial combat game, and a first person shooter game in which a participant’s current state was dependent on prior events. The results revealed that the model predicted behavior changes well, taking into account both the theoretically defined structure of the described events, as well as a person’s prior experience. Thus, theories of event cognition can benefit from efforts that take into account not only how events in the world are structured, but also how people experience those events.

This paper presents affect data collected from periodic emotion detection surveys throughout an introductory Statistics MOOC called "I Heart Stats." This is the first MOOC, to our knowledge, to capture valuable student affect data through self-reported surveys. To collect student affect, we used two self-reporting methods: (1) The Self-Assessment Manikin and (2) A discrete emotion list. We found that the most common reported MOOC emotion was Hope followed by Enjoyment and Contentment. There were substantial shifts in affective states over the course, notably with Anxiety and Pride. The most valuable result of our study is a preliminary description of the methods for collecting self-reported student affect at scale in a MOOC setting.

We present AutoTutor and Affective AutoTutor as examples of innovative 21 st century interactive intelligent systems that promote learning and engagement. AutoTutor is an intelligent tutoring system that helps students compose explanations of difficult concepts in Newtonian physics and enhances computer literacy and critical thinking by interacting with them in natural language with adaptive dialog moves similar to those of human tutors. AutoTutor constructs a cognitive model of students' knowledge levels by analyzing the text of their typed or spoken responses to its questions. The model is used to dynamically tailor the interaction toward individual students' zones of proximal development. Affective AutoTutor takes the individualized instruction and human-like interactivity to a new level by automatically detecting and responding to students' emotional states in addition to their cognitive states. Over 20 controlled experiments comparing AutoTutor with ecological and experimental controls such reading a textbook have consistently yielded learning improvements of approximately one letter grade after brief 30--60-minute interactions. Furthermore, Affective AutoTutor shows even more dramatic improvements in learning than the original AutoTutor system, particularly for struggling students with low domain knowledge. In addition to providing a detailed description of the implementation and evaluation of AutoTutor and Affective AutoTutor, we also discuss new and exciting technologies motivated by AutoTutor such as AutoTutor-Lite, Operation ARIES, GuruTutor, DeepTutor, MetaTutor, and AutoMentor. We conclude this article with our vision for future work on interactive and engaging intelligent tutoring systems.

Moment-to-moment emotions are affective states that dynamically change during reading and potentially influence comprehension. Researchers have recently identified these emotions and the emotion trajectories in reading, tutoring, and problem solving. The primary learning-centered emotions are boredom, frustration, confusion, flow (engagement), delight, surprise, and anxiety. Emotion transitions occur when the text becomes too difficult or easy for the reader and when conceptual obstacles create cognitive disequilibrium. Teachers and computer environments have the potential to improve reading comprehension by detecting and strategically handling the readers’ boredom, frustration, and confusion. One frontier in reading research is to understand the complex dance between comprehension and emotions.

Folk wisdom holds that being confused is detrimental to learning. However, research on emotions and learning suggest a somewhat more complex relationship between confusion and learning outcomes. In fact, it has been proposed that impasses that trigger states of cognitive disequilibrium and confusion can create opportunities for deep learning of conceptually difficult content. This paper discusses four computer learning environments that either naturally or artificially induce confusion in learners in order to create learning opportunities. First, an Intelligent Tutoring System called AutoTutor that engenders confusion through challenging problems and vague hints is described. The remaining three environments were specifically designed to induce confusion through a number of different interventions. These interventions include device breakdowns, contradictory information, and false feedback. The success and limitations of confusion induction and the impact of confusion resolution on learning are discussed. Potential methods to help learners productively manage their confusion instead of being hopelessly confused are also discussed.

Assessing silent reading fluency in classroom environments is challenging. This article reports on a method of assessing silent reading using underlining, an approach that solves many problems other silent reading fluency assessment measures face. This method computationally monitors readers' silent reading fluency by the speed they underline words in a text. Traditional silent reading fluency measures were compared with the new underlining methodology. Fourth- and sixth-grade students completed silent reading fluency measures (i.e., moving windows, underlining, and paper-and-pencil reading tasks), along with measures of their oral reading fluency, reading comprehension, and vocabulary knowledge. Strong alternate-form reliability coefficients were found for underlining, which significantly correlated with other measures of silent reading fluency and reading comprehension. Underlining methodology also correlated with common language factors, such as word length and word frequency. Together, these results provide support for the psychometric properties of underlining and suggest that it is a promising alternative method of assessing silent reading fluency. © 2012 Wiley Periodicals, Inc.