Search

Here we used deep neural network + hidden Markov model (DNN+HMM) to provide a computational account for the relationship among holistic processing (HP), face scanning pattern and face recognition performance. The model accounted for the positive associations between HP and eyes-focused face scanning pattern/face recognition performance observed in the literature regardless of the version of the composite task used to measure HP. Interestingly, we observed a quadratic relationship between HP and face scanning pattern, where models being highly eyes-focused or highly nose-focused had lower HP. By inspecting fixation locations and associated attention window size in the model and XAI methods, we found that the eyes- and nose-focused models both developed local and holistic internal representations during training, and their difference was in the temporal dynamics of how these representations were used. Our findings demonstrated how computational modeling could unravel the mechanisms underlying cognition not readily observable in human data.

Large language models (LLMs) were trained to predict words without having explicit semantic word representations as humans do. Here we compared LLMs and humans in re-solving semantic ambiguities at the word/token level by ex-amining the case of segmenting overlapping ambiguous strings in Chinese sentence reading, where three characters “ABC” could be segmented in either “AB/C” or “A/BC” depending on the context. We showed that although LLMs performed worse than humans, they demonstrated a similar interaction effect between segmentation structure and word frequency order, suggesting that this effect observed in humans could be accounted for by statistical learning of word/token occurrence regularities without assuming an explicit semantic word representation. Nevertheless, across stimuli LLMs' responses were not correlated with any hu-man performance or eye movement measures, suggesting differences in the underlying processing mechanisms. Thus, it is essential to understand these differences through XAI methods to facilitate LLM adoption.

Here we present a new Explainable AI (XAI) method to probe the functional partition in AI models by comparing features attended to at different layers with human attention driven by diverse task demands. We applied this method to explain an object detector Yolo-v5s in multi-category and single-category object detection tasks. We found that the model's neck showed higher similarity to human attention during object detection, indicating a reliance on diagnostic features in the neck, whereas its backbone showed higher similarity to attention during passive viewing, indicating salient local features encoded. With this understanding of its functional partition, using Yolo-v5s as a model for human cognition, our comparative analysis against human attention when providing explanations for object detection revealed that humans attended to a combination of diagnostic and salient features during explaining multi-category general object detection but attended to mainly diagnostic features when explaining single-category human/vehicle detection in driving scenarios.

The recent surge in online learning demands better ways to monitor students' mind wandering (MW) episodes. We examined whether different eye movement measures were associated with MW in tasks with different cognitive demands. We found that a reduced number of fixations was associated with MW in tasks involving searching for information without clearly defined strategies. A larger variance in pupil diameter, as well as reduced eye movement consistency, were associated with MW when imagining a scenario with a central fixation. Reduced eye movement consistency, as well as reduced joint attention with another participant, were both associated with MW in tasks involving a clearly defined strategy. Interestingly, none of these eye movement measures was associated with MW in tasks involving well-learned visual routines such as face and scene identification, suggesting idiosyncrasy in eye movement behavior in these tasks. These findings have important implications for developing effective methods for detecting MW.

Although object detection AI plays an important role in many critical systems, corresponding Explainable AI (XAI) methods remain very limited. Here we first developed FullGrad-CAM and FullGrad-CAM++ by extending traditional gradient-based methods to generate object-specific explanations with higher plausibility. Since human attention may reflect features more in-terpretable to humans, we explored the possibility to use it as guidance to learn how to combine the explanatory information in the detector model to best present as an XAI saliency map that is interpretable (plausible) to humans. Interestingly, we found that human attention maps had higher faithfulness for explaining the detector model than existing saliency-based XAI methods. By using trainable activation functions and smoothing kernels to maximize the XAI saliency map similarity to human attention maps, the generated map had higher faithfulness and plausibility than both existing XAI methods and human atten-tion maps. The learned functions were model-specific, well generalizable to other databases.

Here we examined whether eye movement measures in global-local perceptual processing tasks, where abnormalities were typically found in individuals with schizophrenia, could be used to predict schizotypy through Eye Movement analysis with Hidden Markov Models (EMHMM). Using both multiple regression analysis and Gaussian process classifier to predict schizotypy, we found that in addition to longer response times in contour integration, a less consistent eye fixation to locate a stimulus and a more consistent subsequent fixation to start engaging local processing in the embedded figures task predicted high schizotypy. These effects may be related to reduced top-down attention control due to deficient global processing and enhanced local processing bias respectively. In addition, performance in embedded figures could further enhance classification accuracy when being used in conjunction with the above predictors, suggesting the multifactorial nature of the identification problem. These predictors may be important endophenotype markers for schizotypal personality.

In face recognition, eye gaze to the eye region is reported tobe associated with better performance than to the center of aface. Nevertheless, Caucasians and Asians differ in how muchthey look at the eyes when they scan a face, but havecomparable identification performance. To resolve this issue,here we test the hypothesis that optimal face recognitionperformance involves a balance between global and local faceprocessing. Thus, Asians may benefit from enhancement oflocal processing and vice versa for Caucasians. We showedthat local attention priming using hierarchical letter stimuliled to more eye-focused eye movement patterns compared toglobal attention priming in both Asians and Caucasians.However, Asians had better performance after local primingthan global priming, whereas Caucasian showed the oppositeeffect. These results suggest that engagement of global/localattention leads to face-center/eye biased eye movementsrespectively, and optimal recognition performance involvesboth global and local processing/gaze transitions between theface center and eyes.

Insomniacs were found to have compromised perception offacial expressions. Through eye movement examinations,here we test the hypothesis that this effect is due to impairedvisual attention functions for retrieving diagnostic features infacial expression judgments. 23 individuals with insomniasymptoms and 23 non-insomniac controls completed a task tocategorize happy, sad, fearful, and angry faces. Theinsomniacs were less accurate to recognize angry faces andmade more “fearful” mistakes than controls. A hiddenMarkov modeling approach for eye movement data analysisrevealed that when recognizing angry faces, more insomniacsadopted an eye movement pattern focusing on the mouthwhile more controls adopted a pattern attending to both theeyes and the mouth. This result is consistent with previousfindings that the primary diagnostic feature for recognizingangry faces is the eyes suggesting that impaired informationselection through visual attention control may account for thecompromised emotion perception in insomniac individuals.

Here we used a hidden Markov model (HMM) based ap-proach to infer individual choices from eye movements inpreference decision-making. We assumed that during a deci-sion making process, participants may switch between explo-ration and decision-making periods, and this behavior can bebetter captured with a Switching HMM (SHMM). Throughclustering individual eye movement patterns described inSHMMs, we automatically discovered two groups of partici-pants with different decision making behavior. One groupshowed a strong and early bias to look more often at the to-bechosen stimulus (i.e., the gaze cascade effect; Shimojo et al.,2003) with a short final decision-making period. The othergroup showed a weaker cascade effect with a longer final de-cision-making period. The SHMMs also showed capable ofinferring participants’ preference choice on each trial withhigh accuracy. Thus, our SHMM approach made it possible toreveal individual differences in decision making and discoverindividual preferences from eye movement data.

Recent research has revealed two different eye movement pat-terns during face recognition: holistic and analytic. The pre-sent study investigated the neural correlates of these two pat-terns through functional magnetic resonance imaging (fMRI).A more holistic pattern was associated with more activation inthe face-selective perceptual areas, including the occipitalface area and fusiform face area. In contrast, participants us-ing a more analytic pattern demonstrated more activation inareas important for top-down control of visual attention, in-cluding the frontal eye field and intraparietal sulcus. In addi-tion, participants using the analytic patterns had better recog-nition performance than those showing holistic patterns. The-se results suggest that analytic eye movement patterns are as-sociated with more engagement of top-down control of visualattention, which may consequently enhance recognition per-formance.

Hidden Markov models (HMM) can describe the spatial andtemporal characteristics of eye-tracking recordings incognitive tasks. Here, we introduce a new HMM approach.We developed HMMs based on fixation locations and we alsoused image information as an input feature. We demonstratethe benefits of the newly proposed model in a facerecognition study wherein an HMM was developed for everysubject. Discovery of regions of interest on facial stimuli isimproved as compared with earlier approaches. Moreover,clustering of the newly developed HMMs lead to very distinctgroups. The newly developed approach also allowsreconstructing image information at each fixation.

The present study investigated the relationship between eyemovement pattern in face recognition and cognitive performanceduring natural aging through modeling and comparingeye movement of young (18-24 years) and older (65-81 years)adults using Hidden Markov Model (HMM) based approach.Young adults recognized faces better than older adults, particularlywhen measured by the false alarm rate. Older adults’recognition performance, on the other hand, correlated withtheir cognitive status assessed by the Montreal Cognitive Assessment(MoCA). Eye movement analysis with HMM revealedtwo different strategies, namely “analytic” and “holistic”.Participants using the analytic strategy had better recognitionperformance (particularly in the false alarm rate) thanthose using the holistic strategy. Significantly more youngadults adopted the analytic strategy; whereas more olderadults adopted the holistic strategy. Interestingly, older adultswith lower cognitive status were associated with higher likelihoodof using the holistic strategy. These results suggest anassociation between holistic eye movement patterns and cognitivedecline in the elderly.

Here we explored eye movement strategies that lead to betterperformance in face recognition with hidden Markov models(HMMs). Participants performed a standard face recognitionmemory task with eye movements recorded. The durationsand locations of the fixations were analyzed using HMMs forboth the study and the test phases. Results showed that in thestudy phase, the participants who looked more often at theeyes and shifted between different regions on the face withlong fixation durations had better performances. The testphase analyses revealed that an efficient, short first orientingfixation followed by a more analytic pattern focusing mainlyon the eyes led to better performances. These strategies couldnot be revealed by analysis methods that do not take individualdifferences in both temporal and spatial dimensions of eyemovements into account, demonstrating the power of theHMM approach.

Adynamic texture is a spatio-temporal generative model for video, which represents video sequences as observations from a linear dynamical system. This work studies the mixture of dynamic textures, a statistical model for an ensemble of video sequences that is sampled from a finite collection of visual processes, each of which is a dynamic texture. An expectation-maximization (EM) algorithm is derived for learning the parameters of the model, and the model is related to previous works in linear systems, machine learning, time-series clustering, control theory, and computer vision. Through experimentation, it is shown that the mixture of dynamic textures is a suitable representation for both the appearance and dynamics of a variety of visual processes that have traditionally been challenging for computer vision (for example, fire, steam, water, vehicle and pedestrian traffic, and so forth). When compared with state-of-the-art methods in motion segmentation, including both temporal texture methods and traditional representations (for example, optical flow or other localized motion representations), the mixture of dynamic textures achieves superior performance in the problems of clustering and segmenting video of such processes.

A probabilistic formulation for semantic image annotation and retrieval is proposed. Annotation and retrieval are posed as classification problems where each class is defined as the group of database images labeled with a common semantic label. It is shown that, by establishing this one-to-one correspondence between semantic labels and semantic classes, a minimum probability of error annotation and retrieval are feasible with algorithms that are 1) conceptually simple, 2) computationally efficient, and 3) do not require prior semantic segmentation of training images. In particular, images are represented as bags of localized feature vectors, a mixture density estimated for each image, and the mixtures associated with all images annotated with a common semantic label pooled into a density estimate for the corresponding semantic class. This pooling is justified by a multiple instance learning argument and performed efficiently with a hierarchical extension of expectation-maximization. The benefits of the supervised formulation over the more complex, and currently popular, joint modeling of semantic label and visual feature distributions are illustrated through theoretical arguments and extensive experiments. The supervised formulation is shown to achieve higher accuracy than various previously published methods at a fraction of their computational cost. Finally, the proposed method is shown to be fairly robust to parameter tuning.