Search

One approach to studying the recognition of scenes and objects relies on the comparison of eye movement patterns during encoding and recognition. Past studies typically analyzed the perception of flat stimuli of limited extent presented on a computer monitor that did not require head movements. In contrast, participants in the present study saw omnidirectional panoramic scenes through an immersive 3D virtual reality viewer, and they could move their head freely to inspect different parts of the visual scenes. This allowed us to examine how unconstrained observers use their head and eyes to encode and recognize visual scenes. By studying head and eye movement within a fully immersive environment, and applying cross-recurrence analysis, we found that eye movements are strongly influenced by the content of the visual environment, as are head movements—though to a much lesser degree. Moreover, we found that the head and eyes are linked, with the head supporting, and by and large mirroring the movements of the eyes, consistent with the notion that the head operates to support the acquisition of visual information by the eyes.

Gaze behaviour is an important component of successful social interactions. Existing research on social gaze and attention has largely focused on gaze detection and following, rather than the two-way communicative component of gaze that operates between individuals. The present study sought to address this in two experiments. First, "hiders" were eye-tracked while they selected hiding places among a grid of boxes on a computer screen; these boxes were either homogeneous or contained a visually unique pop-out item. Importantly, sometimes hiders believed that their gaze would be seen by hypothetical "seekers" who they might wish to deceive or communicate truthful information to; and sometime hiders believed that their gaze would be concealed. In a second experiment, seekers were asked to select the hiders' locations after viewing the hiders' gaze behaviour, including the eye movements that hiders had been (falsely) told would be concealed. Results indicate that seekers are most accurate when hiders use their gaze to truthfully communicate their selected locations and least accurate when hiders aim to deceive. Notably, both communication and interpretation strategies were affected by the visual display type (e.g., hiders looked to and preferentially selected pop-out items when communicating truthfully while seekers interpreted gaze differently when allocated to these pop-out items), indicating that the visual context can be integrated with gaze to facilitate mis/communication. Our study illuminates how the gaze of an individual acquires and signals information, and that individuals will spontaneously adjust the balance between these two functions based on their current goal and visual environment.

It has long been thought that visual perception is represented in sensorimotor processes that unfold over time. One prominent theory predicts that our memory for a scene consists of both the scene content and the motor commands (i.e., eye movements) used to explore that scene. This Scanpath Theory (Noton & Stark, Science 171 (1971) 308-311) has long been contested, with many studies providing evidence both for, and against it. That past work, however, has failed to account for the fact that visual perception is embodied within an active system of effectors, namely, that people routinely move both their eyes and head to explore visible space. In the present work we tested Scanpath Theory while observers were free to move within a 360-degree VR environment. Their task was to encode and later recognise panoramic scenes within this fully immersive world. During both encoding and recognition, we recorded their eye and head movements using a VR headset equipped with eye and head tracking. Our results reveal that eye and head movement patterns are diagnostic of memory performance; and that scene recognition improves when certain movements that had occurred during encoding are repeated. Finally, including head movement measures enhances performance prediction, strengthening the evidence for Scanpath Theory, and reinforcing the fact that the head moves in service of the eyes in allocating attention.

During scene viewing, semantic information in the scene has been shown to play a dominant role in guiding fixations compared to visual salience (e.g., HendersonHayes, 2017). However, scene viewing is sometimes disrupted by cognitive processes unrelated to the scene. For example, viewers sometimes engage in mind-wandering, or having thoughts unrelated to the current task. How do meaning and visual salience account for fixation allocation when the viewer is mind-wandering, and does it differ from when the viewer is on-task? We asked participants to study a series of real-world scenes in preparation for a later memory test. Thought probes occasionally occurred after a subset of scenes to assess whether participants were on-task or mind-wandering. We used salience maps (Graph-Based Visual Saliency; Harel, Koch,Perona, 2007) and meaning maps (HendersonHayes, 2017) to represent the distribution of visual salience and semantic richness in the scene, respectively. Because visual salience and meaning were represented similarly, we could directly compare how well they predicted fixation allocation. Our results indicate that fixations prioritized meaningful over visually salient regions in the scene during mind-wandering just as during attentive viewing. These results held across the entire viewing time. A re-analysis of an independent study (Krasich, Huffman, Faber,Brockmole Journal of Vision, 20(9), 10, 2020) showed similar results. Therefore, viewers appear to prioritize meaningful regions over visually salient regions in real-world scenes even during mind-wandering.

Recent debates in the literature discuss commonalities between Attention-Deficit/Hyperactivity Disorder (ADHD) and Autism Spectrum Disorder (ASD) at multiple levels of putative causal networks. This debate requires systematic comparisons between these disorders that have been studied in isolation in the past, employing potential markers of each disorder to be investigated in tandem. The present study, choose superior local processing, typical to ASD, and increased Intra-Subject Variability (ISV), typical to ADHD, for a head-to-head comparison of the two disorders, while also considering the comorbid cases. It directly examined groups of participants aged 10–13 years with ADHD, ASD with (ASD+) or without (ASD−) comorbid ADHD and a typically developing (TD) group (total N = 85). A visual search task consisting of an array of paired words was designed. The participants needed to find the specific pair of words, where the first word in the pair was the cue word. This visual search task was selected to compare these groups on overall search performance and trial-to-trial variability of search performance (i.e., ISV). Additionally, scanpath analysis was also carried out using Recurrence Quantification Analysis (RQA) and the Multi-Match Model. Results show that only the ASD− group exhibited superior search performance; whereas, only the groups with ADHD symptoms showed increased ISV. These findings point towards a double dissociation between ASD and ADHD, and argue against an overlap between ASD and ADHD.

People naturally move both their head and eyes to attend to information. Yet, little is known about how the head and eyes coordinate in attentional selection due to the relative sparsity of past work that has simultaneously measured head and gaze behaviour. In the present study, participants were asked to view fully immersive 360-degree scenes using a virtual reality headset with built-in eye tracking. Participants viewed these scenes through a small moving window that was yoked either to their head or gaze movements. We found that limiting peripheral information via the head- or gaze-contingent windows affected head and gaze movements differently. Compared with free viewing, gaze-contingent viewing was more disruptive than head-contingent viewing, indicating that gaze-based selection is more reliant on peripheral information than head-based selection. These data dovetail with the nested effectors hypothesis, which proposes that people prefer to use their head for exploration into non-visible space while using their eyes to exploit visible or semi-visible areas of space. This suggests that real-world orienting may be more head-based than previously thought. Our work also highlights the utility, ecological validity, and future potential of unconstrained head and eye tracking in virtual reality.

During scene viewing, semantic information in the scene has been shown to play a dominant role in guiding fixations compared to visual salience (e.g., Henderson & Hayes, 2017). However, scene viewing is sometimes disrupted by cognitive processes unrelated to the scene. For example, viewers sometimes engage in mind-wandering, or having thoughts unrelated to the current task. How do meaning and visual salience account for fixation allocation when the viewer is mind-wandering, and does it differ from when the viewer is on-task? We asked participants to study a series of real-world scenes in preparation for a later memory test. Thought probes occasionally occurred after a subset of scenes to assess whether participants were on-task or mind-wandering. We used salience maps (Graph-Based Visual Saliency; Harel, Koch, & Perona, 2007) and meaning maps (Henderson & Hayes, 2017) to represent the distribution of visual salience and semantic richness in the scene, respectively. Because visual salience and meaning were represented similarly, we could directly compare how well they predicted fixation allocation. Our results indicate that fixations prioritized meaningful over visually salient regions in the scene during mind-wandering just as during attentive viewing. These results held across the entire viewing time. A re-analysis of an independent study (Krasich, Huffman, Faber, & Brockmole Journal of Vision, 20(9), 10, 2020) showed similar results. Therefore, viewers appear to prioritize meaningful regions over visually salient regions in real-world scenes even during mind-wandering.

While passive social information (e.g. pictures of people) routinely draws one's eyes, our willingness to look at live others is more nuanced. People tend not to stare at strangers and will modify their gaze behaviour to avoid sending undesirable social signals; yet they often continue to monitor others covertly “out of the corner of their eyes.” What this means for looks that are being made near to live others is unknown. Will the eyes be drawn towards the other person, or pushed away? We evaluate changes in two elements of gaze control: image-independent principles guiding how people look (e.g. biases to make eye movements along the cardinal directions) and image-dependent principles guiding what people look at (e.g. a preference for meaningful content within a scene). Participants were asked to freely view semantically unstructured (fractals) and semantically structured (rotated landscape) images, half of which were located in the space near to a live other. We found that eye movements were horizontally displaced away from a visible other starting at 1032 ms after stimulus onset when fractals but not landscapes were viewed. We suggest that the avoidance of looking towards live others extends to the near space around them, at least in the absence of semantically meaningful gaze targets.

How do we explore the visual environment around us, and how are head and eye movements coordinated during our exploration? To investigate this question, we had observers look at omni-directional panoramic scenes, composed of both landscape and fractal images, using a virtual-reality (VR) viewer while their eye and head movements were tracked. We analyzed the spatial distribution of eye fixations and the distribution of saccade directions; the spatial distribution of head positions and the distribution of head shifts; as well as the relation between eye and head movements. The results show that, for landscape scenes, eye and head behaviour best fit the allocentric frame defined by the scene horizon, especially when head tilt (i.e., head rotation around the view axis) is considered. For fractal scenes, which have an isotropic texture, eye and head movements were executed primarily along the cardinal directions in world coordinates. The results also show that eye and head movements are closely linked in space and time in a complementary way, with stimulus-driven eye movements predominantly leading the head movements. Our study is the first to systematically examine eye and head movements in a panoramic VRenvironment, and the results demonstrate that a VR environment constitutes a powerful and informative research alternative to traditional methods for investigating looking behaviour.

When interacting with other humans, facial expressions provide valuable information for approach or avoid decisions. Here, we consider facial attractiveness as another important dimension upon which approach-avoidance behaviours may be based. In Experiments 1–3, we measured participants' responses to attractive and unattractive women's faces in an approach-avoidance paradigm in which there was no explicit instruction to evaluate facial attractiveness or any other stimulus attribute. Attractive faces were selected more often, a bias that may be sensitive to response outcomes and was reduced when the faces were inverted. Experiment 4 explored an entirely implicit measure of approach, with participants passively viewing single faces while standing on a force platform. We found greater lean towards attractive faces, with this pattern being most obvious in male participants. Taken together, these results demonstrate that attractiveness activates approach-avoidance tendencies, even in the absence of any task demand.

This chapter is aimed at introducing the reader to current methods for the spatial and temporal analysis of eye movements. There are four main parts. In the first part of the chapter, we introduce the relation between attention and eye movements and then review the foundations of attention research, social attention and the effect of stimulus saliency on attention. In the second part, we first review traditional characterizations of eye movements, specifically measures of fixations and saccades, and then we review the major methods for the spatial analysis of eye movements, including heat maps and area-of-interest analyses. In the third part, we introduce a new temporal analysis of eye movements, recurrence quantification analysis. We review the basic methods and measures of this analysis, and we discuss several applications. In the last part of the chapter, we provide an overview of methods for the comparison of scanpaths and show that recurrence quantification analysis can be generalized successfully to scanpath comparisons.

Rich contextual and semantic information can be extracted from only a brief presentation of a natural scene. This is presumed to be activated quickly enough to guide initial eye movements into a scene. However, early, short-latency eye movements in natural scenes have been shown to be dependent on the salience distribution across the image (Anderson, Ort, Kruijne, Meeter, & Donk, 2015). In the present work, we manipulated the salience distribution across a natural scene by changing the global contrast. We showed participants a brief real or nonsense preview of the scene and examined the time-course of eye movement guidance. A real preview decreased the latency and increased the amplitude of initial saccades into the image, suggesting that the preview allowed observers to obtain additional contextual information that would otherwise not be available. However, the preview did not completely override the initial tendency for short-latency saccades to be guided by the underlying salience distribution of the image. We discuss these findings in the context of oculomotor selection based on the integration of contextual information and low-level features in a natural scene.

How do we explore the visual environment around us, and how are head and eye movements coordinated during our exploration? To investigate this question, we had observers look at omnidirectional panoramic scenes, composed of both landscape and fractal images, using a virtual reality viewer while their eye and head movements were tracked. We analyzed the spatial distribution of eye fixations and the distribution of saccade directions and the spatial distribution of head positions and the distribution of head shifts, as well as the relation between eye and head movements. The results show that, for landscape scenes, eye and head behavior best fit the allocentric frame defined by the scene horizon, especially when head tilt (i.e., head rotation around the view axis) is considered. For fractal scenes, which have an isotropic texture, eye and head movements were executed primarily along the cardinal directions in world coordinates. The results also show that eye and head movements are closely linked in space and time in a complementary way, with stimulus-driven eye movements predominantly leading the head movements. Our study is the first to systematically examine eye and head movements in a panoramic virtual reality environment, and the results demonstrate that a virtual reality environment constitutes a powerful and informative research alternative to traditional methods for investigating looking behavior.

In two experiments, we investigated the role of apparent motion in discriminating left/right gaze deviation judgments. We demonstrated that discrimination accuracy and response confidence was significantly higher when the eyes were moved to the left or right, compared to when the eyes were presented in their final shifted position (static images). To dissociate the role of motion signals from luminance signals, gaze stimuli were also presented in reverse contrast. Replicating past studies polarity reversal had a profound and detrimental effect on gaze discrimination in static images, although, intriguingly, while response confidence remained low, participant performance improved as gaze angle increased. In striking contrast to these data, polarity reversal had no negative effect on performance when the eyes were moved. We discuss these findings in the context of a multiple-cue account of gaze perception.

In the present study, we examined the eye movement behaviour of children and adults looking at five Van Gogh paintings in the Van Gogh Museum, Amsterdam. The goal of the study was to determine the role of top-down and bottom-up attentional processes in the first stages of participants’ aesthetic experience. Bottom-up processes were quantified by determining a salience map for each painting. Top-down processing was manipulated by first allowing participants to view the paintings freely, then providing background information about each painting, and then allowing them to view the paintings a second time. The salience analysis showed differences between the eye movement behaviour of children and adults, and differences between the two phases. In the children, the first five fixations during the free viewing phase were strongly related to visually salient features of the paintings—indicating a strong role for bottom-up factors. In the second phase, after children had received background information, top-down factors played a more prominent role. By contrast, adults’ observed patterns were similar in both phases, indicating that bottom-up processes did not play a major role when they viewed the paintings. In the second phase, children and adults both spent more time looking at regions that were mentioned in the background information. This effect was greater for adults than for children, confirming the notion that adults, when viewing paintings, rely much more on top-down processing than children.

A change to an object in natural scenes attracts attention when it occurs during a fixation. However, when a change occurs during a saccade, and is masked by saccadic suppression, it typically does not capture the gaze in a bottom-up manner. In the present work, we investigated how the type and direction of salient changes to objects affect the prioritization and targeting of objects in natural scenes. We asked observers to look around a scene in preparation for a later memory test. After a period of time, an object in the scene was increased or decreased in salience either during a fixation (with a transient signal) or during a saccade (without transient signal), or it was not changed at all. Changes that were made during a fixation attracted the eyes both when the change involved an increase and a decrease in salience. However, changes that were made during a saccade only captured the eyes when the change was an increase in salience, relative to the baseline no-change condition. These results suggest that the prioritization of object changes can be influenced by the underlying salience of the changed object. In addition, object changes that occurred with a transient signal (which is itself a salient signal) resulted in more central object targeting. Taken together, our results suggest that salient signals in a natural scene are an important component in both object prioritization and targeting in natural scene viewing, insofar as they align with object locations.

Interest has flourished in studying both the spatial and temporal aspects of eye movement behavior. This has sparked the development of a large number of new methods to compare scanpaths. In the present work, we present a detailed overview of common scanpath comparison measures. Each of these measures was developed to solve a specific problem, but quantifies different aspects of scanpath behavior and requires different data-processing techniques. To understand these differences, we applied each scanpath comparison method to data from an encoding and recognition experiment and compared their ability to reveal scanpath similarities within and between individuals looking at natural scenes. Results are discussed in terms of the unique aspects of scanpath behavior that the different methods quantify. We conclude by making recommendations for choosing an appropriate scanpath comparison measure.