Search

Animals benefit from knowing if and how they are moving. Across the animal kingdom, sensory information in the form of optic flow over the visual field is used to estimate self-motion. However, different species exhibit strong spatial biases in how they use optic flow. Here, we show computationally that noisy natural environments favor visual systems that extract spatially biased samples of optic flow when estimating self-motion. The performance associated with these biases, however, depends on interactions between the environment and the animal's brain and behavior. Using the larval zebrafish as a model, we recorded natural optic flow associated with swimming trajectories in the animal's habitat with an omnidirectional camera mounted on a mechanical arm. An analysis of these flow fields suggests that lateral regions of the lower visual field are most informative about swimming speed. This pattern is consistent with the recent findings that zebrafish optomotor responses are preferentially driven by optic flow in the lateral lower visual field, which we extend with behavioral results from a high-resolution spherical arena. Spatial biases in optic-flow sampling are likely pervasive because they are an effective strategy for determining self-motion in noisy natural environments.

A current limitation in our understanding of the visual system is its function under natural viewing conditions, especially in the context of dynamic, human behavior.While there have been many advances in understanding the spatial response properties of visual neurons in relation to static stimuli such as natural images, understanding of the corresponding temporal properties has been limited by the lack of high-fidelity datasets that document the properties of the signal that reaches the human retina under natural conditions. In this thesis, I describe the design and construction of a mobile tracking device that can re-create the signal present on the retina of a human as they perform everyday tasks. This device is used to collect high fidelity video and tracking data from human subjects performing a set of tasks that sample the everyday human environment and behavioral repertoire. This new dataset makes it possible to characterize the spatiotemporal statistics of natural time-varying signals as they occur on the retina. Here I examine the spatio-temporal power spectrum, which is of interest as a natural scene statistic in part because it is the Fourier transform of the autocorrelation function. In the absence of ego motion (movement of head and body), the spatiotemporal power spectrum of the dynamic environment has similar power-law structure to that previously reported for Hollywood movies. Head and eye motion modulate the spatiotemporal signal, boosting mid- and high-range temporal frequencies, such that the visual input on the retina is nearly whitened. This can be beneficial for reducing signal redundancy and maximizing the use of available bandwidth in the optic nerve. The phase spectrum, which compliments the power spectrum, also carries relevant information about natural image statistics. Despite the strong perceptual signal carried by the classically defined global phase, I show that it has limited utility to differentiate natural images from noise. However

Acute sleep loss influences visual processes in humans, such as recognizing facial emotions. However, to the best of our knowledge, no study till date has examined whether acute sleep loss alters visual comfort when looking at images. One image statistic that can be used to investigate the level of visual comfort experienced under visual encoding is the slope of the amplitude spectrum, also referred to as the slope constant. The slope constant describes the spatial distribution of pixel intensities and deviations from the natural slope constant can induce visual discomfort. In the present counterbalanced crossover design study, 11 young men with normal or corrected-to-normal vision participated in two experimental conditions: one night of sleep loss and one night of sleep. In the morning after each intervention, subjects performed a computerized psychophysics task. Specifically, they were required to adjust the slope constant of images depicting natural landscapes and close-ups with a randomly chosen initial slope constant until they perceived each image as most natural looking. Subjects also rated the pleasantness of each selected image. Our analysis showed that following sleep loss, higher slope constants were perceived as most natural looking when viewing images of natural landscapes. Images with a higher slope constant are generally perceived as blurrier. The selected images were also rated as less pleasant after sleep loss. No such differences between the experimental conditions were noted for images of close-ups. The results suggest that sleep loss induces signs of visual discomfort in young men. Possible implications of these findings are discussed.

Acute sleep loss influences visual processes in humans, such as recognizing facial emotions. However, to the best of our knowledge, no study till date has examined whether acute sleep loss alters visual comfort when looking at images. One image statistic that can be used to investigate the level of visual comfort experienced under visual encoding is the slope of the amplitude spectrum, also referred to as the slope constant. The slope constant describes the spatial distribution of pixel intensities and deviations from the natural slope constant can induce visual discomfort. In the present counterbalanced crossover design study, 11 young men with normal or corrected-to-normal vision participated in two experimental conditions: one night of sleep loss and one night of sleep. In the morning after each intervention, subjects performed a computerized psychophysics task. Specifically, they were required to adjust the slope constant of images depicting natural landscapes and close-ups with a randomly chosen initial slope constant until they perceived each image as most natural looking. Subjects also rated the pleasantness of each selected image. Our analysis showed that following sleep loss, higher slope constants were perceived as most natural looking when viewing images of natural landscapes. Images with a higher slope constant are generally perceived as blurrier. The selected images were also rated as less pleasant after sleep loss. No such differences between the experimental conditions were noted for images of close-ups. The results suggest that sleep loss induces signs of visual discomfort in young men. Possible implications of these findings are discussed.

Acute sleep loss influences visual processes in humans, such as recognizing facial emotions. However, to the best of our knowledge, no study till date has examined whether acute sleep loss alters visual comfort when looking at images. One image statistic that can be used to investigate the level of visual comfort experienced under visual encoding is the slope of the amplitude spectrum, also referred to as the slope constant. The slope constant describes the spatial distribution of pixel intensities and deviations from the natural slope constant can induce visual discomfort. In the present counterbalanced crossover design study, 11 young men with normal or corrected-to-normal vision participated in two experimental conditions: one night of sleep loss and one night of sleep. In the morning after each intervention, subjects performed a computerized psychophysics task. Specifically, they were required to adjust the slope constant of images depicting natural landscapes and close-ups with a randomly chosen initial slope constant until they perceived each image as most natural looking. Subjects also rated the pleasantness of each selected image. Our analysis showed that following sleep loss, higher slope constants were perceived as most natural looking when viewing images of natural landscapes. Images with a higher slope constant are generally perceived as blurrier. The selected images were also rated as less pleasant after sleep loss. No such differences between the experimental conditions were noted for images of close-ups. The results suggest that sleep loss induces signs of visual discomfort in young men. Possible implications of these findings are discussed.

Images derived from a "virtual phantom" can be useful in characterizing the performance of imaging systems. This has driven the development of virtual breast phantoms implemented in simulation environments. In breast imaging, several such phantoms have been proposed. We analyze the non-Gaussian statistical properties from three classes of virtual breast phantoms and compare them to similar statistics from a database of breast images. These include clustered-blob lumpy backgrounds (CBLBs), truncated binary textures, and the UPenn virtual breast phantoms. We use Laplacian fractional entropy (LFE) as a measure of the non-Gaussian statistical properties of each simulation procedure. Our results show that, despite similar power spectra, the simulation approaches differ considerably in LFE with very low scores for the CBLB to high values for the UPenn phantom at certain frequencies. These results suggest that LFE may have value in developing and tuning virtual phantom simulation procedures.

Acute sleep loss influences visual processes in humans, such as recognizing facial emotions. However, to the best of our knowledge, no study till date has examined whether acute sleep loss alters visual comfort when looking at images. One image statistic that can be used to investigate the level of visual comfort experienced under visual encoding is the slope of the amplitude spectrum, also referred to as the slope constant. The slope constant describes the spatial distribution of pixel intensities and deviations from the natural slope constant can induce visual discomfort. In the present counterbalanced crossover design study, 11 young men with normal or corrected-to-normal vision participated in two experimental conditions: one night of sleep loss and one night of sleep. In the morning after each intervention, subjects performed a computerized psychophysics task. Specifically, they were required to adjust the slope constant of images depicting natural landscapes and close-ups with a randomly chosen initial slope constant until they perceived each image as most natural looking. Subjects also rated the pleasantness of each selected image. Our analysis showed that following sleep loss, higher slope constants were perceived as most natural looking when viewing images of natural landscapes. Images with a higher slope constant are generally perceived as blurrier. The selected images were also rated as less pleasant after sleep loss. No such differences between the experimental conditions were noted for images of close-ups. The results suggest that sleep loss induces signs of visual discomfort in young men. Possible implications of these findings are discussed.

Acute sleep loss influences visual processes in humans, such as recognizing facial emotions. However, to the best of our knowledge, no study till date has examined whether acute sleep loss alters visual comfort when looking at images. One image statistic that can be used to investigate the level of visual comfort experienced under visual encoding is the slope of the amplitude spectrum, also referred to as the slope constant. The slope constant describes the spatial distribution of pixel intensities and deviations from the natural slope constant can induce visual discomfort. In the present counterbalanced crossover design study, 11 young men with normal or corrected-to-normal vision participated in two experimental conditions: one night of sleep loss and one night of sleep. In the morning after each intervention, subjects performed a computerized psychophysics task. Specifically, they were required to adjust the slope constant of images depicting natural landscapes and close-ups with a randomly chosen initial slope constant until they perceived each image as most natural looking. Subjects also rated the pleasantness of each selected image. Our analysis showed that following sleep loss, higher slope constants were perceived as most natural looking when viewing images of natural landscapes. Images with a higher slope constant are generally perceived as blurrier. The selected images were also rated as less pleasant after sleep loss. No such differences between the experimental conditions were noted for images of close-ups. The results suggest that sleep loss induces signs of visual discomfort in young men. Possible implications of these findings are discussed.

The ability of human visual systems to estimate 3D surface orientation from 2D retinal images is critical. But the computation to calculate 3D orientation in real-world scenes is not fully understood. A Bayes optimal model grounded in natural statistics has explained 3D surface tilt estimation of human observers in natural scenes (Kim and Burge, 2018). However, the model is limited because it estimates only unsigned tilt (tilt modulo 180deg). We extend the model to predict signed tilt estimates and compared with human signed estimates. The model takes image pixels as input and produces optimal estimates of tilt as output, using the joint statistics of tilt and image cues in natural scenes. The image cues to tilt are the directions of luminance, texture, and disparity gradients in a local area on the image. To estimate signed tilt, the disparity cue is used as a signed tilt cue, and the luminance and texture cues are used as unsigned tilt cues. Given a particular set of local image cues, the model computes the minimum mean squared error (MMSE) estimate, which is equal to the posterior mean over signed tilt. We found that the signed MMSE estimates were well aligned with human signed tilt estimates on the identical set of stimuli. Next, we pooled the local MMSE estimates across the space to obtain a global tilt estimate. Given that local MMSE estimates are unbiased predictor of groundtruth tilt with nearly equal reliability, the global pooled estimates are also near-optimal. The global estimates even better explained human tilt estimation. We conclude that this computational model provides a tool to understand how human visual systems make the best use of 2D image information to compute local estimates and integrate a global estimate of 3D surface tilt in complex natural scenes using the local estimates.

Natural scenes vary considerably — for instance beaches, forests, and canyons do not share obvious perceptual similarities. However, across natural scenes a common distribution of amplitude variations exists across both spatial and temporal scales called the 1/fα amplitude spectrum (α ≈ 0.6 — 1.6). This property is also considered to underlie the scale-invariant, fractal properties of natural scenes. Preferential tuning toward natural 1/fα spectra has been well documented using psychophysics in the spatial domain, however it has not been thoroughly characterised at the cortical level. It is also currently unknown whether the tuning observed in the spatial domain extends to the temporal domain. To address these gaps, we first used fMRI to measure BOLD responses across a wide range of α values (0.25, 0.75. 1.25, 1.75, 2.25) to fully characterise the tuning of early visual cortex (V1, V2, V3) toward the spatial 1/fα amplitude spectrum. For this, greyscale noise stimuli were presented across low and high contrast levels in the absence and presence of attention (Chapter 2). All visual areas exhibited an inverted U-shape response profile peaking for the most natural α in our set — 1.25 — irrespective of contrast or attention. We subsequently investigated whether BOLD responses were driven by the natural distribution of luminance intensities or the fractal structure contained in our stimuli (Chapter 3 & 4). For this, we used the original greyscale stimuli to create thresholded and edge extracted variants containing the same fractal structure but vastly different luminance intensity distributions. The same pattern of BOLD responses was observed across both thresholded and edge variants — an inverted U-shape peaking at 1.25 — suggesting that early visual cortex is primarily tuned to natural fractal structure. The tuning toward the temporal 1/fα amplitude spectrum was subsequently investigated using a range of greyscale noise movies varying in their spatial (0.25, 1.25, 2.25)

The ability of human visual systems to estimate 3D surface orientation from 2D retinal images is critical. But the computation to calculate 3D orientation in real-world scenes is not fully understood. A Bayes optimal model grounded in natural statistics has explained 3D surface tilt estimation of human observers in natural scenes (Kim and Burge, 2018). However, the model is limited because it estimates only unsigned tilt (tilt modulo 180deg). We extend the model to predict signed tilt estimates and compared with human signed estimates. The model takes image pixels as input and produces optimal estimates of tilt as output, using the joint statistics of tilt and image cues in natural scenes. The image cues to tilt are the directions of luminance, texture, and disparity gradients in a local area on the image. To estimate signed tilt, the disparity cue is used as a signed tilt cue, and the luminance and texture cues are used as unsigned tilt cues. Given a particular set of local image cues, the model computes the minimum mean squared error (MMSE) estimate, which is equal to the posterior mean over signed tilt. We found that the signed MMSE estimates were well aligned with human signed tilt estimates on the identical set of stimuli. Next, we pooled the local MMSE estimates across the space to obtain a global tilt estimate. Given that local MMSE estimates are unbiased predictor of groundtruth tilt with nearly equal reliability, the global pooled estimates are also near-optimal. The global estimates even better explained human tilt estimation. We conclude that this computational model provides a tool to understand how human visual systems make the best use of 2D image information to compute local estimates and integrate a global estimate of 3D surface tilt in complex natural scenes using the local estimates.

A primary focus of neuroscience is understanding how information about the world is encoded in the activity of neurons. The classic approach to studying neural representation is to observe how activity in a neural system is modulated when presented with stimuli that vary along a specific, hypothesis-driven dimension (e.g. luminance, contrast, orientation, spatial frequency). This approach offers straight-forward interpretation and has allowed neuroscientists to develop computational models of the stimulus-response function for neurons in early processing stages of the visual system. However, the interpretations of these studies often do not generalize to other stimuli such as natural images, and the corresponding computational models often fail to explain later processing stages where the stimulus-response function is strikingly nonlinear. What is needed is an alternative approach that can a) inform new hypotheses without relying on the intuitions of the experimenter, b) generate tractable and interpretable nonlinear models of neural function based on these hypotheses, and c) directly assess model accuracy at predicting neural function under natural conditions. In this dissertation I propose a simple but powerful framework that uses statistical feature learning to automatically derive potential forms of neural representation and generates predictive and testable models of neural function based on these potential representations. The framework is general and can potentially be applied to data acquired using any physiological measurement technique (e.g. neurophysiology, neuroimaging, behavioral) in order to model neural representation throughout the brain. I demonstrate the potential of the proposed framework by first modeling the neural representation of image structure in early cortical stages of the macaque visual system (areas V1 and V2). I then apply the framework to investigate high-level scene processing in the human visual system. In both case studies, I am ab

Sighted animals must survive in an environment that is diverse yet highly structured. Neural-coding models predict that the visual system should allocate its computational resources to exploit regularities in the environment, and that this allocation should facilitate perceptual judgments. Here we use three approaches (natural scenes statistical analysis, a reanalysis of single-unit data from alert behaving macaque, and a behavioral experiment in humans) to address the question of how the visual system maximizes behavioral success by taking advantage of low-level regularities in the environment. An analysis of natural scene statistics reveals that the probability distributions for light increments and decrements are biased in a way that could be exploited by the visual system to estimate depth from relative luminance. A reanalysis of neurophysiology data from Samonds et al. (2012) shows that the previously reported joint tuning of V1 cells for relative luminance and binocular disparity is well matched to a predicted distribution of binocular disparities produced by natural scenes. Finally, we show that a percept of added depth can be elicited in images by exaggerating the correlation between luminance and depth. Together, the results from these three approaches provide further evidence that the visual system allocates its processing resources in a way that is driven by the statistics of the natural environment.

[EN] When comparing the performance of video coding approaches, evaluating different commercial video encoders, or measuring the perceived video quality in a wireless environment, Rate/distortion analysis is commonly used, where distortion is usually measured in terms of PSNR values. However, PSNR does not always capture the distortion perceived by a human being. As a consequence, significant efforts have focused on defining an objective video quality metric that is able to assess quality in the same way as a human does. We perform a study of some available objective quality assessment metrics in order to evaluate their behavior in two different scenarios. First, we deal with video sequences compressed by different encoders at different bitrates in order to properly measure the video quality degradation associated with the encoding system. In addition, we evaluate the behavior of the quality metrics when measuring video distortions produced by packet losses in mobile ad hoc network scenarios with variable degrees of network congestion and node mobility. Our purpose is to determine if the analyzed metrics can replace the PSNR while comparing, designing, and evaluating video codec proposals, and, in particular, under video delivery scenarios characterized by bursty and frequent packet losses, such as wireless multihop environments.

[EN] When comparing the performance of video coding approaches, evaluating different commercial video encoders, or measuring the perceived video quality in a wireless environment, Rate/distortion analysis is commonly used, where distortion is usually measured in terms of PSNR values. However, PSNR does not always capture the distortion perceived by a human being. As a consequence, significant efforts have focused on defining an objective video quality metric that is able to assess quality in the same way as a human does. We perform a study of some available objective quality assessment metrics in order to evaluate their behavior in two different scenarios. First, we deal with video sequences compressed by different encoders at different bitrates in order to properly measure the video quality degradation associated with the encoding system. In addition, we evaluate the behavior of the quality metrics when measuring video distortions produced by packet losses in mobile ad hoc network scenarios with variable degrees of network congestion and node mobility. Our purpose is to determine if the analyzed metrics can replace the PSNR while comparing, designing, and evaluating video codec proposals, and, in particular, under video delivery scenarios characterized by bursty and frequent packet losses, such as wireless multihop environments.

A primary focus of neuroscience is understanding how information about the world is encoded in the activity of neurons. The classic approach to studying neural representation is to observe how activity in a neural system is modulated when presented with stimuli that vary along a specific, hypothesis-driven dimension (e.g. luminance, contrast, orientation, spatial frequency). This approach offers straight-forward interpretation and has allowed neuroscientists to develop computational models of the stimulus-response function for neurons in early processing stages of the visual system. However, the interpretations of these studies often do not generalize to other stimuli such as natural images, and the corresponding computational models often fail to explain later processing stages where the stimulus-response function is strikingly nonlinear. What is needed is an alternative approach that can a) inform new hypotheses without relying on the intuitions of the experimenter, b) generate tractable and interpretable nonlinear models of neural function based on these hypotheses, and c) directly assess model accuracy at predicting neural function under natural conditions. In this dissertation I propose a simple but powerful framework that uses statistical feature learning to automatically derive potential forms of neural representation and generates predictive and testable models of neural function based on these potential representations. The framework is general and can potentially be applied to data acquired using any physiological measurement technique (e.g. neurophysiology, neuroimaging, behavioral) in order to model neural representation throughout the brain. I demonstrate the potential of the proposed framework by first modeling the neural representation of image structure in early cortical stages of the macaque visual system (areas V1 and V2). I then apply the framework to investigate high-level scene processing in the human visual system. In both case studies, I am ab

Sighted animals must survive in an environment that is diverse yet highly structured. Neural-coding models predict that the visual system should allocate its computational resources to exploit regularities in the environment, and that this allocation should facilitate perceptual judgments. Here we use three approaches (natural scenes statistical analysis, a reanalysis of single-unit data from alert behaving macaque, and a behavioral experiment in humans) to address the question of how the visual system maximizes behavioral success by taking advantage of low-level regularities in the environment. An analysis of natural scene statistics reveals that the probability distributions for light increments and decrements are biased in a way that could be exploited by the visual system to estimate depth from relative luminance. A reanalysis of neurophysiology data from Samonds et al. (2012) shows that the previously reported joint tuning of V1 cells for relative luminance and binocular disparity is well matched to a predicted distribution of binocular disparities produced by natural scenes. Finally, we show that a percept of added depth can be elicited in images by exaggerating the correlation between luminance and depth. Together, the results from these three approaches provide further evidence that the visual system allocates its processing resources in a way that is driven by the statistics of the natural environment.

There have been several major advances in autonomous navigation for unmanned ground vehicles in controlled urban environments in recent years. However, off-road environments still pose several perception and classification challenges. This paper addresses two of decisions must be made about obstacles in the vehicle's path. The most common obstacle is vegetation, but some vegetation may be traversable, depending on the size of the vehicle and the type of vegetation. However, man-made objects should generally be detected and avoided in navigation. We present recent research towards the goal of vegetation and man-made object detection in the visible spectrum. First, we look at a state-of-the-art approaches to image segmentation and image saliency using natural scene statistics. Then we apply recent work in multi-class image labeling to several images taken from a small unmanned ground vehicle (UGV). This work will attempt to highlight the recent advances and challenges that lie ahead in the ultimate goal of vegetation and man-made object detection and classification in the visual spectrum from UGV.