Search

We describe the design and performance of a high-fidelity wearable head-, body-, and eye-tracking system that offers significant improvement over previous such devices. This devices sensors include a binocular eye tracker, an RGB-D scene camera, a high-frame-rate scene camera, and two visual odometry sensors, for a total of ten cameras, which we synchronize and record from with a data rate of over 700 MB/s. The sensors are operated by a mini-PC optimized for fast data collection, and powered by a small battery pack. The device records a subjects eye, head, and body positions, simultaneously with RGB and depth data from the subjects visual environment, measured with high spatial and temporal resolution. The headset weighs only 1.4 kg, and the backpack with batteries 3.9 kg. The device can be comfortably worn by the subject, allowing a high degree of mobility. Together, this system overcomes many limitations of previous such systems, allowing high-fidelity characterization of the dynamics of natural vision.

No-reference image quality assessment aims to evaluate image quality based on human subjective perceptions. Current methods face challenges with insufficient ability to focus on global and local information simultaneously and information loss due to image resizing. To address these issues, we propose a model that combines Swin-Transformer and natural scene statistics. The model utilizes Swin-Transformer to extract multi-scale features and incorporates a feature enhancement module and deformable convolution to improve feature representation, adapting better to structural variations in images, apply dual-branch attention to focus on key areas, and align the assessment more closely with human visual perception. The Natural Scene Statistics compensates information loss caused by image resizing. Additionally, we use a normalized loss function to accelerate model convergence and enhance stability. We evaluate our model on six standard image quality assessment datasets (both synthetic and authentic), and show that our model achieves advanced results across multiple datasets. Compared to the advanced DACNN method, our model achieved Spearman rank correlation coefficients of 0.922 and 0.923 on the KADID and KonIQ datasets, respectively, representing improvements of 1.9% and 2.4% over this method. It demonstrated outstanding performance in handling both synthetic and authentic scenes. [ABSTRACT FROM AUTHOR]

Sensory perception is widely considered an inference process that reflects the best guess of a stimulus feature based on uncertain sensory information. Here we challenge this reductionist view and propose that perception is rather a holistic inference process that operates not only at the feature but jointly across all levels of the representational hierarchy. We test this hypothesis in the context of a commonly used psychophysical matching task in which subjects are asked to report their perceived orientation of a test stimulus by adjusting a probe stimulus (method-of-adjustment). We introduce a holistic matching model that assumes that subjects' reports reflect an optimal match between the test and probe stimulus, both in terms of their inferred feature (orientation) and also their higher level representation (orientation category). Validation against several existing data sets demonstrates that the model accurately and comprehensively predicts subjects' response behavior and outperforms previous models both qualitatively and quantitatively. Moreover, the model generalizes to other feature domains and offers an alternative account for categorical color perception. Our results suggest that categorical effects in sensory perception are ubiquitous and can be parsimoniously explained as optimal behavior based on holistic sensory representations. [ABSTRACT FROM AUTHOR]

A growing research is focusing on approaches for assessing image quality as a result of advancements in digital imaging. Thus, there is an increasing demand for efficient no-reference image quality assessment methods, as many real-world, everyday applications lack distortion-free, i.e., pristine versions of images. This paper presents a new no-reference image quality outlier entropy perception evaluator method for the objective evaluation of real-world distorted images based on natural scene statistics and mean subtracted and contrast normalized coefficients. Distribution of the coefficients is found useful for no-reference image quality assessment, where their characteristics are investigated here. Moreover, entropies such as Shannon and approximate entropies are found suitable for quality estimation. Recent studies show perception-based approaches that demonstrate differences in correlation with subjective assessments. Similar variations are exhibited in entropy domain showing sample outliers compared to other distorted images with different distortion levels. In order to address these variations, this work presents outlier entropy perception evaluator model based on machine learning in order to describe the diversity of distortions affecting entropy and subjective scoring. Patch extraction is employed in the approach, where distortion level is estimated. The evaluatior model is found to be efficient presenting advantages using Shannon and approximate entropies and outlier detection over available perception-based image quality evaluators. The obtained results using proposed model show significant improvements in the correlation with human perceptual quality ratings. [ABSTRACT FROM AUTHOR]

Animals benefit from knowing if and how they are moving. Across the animal kingdom, sensory information in the form of optic flow over the visual field is used to estimate self-motion. However, different species exhibit strong spatial biases in how they use optic flow. Here, we show computationally that noisy natural environments favor visual systems that extract spatially biased samples of optic flow when estimating self-motion. The performance associated with these biases, however, depends on interactions between the environment and the animal's brain and behavior. Using the larval zebrafish as a model, we recorded natural optic flow associated with swimming trajectories in the animal's habitat with an omnidirectional camera mounted on a mechanical arm. An analysis of these flow fields suggests that lateral regions of the lower visual field are most informative about swimming speed. This pattern is consistent with the recent findings that zebrafish optomotor responses are preferentially driven by optic flow in the lateral lower visual field, which we extend with behavioral results from a high-resolution spherical arena. Spatial biases in optic-flow sampling are likely pervasive because they are an effective strategy for determining self-motion in noisy natural environments.

This study addresses the ongoing challenge for learning-based methods to achieve accurate object detection in foggy conditions. In response to the scarcity of foggy traffic image datasets, we propose a foggy weather simulation algorithm based on monocular depth estimation. The algorithm involves a multi-step process: a self-supervised monocular depth estimation network generates a relative depth map and then applies dense geometric constraints for scale recovery to derive an absolute depth map. Subsequently, the visibility of the simulated image is defined to generate a transmittance map. The dark channel map is then used to distinguish sky regions and estimate atmospheric light values. Finally, the atmospheric scattering model is used to generate fog simulation images under specified visibility conditions. Experimental results show that more than 90% of fog images have AuthESI values of less than 2, which indicates that their non-structural similarity (NSS) characteristics are very close to those of natural fog. The proposed fog simulation method is able to convert clear images in natural environments, providing a solution to the problem of lack of foggy image datasets and incomplete visibility data. [ABSTRACT FROM AUTHOR]

We propose a blind image quality assessment (BIQA) method of using the multitask-learningbased end-to-end convolutional neural network (CNN) approach. The architecture of the proposed method is integrated by two streams. In the first stream, multiscale image features are extracted by using the inception and pyramid pooling modules. Natural scene statistics (NSS)-based features are extracted in the second stream. The two streams are then integrated into fully connected layers to estimate the image quality score. The performance of the proposed method is validated with four public IQA databases and the obtained experimental results show the superiority of the proposed method over conventional IQA methods. [ABSTRACT FROM AUTHOR]

Objective assessment of image quality seeks to predict image quality without human perception. Given that the ultimate goal of a blind/no-reference image quality assessment (BIQA) algorithm is to provide a score consistent with the subject's prediction, it makes sense to design an algorithm that resembles human behavior. Recently, a large number of image features have been introduced to image quality assessment. However, only a few of these features are generated by using the computational mechanisms of the visual cortex. In this paper, we propose bioinspired algorithms to extract image features for BIQA by simulating the visual cortex. We extract spatial features like texture and energy from images by mimicking the retinal circuit. We extract spatial-frequency features from images by imitating the simple cell of the primary visual cortex. And we extract color features from images by employing the color opponent mechanism of the biological vision system. Then, by using the statistical features derived from these physiologically plausible features, we train a support vector regression model to predict image quality. The experimental results show that the proposed algorithm is more consistent with subjective evaluations than the comparison algorithms in predicting image quality. [ABSTRACT FROM AUTHOR]

Due to the deficient knowledge of binocular vision properties, how to effectively evaluate stereoscopic images still remains a challenging task. Inspired by multichannel processing of human visual system (HVS), we propose a blind method for stereoscopic image quality assessment (SIQA) by extracting quality related features in sub-bands of the image. First of all, we introduce the shearlet transform to decompose the left- and right-view images into multiple sub-bands content with diverse combinations of scales and orientations, and obtain the combined view based on energy-weighted summation of the corresponding sub-bands of two eye views. Then, natural scene statistics (NSS) of the original left and right images are obtained as quality-sensitive features, followed by extracting NSS features of the sub-bands of left, right and combined views. Moreover, we calculate the gradient similarity between each sub-band pair to denote the asymmetric distortion and disparity information. Finally, all the extracted features are mapped into a quality score by support vector regression (SVR). experimental results on multiple benchmark databases verify the superiority of our method.

Recently, stereoscopic image quality assessment has attracted a lot attention. However, compared with 2D image quality assessment, it is much more difficult to assess the quality of stereoscopic images due to the lack of understanding of 3D visual perception. This paper proposes a novel no-reference quality assessment metric for stereoscopic images using natural scene statistics with consideration of both the quality of the cyclopean image and 3D visual perceptual information (binocular fusion and binocular rivalry). In the proposed method, not only is the quality of the cyclopean image considered, but binocular rivalry and other 3D visual intrinsic properties are also exploited. Specifically, in order to improve the objective quality of the cyclopean image, features of the cyclopean images in both the spatial domain and transformed domain are extracted based on the natural scene statistics (NSS) model. Furthermore, to better comprehend intrinsic properties of the stereoscopic image, in our method, the binocular rivalry effect and other 3D visual properties are also considered in the process of feature extraction. Following adaptive feature pruning using principle component analysis, improved metric accuracy can be found in our proposed method. The experimental results show that the proposed metric can achieve a good and consistent alignment with subjective assessment of stereoscopic images in comparison with existing methods, with the highest SROCC (0.952) and PLCC (0.962) scores being acquired on the LIVE 3D database Phase I. [ABSTRACT FROM AUTHOR]

As a fundamental research problem, blind image quality assessment (BIQA) has attracted increasing interest in recent years. Although great progress has been made, BIQA still remains a challenge. To better understand the research progress and challenges in this field, we review BIQA methods in this paper. First, we introduce the BIQA problem definition and related methods. Second, we provide a detailed review of the existing BIQA methods in terms of representative hand-crafted features, learning-based features and quality regressors for two-stage methods, as well as one-stage DNN models with various architectures. Moreover, we also present and analyze the performance of competing BIQA methods on six public IQA datasets. Finally, we conclude our paper with possible future research directions based on a performance analysis of the BIQA methods. This review will provide valuable references for researchers interested in the BIQA problem. [ABSTRACT FROM AUTHOR]

Images derived from a "virtual phantom" can be useful in characterizing the performance of imaging systems. This has driven the development of virtual breast phantoms implemented in simulation environments. In breast imaging, several such phantoms have been proposed. We analyze the non-Gaussian statistical properties from three classes of virtual breast phantoms and compare them to similar statistics from a database of breast images. These include clustered-blob lumpy backgrounds (CBLBs), truncated binary textures, and the UPenn virtual breast phantoms. We use Laplacian fractional entropy (LFE) as a measure of the non-Gaussian statistical properties of each simulation procedure. Our results show that, despite similar power spectra, the simulation approaches differ considerably in LFE with very low scores for the CBLB to high values for the UPenn phantom at certain frequencies. These results suggest that LFE may have value in developing and tuning virtual phantom simulation procedures.

Despite the enormous performance of deep neural networks (DNNs), recent studies have shown their vulnerability to adversarial examples (AEs), i.e., carefully perturbed inputs designed to fool the targeted DNN. Currently, the literature is rich with many effective attacks to craft such AEs. Meanwhile, many defense strategies have been developed to mitigate this vulnerability. However, these latter showed their effectiveness against specific attacks and does not generalize well to different attacks. In this paper, we propose a framework for defending DNN classifier against adversarial samples. The proposed method is based on a two-stage framework involving a separate detector and a denoising block. The detector aims to detect AEs by characterizing them through the use of natural scene statistic (NSS), where we demonstrate that these statistical features are altered by the presence of adversarial perturbations. The denoiser is based on block matching 3D (BM3D) filter fed by an optimum threshold value estimated by a convolutional neural network (CNN) to project back the samples detected as AEs into their data manifold. We conducted a complete evaluation on three standard datasets, namely MNIST, CIFAR-10 and Tiny-ImageNet. The experimental results show that the proposed defense method outperforms the state-of-the-art defense techniques by improving the robustness against a set of attacks under black-box, gray-box and white-box settings. The source code is available at: https://github.com/kherchouche-anouar/2DAE. [ABSTRACT FROM AUTHOR]

To improve the viewer’s Quality of Experience (QoE) and optimize computer graphics applications, 3D model quality assessment (3D-QA) has become an important task in the multimedia area. Point cloud and mesh are the two most widely used digital representation formats of 3D models, the visual quality of which is quite sensitive to lossy operations like simplification and compression. Therefore, many related studies such as point cloud quality assessment (PCQA) and mesh quality assessment (MQA) have been carried out to measure the visual quality of distorted 3D models. However, most previous studies utilize full-reference (FR) metrics, which indicates they can not predict the quality level in the absence of the reference 3D model. Furthermore, few 3D-QA metrics consider color information, which significantly restricts their effectiveness and scope of application. In this paper, we propose a no-reference (NR) quality assessment metric for colored 3D models represented by both point cloud and mesh. First, we project the 3D models from 3D space into quality-related geometry and color feature domains. Then, the 3D natural scene statistics (3D-NSS) and entropy are utilized to extract quality-aware features. Finally, a support vector regression (SVR) model is employed to regress the quality-aware features into visual quality scores. Our method is validated on the colored point cloud quality assessment database (SJTU-PCQA), the Waterloo point cloud assessment database (WPC), and the colored mesh quality assessment database (CMDM). The experimental results show that the proposed method outperforms most compared NR 3D-QA metrics with competitive computational resources and greatly reduces the performance gap with the state-of-the-art FR 3D-QA metrics. The code of the proposed model is publicly available now at https://github.com/zzc-1998/NR-3DQA. [ABSTRACT FROM AUTHOR]

This paper proposes an approach to improve the performance of no‐reference video quality assessment for sports videos with dynamic motion scenes using an efficient spatiotemporal model. In the proposed method, we divide the video sequences into video blocks and apply a 3D shearlet transform that can efficiently extract primary spatiotemporal features to capture dynamic natural motion scene statistics from the incoming video blocks. The concatenation of a deep residual bidirectional gated recurrent neural network and logistic regression is used to learn the spatiotemporal correlation more robustly and predict the perceptual quality score. In addition, conditional video block‐wise constraints are incorporated into the objective function to improve quality estimation performance for the entire video. The experimental results show that the proposed method extracts spatiotemporal motion information more effectively and predicts the video quality with higher accuracy than the conventional no‐reference video quality assessment methods.

Blind or no-reference video quality assessment of user-generated content (UGC) has become a trending, challenging, heretofore unsolved problem. Accurate and efficient video quality predictors suitable for this content are thus in great demand to achieve more intelligent analysis and processing of UGC videos. Previous studies have shown that natural scene statistics and deep learning features are both sufficient to capture spatial distortions, which contribute to a significant aspect of UGC video quality issues. However, these models are either incapable or inefficient for predicting the quality of complex and diverse UGC videos in practical applications. Here we introduce an effective and efficient video quality model for UGC content, which we dub the Rapid and Accurate Video Quality Evaluator (RAPIQUE), which we show performs comparably to state-of-the-art (SOTA) models but with orders-of-magnitude faster runtime. RAPIQUE combines and leverages the advantages of both quality-aware scene statistics features and semantics-aware deep convolutional features, allowing us to design the first general and efficient spatial and temporal (space-time) bandpass statistics model for video quality modeling. Our experimental results on recent large-scale UGC video quality databases show that RAPIQUE delivers top performances on all the datasets at a considerably lower computational expense. We hope this work promotes and inspires further efforts towards practical modeling of video quality problems for potential real-time and low-latency applications.

Perceptual confidence typically corresponds to accuracy. However, observers can be overconfident relative to accuracy, termed "subjective inflation." Inflation is stronger in the visual periphery relative to central vision, especially under conditions of peripheral inattention. Previous literature suggests inflation stems from errors in estimating noise (i.e., "variance misperception"). However, despite previous Bayesian hypotheses about metacognitive noise estimation, no work has systematically explored how noise estimation may critically depend on empirical noise statistics, which may differ across the visual field, with central noise distributed symmetrically but peripheral noise positively skewed. Here, we examined central and peripheral vision predictions from five Bayesian-inspired noise-estimation algorithms under varying usage of noise priors, including effects of attention. Models that failed to optimally estimate noise exhibited peripheral inflation, but only models that explicitly used peripheral noise priors—but used them incorrectly—showed increasing peripheral inflation under increasing peripheral inattention. Further, only one model successfully captured previous empirical results, which showed a selective increase in confidence in incorrect responses under performance reductions due to inattention accompanied by no change in confidence in correct responses; this was the model that implemented Bayesian estimation of peripheral noise, but using an (incorrect) symmetric rather than the correct positively skewed peripheral noise prior. Our findings explain peripheral inflation, especially under inattention, and suggest future experiments that might reveal the noise expectations used by the visual metacognitive system. [ABSTRACT FROM AUTHOR]

Deep neural networks targeting stereo disparity estimation have recently surpassed the performance of hand-crafted traditional models. However, training these networks require large labeled databases for obtaining accurate disparity estimates. In this letter, we address the large data requirement by generating synthetic data using natural image statistics. Images generated using dead leaves model have been shown to share many statistical characteristics commonly seen in natural images. In this work, we created a synthetic dataset using the 3D dead leaves model consisting of occluding spheres, and projected them onto parallel camera planes to obtain stereo image pairs along with ground-truth disparity map. This generated data was subsequently used to train a deep neural network in a supervised manner to estimate disparity. Through experiments we show that this trained model achieves competitive performance across real-world and synthetic stereo datasets, even without any additional fine-tuning. The proposed method for dataset generation is simplistic in nature, computationally inexpensive and can be easily scaled for large scale data generation. [ABSTRACT FROM AUTHOR]

We develop a new no-reference (NR) stereoscopic video quality assessment model that adopts oriented local gravitational force (OLGF) statistics in the space-time domain. The OLGF is a novel extension of an existing local gravitational force descriptor and includes two new components: relative local gravitational force magnitude and relative local gravitational force orientation. The resulting algorithm, called Stereoscopic Video Integrity Predictor using OLGF Statistics (SVIPOS), first uses our previous work to generate a cyclopean image and a product image in spatial domain from a stereoscopic video sequence pair, and calculates a frame difference image in temporal domain from a left video sequence. Then the OLGF model is deployed to compute various local gravitational force responses on the cyclopean image, the product image and the frame difference image. Finally, a support vector regression is used to map statistical features extracted from the computed gravitational force responses to stereoscopic video quality predications. SVIPOS is thoroughly tested on three publicly stereoscopic video quality databases. The experimental results show that SVIPOS outperforms state-of-the-art stereoscopic video quality methods.

Sighted animals must survive in an environment that is diverse yet highly structured. Neural-coding models predict that the visual system should allocate its computational resources to exploit regularities in the environment, and that this allocation should facilitate perceptual judgments. Here we use three approaches (natural scenes statistical analysis, a reanalysis of single-unit data from alert behaving macaque, and a behavioral experiment in humans) to address the question of how the visual system maximizes behavioral success by taking advantage of low-level regularities in the environment. An analysis of natural scene statistics reveals that the probability distributions for light increments and decrements are biased in a way that could be exploited by the visual system to estimate depth from relative luminance. A reanalysis of neurophysiology data from Samonds et al. (2012) shows that the previously reported joint tuning of V1 cells for relative luminance and binocular disparity is well matched to a predicted distribution of binocular disparities produced by natural scenes. Finally, we show that a percept of added depth can be elicited in images by exaggerating the correlation between luminance and depth. Together, the results from these three approaches provide further evidence that the visual system allocates its processing resources in a way that is driven by the statistics of the natural environment.

Although feedforward activity may suffice for recognizing objects in isolation, additional visual operations that aid object recognition might be needed for real-world scenes. One such additional operation is figure-ground segmentation, extracting the relevant features and locations of the target object while ignoring irrelevant features. In this study of 60 human participants (female and male), we show objects on backgrounds of increasing complexity to investigate whether recurrent computations are increasingly important for segmenting objects from more complex backgrounds. Three lines of evidence show that recurrent processing is critical for recognition of objects embedded in complex scenes. First, behavioral results indicated a greater reduction in performance after masking objects presented on more complex backgrounds, with the degree of impairment increasing with increasing background complexity. Second, electroencephalography (EEG) measurements showed clear differences in the evoked response potentials between conditions around time points beyond feedforward activity, and exploratory object decoding analyses based on the EEG signal indicated later decoding onsets for objects embedded in more complex backgrounds. Third, deep convolutional neural network performance confirmed this interpretation. Feedforward and less deep networks showed a higher degree of impairment in recognition for objects in complex backgrounds compared with recurrent and deeper networks. Together, these results support the notion that recurrent computations drive figure-ground segmentation of objects in complex scenes. [ABSTRACT FROM AUTHOR]

This paper proposes an approach to improve the performance of no‐reference video quality assessment for sports videos with dynamic motion scenes using an efficient spatiotemporal model. In the proposed method, we divide the video sequences into video blocks and apply a 3D shearlet transform that can efficiently extract primary spatiotemporal features to capture dynamic natural motion scene statistics from the incoming video blocks. The concatenation of a deep residual bidirectional gated recurrent neural network and logistic regression is used to learn the spatiotemporal correlation more robustly and predict the perceptual quality score. In addition, conditional video block‐wise constraints are incorporated into the objective function to improve quality estimation performance for the entire video. The experimental results show that the proposed method extracts spatiotemporal motion information more effectively and predicts the video quality with higher accuracy than the conventional no‐reference video quality assessment methods. [ABSTRACT FROM AUTHOR]

The 1/fα amplitude spectrum is a statistical property of natural scenes characterising a specific distribution of spatial and temporal frequencies and their associated luminance intensities. This property has been studied extensively in the spatial domain whereby sensitivity and visual preference overlap and peak for slopes within the natural range (α ≈ 1), but remains relatively less studied in the temporal domain. Here, we used a 4AFC task to measure sensitivity and a 2AFC task to measure visual preference and across a wide range of spatial (α = 0.25, 1.25, 2.25) and temporal (α = 0.25 to 2.50, step size: 0.25) slope conditions. Stimuli with a shallow temporal slope modulate rapidly (e.g. 0.25), whereas stimuli with a steep slope modulate slowly (e.g. 2.25). Interestingly, sensitivity and visual preference did not closely overlap. While the sensitivity of the visual system is highest for our stimulus with an intermediate modulation rate (1.25), which is most abundant in nature, the stimulus with the slowest modulation rate (2.25) was most preferred. It seems sensible for the visual system to be sensitive to spatiotemporal spectra that most commonly exist in nature (α ≈ 1). However, it is possible that preference might be related to what these properties signal in the natural world. Consider the cases of waves slowly vs. rapidly crashing on a beach or fast vs. slow animals. In both instances the slowest option is often the safest and preferential, suggesting that the temporal 1/fα amplitude spectrum provides additional information that may indicate preferred environmental conditions. [ABSTRACT FROM AUTHOR]

Monocular depth estimation (MDE), which is the task of using a single image to predict scene depths, has gained considerable interest, in large part owing to the popularity of applying deep learning methods to solve "computer vision problems". Monocular cues provide sufficient data for humans to instantaneously extract an understanding of scene geometries and relative depths, which is evidence of both the processing power of the human visual system and the predictive power of the monocular data. However, developing computational models to predict depth from monocular images remains challenging. Hand-designed MDE features do not perform particularly well, and even current "deep" models are still evolving. Here we propose a novel approach that uses perceptually-relevant natural scene statistics (NSS) features to predict depths from monocular images in a simple, scale-agnostic way that is competitive with state-of-the-art systems. While the statistics of natural photographic images have been successfully used in a variety of image and video processing, analysis, and quality assessment tasks, they have never been applied in a predictive end-to-end deep-learning model for monocular depth. Correspondingly, no previous work has explicitly incorporated perceptual features in a monocular depth-prediction approach. Here we accomplish this by developing a new closed-form bivariate model of image luminances and use features extracted from this model and from other NSS models to drive a novel deep learning framework for predicting depth given a single image. [ABSTRACT FROM AUTHOR]

The amplitude of Fourier spectra for natural scenes falls with spatial frequency (f) and is described by the equation, 1/fα, where exponent α corresponds to the slope of the spectral drop-off. For natural scenes α takes on intermediate values ~1.25, reflecting their scale invariance. It is also well-established that, on average, images with natural scene statistics are preferred to those that deviate from these properties. Although this average pattern of preference for images with the intermediate values of α is robust, there are also marked individual differences in preference for different levels of α. This study investigated the effects of adaptation on average and individual visual preferences for synthetic filtered noise images varying in α. Participant preferences (N = 58) were measured via a 2AFC task prior to adaptation (baseline) and post-adaptation There were 3 adaptation conditions (α = 0.25, 1.25, 2.25) and 5 test levels of α (0.25, 0.75, 1.25, 1.75, 2.25). On average, the adaptation elevated preferences for test images with α matching the adaptor conditions, especially in adaptor conditions, α = 0.25 and 2.25. We also observed marked individual differences in preference for different levels of α. These different preference profiles remained stable throughout the experiment and affected the levels of adaptation observed in different adaptation conditions. [ABSTRACT FROM AUTHOR]

A fundamental natural visual task is the identification of specific target objects in the environments that surround us. It has long been known that some properties of the background have strong effects on target visibility. The most well-known properties are the luminance, contrast, and similarity of the background to the target. In previous studies, we found that these properties have highly lawful effects on detection in natural backgrounds. However, there is another important factor affecting detection in natural backgrounds that has received little or no attention in the masking literature, which has been concerned with detection in simpler backgrounds. Namely, in natural backgrounds the properties of the background often vary under the target, and hence some parts of the target are masked more than others. We began studying this factor, which we call the "partial masking factor," by measuring detection thresholds in backgrounds of contrastmodulated white noise that was constructed so that the standard template-matching (TM) observer performs equally well whether or not the noise contrast modulates in the target region. If noise contrast is uniform in the target region, then this TM observer is the Bayesian optimal observer. However, when the noise contrast modulates then the Bayesian optimal observer weights the template at each pixel location by the estimated reliability at that location. We find that human performance for modulated noise backgrounds is predicted by this reliability-weighted TM (RTM) observer. More surprisingly, we find that human performance for natural backgrounds is also predicted by the RTM observer. [ABSTRACT FROM AUTHOR]

The statistics of our environment impact not only our behavior, but also the selectivity and connectivity of the early sensory cortices. Over the last fifty years, powerful theories such as efficient coding, sparse coding, and the infomax principle have been proposed to explain the nature of this influence. Numerous computational and theoretical studies have since demonstrated solid, testable evidence in support of these theories, especially in the visual domain. However, most such work has concentrated on monocular, luminance-field descriptions of natural scenes, and studies that systematically focus on binocular processing of realistic visual input have only been conducted over the past two decades. In this review, we discuss the most recent of these binocular computational studies, with particular emphasis on disparity selectivity. We begin with a report of the relevant literature demonstrating concrete evidence for the relationship between natural disparity statistics, neural selectivity, and behavior. This is followed by a discussion of supervised and unsupervised computational studies. For each study, we include a description of the input data, theoretical principles employed in the models, and the contribution of the results in explaining biological data (neural and behavioral). In the discussion, we compare these models to the binocular energy model, and examine their application to the modelling of normal and abnormal development of vision. We conclude with a short description of what we believe are the most important limitations of the current state-of-the-art, and directions for future work which could address these shortcomings and enrich current and future models. [ABSTRACT FROM AUTHOR]

Most of no-reference image quality assessment (NR-IQA) techniques reported in literature have utilized transform coefficients, which are modeled using curve fitting to extract features based on natural scene statistics (NSS). The performance of NR-IQA techniques that utilize curve-fitting suffers from degradation in performance because the distribution of curve fitted NSS features deviate from the statistical distribution of a distorted image. Although deep convolutional neural networks (DCNNs) have been used for NR-IQA that are NSS model-independent but their performance is dependent upon the size of training data. The available datasets for NR-IQA are small, therefore data augmentation is used that affects the performance of DCNN based NR-IQA techniques and is also computationally expensive. This work proposes a new patch-based NR-IQA technique, which utilizes features extracted from discrete cosine transform coefficients. The proposed technique is curve fitting independent and helps in avoiding errors in the statistical distribution of NSS features. It relies on global statistics to estimate image quality based on local patches, which allow us to decompose the statistics of images. The proposed technique divides the image into patches and extracts nine handcrafted features i.e., entropy, mean, variance, skewness, kurtosis, mobility, band power, energy, complexity, and peak to peak value. The extracted features are used with a support vector regression model to predict the image quality score. The experimental results have shown that the proposed technique is database and image content-independent. It shows better performance over a majority of distortion types and on images taken in real-time. [ABSTRACT FROM AUTHOR]

An ideal quality assessment model should simulate the properties of the visual brain to be consistent with human evaluation. The visual brain appears to have both evolved to seek an efficient, decorrelated representation of image information and to “match” the statistics of the natural image. On one hand, the theoretical studies suggest that sparse representation resembles the strategy in the primary visual cortex of brain for representing natural images. On the other hand, the natural scene statistics have driven the evolution of human visual system and have also inspired the understanding and simulating of visual perception. Inspired by these observations, in this paper, we propose a novel reduced-reference stereoscopic image quality assessment metric using sparse representation and natural scene statistics to simulate the visual perception of the brain. Specifically, the distribution statistics of the classified visual primitives extracted by sparse representation are used to measure the visual information, which is closely related to the hierarchical progressive process of human visual perception. Particularly, the mutual information of classified primitives between two view images is derived as a binocular cue to simulate the binocular fusion process. The maximum mechanism that is applied to select the visual information is a pooling mechanism with which complex cells use the maximal stimuli from a group of simple cells during the transfer process in the primary visual cortex. The natural scene statistics of locally normalized luminance coefficients are used to evaluate the natural losses due to the presence of distortions. The differences of the visual information and the natural scene statistics between the original and distorted images are used to compute the quality score by a prediction function which is trained using support vector regression. Experimental results show that the proposed metric outperforms the state-of-the-art stereoscopic image quality assessment metrics on LIVE 3D IQA database and NBU-MDSID Phase-II database, and delivers competitive performance on Waterloo IVC 3D database. [ABSTRACT FROM AUTHOR]

It has been suggested that neurons can represent sensory input using probability distributions and neural circuits can perform probabilistic inference. Lateral connections between neurons have been shown to have non-random connectivity and modulate responses to stimuli within the classical receptive field. Large-scale efforts mapping local cortical connectivity describe cell type specific connections from inhibitory neurons and like-to-like connectivity between excitatory neurons. To relate the observed connectivity to computations, we propose a neuronal network model that approximates Bayesian inference of the probability of different features being present at different image locations. We show that the lateral connections between excitatory neurons in a circuit implementing contextual integration in this should depend on correlations between unit activities, minus a global inhibitory drive. The model naturally suggests the need for two types of inhibitory gates (normalization, surround inhibition). First, using natural scene statistics and classical receptive fields corresponding to simple cells parameterized with data from mouse primary visual cortex, we show that the predicted connectivity qualitatively matches with that measured in mouse cortex: neurons with similar orientation tuning have stronger connectivity, and both excitatory and inhibitory connectivity have a modest spatial extent, comparable to that observed in mouse visual cortex. We incorporate lateral connections learned using this model into convolutional neural networks. Features are defined by supervised learning on the task, and the lateral connections provide an unsupervised learning of feature context in multiple layers. Since the lateral connections provide contextual information when the feedforward input is locally corrupted, we show that incorporating such lateral connections into convolutional neural networks makes them more robust to noise and leads to better performance on noisy versions of the MNIST dataset. Decomposing the predicted lateral connectivity matrices into low-rank and sparse components introduces additional cell types into these networks. We explore effects of cell-type specific perturbations on network computation. Our framework can potentially be applied to networks trained on other tasks, with the learned lateral connections aiding computations implemented by feedforward connections when the input is unreliable and demonstrate the potential usefulness of combining supervised and unsupervised learning techniques in real-world vision tasks. [ABSTRACT FROM AUTHOR]

Video quality assessment (VQA) remains an important and challenging problem that affects many applications at the widest scales. Recent advances in mobile devices and cloud computing techniques have made it possible to capture, process, and share high resolution, high frame rate (HFR) videos across the Internet nearly instantaneously. Being able to monitor and control the quality of these streamed videos can enable the delivery of more enjoyable content and perceptually optimized rate control. Accordingly, there is a pressing need to develop VQA models that can be deployed at enormous scales. While some recent effects have been applied to full-reference (FR) analysis of variable frame rate and HFR video quality, the development of no-reference (NR) VQA algorithms targeting frame rate variations has been little studied. Here, we propose a first-of-a-kind blind VQA model for evaluating HFR videos, which we dub the Framerate-Aware Video Evaluator w/o Reference (FAVER). FAVER uses extended models of spatial natural scene statistics that encompass space–time wavelet-decomposed video signals, and leverages the advantages of the deep neural network to provide motion perception, to conduct efficient frame rate sensitive quality prediction. Our extensive experiments on several HFR video quality datasets show that FAVER outperforms other blind VQA algorithms at a reasonable computational cost. To facilitate reproducible research and public evaluation, an implementation of FAVER is being made freely available online: https://github.com/uniqzheng/HFR-BVQA. [ABSTRACT FROM AUTHOR]

Perceptual stereo image quality assessment (SIQA) aims to design computational models to measure the stereo image quality in accordance with human opinions. In this paper, a novel reduced-reference (RR) SIQA is proposed by characterizing the statistical and perceptual properties of the stereo image in both the spatial and gradient domains. To be specific, in the spatial domain, we extract the parameters of the generalized Gaussian distribution fits of luminance wavelet coefficients to form the underlying features. In the gradient domain, the modified gradient magnitudes maps are generated by jointly considering human visual system's contrast sensitivity and neighborhood gradient information to weight the gradient magnitudes in a locally adaptive manner. Afterward, perceptual features are extracted based on the entropy of discrete wavelet transform coefficients of modified gradient magnitudes. Furthermore, we consolidate the left and right features into a single set of features per stereo image pair. Finally, the qualities of both the spatial and gradient domains are combined to obtain the overall quality of stereo image. Extensive experiments performed on popular data sets demonstrate that the proposed RR-SIQA method achieves highly competitive performance as compared with the state-of-the-art RR-SIQA models as well as full-reference ones for both symmetric and asymmetric distortions.

Image quality assessment (IQA) technology is facing a new challenge due to the large amount of demand for high-quality 3-D-synthesized views stimulated by the rapid development of virtual reality applications. Since 3-D synthesis views commonly rely on the depth-image-based rendering technology to synthesize virtual images (without reference images in reality and containing geometric distortion), only the no-reference (NR) quality assessment method can meet the requirements. However, most of the current IQA methods for 3-D-synthesized views are full-reference methods. So far, only one specialized NR IQA method for 3-D-synthesized views has been proposed, but its computation is too expensive. For this reason, we have previously proposed a method for extracting geometric distortion regions using the Joint Photographic Experts Group (JPEG) image compression technology to evaluate image quality. In this paper, we consider that although heavy JPEG compression can effectively extract the geometric distortion area, it will ignore the image quality degradation caused by other distortions. Therefore, we have improved our previous work to extract geometric distortions and non-geometric distortions by high-level JPEG compression and lowlevel JPEG compression, respectively. The overall image quality score was obtained by the fusion of the two results. Experiments indicate that the proposed blind quality model is superior to modern full-, reduced-, and no-reference methods. Compared with our previous work, the performance of the new algorithm has been greatly improved. At the same time, compared with the existing dedicated NR IQA method, the performance is similar but the calculation speed has obvious advantages.

Sparse coding models of natural images and sounds have been able to predict several response properties of neurons in the visual and auditory systems. While the success of these models suggests that the structure they capture is universal across domains to some degree, it is not yet clear which aspects of this structure are universal and which vary across sensory modalities. To address this, we fit complete and highly overcomplete sparse coding models to natural images and spectrograms of speech and report on differences in the statistics learned by these models. We find several types of sparse features in natural images, which all appear in similar, approximately Laplace distributions, whereas the many types of sparse features in speech exhibit a broad range of sparse distributions, many of which are highly asymmetric. Moreover, individual sparse coding units tend to exhibit higher lifetime sparseness for overcomplete models trained on images compared to those trained on speech. Conversely, population sparseness tends to be greater for these networks trained on speech compared with sparse coding models of natural images. To illustrate the relevance of these findings to neural coding, we studied how they impact a biologically plausible sparse coding network's representations in each sensory modality. In particular, a sparse coding network with synaptically local plasticity rules learns different sparse features from speech data than are found by more conventional sparse coding algorithms, but the learned features are qualitatively the same for these models when trained on natural images.

A core goal of visual neuroscience is to predict human perceptual performance from natural signals. Performance in any natural task can be limited by at least three sources of uncertainty: stimulus variability, internal noise, and suboptimal computations. Determining the relative importance of these factors has been a focus of interest for decades but requires methods for predicting the fundamental limits imposed by stimulus variability on sensory-perceptual precision. Most successes have been limited to simple stimuli and simple tasks. But perception science ultimately aims to understand how vision works with natural stimuli. Successes in this domain have proven elusive. Here, we develop a model of humans based on an image-computable (images in, estimates out) Bayesian ideal observer. Given biological constraints, the ideal optimally uses the statistics relating local intensity patterns in moving images to speed, specifying the fundamental limits imposed by natural stimuli. Next, we propose a theoretical link between two key decision-theoretic quantities that suggests how to experimentally disentangle the impacts of internal noise and deterministic suboptimal computations. In several interlocking discrimination experiments with three male observers, we confirm this link and determine the quantitative impact of each candidate performance-limiting factor. Human performance is near-exclusively limited by natural stimulus variability and internal noise, and humans use near-optimal computations to estimate speed from naturalistic image movies. The findings indicate that the partition of behavioral variability can be predicted from a principled analysis of natural images and scenes. The approach should be extendable to studies of neural variability with natural signals. [ABSTRACT FROM AUTHOR]

No-reference image quality assessment (NR-IQA) is a non-trivial task, because it is hard to find a pristine counterpart for an image in real applications, such as image selection, high quality image recommendation, etc. In recent years, deep learning-based NR-IQA methods emerged and achieved better performance than previous methods. In this paper, we present a novel deep neural networks-based multi-task learning approach for NR-IQA. Our proposed network is designed by a multi-task learning manner that consists of two tasks, namely, natural scene statistics (NSS) features prediction task and the quality score prediction task. NSS features prediction is an auxiliary task, which helps the quality score prediction task to learn better mapping between the input image and its quality score. The main contribution of this work is to integrate the NSS features prediction task to the deep learning-based image quality prediction task to improve the representation ability and generalization ability. To the best of our knowledge, it is the first attempt. We conduct the same database validation and cross database validation experiments on LIVE, TID2013, CSIQ, LIVE multiply distorted image quality database (LIVE MD), CID2013, and LIVE in the wild image quality challenge (LIVE challenge) databases to verify the superiority and generalization ability of the proposed method. Experimental results confirm the superior performance of our method on the same database validation; our method especially achieves 0.984 and 0.986 on the LIVE image quality assessment database in terms of the Pearson linear correlation coefficient (PLCC) and Spearman rank-order correlation coefficient (SROCC), respectively. Also, experimental results from cross database validation verify the strong generalization ability of our method. Specifically, our method gains significant improvement up to 21.8% on unseen distortion types. [ABSTRACT FROM AUTHOR]

The predictive coding account of psychosis postulates the abnormal formation of prior beliefs in schizophrenia, resulting in psychotic symptoms. One domain in which priors play a crucial role is visual perception. For instance, our perception of brightness, line length, and motion direction are not merely based on a veridical extraction of sensory input but are also determined by expectation (or prior) of the stimulus. Formation of such priors is thought to be governed by the statistical regularities within natural scenes. Recently, the use of such priors has been attributed to a specific set of well-documented visual illusions, supporting the idea that perception is biased toward what is statistically more probable within the environment. The Predictive Coding account of psychosis proposes that patients form abnormal representations of statistical regularities in natural scenes, leading to altered perceptual experiences. Here we use classical vision experiments involving a specific set of visual illusions to directly test this hypothesis. We find that perceptual judgments for both patients and control participants are biased in accordance with reported probability distributions of natural scenes. Thus, despite there being a suggested link between visual abnormalities and psychotic symptoms in schizophrenia, our results provide no support for the notion that altered formation of priors is a general feature of the disorder. These data call for a refinement in the predictions of quantitative models of psychosis. [ABSTRACT FROM AUTHOR]