Search

Stereo matching is a popular topic in the image processing and computer vision fields. Although deep learning-based stereo matching approaches have achieved remarkable performance with respect to both estimation accuracy and computation efficiency, textureless and occluded regions and regions with edge blurring remain significant challenges for most existing stereo matching methods. To address these issues, we propose a geometric prior-guided stereo matching method with disparity fusion refinement, named GPDF-Net, in this paper. First, we exploit a geometric prior guidance module in the feature extraction part to enable features to obtain the global information of cross-view interactions and pay attention to the structural information contained in the input image. Second, we construct cost volume with disparity fusion refinement that introduces disparity-related geometric features to the concatenation volume and suppresses its redundant information to provide a better similarity measure. Third, we explore a method for replacing 3D convolution, making the cost aggregation module lighter and more efficient. Finally, we compare the proposed method with several state-of-the-art approaches on the Scene Flow and KITTI test databases. The experimental results demonstrate that the proposed method achieves competitive performance with respect to both accuracy and robustness, and it produces better results than those of other methods when given blurred edges and textureless and occluded areas. (The code is available at https://github.com/PCwenyue.) [ABSTRACT FROM AUTHOR]

Visual discomfort significantly limits the broader application of stereoscopic display technology. Hence, the accurate assessment of stereoscopic visual discomfort is a crucial topic in this field. Electroencephalography (EEG) data, which can reflect changes in brain activity, have received increasing attention in objective assessment research. However, inaccurately labeled data, resulting from the presence of individual differences, restrict the effectiveness of the widely used supervised learning methods in visual discomfort assessment tasks. Simultaneously, visual discomfort assessment methods should pay greater attention to the information provided by the visual cortical areas of the brain. To tackle these challenges, we need to consider two key aspects: maximizing the utilization of inaccurately labeled data for enhanced learning and integrating information from the brain’s visual cortex for feature representation purposes. Therefore, we propose the weakly supervised graph convolution neural network for visual discomfort (WSGCN-VD). In the classification part, a center correction loss serves as a weakly supervised loss, employing a progressive selection strategy to identify accurately labeled data while constraining the involvement of inaccurately labeled data that are influenced by individual differences during the model learning process. In the feature extraction part, a feature graph module pays particular attention to the construction of spatial connections among the channels in the visual regions of the brain and combines them with high-dimensional temporal features to obtain visually dependent spatio-temporal representations. Through extensive experiments conducted in various scenarios, we demonstrate the effectiveness of our proposed model. Further analysis reveals that the proposed model mitigates the impact of inaccurately labeled data on the accuracy of assessment. [ABSTRACT FROM AUTHOR]

In recent years, convolutional‐neural‐network based stereo matching methods have achieved significant gains compared to conventional methods in terms of both speed and accuracy. Current state‐of‐the‐art disparity estimation algorithms require many parameters and large amounts of computational resources and are not suited for applications on edge devices. In this paper, an end‐to‐end light‐weight network (LWNet) for fast stereo matching is proposed, which consists of an efficient backbone with multi‐scale feature fusion for feature extraction, a 3D U‐Net aggregation architecture for disparity computation, and color guidance in a 2D convolutional neural network (CNN) for disparity refinement. MobileNetV2 is adopted as an efficient backbone in feature extraction. The channel attention module is applied to improve the representational capacity of features and multi‐resolution information is adaptively incorporated into the cost volume via cross‐scale connections. Further, a left‐right consistency check and color guidance refinement are introduced and a robust disparity refinement network is designed with skip connections and dilated convolutions to capture global context information and improve disparity estimation accuracy with little computational cost and memory space. Extensive experiments on Scene Flow, KITTI 2015, and KITTI 2012 demonstrate that the proposed LWNet achieves competitive accuracy and speed when compared with state‐of‐the‐art stereo matching methods. [ABSTRACT FROM AUTHOR]

Abstract Stereo image pairs can effectively enhance the performance of super‐resolution (SR) since both intra‐view and cross‐view information can be used. However, exploiting cross‐view information accurately is extremely challenging. Most recent methods use the attention mechanism to get stereo correspondence. But these methods ignore the high‐frequency information since they only utilise first‐order statistics, which leads to reducing the discriminative ability of the network. To address this issue, in this work, a parallax‐based second‐order mixed attention stereo SR network (PSMASSRnet) is proposed to integrate the cross‐view information from a stereo image pair for SR. Specifically, a novel parallax‐based second‐order mixed attention module (PSMAM) is developed to combine second‐order channel features and spatial features to obtain more discriminative representations. Furthermore, a dense cross‐atrous spatial pyramid pooling (ASPP) module is also presented, which can effectively explore the local and the multi‐scale features with different dilation rates to extract more discriminative features with fewer parameters and less execution. The extensive experiments on the KITTI2012, KITTI2015 and Middlebury datasets have demonstrated the superiority of the proposed PSMASSRnet over the state‐of‐the‐art methods in the aspects of both the quantitative metrics and the visual quality.

Modeling 3D visual saliency has received great attention due to the development of emerging 3D display technologies. Traditional methods relying on low-level features may not be efficient in interpreting 3D visual content from high-level semantic perspective. Despite numerous efforts dedicated to this area, existing 3D visual saliency detection methods do not necessarily excel in exploring the stereoscopic image saliency driven by the intra-view and inter-view dependencies among left and right views. In this paper, we propose a visual saliency detection method for stereoscopic images grounded on adaptive viewpoint feature enhancement via binocular vision. More specifically, the correlation among left and right views is investigated through a delicately designed binocular stereoscopic saliency feature aggregation module, enabling the generation of more representative saliency features towards binocular vision. Subsequently, to further aggregate the saliency features in multiple scales, we design a progressive attention-based saliency feature pyramid extraction module to effectively integrate the features from top-level to down-level based on the network hierarchy mechanism. The saliency maps are ultimately produced for stereoscopic images by evaluating the obtained saliency features. In addition, we create a stereoscopic image saliency dataset (SIS-3D) that includes 1086 stereoscopic image pairs with various content and their corresponding human eye fixation annotations, aiming to further facilitate the research on visual saliency detection for stereoscopic images. Extensive experiments demonstrate that our proposed method improves CC by an average of 4.02% compared to representative counterparts on the newly built saliency dataset and another publicly available dataset. [ABSTRACT FROM AUTHOR]

Existing forgery image localization methods have achieved impressive performance on monocular images but struggle to maintain comparable performance on stereo super-resolution (SR) images. The stereo image SR process usually introduces additional noise, i.e., reconstruction artifacts, which interferes with the extraction of forgery clues and thus undermines the localization performance. To mitigate this issue, we present a multi-scale attention framework named SSR-IFL for stereo SR image forgery localization, which explores multi-scale forgery clues in stereo SR images to minimize disturbances caused by reconstruction artifacts. Specifically, the novel framework integrates two significant components: the multi-scale feature extraction (MSFE) network to thoroughly mine forgery clues against the inference of reconstruction artifacts, and the bidirectional progressive feature fusion (BPFF) network to effectively reason the learned forgery clues and obtain robust forgery feature representation. Under the MSFE network, the multi-dilated self-attention (MDSA) block makes full use of multi-resolution pixel correlation to adapt to changes in pixel correlation distributions induced by the stereo SR reconstruction process, thus extracting more precise multi-scale forgery clues. The BPFF network introduces a bidirectional feature fusion strategy to fuse multi-scale forgery clues, effectively eliminating redundant noise such as reconstruction artifacts and enhancing the robustness of forgery feature representation. Experimental results on public image datasets demonstrate that our method can achieve not only competitive performance on stereo SR images but also on monocular images. • Multi-scale attention framework for robust forgery localization in stereo SR images. • The multi-scale feature extraction network is introduced to against reconstruction artifacts. • The multi-dilated self-attention block adapts to pixel correlation changes for precise multi-scale clues. • The bidirectional progressive feature fusion network is proposed to enhance model robustness. • Competitive performance on both stereo SR and monocular images in experiments. [ABSTRACT FROM AUTHOR]

Existing stereo matching methods achieve satisfactory average accuracy on a whole predicted disparity map under common global metrics, but ignore the fine-grained performance at region level, especially for far regions in the scene, which is more crucial in actual auto-driving scenarios. There are two factors accounting for this problem: 1) Depth resolution. Existing methods use disparity-based sampling to extract matching candidates uniformly according to the disparity range, but leads to sparser sampling density at far regions than that of close regions in terms of depth range, resulting in low depth resolution in far regions. 2) Feature discriminability. Limited image resolution and inferior feature extraction at far regions result in the obtained features with low discrimination, which influences the subsequent matching process between stereo images. To improve the estimation accuracy of far regions and thus achieve a balanced performance at region level, we design a novel two-stage Balanced Stereo Matching Network (BSMNet) to address the above problems. The coarse stage of BSMNet introduces a direct depth-based sampling strategy, which generates matching candidates according to scene depth instead of disparity, thus improving the depth resolution and obtaining initial depth map with more balanced accuracy. Then, a depth refinement stage is proposed to solve the problem of low feature discriminability and further optimizes the initial depth map obtained from the coarse stage. It selects matching candidates and computes their similarity scores from a carefully designed adaptive feature volume guided by a learnable scale map, thus making the final estimation more accurate. Different from the existing methods that construct stereo matching based on disparity prediction, our proposed pipeline is to directly optimize on depth information. Experiments show that our BSMNet can obtain an obvious performance improvement at far regions without discarding that at close regions, so as to largely outperform existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In contemporary society full of stereoscopic images, how to assess visual quality of 3D images has attracted an increasing attention in field of Stereoscopic Image Quality Assessment (SIQA). Compared with 2D-IQA, SIQA is more challenging because some complicated features of Human Visual System (HVS), such as binocular interaction and binocular fusion, must be considered. In this paper, considering both binocular interaction and fusion mechanisms of the HVS, a hierarchical no-reference stereoscopic image quality assessment network (StereoIF-Net) is proposed to simulate the whole quality perception of 3D visual signals in human cortex, including two key modules: BIM and BFM. In particular, Binocular Interaction Modules (BIMs) are constructed to simulate binocular interaction in V2-V5 visual cortex regions, in which a novel cross convolution is designed to explore the interaction details in each region. In the BIMs, different output channel numbers are designed to imitate various receptive fields in V2-V5. Furthermore, a Binocular Fusion Module (BFM) with automatic learned weights is proposed to model binocular fusion of the HVS in higher cortex layers. The verification experiments are conducted on the LIVE 3D, IVC and Waterloo-IVC SIQA databases and three indices including PLCC, SROCC and RMSE are employed to evaluate the assessment consistency between StereoIF-Net and the HVS. The proposed StereoIF-Net achieves almost the best results compared with advanced SIQA methods. Specifically, the metric values on LIVE 3D, IVC and WIVC-I are the best, and are the second-best on the WIVC-II. [ABSTRACT FROM AUTHOR]

Stereoscopic omnidirectional content, as a novel visual media, has drawn wide attention in recent years due to its ability in providing strong immersive experience. Since Stereoscopic Omnidirectional Images (SOIs) involve the properties from panoramic and stereoscopic visual perception, it is very challenging to establish an efficient and effective visual quality evaluation model for SOIs. To better measure the user’s experience in virtual reality, we put forward a novel deep learning framework to assess the quality of SOIs in this paper. Firstly, the deformable convolutions instead of standard convolutions are adopted to ensure the invariant receptive fields of convolutional kernels on Equi-Rectangular Projection (ERP). Secondly, according to the stereoscopic property, we use binocular-difference information and a coarse-to-fine mechanism to construct the binocular feature extraction network. Thirdly, a three-channel network involving left-view, right-view and binocular-difference channels is presented to simulate the process of monocular and binocular interactions, in which independent quality labels are provided for each channel to reflect the individual effect of monocular and binocular visions on the whole visual quality. Finally, experimental results on two available benchmark databases demonstrate the superiority of the proposed metric over the state-of-the-art blind quality assessment models in predicting the quality of SOIs. Moreover, our model is efficient in computational cost as the feature extraction is directly applied on ERP images. [ABSTRACT FROM AUTHOR]

Stereoscopic image retargeting aims at converting stereoscopic images to the target resolution adaptively. Different from 2D image retargeting, stereoscopic image retargeting needs to preserve both the shape structure of salient objects and depth consistency of 3D scenes. In this paper, we present a stereoscopic image retargeting method based on deep convolutional neural network to obtain high-quality retargeted images with both object shape preservation and scene depth preservation. First, a cross-attention extraction mechanism is constructed to generate attention map, which contains the valuable attention features of the left and right images and the common attention features between them. Second, since the disparity map can provide accurate depth information of objects in 3D scenes, a disparity-assisted 3D significance map generation module is utilized to further preserve the valuable depth information of stereoscopic images. Finally, in order to predict the retargeted stereoscopic images accurately, an image consistency loss is developed to preserve the geometric structure of salient objects, and a disparity consistency loss is introduced to eliminate depth distortions. Experimental results demonstrate that the proposed deep convolutional neural network can provide favorable stereoscopic image retargeting results. [ABSTRACT FROM AUTHOR]

Compared with traditional 2D images, stereoscopic omnidirectional images (SOIs) usually have more complex perceptual factors due to the particularities of imaging and display, making the objective quality assessment of SOIs challenging. In this paper, we construct a large and diverse subjective SOIs database named as NBU-SOID for further research demand. And then, we propose a viewport perception based blind SOIs quality assessment (VP-BSOIQA) method by considering the impacts of viewport, user behavior and stereoscopic perception on human visual system, which is mainly composed of binocular perception model (BPM) and omnidirectional perception model (OPM). In the BPM, a binocular combination perception map is generated by the dimension reduction of stereopair and the weighting of binocular energy to reflect the binocular masking effect. In the OPM, several viewports are first created to ensure the consistency of evaluation objects. Then, the intra-viewport and inter-viewport weighting factors are designed with the common influences of visual attention and peripheral vision sensitivity to aggregate the novel multi-orientation structural features extracted from all potential viewports. Experimental results on the NBU-SOID and SOLID databases demonstrate that BPM and OPM can be robustly combined with the existing 2D image quality assessment (IQA) methods, thus averagely achieving 10.2% and 12.2% performance gain in terms of SRCC, respectively. In addition, the proposed VP-BSOIQA method outperforms the state-of-the-art blind IQA methods in predicting the quality of SOIs. [ABSTRACT FROM AUTHOR]

Deep learning-based methods have achieved remarkable performance in single image super-resolution. However, these methods cannot be effectively applied in stereoscopic image super-resolution without considering the characteristics of stereoscopic images. In this article, an interaction module-based stereoscopic image super-resolution network (IMSSRnet) is proposed to effectively utilize the correlation information in stereoscopic images. The key insight of the network lies with how to explore the complementary information of one view to help the reconstruction of another view. Thus, an interaction module is designed to acquire the enhanced features by utilizing complementary information between different views. Specifically, the interaction module is composed of a series of interaction units with a residual structure. In addition, the single image features of left and right views are obtained by a spatial feature extraction module, which can be realized by any existing single image super-resolution models. In order to obtain high-quality stereoscopic images, a gradient loss is introduced to preserve the texture details in a view, and a disparity loss is developed to constrain the disparity relationship between different views. Experimental results demonstrate that the proposed method achieves a promising performance and outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Existing RGB-D Salient Object Detection (SOD) methods take advantage of depth cues to improve the detection accuracy, while pay insufficient attention to the quality of depth information. In practice, a depth map is often with uneven quality and sometimes suffers from distractors, due to various factors in the acquisition procedure. In this article, to mitigate distractors in depth maps and highlight salient objects in RGB images, we propose a Hierarchical Alternate Interactions Network (HAINet) for RGB-D SOD. Specifically, HAINet consists of three key stages: feature encoding, cross-modal alternate interaction, and saliency reasoning. The main innovation in HAINet is the Hierarchical Alternate Interaction Module (HAIM), which plays a key role in the second stage for cross-modal feature interaction. HAIM first uses RGB features to filter distractors in depth features, and then the purified depth features are exploited to enhance RGB features in turn. The alternate RGB-depth-RGB interaction proceeds in a hierarchical manner, which progressively integrates local and global contexts within a single feature scale. In addition, we adopt a hybrid loss function to facilitate the training of HAINet. Extensive experiments on seven datasets demonstrate that our HAINet not only achieves competitive performance as compared with 19 relevant state-of-the-art methods, but also reaches a real-time processing speed of 43 fps on a single NVIDIA Titan X GPU. The code and results of our method are available at https://github.com/MathLee/HAINet. [ABSTRACT FROM AUTHOR]

Stereoscopic image quality measurement (SIQM) has become increasingly important for guiding stereo image processing and commutation systems due to the widespread usage of 3-D contents. Compared with conventional methods that are relied on handcrafted features, deep-learning-oriented measurements have achieved remarkable performance in recent years. However, most existing deep SIQM evaluators are not specifically built for stereoscopic contents and consider little prior domain knowledge of the 3-D human visual system (HVS) in network design. In this article, we develop a Predictive Auto-encoDing Network (PAD-Net) for blind/no-reference SIQM. In the first stage, inspired by the predictive coding theory that the cognition system tries to match bottom–up visual signal with top–down predictions, we adopt the encoder–decoder architecture to reconstruct the distorted inputs. Besides, motivated by the binocular rivalry phenomenon, we leverage the likelihood and prior maps generated from the predictive coding process in the Siamese framework for assisting SIQM. In the second stage, a quality regression network is applied to the fusion image for acquiring the perceptual quality prediction. The performance of PAD-Net has been extensively evaluated on three benchmark databases and the superiority has been well validated on both symmetrically and asymmetrically distorted stereoscopic images under various distortion types. [ABSTRACT FROM AUTHOR]

The satisfied user ratio (SUR) for a given distortion level is the fraction of subjects that cannot perceive a quality difference between the original image and its compressed version. By predicting the SUR, one can determine the highest distortion level which allows to save bit rate while guaranteeing a good visual quality. We propose the first method to predict the SUR for symmetrically and asymmetrically compressed stereoscopic images. Unlike SUR prediction techniques for two-dimensional images and videos, our method exploits the properties of binocular vision. We first extract features that characterize image quality and image content. Then, we use gradient boosting decision trees to reduce the number of features and train a regression model that learns a mapping function from the features to the SUR values. Experimental results on the SIAT-JSSI and SIAT-JASI datasets show high SUR prediction accuracy for H.265 All-Intra and JPEG2000 symmetrically and asymmetrically compressed stereoscopic images. [ABSTRACT FROM AUTHOR]

In recent years, consumer depth cameras have been widely used in digital entertainment and human‐machine interaction due to the advantages of real‐time performance and low cost. Facial depth maps have shown great potential in 3D‐face‐related studies. However, disadvantages of low resolution and precision limit its further applications. In this work, the authors propose an edge‐guided convolutional neural network for single facial depth map super‐resolution. It consists of two parts: an edge prediction sub‐network and a depth reconstruction sub‐network. The edge prediction sub‐network generates an edge guidance map to guide the depth reconstruction sub‐network to recover sharp edges and fine structures. Effective data augmentation methods are proposed as well. The network is patch‐based and able to cope with any size of the input depth maps. In addition, it is insensitive to the face pose since the synthetic training dataset they generated covers a wide range of face poses. The proposed method is validated with three datasets including a synthetic facial depth data set, a real Kinect V2 facial depth data set and Middlebury Stereo Data set. Experimental results show that it outperforms the state‐of‐the‐art methods on all the three data sets. [ABSTRACT FROM AUTHOR]

A novel learning‐based end‐to‐end network for stereo matching, named Multi‐path Attention Stereo Matching (MPA‐Net), is introduced in this study. Different from existing methods, the multi‐path attention aggregation module is designed firstly, named MPA, which is a unified structure using three different parallel layers with a respective attention mechanism to extract the multi‐scale informational features. Secondly, the method of cost volume construction, which differs from the traditional stereo matching methods, is extended. And then, the absolute difference between two input features is calculated. Furthermore, a u‐shaped structure with 3D attention gate is selected as the encoder‐decoder module. Specifically, the module is used to fuse the encoding features to their corresponding decoding features under the supervision of the authors' attention gate with skip‐connection, and thus exploit more significant information for matching cost regularisation and disparity prediction. Finally, specific experiments are conducted to evaluate their network on SceneFlow, KITTI2012 and KITTI2015 data sets. The results show that their method achieves a better improvement in disparity maps prediction compared with some existing state‐of‐the‐art methods on KITTI benchmark. [ABSTRACT FROM AUTHOR]

Due to the wide application of stereoscopic display technology, stereoscopic video quality assessment (SVQA) is facing great challenges, but worthwhile. Stereoscopic videos contain a great deal of information, which involves not only the spatial domain but also the spatio-temporal domain. Motion in stereoscopic video plays a critical role in quality perception, while the existing SVQA methods rarely refer to motion factors, and the performance of these methods is restrained. In this article, a novel SVQA based on motion perception is introduced and its performance is superior to that of existing excellent methods. Particularly, to appropriately reduce the amount of data processing, we extract the key-frame sequences according to the influence of movement intensity on binocular visual quality perception. The binocular summation and difference operations are implemented on extracted sequences, and then spatial texture and spatio-temporal texture statistic measurement are extracted simultaneously with local binary patterns from three orthogonal planes (LBP-TOP). Experiments are implemented on two publicly available databases and the results demonstrate the effectiveness and robustness of our algorithm for various categories of distortion stereoscopic video pairs. [ABSTRACT FROM AUTHOR]

In this paper, we devise a saliency prediction model for stereoscopic videos that learns to explore saliency inspired by the component-based interactions including spatial, temporal, as well as depth cues. The model first takes advantage of specific structure of 3D residual network (3D-ResNet) to model the saliency driven by spatio-temporal coherence from consecutive frames. Subsequently, the saliency inferred by implicit-depth is automatically derived based on the displacement correlation between left and right views by leveraging a deep convolutional network (ConvNet). Finally, a component-wise refinement network is devised to produce final saliency maps over time by aggregating saliency distributions obtained from multiple components. In order to further facilitate research towards stereoscopic video saliency, we create a new dataset including 175 stereoscopic video sequences with diverse content, as well as their dense eye fixation annotations. Extensive experiments support that our proposed model can achieve superior performance compared to the state-of-the-art methods on all publicly available eye fixation datasets. [ABSTRACT FROM AUTHOR]

For effective in situ endoscopic diagnosis and treatment, measurement of polyp sizes is important. For this purpose, 3D endoscopic systems have been researched. Among such systems, an active stereo technique, which projects a special pattern wherein each feature is coded, is a promising approach because of simplicity and high precision. However, previous works of this approach have problems. First, the quality of 3D reconstruction depended on the stabilities of feature extraction from the images captured by the endoscope camera. Second, due to the limited pattern projection area, the reconstructed region was relatively small. In this Letter, the authors propose a learning-based technique using convolutional neural networks to solve the first problem and an extended bundle adjustment technique, which integrates multiple shapes into a consistent single shape, to address the second. The effectiveness of the proposed techniques compared to previous techniques was evaluated experimentally.

As a significant branch of virtual reality, stereoscopic image stitching aims to generating wide perspectives and natural-looking scenes. Existing 2D image stitching methods cannot be successfully applied to the stereoscopic images without considering the disparity consistency of stereoscopic images. To address this issue, this paper presents a stereoscopic image stitching method based on disparity-constrained warping and blending, which could avoid visual distortion and preserve disparity consistency. First, a point-line-driven homography based disparity minimization method is designed to pre-align the left and right images and reduce vertical disparity. Afterwards, a multi-constraint warping is proposed to further align the left and right images, where the initial disparity map is introduced to control the consistency of disparities. Finally, a disparity consistency seam-cutting and blending method is presented to determine the optimal seam and conduct stereoscopic image stitching. Experimental results demonstrate that the proposed method achieves competitive performance compared with other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

For image matching, the scale invariant feature transform (SIFT) algorithm is a commonly used one. They are invariant to image rotation, scale zooming, and partially invariant to change in illumination and 3D camera viewpoint. Affine SIFT (ASIFT) is an extension of SIFT, which solves the problem when images are captured at different angles. However, ASIFT has higher computational complexity than SIFT, due to a huge amount of features in the images. Therefore, in this study, a Hadoop‐based image retrieval system is proposed to solve the ASIFT shortcomings of high computation by the MapReduce technology. The system uses a combination of the Bag‐of‐Words method and support vector machine. Finally, the experimental results verify that the proposed method is more effective than the other state‐of‐the‐art methods for a variety of datasets. [ABSTRACT FROM AUTHOR]

Visual attention is an important mechanism in the human visual system (HVS) and there have been numerous saliency detection algorithms designed for 2D images/video recently. However, the research for fixation detection of stereoscopic video is still limited and challenging due to the complicated depth and motion information. In this paper, we design a novel multi-module fully convolutional network (MM-FCN) for fixation detection of stereoscopic video. Specifically, we design a fully convolutional network for spatial saliency prediction (S-FCN), where the initial spatial saliency map of stereoscopic video is learned by image database of object detection. Furthermore, the fully convolutional network for temporal saliency prediction (T-FCN) is constructed by combining saliency results from S-FCN and motion information from video frames. Finally, the fully convolutional network for depth fixation prediction (D-FCN) is designed to compute the final fixation map of stereoscopic video by learning depth features with spatiotemporal features from T-FCN. The experimental results show that the proposed MM-FCN can predict fixation results for stereoscopic video more effectively and efficiently than other related fixation prediction methods. [ABSTRACT FROM AUTHOR]

For visual discomfort prediction (VDP) of stereoscopic 3D images, a common two-stage framework is to first extract features that are predictive of the experienced visual discomfort level when viewing stereoscopic images and then use typical regression tools to learn the mapping from the extracted features to visual discomfort scores. Most existing approaches for stereoscopic 3D VDP focus on the former stage, i.e., feature extraction, while limited efforts have dedicated to exploiting more powerful and robust learning algorithms in this field. In this letter, inspired by the pairwise comparison-based subjective evaluation methodology, we propose a novel Risk-Aware Pairwise Rank Learning (RAPRL) approach to further improve the prediction accuracy. Unlike the traditional VDP approaches using different regression tools for feature-score mapping, our proposed RARL method addresses this problem based on a completely different pairwise rank learning framework with a risk-aware constraint. Experiments have verified the effectiveness and robustness of our proposed VDP model using RAPRL as the learning algorithm. [ABSTRACT FROM AUTHOR]

In this paper, we proposed an effective 3-D image-quality assessment method based on an adaptive cyclopean image by using ensemble learning. Our cyclopean image is not only suitable for a symmetrical distortion image, but also especially suitable for an asymmetrical distortion image. This adaptivity of our cyclopean image can be attributed to the consideration of gain control and gain enhancement in a binocular rivalry visual mechanism. In addition, we use a salient map to modify our cyclopean image to let the salient area of our cyclopean become more attractive. As a result, we can get better results. To remove redundant information out from our cyclopean, the sparse representation is applied to extract essential features. Finally, to get better regression accuracy on extracted feature, we use ensemble learning to get the final quality score of a stereoscopic image. The ensemble learner can improve the regression accuracy by 2% than a single learner. Experimental results show that the proposed algorithm outperforms the state-of-the-art methods on two publicly available stereoscopic image-quality assessment databases LIVE I and LIVE II. [ABSTRACT FROM AUTHOR]

We present an unsupervised segmentation framework for stereo videos using stereoscopic trajectories. The proposed stereo trajectory shows favorable properties for modeling the long-term motion information through the whole sequence and explicitly capturing the corresponding relationships between two stereo views. The stereo prior is important for inferring the desired object and guarantees the consistent spatial-temporal segmentation, which contributes to an enjoyable stereo experience. We start by deriving stereo trajectories from left and right views simultaneously, which are represented via a graph structure. Then we detect object-like stereo trajectories via the graph structure to efficiently infer the desired object. Finally, an energy optimization function is proposed to produce the stereo segmentation results via leveraging the object information from stereo trajectories. To benefit potential research, we collected a new stereoscopic video benchmark, which consists of a total of 50 stereo video clips and includes many challenges in segmentation. Extensive experimental results demonstrate that our stereo segmentation method achieves higher performance and preserves better stereo structures, compared with prevailing competitors. The source code and results are available at: https://github.com/shenjianbing/StereoSeg. [ABSTRACT FROM AUTHOR]

Most previous 2-D and 3-D image quality evaluators were based on shallow architectures. Their shallow architectures cannot model phenomenon occurring in human visual systems sufficiently. Disparities between left and right views have been importantly used for 3-D image quality assessment (IQA), but single disparity, depth, or cyclopean maps made from the disparities cannot thoroughly reflect the depth sense. In this paper, we propose a blind stereoscopic image quality evaluator using stacked auto-encoders (SAE). The proposed method is based on two theories on initial stages of stereopsis. One is cyclopean channel theory and the other is binocular summation/difference channels theory. Especially, a cyclopean image that models the former theory is computed to consider binocular suppression, whereas summation and difference images that model the latter one are utilized to treat the depth sense. We train three SAEs for quality-aware features from the three images in an unsupervised manner. Through the SAEs, the features are transformed into more meaningful features, and they are used to train two regressors. The regressors are used to obtain a final predicted score. Experimental results conducted on popular 3-D IQA databases prove that the proposed algorithm outperforms state-of-the-art 3-D IQA methods. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel deep learning-based visual comfort assessment (VCA) for stereoscopic images. To assess the overall degree of visual discomfort in stereoscopic viewing, we devise a binocular fusion deep network (BFN) learning binocular characteristics between stereoscopic images. The proposed BFN learns the latent binocular feature representations for the visual comfort score prediction. In the BFN, the binocular feature is encoded by fusing the spatial features extracted from left and right views. Finally, the visual comfort score is predicted by projecting the binocular feature onto the subjective score space. In addition, we devise a disparity regularization network (DRN) for improving the prediction results. The proposed DRN takes the binocular feature from the BFN and estimates disparity maps from the feature in order to embed disparity relations between left and right views into the deep network. The proposed deep network with BFN and DRN is end-to-end trained in a unified framework in which the DRN acts as disparity regularization. We evaluated the prediction performance of the proposed deep network for VCA by the comparison of existing objective VCA metrics. Further, we demonstrated that the proposed BFN showed various factors causing visual discomfort by using network visualization. [ABSTRACT FROM AUTHOR]

This study proposes a robust method for the face recognition from low-resolution red, green, and blue-depth (RGB-D) cameras acquired images which have a wide range of variations in head pose, illumination, facial expression, and occlusion in some cases. The local binary pattern (LBP) of the RGB-D images with the suitable feature dimension of Depth image is employed to extract the facial features. On the basis of error correcting output codes, they are fed to multiclass support vector machines (MSVMs) for the off-line training and validation, and then the online classification. The proposed method is called as the LBP-RGB-D-MSVM with the suitable feature dimension of the depth image. The effectiveness of the proposed method is evaluated by the four databases: Indraprastha Institute of Information Technology, Delhi (IIIT-D) RGB-D, visual analysis of people (VAP) RGB-D-T, EURECOM, and the authors. In addition, an extended database merged by the first three databases is employed to compare among the proposed method and some existing two-dimensional (2D) and 3D face recognition algorithms. The proposed method possesses satisfactory performance (as high as 99.10 ± 0.52% for Rank 5 recognition rate in their database) with low computation (62 ms for feature extraction) which is desirable for real-time applications.

This paper proposes a stable method of generating a stereoscopic panoramic video with the omnidirectional stereo (ODS) format. Different from the traditional image stitching method which can only be applied to generate a monocular panorama, we adopt an optical flow-based blending method to create two panoramas for binocular vision. In addition, traditional image stitching methods based on seam-finding tend to cause the problem of temporal flicker. We address this problem by restricting the optical flow field of the new frame with its previous frame's optical flow field. Thus, the generated video is stable and out of temporal flicker. There are four key operations in our approach. First, we adopt the ODS format which is the basis of stereoscopic panorama. Second, we do effective exposure compensation, making the brightness of the two eyes' panoramas consistent. Third, we employ an optical flow-based blending method to synthesis the final panorama effectively. Fourth, we take the previous frame's optical flow field as the restriction of the present frame's optical flow field to acquire a stable video. The final output videos can deliver a pleasant and impressive stereoscopic viewing experience to the audience when the audience watches the videos in the virtual reality headset.

Hyperspectral images can present low contrast, noisy pixels, and illumination variation among bands, which complicates the extraction of interest points and reduces the number of reliable image matches affecting subsequent tasks as band registration and bundle adjustment. Once matched points have been determined, a technique to select correct matches in sets with outliers is required, as well as to fix mismatches. In this letter, we apply a filtering technique that uses a majority voting algorithm combined with a 2-D Helmert geometric transformation to identify consistent matches. The correct matches also allow the estimation of parameters of a geometric transformation, which enables point transfer between images. Thus, mismatches can be fixed to their correct positions. Experiments were performed with the proposed technique using hyperspectral images that were collected with a lightweight camera using the time-sequential principle, while onboard an unmanned aerial vehicle. Scale-invariant feature transform was used for both keypoint extraction and image matching. Reliable matches were extracted from the sets with outliers, and incorrect matches were fixed. The results of the technique were compared with an algorithm based on random sample consensus. In the comparison, the proposed technique was efficient in extracting a larger number of correct matches. In addition, 85% of the incorrect matches were recovered, which significantly increased the density of matched pairs. [ABSTRACT FROM AUTHOR]

In the field of 3-D technology, it is interesting as well as meaningful issue to control object depth in 3-D space. Recently, some depth control methods for stereoscopic images have been proposed, which usually employ depth map or directly process color images to implement depth control. There are two main disadvantages for these methods. First, the results of these methods usually suffer from object deformation and holes. Second, these methods are prone to cause undesired object size changing in 3-D space. To address these issues, we propose a shape-preserving object depth control method for stereoscopic images. First, a novel depth mapping model is presented for calculating the ideal coordinates of the key points in depth control, so that the shape of the object can be well preserved. Afterward, the image content-based constraints are used to further preserve the structure of the object and its background. Finally, the warping technology is introduced to deal with images optimally as well as to avoid holes. Experimental results show that the proposed method can control object depth and preserve the shape of the object effectively without sensible background distortion. [ABSTRACT FROM AUTHOR]

Quality prediction for asymmetric multiply-distorted stereoscopic images (MDSIs) confronts more challenges than previous stereoscopic image quality assessment (SIQA) issues, whereas the existing no-reference SIQA methods have been limited to understand the asymmetric distortions and multiple distortions simultaneously for general-purpose blind quality prediction. In this paper, we propose a multistage pooling (MUSP) model for quality prediction of asymmetric MDSIs. In the training stage, we establish multimodal sparse representation framework for phase and amplitude components, respectively. In the testing stage, we use an MUSP strategy to simulate the pooling procedure undergoing multimodal quality pooling, feature pooling, binocular pooling, and phase-amplitude quality pooling in order. Experimental results on our new established database (NBU-MDSID Phase-II) demonstrate the effectiveness of our blind metric. [ABSTRACT FROM AUTHOR]

Disparity refinement is an important step to enhance the accuracy of stereo matching. This paper extends the scheme of a recent successful approach, namely the nonlocal disparity refinement algorithm, to exploit the initial disparity map in the aggregation phase of disparity refinement, in addition to the information of spatial distance and intensity difference. In addition, we propose a constraint function applied to the matching cost that constrains the scope of dissimilarity measures to further improve the accuracy of disparity refinement. Extensive experimental comparisons with several state-of-the-art methods using the Middlebury Stereo Evaluation version 3 datasets show that the proposed scheme has a great advantage in disparity refinement. ©2018 SPIE and IS&T [DOI: 10.1117/1.JEI.27.2.023031] [ABSTRACT FROM AUTHOR]

In this paper, an online adaptive model-free tracker is proposed to track single objects in video sequences to deal with real-world tracking challenges like low-resolution, object deformation, occlusion and motion blur. The novelty lies in the construction of a strong appearance model that captures features from the initialized bounding box and then are assembled into anchor point features. These features memorize the global pattern of the object and have an internal star graph-like structure. These features are unique and flexible and help tracking generic and deformable objects with no limitation on specific objects. In addition, the relevance of each feature is evaluated online using short-term consistency and long-term consistency. These parameters are adapted to retain consistent features that vote for the object location and that deal with outliers for long-term tracking scenarios. Additionally, voting in a Gaussian manner helps in tackling inherent noise of the tracking system and in accurate object localization. Furthermore, the proposed tracker uses pairwise distance measure to cope with scale variations and combines pixel-level binary features and global weighted color features for model update. Finally, experimental results on a visual tracking benchmark dataset are presented to demonstrate the effectiveness and competitiveness of the proposed tracker. [ABSTRACT FROM AUTHOR]

This paper presents a new stereo image (SI) retrieval method based on a statistical model of complex wavelet coefficients subbands. In this context, a Gaussian copula-based multivariate model is used to capture the dependence between complex wavelet coefficients of both left and right images, and a non-Gaussian univariate model is used to characterize the statistical behavior of the disparity map. Thanks to its flexibility, the copula tool allows us to choose several marginal densities while keeping the multivariate properties. Features are extracted by estimating parameters for both multivariate and univariate models. Finally, a weighted Jeffrey divergence (JD) is used as a similarity measurement between the underlying models. Experimental results on a stereo image database demonstrate the performance of the proposed method in terms of the retrieval rates as well as the computational time. [ABSTRACT FROM AUTHOR]

Visual comfort and depth sensation are two important incongruent counterparts in determining the overall stereoscopic 3-D experience. In this paper, we proposed a novel simultaneous visual comfort and depth sensation optimization approach for stereoscopic images. The main motivation of the proposed optimization approach is to enhance the overall stereoscopic 3-D experience. Toward this end, we propose a two-stage solution to address the optimization problem. In the first layer-independent disparity adjustment process, we iteratively adjust the disparity range of each depth layer to satisfy with visual comfort and depth sensation constraints simultaneously. In the following layer-dependent disparity process, disparity adjustment is implemented based on a defined total energy function built with intra-layer data, inter-layer data and just noticeable depth difference terms. Experimental results on perceptually uncomfortable and comfortable stereoscopic images demonstrate that in comparison with the existing methods, the proposed method can achieve a reasonable performance balance between visual comfort and depth sensation, leading to promising overall stereoscopic 3-D experience. [ABSTRACT FROM PUBLISHER]

The depth measurement error of bionic eyes consisting of two active cameras is derived by considering the eyes' rotation angle errors and image feature extraction errors. The effect factors of the depth measurement are obtained from the calculation formula of depth error. Based on the analysis of these effect factors, some effective guidelines for bionic eyes are proposed to reduce the measurement errors. The guidelines suggest keeping long baseline, observing the target as nearly as possible, controlling two active cameras with the same angular velocity and keeping the target on Zw axis if possible. The simulation experiments and practical experiments in bionic eyes platform validate the effectiveness of the proposed guidelines. [ABSTRACT FROM AUTHOR]

The problem of stereoscopic image quality assessment, which finds applications in 3D visual content delivery such as 3DTV, is investigated in this work. Specifically, we propose a new ParaBoost (parallel-boosting) stereoscopic image quality assessment (PBSIQA) system. The system consists of two stages. In the first stage, various distortions are classified into a few types, and individual quality scorers targeting at a specific distortion type are developed. These scorers offer complementary performance in face of a database consisting of heterogeneous distortion types. In the second stage, scores from multiple quality scorers are fused to achieve the best overall performance, where the fuser is designed based on the parallel boosting idea borrowed from machine learning. Extensive experimental results are conducted to compare the performance of the proposed PBSIQA system with those of existing stereo image quality assessment (SIQA) metrics. The developed quality metric can serve as an objective function to optimize the performance of a 3D content delivery system. [ABSTRACT FROM AUTHOR]

With widespread applications of three-dimensional (3-D) technology, measuring quality of experience for 3-D multimedia content plays an increasingly important role. In this paper, we propose a full reference stereo image quality assessment (SIQA) framework which focuses on the innovation of binocular visual properties and applications of low-level features. On one hand, based on the fact that human visual system understands an image mainly according to its low-level features, local phase and local amplitude extracted from phase congruency measurement are employed as primary features. Considering the less prominent performance of amplitude in IQA, visual saliency is applied into the modification on amplitude. On the other hand, by fully considering binocular rivalry phenomena, we create the cyclopean amplitude map and cyclopean phase map. With this method, both image features and binocular visual properties are mutually combined with each other. Meanwhile, a novel binocular modulation function in spatial domain is also adopted into the overall quality prediction of amplitude and phase. Extensive experiments demonstrate that the proposed framework achieves higher consistency with subjective tests than relevant SIQA metrics. [ABSTRACT FROM PUBLISHER]

Perceptual quality prediction for stereoscopic images is of fundamental importance in determining the level of quality perceived by humans in terms of the 3D viewing experience. However, the existing no-reference quality assessment (NR-IQA) framework has its limitation in addressing binocular combination for stereoscopic images. In this paper, we propose a new NR-IQA for stereoscopic images using joint sparse representation. We analyze the relationship between left and right quality predictors, and formulate stereoscopic quality prediction as a combination of feature-prior and feature-distribution. Based on this finding, we extract feature vector that handles different features to be interacted by joint sparse representation, and use support vector regression to characterize feature-prior. Meanwhile, we implement feature-distribution using sparsity regularization as the basis of weights for binocular combination to derive the overall quality score. Experimental results on five public 3D IQA databases demonstrate that in comparison with the existing methods, the devised algorithm achieves high consistent alignment with subjective assessment. [ABSTRACT FROM PUBLISHER]

Pedestrian detection is an important yet challenging task. In this paper, a pedestrian codetection framework for binocular stereo sequences is proposed. Binocular vision and consecutive frames can provide more information than a single image. That information can be used to improve detection performance by reducing the number of candidate detection windows with low confidence or enhancing the high-confidence candidates. To design the framework, we follow the intuition that a pedestrian has consistent appearance when observed from the same or different viewpoints. First, a baseline detector is used in both stereo images with a conservative threshold in order to expand the set of detection candidates. Before the detection process, an assisted stixel world model is computed for both left and right frames. Thus, the search range of the detector is greatly reduced with the help of the stixel world. Second, adjacency-constrained patch matching is proposed to build correspondence between two candidates in both intra and inter sequences of binocular vision. Finally, we establish a mechanism to update the score of the detection aided by the corresponding candidates. The experimental results show that our framework significantly improves the performance of the evaluated baseline pedestrian approaches. [ABSTRACT FROM AUTHOR]

Recently deep learning‐based methods for dense depth completion from sparse depth data have shown superior performance than traditional techniques. However, sparse depth data lose the details of the scenes, for instance, the spatial and texture information. To overcome this problem, additional single image is introduced and a multi‐scale features fusion scheme to learn more correlations of the two different data is proposed. Furthermore, sparse convolution operation to improve feature robustness for sparse depth data is exploited. Experiments demonstrate that the approach obviously improves the performance for depth completion and outperforms all the previous published methods. The authors believe their works also have the guidance significance for stereo images depth estimation fused with sparse LiDAR depth data. [ABSTRACT FROM AUTHOR]

Recently, the stereo matching task has been dramatically promoted by the deep learning methods. Specifically, the encoder–decoder framework with skip connection achieves outstanding performance over others. The skip connection scheme can bring detailed or in other words, residual information for the final prediction, thus improves the performance, which is successfully applied in many other pixel‐wise prediction tasks, such as semantic segmentation, depth estimation and so on. In contrast to other tasks, the authors can explicitly obtain the residual information for stereo matching, which is achieved by back‐warping the right image and calculating the reconstruction error. The reconstruction error is successfully used as unsupervised loss, but has not been explored for skip connection. In this Letter, the authors show that the reconstruction error in the feature space is very helpful to bring residual information for the final prediction. They validate the effectiveness of using reconstruction error for skip connection by conducting experiments on the KITTI 2015 and Scene Flow datasets. Experiments show that the proposed scheme can improve the performance by a notable margin and achieves the state‐of‐the‐art performance with very fast processing time. [ABSTRACT FROM AUTHOR]

Predicting the degree of experienced visual comfort in the context of stereoscopic 3-D (S3D) viewing is particularly challenging. In this letter, a simple yet effective visual comfort assessment (VCA) approach for stereoscopic images is proposed from the perspective of learning to rank (L2R). The proposed L2R-based VCA (L2R-VCA) approach is inspired by the traditional absolute categorical rating (ACR) methodology in subjective study and is to characterize the qualitative description behavior of human subjective study. Experimental results on our recently built database confirm the promising performance of the proposed L2R-VCA approach, yielding higher consistency with human subject judgment results. [ABSTRACT FROM PUBLISHER]

We describe a method to obtain a pixel-wise segmentation and pose estimation of multiple people in stereoscopic videos. This task involves challenges such as dealing with unconstrained stereoscopic video, non-stationary cameras, and complex indoor and outdoor dynamic scenes with multiple people. We cast the problem as a discrete labelling task involving multiple person labels, devise a suitable cost function, and optimize it efficiently. The contributions of our work are two-fold: First, we develop a segmentation model incorporating person detections and learnt articulated pose segmentation masks, as well as colour, motion, and stereo disparity cues. The model also explicitly represents depth ordering and occlusion. Second, we introduce a stereoscopic dataset with frames extracted from feature-length movies “StreetDance 3D” and “Pina”. The dataset contains 587 annotated human poses, 1,158 bounding box annotations and 686 pixel-wise segmentations of people. The dataset is composed of indoor and outdoor scenes depicting multiple people with frequent occlusions. We demonstrate results on our new challenging dataset, as well as on the H2view dataset from (Sheasby et al. ACCV 2012). [ABSTRACT FROM PUBLISHER]

Finding feature correspondences between a pair of stereo images is a key step in computer vision for 3D reconstruction and object recognition. In practice, a larger number of correct correspondences and a higher percentage of correct matches are beneficial. Previous researches show that the spatial distribution of correspondences are also very important especially for fundamental matrix estimation. So far, no existing feature matching method considers the spatial distribution of correspondences. In our research, we develop a new algorithm to find good correspondences in all the three aspects mentioned, i.e., larger number of correspondences, higher percentage of correct correspondences, and better spatial distribution of correspondences. Our method consists of two processes: an adaptive disparity smoothing filter to remove false correspondences based on the disparities of neighboring correspondences and a matching exploration algorithm to find more correspondences according to the spatial distribution of correspondences so that the correspondences are as uniformly distributed as possible in the images. To find correspondences correctly and efficiently, we incorporate the cheirality constraint under an epipole polar transformation together with the epipolar constraint to predict the potential location of matching point. Experiments demonstrate that our method performs very well on both the number of correct correspondences and the percentage of correct correspondences; and the obtained correspondences are also well distributed over the image space. [ABSTRACT FROM AUTHOR]