Search

Object detection in aerial images has achieved remarkable progress with the advent of deep convolutional neural networks (CNNs). It is, however, still a challenging task since the objects in aerial images are arbitrarily oriented and often densely packed. In this letter, we propose a novel method for oriented object detection in aerial images that represents objects as rotation equivariant semantic keypoints. Unlike conventional methods that represent object rotation according to angles from each axis in the Cartesian coordinate system, we represent object using a canonical orientation to ensure rotation equivariance. We accomplish this by representing an object as semantic keypoints, where each keypoint of the object consistently corresponds to the semantic part, regardless of rotation variation. To this end, we define the “head” point of the object as the canonical orientation and the remaining bounding box vectors as semantic keypoints in clockwise order. To discriminate visual attributes between different categories, we further use category-specific semantic keypoints, so that object classification and localization can be jointly solved in a cooperative manner. Our experiments demonstrate the effectiveness of rotation equivariant semantic keypoints on oriented object detection. [ABSTRACT FROM AUTHOR]

Recent deep neural networks based methods have achieved state-of-the-art performance on various facial expression recognition tasks. Despite such progress, previous researches for facial expression recognition have mainly focused on analyzing color recording videos only. However, the complex emotions expressed by people with different skin colors under different lighting conditions through dynamic facial expressions can be fully understandable by integrating information from multi-modal videos. We present a novel method to estimate dimensional emotion states, where color, depth, and thermal recording videos are used as a multi-modal input. Our networks, called multi-modal recurrent attention networks (MRAN), learn spatiotemporal attention volumes to robustly recognize the facial expression based on attention-boosted feature volumes. We leverage the depth and thermal sequences as guidance priors for color sequence to selectively focus on emotional discriminative regions. We also introduce a novel benchmark for multi-modal facial expression recognition, termed as multi-modal arousal-valence facial expression recognition (MAVFER), which consists of color, depth, and thermal recording videos with corresponding continuous arousal-valence scores. The experimental results show that our method can achieve the state-of-the-art results in dimensional facial expression recognition on color recording datasets including RECOLA, SEWA and AFEW, and a multi-modal recording dataset including MAVFER. [ABSTRACT FROM AUTHOR]

The sensor response function of a color camera is very essential to understand an overall camera imaging pipeline and to process captured images. It is also true when we characterize underlying imaging behaviors of the electron multiplying charge coupled device (EMCCD) camera, which was recently proposed to acquire color images in low-light-level conditions. Unlike existing CCD cameras, the EMCCD camera contains partially uni-modal spectral sensitivity functions (SSFs), some of which consists of two base (bi-modal) functions, thus often leading to a serious color distortion in existing characterization techniques. To address this problem, we propose a novel method that corrects output colors of the EMCCD camera by analyzing its partially uni-modal characteristics. Specifically, our goal is to correct such color distortion by adjusting bi-modal SSFs to be uni-modal. We remove the secondary region of bi-modal channels using a pre-calculated spectral sensitivity of the EMCCD camera. Experimental results demonstrate that the proposed method reduces the color distortion as well as enlarges a color gamut, which is crucial to color reproduction. [ABSTRACT FROM AUTHOR]

The human category is among the most commonly captured and therefore important subject in digital cameras. As a result, human segmentation also plays an important role in the camera system. This paper presents a framework of non-parametric learning that achieves human segmentation using support vector machine (SVM). In the training stage, the training human windows are warped to a normalized size and oversegmented into regular superpixels. A two-dimensional (2D) array of SVM models is then trained by extracting various edge and color features from each superpixel. In this process of SVM training, the 2D array of SVM models automatically and effectively learns the characteristics of the human shape. Given an affinely warped human window for testing, the proposed method calculates superpixels' initial scores of belonging to a human (foreground) using the trained SVM models. Finally, the initial prediction scores are effectively propagated by optimizing a well-defined energy function using an estimated confidence map. In experiments on a publicly available challenging dataset, the proposed framework rapidly yields excellent results in human segmentation both qualitatively and quantitatively1. [ABSTRACT FROM PUBLISHER]

A robust similarity measure called the Mahalanobis distance cross-correlation (MDCC) is proposed for illumination-invariant stereo matching, which uses a local color distribution within support windows. It is shown that the Mahalanobis distance between the color itself and the average color is preserved under affine transformation. The MDCC converts pixels within each support window into the Mahalanobis distance transform (MDT) space. The similarity between MDT pairs is then computed using the cross-correlation with an asymmetric weight function based on the Mahalanobis distance. The MDCC considers correlation on cross-color channels, thus providing robustness to affine illumination variation. Experimental results show that the MDCC outperforms state-of-the-art similarity measures in terms of stereo matching for image pairs taken under different illumination conditions. [ABSTRACT FROM PUBLISHER]

Quality perception of 3-D images is one of the most important parameters for accelerating advances in 3-D imaging fields. Despite active research in recent years for understanding the quality perception of 3-D images, binocular quality perception of asymmetric distortions in stereoscopic images is not thoroughly comprehended. In this paper, we explore the relationship between the perceptual quality of stereoscopic images and visual information, and introduce a model for binocular quality perception. Based on this binocular quality perception model, a no-reference quality metric for stereoscopic images is proposed. The proposed metric is a top-down method modeling the binocular quality perception of the human visual system in the context of blurriness and blockiness. Perceptual blurriness and blockiness scores of left and right images were computed using local blurriness, blockiness, and visual saliency information and then combined into an overall quality index using the binocular quality perception model. Experiments for image and video databases show that the proposed metric provides consistent correlations with subjective quality scores. The results also show that the proposed metric provides higher performance than existing full-reference methods even though the proposed method is a no-reference approach. [ABSTRACT FROM AUTHOR]

Color-plus-depth video format has been increasingly popular in 3-D video applications, such as auto-stereoscopic 3-D TV and freeview TV. The performance of these applications is heavily dependent on the quality of depth maps since intermediate views are synthesized using the corresponding depth maps. This paper presents a novel framework for obtaining high-quality multiview color-plus-depth video using a hybrid sensor, which consists of multiple color cameras and depth sensors. Given multiple high-resolution color images and low quality depth maps obtained from the color cameras and depth sensors, we improve the quality of the depth map corresponding to each color view by increasing its spatial resolution and enforcing interview coherence. Specifically, a new up-sampling method considering the interview coherence is proposed to enhance multiview depth maps. This approach can improve the performance of the existing up-sampling algorithms, such as joint bilateral up-sampling and weighted mode filtering, which have been developed to enhance a single-view depth map only. In addition, an adaptive approach of fusing multiple input low-resolution depth maps is proposed based on the reliability that considers camera geometry and depth validity. The proposed framework can be extended into the temporal domain for temporally consistent depth maps. Experimental results demonstrate that the proposed method provides better multiview depth quality than the conventional single-view-based methods. We also show that it provides comparable results, yet much more efficiently, to other fusion approaches that employ both depth sensors and stereo matching algorithm together. Moreover, it is shown that the proposed method significantly reduces bit rates required to compress the multiview color-plus-depth video. [ABSTRACT FROM AUTHOR]