Search

In general, vehicle images have varying resolutions due to vehicles’ movements and different camera settings. However, most existing vehicle re-identification models are single-resolution deep networks trained with pre-uniformly resizing vehicle images, which underestimates adverse effects of varying resolutions and leads to unsatisfactory performance. A straightforward solution for dealing with varying resolutions is to train multiple vehicle re-identification models. Each model is independently trained with images of a specific resolution. However, this straightforward solution requires significant overhead and ignores intrinsic associations among different resolution images. For that, an efficient multi-resolution network (EMRN) is proposed for vehicle re-identification in this paper. Firstly, EMRN embeds a newly-designed multi-resolution feature dimension uniform module (MR-FDUM) behind a traditional backbone network (i.e., ResNet-50). As a result, the whole model can extract fixed dimensional features from different resolution images so that it can be trained with one loss function of fixed dimensional parameters rather than training multiple models. Secondly, a multi-resolution image randomly feeding strategy is designed to train EMRN, making each mini-batch data of a random resolution during the training process. Consequently, EMRN can implicitly learn collaborative multi-resolution features via only a unitary deep network. Experiments on three large scale datasets, i.e., VeRi776, VehicleID, and VRIC, demonstrate that EMRN is superior to state-of-the-art vehicle re-identification methods.

A densely-sampled light field (LF) is highly desirable in various applications, such as 3-D reconstruction, post-capture refocusing and virtual reality. However, it is costly to acquire such data. Although many computational methods have been proposed to reconstruct a densely-sampled LF from a sparsely-sampled one, they still suffer from either low reconstruction quality, low computational efficiency, or the restriction on the regularity of the sampling pattern. To this end, we propose a novel learning-based method, which accepts sparsely-sampled LFs with irregular structures, and produces densely-sampled LFs with arbitrary angular resolution accurately and efficiently. We also propose a simple yet effective method for optimizing the sampling pattern. Our proposed method, an end-to-end trainable network, reconstructs a densely-sampled LF in a coarse-to-fine manner. Specifically, the coarse sub-aperture image (SAI) synthesis module first explores the scene geometry from an unstructured sparsely-sampled LF and leverages it to independently synthesize novel SAIs, in which a confidence-based blending strategy is proposed to fuse the information from different input SAIs, giving an intermediate densely-sampled LF. Then, the efficient LF refinement module learns the angular relationship within the intermediate result to recover the LF parallax structure. Comprehensive experimental evaluations demonstrate the superiority of our method on both real-world and synthetic LF images when compared with state-of-the-art methods. In addition, we illustrate the benefits and advantages of the proposed approach when applied in various LF-based applications, including image-based rendering and depth estimation enhancement., 17 pages, 11 figures, 10 tables

This paper presents a new full-reference image quality assessment (IQA) method for conducting the perceptual quality evaluation of the light field (LF) images, called the symmetry and depth feature-based model (SDFM). Specifically, the radial symmetry transform is first employed on the luminance components of the reference and distorted LF images to extract their symmetry features for capturing the spatial quality of each view of an LF image. Second, the depth feature extraction scheme is designed to explore the geometry information inherited in an LF image for modeling its LF structural consistency across views. The similarity measurements are subsequently conducted on the comparison of their symmetry and depth features separately, which are further combined to achieve the quality score for the distorted LF image. Note that the proposed SDFM that explores the symmetry and depth features is conformable to the human vision system, which identifies the objects by sensing their structures and geometries. Extensive simulation results on the dense light fields dataset have clearly shown that the proposed SDFM outperforms multiple classical and recently developed IQA algorithms on quality evaluation of the LF images.

Pedestrian gender recognition plays an important role in smart city. To effectively improve the pedestrian gender recognition performance, a new method, called cascading scene and viewpoint feature learning (CSVFL), is proposed in this article. The novelty of the proposed CSVFL lies on the joint consideration of two crucial challenges in pedestrian gender recognition, namely, scene and viewpoint variation. For that, the proposed CSVFL starts with the scene transfer (ST) scheme, followed by the viewpoint adaptation (VA) scheme in a cascading manner. Specifically, the ST scheme exploits the key pedestrian segmentation network to extract the key pedestrian masks for the subsequent key pedestrian transfer generative adversarial network, with the goal of encouraging the input pedestrian image to have the similar style to the target scene while preserving the image details of the key pedestrian as much as possible. Afterward, the obtained scene-transferred pedestrian images are fed to train the deep feature learning network with the VA scheme, in which each neuron will be enabled/disabled for different viewpoints depending on whether it has contribution on the corresponding viewpoint. Extensive experiments conducted on the commonly used pedestrian attribute data sets have demonstrated that the proposed CSVFL approach outperforms multiple recently reported pedestrian gender recognition methods.

Vehicle re-identification (Re-ID) technology plays an important role in the intelligent transportation system for smart city. Due to various uncertain factors in the real-world scenarios, (e.g., resolution variation, viewpoint variation, illumination changes, occlusion, etc., vehicle Re-ID is a very challenging task. To resist the adverse effect of resolution variation, a joint pyramid feature representation network (JPFRN) for vehicle Re-ID is proposed in this paper. Based on the consideration that various convolution blocks with different depths hold different resolutions and semantic information of the vehicle image, the proposed JPFRN method employs a base network to obtain multi-resolution vehicle features in the first stage. Then, a pyramid feature representation scheme is developed to reconstruct and integrate the obtained multi-resolution vehicle features together. Finally, these pyramid features are jointly represented for learning a more discriminative feature under the supervision of joint Triplet loss and softmax loss. Extensive experimental results on two commonly-used vehicle databases (i.e., VehicleID and VeRi) show that the proposed JPFRN is superior to multiple recently-developed vehicle Re-ID methods.

Object reidentification with the goal of matching pedestrian or vehicle images captured from different camera viewpoints is of considerable significance to public security. Quadruple directional deep learning features (QD-DLFs) can comprehensively describe object images. However, the correlation among QD-DLFs is an unavoidable problem, since QD-DLFs are learned with quadruple independent directional deep networks (QIDDNs) driven with the same training data, and each network holds the same basic deep feature learning architecture (BDFLA). The correlation among QD-DLFs is harmful to the complementarity of QD-DLFs, restricting the object reidentification performance. For that, we propose joint quadruple decorrelation directional deep networks (JQD3Ns) to reduce the correlation among the learned QD-DLFs. In order to jointly train JQD3Ns, besides the softmax loss functions, a parameter correlation cost function is proposed to indirectly reduce the correlation among QD-DLFs by enlarging the dissimilarity among the parameters of JQD3Ns. Extensive experiments on three publicly available large-scale data sets demonstrate that the proposed JQD3Ns approach is superior to multiple state-of-the-art object reidentification methods.

In recent years, deep learning (DL) has been successfully and widely applied in the person reidentification (Re-ID). However, the DL-based person Re-ID methods face a bottleneck that the scales of most existing person Re-ID databases are not large enough for training very deep models. To address this problem, a body symmetry and part-locality-guided direct nonparametric deep feature enhancement (DNDFE) method is proposed in this article. Based on the observation that the body symmetry and part locality are two important appearance properties inherited in the upright walking persons, the proposed method designs two nonparametric layers, namely, the body symmetry average pooling and local normalization layers, to construct a DNDFE module to well explore the body symmetry and part locality properties. The proposed DNDFE module could be directly embedded between the traditional deep feature learning module and similarity learning module to enhance the DL features so as to improve the person Re-ID performance. The experimental results have shown that the proposed DNDFE method is superior to multiple state-of-the-art person Re-ID methods in terms of accuracy and efficiency.

This paper investigates the 4-D light field (LF) reconstruction from 2-D measurements captured by the coded aperture camera. To tackle such an ill-posed inverse problem, we propose a cycle-consistent reconstruction network (CR-Net). To be specific, based on the intrinsic linear imaging model of the coded aperture, CR-Net reconstructs an LF through progressively eliminating the residuals between the projected measurements from the reconstructed LF and input measurements. Moreover, to address the crucial issue of extracting representative features from high-dimensional LF data efficiently and effectively, we formulate the problem in a probability space and propose to approximate a posterior distribution of a set of carefully-defined LF processing events, including both layer-wise spatial-angular feature extraction and network-level feature aggregation. Through droppath from a densely-connected template network, we derive an adaptively learned spatial-angular fusion strategy, which is sharply contrasted with existing manners that combine spatial and angular features empirically. Extensive experiments on both simulated measurements and measurements by a real coded aperture camera demonstrate the significant advantage of our method over state-of-the-art ones, i.e., our method improves the reconstruction quality by 4.5 dB.

In this paper, a new full-reference image quality assessment (IQA) method for performing perceptual quality evaluation on the light field (LF) image is proposed, called the contourlet transform-based model (CTM). The LF image consists of a set of sub-aperture images (SAIs) that have similar image content but with small angular deviations due to different viewing angles. Hence, the abundant image details can be extracted from the SAIs. To fulfil this goal, the contourlet transform is used to extract the multi-scale spatial features of the reference and distorted SAIs, respectively. Based on our proposed IQA metric, the degree of similarity can be computed based on the above-measured quantities for arriving at the final IQA score of the distorted LF image under evaluation. Experimental simulation results obtained from the dense light fields datasets clearly show that the proposed CTM algorithm is more in line with the quality assessment of the LF images perceived by the human visual system (HVS) when compared with that of using other state-of-the-art IQA algorithms.

The ultra high-definition (UHD) video format, which has recently become popular, aims to provide high spatial resolution, high temporal frame rate, high sample bit-depth, and wide pixel color gamut. Despite the continued development of global network capacities, it inevitably causes the increased bandwidth cost of catering to the requirement of delivering UHD video services. To address such challenges, this paper presents an improved super coding unit (SCU) method for UHD video coding in High Efficiency Video Coding (HEVC). Initially, the medium coding unit (MCU) is proposed to avoid unnecessary brute-force coding unit (CU) partitions of SCU. Furthermore, the SCU is proposed to be encoded by Direct-MCU and SCU-to-MCU modes: the Direct-MCU mode is intended to better adapt to the texture-rich region, which guarantees the compression efficiency by avoiding extra-size CU partition; the SCU-to-MCU mode is designed for the homogeneous region of UHD content, which saves the encoding time by skipping fine-grained CU partition search. Moreover, a learning-based fast SCU decision approach is proposed to speed up the determination process of Direct-MCU and SCU-to-MCU, where three representative handcrafted features are extracted. Experimental results show that our method achieves an affordable complexity and excellent coding efficiency (up to 7.30% Bjontegaard Delta rate savings) in UHD video coding compared to recent HEVC reference software.

Vehicle re-identification (Re-ID) plays an important role in intelligent transportation systems. It usually suffers from various challenges encountered on the real-life environments, such as viewpoint variations, illumination changes, object occlusions, and other complicated scenarios. To effectively improve the vehicle Re-ID performance, a new method, called the deep quadruplet appearance learning (DQAL), is proposed in this paper. The novelty of the proposed DQAL lies on the consideration of the special difficulty in vehicle Re-ID that the vehicles with the same model and color but different identities (IDs) are highly similar to each other. For that, the proposed DQAL designs the concept of quadruplet and forms the quadruplets as the input, where each quadruplet is composed of the anchor (or target), positive, negative, and the specially considered high-similar (i.e., the same model and color but different IDs with respect to the anchor) vehicle samples. Then, the quadruplet network with the incorporation of the proposed quadruplet loss and softmax loss is developed to learn a more discriminative feature for vehicle Re-ID, especially discerning those difficult high-similar cases. Extensive experiments conducted on two commonly used datasets VeRi-776 and VehicleID have demonstrated that the proposed DQAL approach outperforms multiple recently reported vehicle Re-ID methods.

The just noticeable difference (JND) models in pixel domain are generally composed of luminance adaptation (LA) and contrast masking (CM), which takes edge masking (EM) and texture masking (TM) into consideration. However, in existing pixel-wise JND models, CM is not evaluated appropriately since they overestimate the masking effect of regular oriented texture regions and neglect the visual attention characteristic of human eyes for the real image. In this work, a novel JND model in pixel domain is proposed, where orderly texture masking (OTM) for regular texture areas (also called orderly texture regions) and disorderly texture masking (DTM) for complex texture areas (also called disorderly texture regions) are presented based on the orientation complexity. Meanwhile, the visual saliency is set as the weighting factor and is incorporated into CM evaluation to enhance JND thresholds. Experimental results indicate that compared with existing relevant JND profiles, the proposed JND model tolerates more distortion in the same perceptual quality, and brings better visual perception in the same level of the injected JND-noise energy. Published version

Due to the effectiveness of anomaly/outlier detection, one-class algorithms have been extensively studied in the past. The representatives include the shallow-structure methods and deep networks, such as the one-class support vector machine (OC-SVM), one-class extreme learning machine (OC-ELM), deep support vector data description (Deep SVDD), and multilayer OC-ELM (ML-OCELM/MK-OCELM). However, existing algorithms are generally built on the minimum mean-square-error (mse) criterion, which is robust to the Gaussian noises but less effective in dealing with large outliers. To alleviate this deficiency, a robust maximum correntropy criterion (MCC)-based OC-ELM (MC-OCELM) is first proposed and then further extended to a hierarchical network to enhance its capability in characterizing complex and large data (named HC-OCELM). The gradient derivation combining with a fixed-point iterative updation scheme is adopted for the output weight optimization. Experiments on many benchmark data sets are conducted for effectiveness validation. Comparisons to many state-of-the-art approaches are provided for the superiority demonstration.

The light field (LF) images have received significant attentions from both academia and industry, which allow a wide range of applications in the areas of computer vision and computer graphics, such as three-dimensional reconstruction, virtual reality, and so on. Considering that the human eyes are the final receivers of the LF images and the LF images are inevitably suffered from various distortions, image quality assessment (IQA) becomes an important issue for the LF images with the goal of evaluating the image quality of the LF images in accordance with the human visual system. This paper presents an up-to-date overview of the IQA specifically for the LF images, focusing on the formation of the LF images, the LF IQA databases and the existing LF IQA models.

This paper explores the problem of hyperspectral image (HSI) super-resolution that merges a low resolution HSI (LR-HSI) and a high resolution multispectral image (HR-MSI). The cross-modality distribution of the spatial and spectral information makes the problem challenging. Inspired by the classic wavelet decomposition-based image fusion, we propose a novel \textit{lightweight} deep neural network-based framework, namely progressive zero-centric residual network (PZRes-Net), to address this problem efficiently and effectively. Specifically, PZRes-Net learns a high resolution and \textit{zero-centric} residual image, which contains high-frequency spatial details of the scene across all spectral bands, from both inputs in a progressive fashion along the spectral dimension. And the resulting residual image is then superimposed onto the up-sampled LR-HSI in a \textit{mean-value invariant} manner, leading to a coarse HR-HSI, which is further refined by exploring the coherence across all spectral bands simultaneously. To learn the residual image efficiently and effectively, we employ spectral-spatial separable convolution with dense connections. In addition, we propose zero-mean normalization implemented on the feature maps of each layer to realize the zero-mean characteristic of the residual image. Extensive experiments over both real and synthetic benchmark datasets demonstrate that our PZRes-Net outperforms state-of-the-art methods to a \textit{significant} extent in terms of both 4 quantitative metrics and visual quality, e.g., our PZRes-Net improves the PSNR more than 3dB, while saving 2.3$\times$ parameters and consuming 15$\times$ less FLOPs. The code is publicly available at https://github.com/zbzhzhy/PZRes-Net .

Vehicle re-identification (Re-ID) technology plays an important role in intelligent video surveillance systems. Due to various factors, e.g., resolution variation, viewpoint variation, illumination changes, occlusion, etc., vehicle Re-ID is a very challenging computer vision task. In order to solve this problem, a joint pyramid feature representation network (JPFRN) is proposed in this paper. Based on the consideration that various convolution blocks with different depths hold various resolution and semantic information of the vehicle image, which can help to effectively identify the vehicle, the proposed JPFRN method obtains four vehicle feature blocks with different depths by designing pyramidal feature fusion of each convolution block in a basic network. After that, a joint representation of these pyramidal features is feed into the loss function for learning discriminative features for vehicle Re-ID. We validated the proposed approach on a commonly used vehicle database i.e., VehicleID. Extensive experimental results show that the proposed method is superior to multiple state-of-the-art vehicle Re-ID methods.

This paper presents an image quality assessment (IQA) model exploring the multi-order derivative feature, called Multi-order Derivative Feature-based Model (MDFM), for evaluating the perceptual quality of light field image (LFI). In our approach, for the input reference and distorted LFIs, the multi-order derivative features are extracted by using the discrete derivative filter to represent the image details in different degrees. Then, the similarities of the extracted derivative features are measured independently. Finally, the weight map is established through the maximum value of the second-order derivative feature of reference and distorted LFIs, which is further utilized to pool the similarity map for generating the final score. Extensive simulation results have demonstrated that the proposed MDFM is more consistent with the perception of the HVS on the evaluation of LFI than the classical and state-of-the-art IQA methods.

Pedestrian gender recognition is a very challenging problem, since the viewpoint variations, illumination changes, occlusion, and poor quality are usually encountered in the pedestrian images. To address this problem, an effective HOG-assisted deep feature learning (HDFL) method is proposed in this paper. The key novelty lies in the design of HDFL network to effectively explore both deep-learned feature and weighted histogram of oriented gradient (HOG) feature for the pedestrian gender recognition. Specifically, the deep-learned and weighted HOG feature extraction branches are simultaneously performed on the input pedestrian image. A feature fusion process is subsequently conducted to obtain a more robust and discriminative feature, which is then fed to a softmax classifier for pedestrian gender recognition. Extensive experiments on multiple existing pedestrian image datasets have shown that the proposed HDFL method is able to effectively recognize the pedestrian gender, and consistently outperforms the state-of-the-art methods.

In this paper, a joint feature and similarity deep learning (JFSDL) method for vehicle re-identification is proposed. The proposed JFSDL method applies a siamese deep network to extract deep learning features for an input vehicle image pair simultaneously. The siamese deep network is learned under the joint identification and verification supervision. The joint identification and verification supervision is realized by linearly combining two softmax functions and one hybrid similarity learning function. Moreover, based on the hybrid similarity learning function, the similarity score between the input vehicle image pair is also obtained by simultaneously projecting the element-wise absolute difference and multiplication of the corresponding deep learning feature pair with a group of learned weight coefficients. Extensive experiments show that the proposed JFSDL method is superior to multiple state-of-the-art vehicle re-identification methods on both the VehicleID and VeRi data sets.

To reduce the computational complexity of screen content video coding (SCC), a fast algorithm based on gray level co-occurrence matrix and Gabor feature model for HEVC-SCC, denoted as GGM, is proposed in this paper. By studying the correlation of non-zero number in gray level co-occurrence matrix with different partitioning depth, the coding unit (CU) size of intra coding can be prejudged, which selectively skips the intra prediction process of CU in other depth. With Gabor filter, the edge information reflecting the features of screen content images to the human visual system (HVS) are extracted. According to Gabor feature, CUs are classified into natural content CUs (NCCUs), smooth screen content CUs (SSCUs) and complex screen content CUs (CSCUs), with which, the calculation and judgment of unnecessary intra prediction modes are skipped. Under all-intra (AI) configuration, experimental results show that the proposed algorithm GGM can achieve encoding time saving by 42.13% compared with SCM-8.3, and with only 1.85% bit-rate increasement.

The homework tidiness assessment aims to auto evaluate the writing tidiness of homework, playing an important role in daily teaching. However, there is still no comprehensive basis for homework tidiness assessment. For this, a benchmark for homework tidiness assessment (HTA) is proposed. Firstly, a database named HTA 1.0 containing 1000 homework images is collected. Each image is manually annotated by multiple volunteers. Secondly, a comprehensive evaluation protocol is designed, using mean absolute error (MAE), root mean square error (RMSE), mean absolute percentage error (MAPE) and accuracy (Acc) as performance indicators. Finally, three deep learning models (i.e., LeNet, AlexNet and VGGNet) are applied as baseline methods and the results are reported and analyzed.

In this paper, an image quality assessment (IQA) model for conducting objective evaluations of screen content images (S-CIs) is proposed, called the log-Gabor feature-based model (LGFM). From the standpoint of signal representation, the log-Gabor filters outperform the classical Gabor filters since the outputs of the log-Gabor filters are more consistent with the perception of visual cortex in human visual system (HVS). Furthermore, the following two remarkable characteristics of the log-Gabor filters are highly beneficial to develop a more accurate IQA model; i.e., (i) zero response at the DC, and (ii) stronger response at high frequencies. In our proposed L-GFM, the log-Gabor filters are used to extract features from the luminance of the reference SCIs and that of the distorted SCIs for measuring their degree of similarity. Together with the measurements from the other two chrominance components, the final LGFM score will be arrived at the output of the pooling stage. Extensive simulation results have shown that our proposed LGFM is highly consistent with the human perception, compared to other state-of-the-art IQA models.

The Vehicle Logo Recognition (VLR) is becoming more and more important in the Automated Vehicle Classification (AVC) research area since it can effectively assist vehicle classification. However, existing open-access vehicle logo datasets have many issues, which are hard to represent real-world conditions such as occlusions, low image resolutions and varying lighting conditions, thus heavily hindering the VLR application in reality. For this, a large-scale vehicle logo benchmark is proposed in this paper. Firstly, a large-scale vehicle logo dataset namely VLD 1.0 is collected, which contains 25,189 images of 66 classes. Each type of cars is collected with multiple images from various conditions to comprehensively represent real-world conditions. Secondly, a vehicle logo recognition evolution protocol is designed to fully evaluate performance, i.e., Mean Average Precision (MAP) under different Intersection Over Union (IOU) values and Speed. Thirdly, the YOLOv3 deep learning model is applied as a baseline method and its performance on the VLD 1.0 is reported.

Fabric image retrieval aims to match a probe fabric image from a gallery fabric image set, playing a great role in the textile industry. However, there is a lack of one comprehensive benchmark for fabric image retrieval. For this, a large benchmark for fabric image retrieval is proposed in this paper. Firstly, a dataset namely Fabric 1.0 is proposed, which contains 46,656 fabric images of 972 subjects. Each subject is collected with multiple images from various angles and both front and back sides. Secondly, a evaluation protocol is designed to evaluate the fabric image retrieval accuracy, which uses the cumulative match characteristic (CMC) curve and the mean average precision (MAP) as performance indicators. Finally, the baseline performance resulted from a non deep learning method (i.e., bag of feature) and three deep learning methods (i.e., LeNet, AlexNet and VGGNet) are reported and analyzed.

This paper proposes a new local texture descriptor, called quad binary pattern (QBP). Compared with local binary pattern (LBP), the QBP is with stronger robustness for feature extraction under complex scene (e.g., luminance change, similar target and background color) and with lower computational complexity. To demonstrate its effectiveness, the proposed QBP is further applied on the mean-shift tracking, in which a joint color-QBP model is developed to effectively represent the color and texture characteristics of the target region. Extensive simulation results have demonstrated that the proposed algorithm is able to improve the tracking speed and accuracy, compared with the standard mean-shift tracking and joint color-LBP model based mean-shift tracking.

In this letter, we make the first attempt to explore the usage of the gradient direction to conduct the perceptual quality assessment of the screen content images (SCIs). Specifically, the proposed approach first extracts the gradient direction based on the local information of the image gradient magnitude, which not only preserves gradient direction consistency in local regions, but also demonstrates sensitivities to the distortions introduced to the SCI. A deviation-based pooling strategy is subsequently utilized to generate the corresponding image quality index. Moreover, we investigate and demonstrate the complementary behaviors of the gradient direction and magnitude for SCI quality assessment. By jointly considering them together, our proposed SCI quality metric outperforms the state-of-the-art quality metrics in terms of correlation with human visual system perception.

With the advances in understanding perceptual properties of the human visual system, perceptual video coding, which aims to incorporate human perceptual mechanisms into video coding for maximizing the perceptual coding efficiency, becomes an essential research topic. Since the newest video coding standard—high efficiency video coding (HEVC) does not fully consider the perceptual characteristic of the input video, a perceptual feature guided rate distortion optimization (RDO) method is presented to improve its perceptual coding performance in this paper. In the proposed method, for each coding tree unit, the spatial perceptual feature (i.e., gradient magnitude ratio) and the temporal perceptual feature (i.e., gradient magnitude similarity deviation ratio) are extracted by considering the spatial and temporal perceptual correlations. These perceptual features are then utilized to guide the RDO process by perceptually adjusting the corresponding Lagrangian multiplier. By incorporating the proposed method into the HEVC, extensive simulation results have demonstrated that the proposed approach can significantly improve the perceptual coding performance and obtain better visual quality of the reconstructed video, compared with the original RDO in HEVC.

Visual information becomes an indispensable part of our life through diversified multimedia applications, such as smart phones, IPAD, high definition TV, three dimensional TV and free viewpoint TV, to name a few. To provide more interactive and realistic multimedia experience, a new set of visual information processing technologies are emerging. These involve several research fields, such as visual information representation, compression, transmission and display technologies for variousmultimedia applications. The goal of this special issue is to present the latest research achievements of the emerging visual information processing technologies for multimedia applications. This special issue includes 16 papers, which will be briefly discussed as follows. Visual object detection, extraction and analysis are very meaningful and have various applications. The paper entitled BComic Storyboard Extraction via Edge Segment Analysis^ (10.1007/s11042-015-2680-8) aims to decompose the comic image into several frames, which plays a key role in producing the digital comic documents suitable for mobile reading. The authors propose new validation rules to reject the false storyboard detections; therefore, their algorithm achieves much better performance than the existing methods. In touch screen manufacturing, the defect inspection is one of the most important processes. The paper entitled BTouch Screen Defect Inspection based on Sparse Representation in Low Resolution Images^ (10.1007/s11042-015-2559-8) proposes a novel defect inspection approach based on sparse representation of low resolution image. Based on their simulation results, the proposed method achieves high inspection accuracy for different defect types. In the paper entitled BHierarchical Salient Object Detection Model using Contrast-based Saliency and Color Spatial Distribution^ (10.1007/s11042-015-2570-0), the authors aim to accurately interpret object saliency in image, in which the distinctive features are investigated from a global perspective. The effectiveness of this proposed model has been validated on MSRA database. Multimed Tools Appl (2016) 75:2633–2636 DOI 10.1007/s11042-016-3314-5

Reversible data hiding in encrypted images (RDH-EI) is a technique that hides additional data into the ciphertext with a capability of recovering the original content losslessly. In recent years, many methods have been proposed on this topic. In this paper, we propose a new RDH-EI approach that can achieve a very large embedding capacity, using adaptive reversible integer transformation. We first divide the original image into non-overlapped blocks. With an algorithm of adaptive reversible integer transformation, we vacate the embedding room for accommodating the auxiliary data and the payload. After image encryption and uploading, the cloud server can hide a large amount of additional data into the ciphertext using the embedded auxiliary data. The process is reversible that the additional data can be extracted exactly and the original image can be recovered without any error. Compared with state-of-the-art RDH-EI methods, the proposed method can achieve a much higher embedding capacity and a better rate distortion performance.

In order to resist the adverse effect of viewpoint variations for improving vehicle re-identification performance, we design quadruple directional deep learning networks to extract quadruple directional deep learning features (QD-DLF) of vehicle images. The quadruple directional deep learning networks are with similar overall architecture, including the same basic deep learning architecture but different directional feature pooling layers. Specifically, the same basic deep learning architecture is a shortly and densely connected convolutional neural network to extract basic feature maps of an input square vehicle image in the first stage. Then, the quadruple directional deep learning networks utilize different directional pooling layers, i.e., horizontal average pooling (HAP) layer, vertical average pooling (VAP) layer, diagonal average pooling (DAP) layer and anti-diagonal average pooling (AAP) layer, to compress the basic feature maps into horizontal, vertical, diagonal and anti-diagonal directional feature maps, respectively. Finally, these directional feature maps are spatially normalized and concatenated together as a quadruple directional deep learning feature for vehicle re-identification. Extensive experiments on both VeRi and VehicleID databases show that the proposed QD-DLF approach outperforms multiple state-of-the-art vehicle re-identification methods., this paper has been submitted to IEEE Transactions on Intelligent Transportation Systems, under review

Just noticeable difference (JND), which reflects the minimum visibility threshold of human visual system, is usually utilized to remove the perceptual redundancies for image/video coding. However, most of the existing DCT-based JND models failed to effectively estimate contrast masking (CM) of image blocks with different complexity. In this paper, we propose an improved CM model via frequency energy of DCT block, which is calculated based on DCT coefficient distribution. Then, by integrating luminance adaptation and contrast sensitivity function, an effective JND model in the DCT domain is realized. Experimental results have confirmed that the proposed method can produce more accurate JND threshold and remove more visual redundancies than the state-of-the-art JND models.

In this paper, we propose a shortly and densely connected convolutional neural network (SDC-CNN) for vehicle re-identification. The proposed SDC-CNN mainly consists of short and dense units (SDUs), necessary pooling and normalization layers. The main contribution lies at the design of short and dense connection mechanism, which would effectively improve the feature learning ability. Specifically, in the proposed short and dense connection mechanism, each SDU contains a short list of densely connected convolutional layers and each convolutional layer is of the same appropriate channels. Consequently, the number of connections and the input channel of each convolutional layer are limited in each SDU, and the architecture of SDC-CNN is simple. Extensive experiments on both VeRi and VehicleID datasets show that the proposed SDC-CNN is obviously superior to multiple state-of-the-art vehicle re-identification methods.

In this paper, an accurate and efficient full-reference image quality assessment (IQA) model using the extracted Gabor features, called Gabor feature-based model (GFM), is proposed for conducting objective evaluation of screen content images (SCIs). It is well-known that the Gabor filters are highly consistent with the response of the human visual system (HVS), and the HVS is highly sensitive to the edge information. Based on these facts, the imaginary part of the Gabor filter that has odd symmetry and yields edge detection is exploited to the luminance of the reference and distorted SCI for extracting their Gabor features, respectively. The local similarities of the extracted Gabor features and two chrominance components, recorded in the LMN color space, are then measured independently. Finally, the Gabor-feature pooling strategy is employed to combine these measurements and generate the final evaluation score. Experimental simulation results obtained from two large SCI databases have shown that the proposed GFM model not only yields a higher consistency with the human perception on the assessment of SCIs but also requires a lower computational complexity, compared with that of classical and state-of-the-art IQA models. 1 1 The source code for the proposed GFM will be available at http://smartviplab.org/pubilcations/GFM.html .

In this paper, a novel image quality assessment (IQA) model for the screen content image s (SCIs) is proposed by using multi-scale difference of Gaussian (MDOG). Motivated by the observation that the human visual system (HVS) is sensitive to the edges while the image details can be better explored in different scales, the proposed model exploits MDOG to effectively characterize the edge information of the reference and distorted SCIs at two different scales, respectively. Then, the degree of edge similarity is measured in terms of the smaller-scale edge map. Finally, the edge strength computed based on the larger-scale edge map is used as the weighting factor to generate the final SCI quality score. Experimental results have shown that the proposed IQA model for the SCIs produces high consistency with human perception of the SCI quality and outperforms the state-of-the-art quality models.

The newest video coding standard--high efficiency video coding (HEVC) optimizes its coding efficiency in terms of sum of square error (SSE), which does not fully consider the perceptual characteristic of the input video. Thus, the HEVC is not effective in the sense of perceptual video coding. To address this problem, an efficient perceptual sensitivity-based rate control (PSRC) method for HEVC is proposed based on the human visual system (HVS) observation that the region with less perceptual sensitivity can tolerate more distortion. In the first stage, the proposed method develops a perceptual sensitivity measurement to evaluate the perceptual sensitivity of each coding tree unit (CTU) and each frame, which is then used to guide the bit allocation so that more bits will be allocated to those regions with higher perceptual sensitivity. To meet the target bits, an improved R-ź model is utilized to determine the quantization parameter (QP). Experimental results have shown that the proposed method is able to improve the perceptual coding performance, compared with the original rate control in HEVC.

During the multi-view video acquisition, color variation across the views tends to be incurred due to different camera positions, orientations, and local lighting conditions. Such color variation will inevitably deteriorate the performance of the follow-up multi-view video processing, such as multi-view video coding (MVC). To address this problem, an effective color correction algorithm, called the SIFT flow-based color correction (SFCC), is proposed in this paper. First, the SIFT-flow technique is used to establish point-to-point correspondences across all the views of the multi-view video. The average color is then computed based on those identified common corresponding points and used as the reference color. By minimizing the energy of the difference yielded between the color of those identified common corresponding points in each view with respect to the reference color, the color correction matrix for each view can be obtained and used to correct its color. Experimental results have shown that the proposed SFCC algorithm is able to effectively eliminate the color variation inherited in multi-view video. By further exploiting the developed SFCC algorithm as a pre-processing for the MVC, extensive simulation results have shown that the coding efficiency of the color-corrected multi-view video can be greatly improved (on average, 0.85dB, 1.27dB and 1.63dB gain for Y, U, and V components, respectively), compared with that of the original multi-view video without color correction. HighlightsPropose to exploit SIFT flow to conduct color correction for multi-view video.Use SIFT flow to identify all common corresponding points across all the views.Compute the reference color by averaging the color of the common corresponding points.Generate color correction matrix by minimizing the energy of color difference.Experimental results clearly justify the effectiveness of the proposed method.

In this paper, a novel image quality assessment (IQA) metric for the multi-exposure image fusion (MEF) is proposed by using contrast information. Specifically, the proposed approach firstly performs the measurements of contrast structure similarity and contrast saturation similarity based on the observation that human perception is sensitive to contrast information inherited in the MEF and reference images. Then, considering that different reference images contribute differently to the MEF image, the weights are adaptively assigned to each reference image according to its relevance to the MEF image. A standard deviation based pooling strategy and multi-scale scheme are subsequently used to generate the final MEF image quality score. Experimental results have shown that the proposed metric produces high consistency with human perception of the MEF image quality and outperforms the state-of-the-art quality metric.

Since the object occlusion, view variation, arbitrary shape of pedestrian, etc., are inherited in the pedestrian images, the pedestrian gender classification is an extremely challenging task in the object recognition field. To address this problem, an effective multi-view joint learning network (MJLN) is proposed for pedestrian gender classification. Based on the observation that the view variation of pedestrian will affect the judgment of pedestrian gender, the proposed MJLN simultaneously performs pedestrian gender learning and pedestrian view learning. Consequently, the determination of pedestrian view (i.e., frontal, rear, and left/right profile) obtained by the pedestrian view learning module will effectively assist the pedestrian gender learning module to yield a more robust and distinguishable feature so as to improve the gender classification performance. Extensive experiments on multiple challenging pedestrian datasets, which includes CUHK, VIPeR, GRID, PRID, and MIT, have demonstrated that the proposed MJLN can effectively promote the gender classification performance, and outperforms multiple state-of-the-art methods.

With the development of urbanization, the underground pipelines play an important role in ensuring the basic conditions of citizen life. However, construction activities are continuously increasing in the mainland of China nowadays, which lead to serious damages and threatens to the underground pipelines by excavation equipment. To this end, we focus on developing an automatically monitoring system for underground pipeline network protection in this paper. The acoustic characteristics of excavation equipment are first analyzed. Then, a recognition framework combining with the feature extraction algorithm of Mel-Frequency Linear Prediction Cepstral Coefficients (MF-PLPCC) and the classification algorithm of Regularized Extreme Learning Machine (RELM) is developed for excavation equipment. The MF-PLPCC processing includes the use of Mel filter bank that simulates the acoustic perception of human ear to different frequency, and the use of a power-law nonlinearity that replaces the traditional log nonlinearity used in MFCC. Experimental results demonstrate that the method of this paper provides substantial improvements in recognition accuracy comparing to other state-of-the-art methods.

In this paper, an accurate full-reference image quality assessment (IQA) model developed for assessing screen content images (SCIs), called the edge similarity (ESIM), is proposed. It is inspired by the fact that the human visual system (HVS) is highly sensitive to edges that are often encountered in SCIs; therefore, essential edge features are extracted and exploited for conducting IQA for the SCIs. The key novelty of the proposed ESIM lies in the extraction and use of three salient edge features—i.e., edge contrast , edge width , and edge direction . The first two attributes are simultaneously generated from the input SCI based on a parametric edge model, while the last one is derived directly from the input SCI. The extraction of these three features will be performed for the reference SCI and the distorted SCI, individually. The degree of similarity measured for each above-mentioned edge attribute is then computed independently, followed by combining them together using our proposed edge-width pooling strategy to generate the final ESIM score. To conduct the performance evaluation of our proposed ESIM model, a new and the largest SCI database (denoted as SCID) is established in our work and made to the public for download. Our database contains 1800 distorted SCIs that are generated from 40 reference SCIs. For each SCI, nine distortion types are investigated, and five degradation levels are produced for each distortion type. Extensive simulation results have clearly shown that the proposed ESIM model is more consistent with the perception of the HVS on the evaluation of distorted SCIs than the multiple state-of-the-art IQA methods.

In this paper, we propose an effective deep-learned and hand-crafted features fusion network (DHFFN) for pedestrian gender recognition. In the proposed DHFFN, the deep-learned and hand-crafted (i.e., HOG) features are extracted for the input image, followed by the feature fusion process that is to combine these two features together for fully exploring the merits from both deep-learned and HOG features. Extensive experiments on multiple public datasets have demonstrated that the proposed DHFFN method is superior to the state-of-the-art pedestrian gender recognition methods.

Person Re-IDentification (Re-ID) aims to match person images captured from two non-overlapping cameras. In this paper, a deep hybrid similarity learning (DHSL) method for person Re-ID based on a convolution neural network (CNN) is proposed. In our approach, a CNN learning feature pair for the input image pair is simultaneously extracted. Then, both the element-wise absolute difference and multiplication of the CNN learning feature pair are calculated. Finally, a hybrid similarity function is designed to measure the similarity between the feature pair, which is realized by learning a group of weight coefficients to project the element-wise absolute difference and multiplication into a similarity score. Consequently, the proposed DHSL method is able to reasonably assign parameters of feature learning and metric learning in a CNN so that the performance of person Re-ID is improved. Experiments on three challenging person Re-ID databases, QMUL GRID, VIPeR and CUHK03, illustrate that the proposed DHSL method is superior to multiple state-of-the-art person Re-ID methods., 10 pages, 12 figures, fixed typo errors in Fig.8

In this paper, a perceptual-based rate control algorithm for the high efficiency video coding (HEVC) is proposed. Since the human visual system (HVS) is highly sensitive to edge information inherited in image and video contents, gradient magnitude is considered a very effective attribute on the computation of the spatial perceptual characteristics (SPC) and the temporal perceptual characteristic (TPC). The latter is further computed based on the gradient magnitude similarity deviation (GMSD). The combined perceptual characteristic of SPC and TPC is then used to guide the bit allocation process for each CTU. Experimental results have clearly shown that the proposed perceptual rate control scheme can achieve superior perceptual coding performance, compared with the original R-A rate control adopted in the HEVC.

This paper proposes a multiple description image coding scheme based on 2D dual-tree transform and the enhanced x-tree encoding method. The input image is firstly mapped into 2D dual-tree discrete wavelet domain to form 2 wavelet coefficient trees. A sparse algorithm is then used to remove the most redundant wavelet coefficients resulted from dual-tree discrete wavelet transform (DDWT), forming the basic component of both descriptions, respectively. In order to improve the quality of the side reconstruction, a side sparse algorithm is then imposed on two sparse coefficient trees to produce the additional for both side decoding. The basic information from one tree and additional information from the other are sent to an enhanced x-tree encoder, which is proposed to exploit the strong correlation between two wavelet trees resulted from DDWT, forming the bitstream of a description. Since each description includes the basic information and part of details of the input image, even one of the descriptions gets lost, the reconstructed image can still keep acceptable quality. Simulation results have verified that the proposed algorithm has good coding performance and error resilient ability.

The latest high efficiency video coding-based three dimensional video coding (3D-HEVC) exploits sophisticated intra prediction scheme to improve the coding performance of the depth video, but incurring heavy computational complexity. To address this problem, a low complexity depth intra coding method is presented for 3D-HEVC based on depth classification. Firstly, a database of depth prediction units (PUs) with three kinds of complexities is collected based on their optimal intra prediction mode. Then, the histogram of oriented gradient (HOG) features are extracted on these established database to train the classifier using support vector machine (SVM). For the current depth PU, the trained classifier is applied to determine its most possible complexity class so as to select the corresponding modes for involving the mode decision process. Experimental results show that the proposed method is able to significantly reduce the computational complexity while keeping almost the same coding performance of depth video and video quality of the synthesized view, compared with the exhaustive mode decision in 3D-HEVC.

As the concept of block level adaptivity becomes an important feature in recent video CODECs, block level adaptive quantization (BLAQ) is being considered in the High Efficiency Video Coding (HEVC) standard. The BLAQ is based on the assumption that each block should have its own quantization parameter (QP), which can adapt to the local content of video sequences much better, and hence the video encoder with adaptive QP can perform a better perceptual quality. However, in the HEVC reference software, the BLAQ is required to obtain a proper QP for each block by the rate distortion optimization (RDO) scheme and so the computational complexity of the encoder increases significantly. In this paper, an improved BLAQ algorithm is proposed to obtain the adaptive QP for each block. The simulation results show that the proposed method can save more bits as well as require lower computational complexity, compared to the traditional method.

In this paper, a novel multiple description coding MDC scheme for multi-view video coding MVC is proposed that can deliver higher coding efficiency. This is achieved by separating one description into two subsequences and directly using the modes and prediction vectors computed during the encoding of one subsequence to the encoding process of the other subsequence from the same description. Such reuse strategy is made possible due to high correlation existing between the two subsequences that were generated from the input multi-view video sequence through the spatial polyphase subsampling and 'cross-interleaved' sample grouping. Due to the data reuse, the required representation bits for storage and transmission are greatly saved. Extensive simulation results experimented on the JMVC codec platform have shown that the performance of the proposed MDC scheme outperforms several state-of-the-art MDC methods for stereoscopic video and multi-view video.

A novel local texture descriptor, called multi-block quad binary pattern (MB-QBP), is proposed in this paper. To demonstrate its effectiveness on local feature representation and potential usage in computer vision applications, the proposed MB-QBP is applied to face detection. Compared with the multi-block local binary pattern (MB-LBP), MB-QBP has more features to conduct a better training process to refine the classifier. Consequently, the over-fitting problem becomes much smaller in the MB-QBP-based classifier. Extensive simulation results conducted by using the test images from the BioID and CMU+MIT databases have clearly shown that the proposed MB-QBP-based face detector outperforms the MB-LBP-based approach by about 6 % on the correct detection rate under the same training conditions.