Search

Stereo matching has been studied for many years and is still a challenge problem. The Markov Random Fields (MRF) model and the Conditional Random Fields (CRF) model based methods have achieved good performance recently. Based on these pioneer works, a deep conditional random fields based stereo matching algorithm is proposed in this paper, which draws a connection between the Convolutional Neural Network (CNN) and CRF. The object knowledge is used as a soft constraint, which can effectively improve the depth estimation accuracy. Moreover, we proposed a CNN potential function that learns the potentials of CRF in a CNN framework. The inference of the CRF model is formulated as a Recurrent Neural Network (RNN). A variety of experiments have been conducted on KITTI and Middlebury benchmark. The results show that the proposed algorithm can produce state-of-the-art results and outperform other MRF-based or CRF-based methods. [ABSTRACT FROM AUTHOR]

It is difficult for us to generate a gradually changed texture by combing two textures when they have very different textures and structures. This is a common problem in constructing panoramas and in pasting from one texture to another. The inconsistency between the sources is what we want to deal with in this paper. We present a new method for synthesizing a transition region between two source textures, such that inconsistent textures and structural properties all change gradually from one source to the other. We first extract the convolutional neural network (CNN) features of two source textures and one initial target texture at convolution and pooling layers. We set the distortion function to be the square of the difference between feature maps of source texture and target texture. And we based on feature maps compute the Gram matrix which is the inner product between feature maps in each layer. We set the second distortion function to be the square of the difference between Gram matrix of second source texture and target texture. Our target function is the weighted sum of two distortion functions we described before. We use Large-scale bound-constrained optimization method to optimize the target function and get the ultimate result. The model provides a new tool to generate a transition region between two source textures by CNN. Our method is robust to various types of images tested. [ABSTRACT FROM AUTHOR]

In this paper, we strive to propose a self-interpretable framework, termed PrimitiveTree , that incorporates deep visual primitives condensed from deep features with a conventional decision tree, bridging the gap between deep features extracted from deep neural networks (DNNs) and trees' transparent decision-making processes. Specifically, we utilize a codebook, which embeds the continuous deep features into a finite discrete space (deep visual primitives) to distill the most common semantic information. The decision tree adopts the spatial location information and the mapped primitives to present the decision-making process of the deep features in a tree hierarchy. Moreover, the trained interpretable PrimitiveTree can inversely explain the constituents of the deep features, highlighting the most critical and semantic-rich image patches attributing to the final predictions of the given DNN. Extensive experiments and visualization results validate the effectiveness and interpretability of our method. [Display omitted] • A codebook distills semantic-rich visual primitives from the continuous feature space. • The proposed PrimitiveTree can interpret deep features and reveal key image constituents. • Extensive experiments on popular datasets demonstrate the effectiveness and interpretability of our method. [ABSTRACT FROM AUTHOR]

Representing contextual features at multiple scales is important for RGB-D SOD. Recently, due to advances in backbone convolutional neural networks (CNNs) revealing stronger multi-scale representation ability, many methods achieved comprising performance. However, most of them represent multi-scale features in a layer-wise manner, which ignores the fine-grained global contextual cues in a single layer. In this paper, we propose a novel global contextual exploration network (GCENet) to explore the performance gain of multi-scale contextual features in a fine-grained manner. Concretely, a cross-modal contextual feature module (CCFM) is proposed to represent the multi-scale contextual features at a single fine-grained level, which can enlarge the range of receptive fields for each network layer. Furthermore, we design a multi-scale feature decoder (MFD) that integrates fused features from CCFM in a top-down way. Extensive experiments on five benchmark datasets demonstrate that the proposed GCENet outperforms the other state-of-the-art (SOTA) RGB-D SOD methods. • A global contextual exploration network (GCENet) is proposed to exploit the role of multi-scale features at a single fine-grained level for RGB-D SOD, where a cross-modal contextual feature module (CCFM) is designed to extract and fuse cross-modal and multi-scale features at a single micro level. • To integrate the multi-scale from multiple blocks of the backbone, we design a multi-scale feature decoder (MFD) to fuse these multi-scale features in a top-down layer-wise strategy, where the top features are embedded into all the features of the lower layer by a fully transferred strategy. • Extend experiments on five benchmark datasets demonstrate that the proposed GCENet outperforms other 18 well-known RGB-D SOD methods and achieves state-of-the-art performance in terms of five widely used evaluation metrics. [ABSTRACT FROM AUTHOR]

Deep neural network models with strong feature extraction capacity are prone to overfitting and fail to adapt quickly to new tasks with few samples. Gradient-based meta-learning approaches can minimize overfitting and adapt to new tasks fast, but they frequently use shallow neural networks with limited feature extraction capacity. We present a simple and effective approach called Meta-Transfer-Adjustment learning (MTA) in this paper, which enables deep neural networks with powerful feature extraction capabilities to be applied to few-shot scenarios while avoiding overfitting and gaining the capacity for quickly adapting to new tasks via training on numerous tasks. Our presented approach is classified into two major parts, the Feature Adjustment (FA) module, and the Task Adjustment (TA) module. The feature adjustment module (FA) helps the model to make better use of the deep network to improve feature extraction, while the task adjustment module (TA) is utilized for further improve the model's fast response and generalization capabilities. The proposed model delivers good classification results on the benchmark small sample datasets MiniImageNet and Fewshot-CIFAR100, as proved experimentally. [ABSTRACT FROM AUTHOR]

Recently, inferring users' personality traits on social media has attracted extensive attention. Existing studies have shown that users' personality traits can be inferred from their preferences for images. However, since users' preferences on images are often affected by multiple factors, some liked images cannot effectively reflect their personality traits. To handle this issue, this paper proposes a personality modeling approach based on image aesthetic attribute-aware graph representation learning, which can leverage aesthetic attributes to refine the liked images that are consistent with users' personality traits. Specifically, we first utilize a Convolutional Neural Network (CNN) to train an aesthetic attribute prediction module. Then, attribute-aware graph representation learning is introduced to refine the images with similar aesthetic attributes from users' liked images. Finally, the aesthetic attributes of all refined images are combined to predict personality traits through a Multi-Layer Perceptron (MLP). Experimental results and visual analysis have shown that the proposed method is superior to state-of-the-art personality modeling methods. • We propose an attribute-aware graph representation learning for personality modeling. • The images with similar aesthetic attributes can be refined from users' liked images. • The combined aesthetic attributes of refined images are used to infer personality. [ABSTRACT FROM AUTHOR]

Recently, very deep convolution neural network (CNN) has shown strong ability in single image super-resolution (SISR) and has obtained remarkable performance. However, most of the existing CNN-based SISR methods rarely explicitly use the high-frequency information of the image to assist the image reconstruction, thus making the reconstructed image looks blurred. To address this problem, a novel contour enhanced Image Super-Resolution by High and Low Frequency Fusion Network (HLFN) is proposed in this paper. Specifically, a contour learning subnetwork is designed to learn the high-frequency information, which can better learn the texture of the image. In order to reduce the redundancy of the contour information learned by the contour learning subnetwork during fusion, the spatial channel attention block (SCAB) is introduced, which can select the required high-frequency information adaptively. Moreover, a contour loss is designed and it is used with the ℓ 1 loss to optimize the network jointly. Comprehensive experiments demonstrate the superiority of our HLFN over state-of-the-art SISR methods. • High-frequency information will be lost in the restoration process of super-resolution reconstruction. • By using image contour information to assist super-resolution reconstruction. • Introducing spatial channel attention to enhance high-frequency information further. • Use proportion of image contour information is controlled by contour loss. [ABSTRACT FROM AUTHOR]

Multiple JPEG compressions leave artifacts in digital images: residual traces that could be exploited in forensics investigations to recover information about the device employed for acquisition or image editing software. In this paper, a novel First Quantization Estimation (FQE) algorithm based on convolutional neural networks (CNNs) is proposed. In particular, a solution based on an ensemble of CNNs was developed in conjunction with specific regularization strategies exploiting assumptions about neighboring element values of the quantization matrix to be inferred. Mostly designed to work in the aligned case, the solution was tested in challenging scenarios involving different input patch sizes, quantization matrices (both standard and custom) and datasets (i.e., RAISE and UCID collections). Comparisons with state-of-the-art solutions confirmed the effectiveness of the presented solution demonstrating for the first time to cover the widest combinations of parameters of double JPEG compressions. [ABSTRACT FROM AUTHOR]

Image steganalysis based on convolutional neural networks(CNN) has attracted great attention. However, existing networks lack attention to regional features with complex texture, which makes the ability of discrimination learning miss in network. In this paper, we described a new CNN designed to focus on useful features and improve detection accuracy for spatial-domain steganalysis. The proposed model consists of three modules: noise extraction module, noise analysis module and classification module. A channel attention mechanism is used in the noise extraction module and analysis module, which is realized by embedding the SE(Squeeze-and-Excitation) module into the residual block. Then, we use convolutional pooling instead of average pooling to aggregate features. The experimental results show that detection accuracy of the proposed model is significantly better than those of the existing models such as SRNet, Zhu-Net and GBRAS-Net. Compared with these models, our model has better generalization ability, which is critical for practical application. • An end-to-end spatial steganalysis network CCNet is proposed. • Channel attention mechanism is introduced to focus on the region suitable for embedding secret information. • Convolution is used instead of average pooling to avoid the loss of steganographic information. • The proposed model is verified to have higher accuracy through comparative experiments. [ABSTRACT FROM AUTHOR]

For reasons of public security, modeling large crowd distributions for counting or density estimation has attracted significant research interests in recent years. Existing crowd counting algorithms rely on predefined features and regression to estimate the crowd size. However, most of them are constrained by such limitations: (1) they can handle crowds with a few tens individuals, but for crowds of hundreds or thousands, they can only be used to estimate the crowd density rather than the crowd count; (2) they usually rely on temporal sequence in crowd videos which is not applicable to still images. Addressing these problems, in this paper, we investigate the use of a deep-learning approach to estimate the number of individuals presented in a mid-level or high-level crowd visible in a single image. Firstly, a ConvNet structure is used to extract crowd features. Then two supervisory signals, i.e., crowd count and crowd density, are employed to learn crowd features and estimate the specific counting. We test our approach on a dataset containing 107 crowd images with 45,000 annotated humans inside, and each with head counts ranging from 58 to 2201. The efficacy of the proposed approach is demonstrated in extensive experiments by quantifying the counting performance through multiple evaluation criteria. [ABSTRACT FROM AUTHOR]

Face recognition systems based on Convolutional Neural Networks (CNNs) or convolutional architectures currently represent the state of the art, achieving an accuracy comparable to that of humans. Nonetheless, there are two issues that might hinder their adoption on distributed battery-operated devices (e.g., visual sensor nodes, smartphones, and wearable devices). First, convolutional architectures are usually computationally demanding, especially when the depth of the network is increased to maximize accuracy. Second, transmitting the output features produced by a CNN might require a bitrate higher than the one needed for coding the input image. Therefore, in this paper we address the problem of optimizing the energy-rate-accuracy characteristics of a convolutional architecture for face recognition. We carefully profile a CNN implementation on a Raspberry Pi device and optimize the structure of the neural network, achieving a 17-fold speedup without significantly affecting recognition accuracy. Moreover, we propose a coding architecture custom-tailored to features extracted by such model. [ABSTRACT FROM AUTHOR]

This paper presents an artificial neural network (ANN) based method to detect random-valued impulse noise (RVIN) in images. The proposed method employs the ANN to decide whether a pixel is corrupted or not with RVIN. The inputs of the ANN are the rank ordered absolute differences (ROAD) and the rank-ordered logarithmic difference (ROLD) values. After the detection process is completed, the corrupted pixels are restored by the edge-preserving regularization (EPR) method which allows edges and noise-free pixels to be preserved. The performance of the proposed method is evaluated on different test images and compared with ten different comparison filters from the literature. Simulation results indicate that the proposed method provides significant improvement over comparison filters especially for high noise densities. [ABSTRACT FROM AUTHOR]

Farmland images recognition and classification are of great significance in farmland environmental perception. Since the open and unstructured farmland environment has complex scenes, and is easily affected by various factors, furthermore, environmental information is uncertain and hard to predict. Based on hue saturation value (HSV), hue saturation lightness (HSL) and hue saturation intensity (HSI) color space models, taking use of image analysis and classification technology, this paper realizes the classification of farmland images in different environments. On the basis of color space, eight color features of the images are extracted. First, we conducted non equal interval quantification and drew the color feature curves, after that, we selected five eigenvectors which can correctly classify the images. Then, principal component analysis (PCA) was used for dimension reduction. Finally, radial basis function (RBF) neural network was joined for the extraction of images in the same scenes and different ones. The performance of the use of multiple color spaces combining with PCA and RBF shows that the average recognition rates of sunny days and cloudy days in the same scenes and different scenes are 100%, 87.36% and 84.58%, 68.11% respectively. Therefore, this method has higher recognition rate than BP neural network. [ABSTRACT FROM AUTHOR]

With the rapid development of artificial intelligence technology, text categorization technology is becoming more and more mature. However, text categorization in real situations still faces various unconstrained conditions. English text is an important part of text information, it is also an important way for people to get information from abroad. How can everyone get the desired content from the massive data quickly and accurately, it has become a hot issue in current research. This paper improves the current text categorization algorithm based on English quality-related text categorization. The design and implementation of text categorization system are illustrated with an example of English quality-related text categorization system, complete the research work of text categorization algorithm. The core work of this paper is to mine, classify and analyze large amounts of data in English text by using the method of combining cyclic neural network with quality. Finally, the essential features of high quality English texts are obtained. Traditional English text categorization algorithm if the amount of training data is large, it is easy to show some defects such as unclear feature items. In view of these problems, in order to improve the accuracy and flexibility of English text categorization, this paper proposes a quality-related English text categorization method based on cyclic neural network. A mechanism combining attention is proposed to improve the problem of label disorder and make the structure of the model more flexible. The model proposed in this paper is compared and optimized. Experiments show that the accuracy of neural text classification based on quality classification can reach about 96%. [ABSTRACT FROM AUTHOR]

Depth estimation from a single RGB image is a challenging task. It is ill-posed since a single 2D image may correspond to various 3D scenes at different scales. On the other hand, estimating the relative depth relationship between two objects in a scene is easier and may yield more reliable results. Thus, in this paper, we propose a novel algorithm for monocular depth estimation using relative depths. First, using a convolutional neural network, we estimate two types of depths at multiple spatial resolutions: ordinary depth maps and relative depth tensors. Second, we restore a relative depth map from each relative depth tensor. A relative depth map is equivalent to an ordinary depth map with global scale information removed. For the restoration, sparse pairwise comparison matrices are constructed from available relative depths, and missing entries are filled in using the alternative least square (ALS) algorithm. Third, we decompose the ordinary and relative depth maps into components and recombine them to yield a final depth map. To reduce the computational complexity, relative depths at fine spatial resolutions are directly used to refine the final depth map. Extensive experimental results on the NYUv2 dataset demonstrate that the proposed algorithm provides state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Object trackers based on Siamese network usually transform the tracking task into a matching problem between the candidate samples and the target template. However, with the increasing depth and width of backbone networks, researches on Siamese trackers using backbone networks are not very advanced. Therefore, it is necessary for us to further investigate the characteristics of backbone network. As a fact, the ability of backbone network to extract features can directly determine the performance of object tracker. Given this, in this paper, we first propose an asymmetric convolutional network to improve the representational capability of backbone network. And then, the strip convolution is employed to enhance the operational capability of square kernel convolution in the backbone network. Besides, we also construct a novel module named Feature Dropblock (i.e., FD) to simulate the occlusion of hidden space, which goal is to improve the performance of backbone network in the target tracking under occlusion. To demonstrate the effectiveness of the proposed tracker, extensive ablation studies are conducted. Better results are obtained on the tracking benchmarks OTB100 and VOT2018, compared to other state-of-the-art trackers. [ABSTRACT FROM AUTHOR]

Recently, Siamese trackers have received widespread attention for visual object tracking owing to their good balance between speed and performance. Those Siamese trackers heavily depend on target template while conventional practice fixes the template to initial frame. This strategy makes it unable to cope with variation of target appearance, which often leads to tracking failures and causes the gap in performance from other tracking methods. Despite the performance gain achieved by few template update methods with target templates generated by the tracked results, these tracked templates are easy to accumulate errors and cause tracking drift. In this paper, we propose two template update mechanisms to effectively adapt the target template during the tracking process which is dubbed as DTDU (Dynamic Template with Dual Update). Unlike predecessors that directly use the tracked template, we use initial template to perform similar transformation to the tracked template. Then the similar transformed template and the tracked template are combined linearly to capture the variation of target appearance. These updated templates are stored in a memory bank and retrieved to generate the final target template. In order to enhance quick update of memory bank to accommodate the target appearance, we use the retrieved template to further update the templates in memory bank for subsequent tracking. Extensive experiments on OTB-2015, VOT2016, VOT2018 and GOT-10k datasets have proved the effectiveness of these two update mechanisms and the proposed tracker achieves a real-time speed of 44 fps. • We update template to integrate initial and variation appearance of target for robustness. • The retrieved template is to update templates in memory bank for more dynamic change. • Extensive evaluations on four benchmarks show the effectiveness of the proposed DTDU. [ABSTRACT FROM AUTHOR]

Vehicle re-identification (V-ReID) aims at discovering an image of a specific vehicle from a set of images typically captured by different cameras. Vehicles are one of the most important objects in cross-camera target recognition systems, and recognizing them is one of the most difficult tasks due to the subtle differences in the visible characteristics of vehicle rigid objects. Compared to various methods that can improve re-identification accuracy, data augmentation is a more straightforward and effective technique. In this paper, we propose a novel data synthesis method for V-ReID based on local-region perspective transformation, transformation state adversarial learning and a candidate pool. Specifically, we first propose a parameter generator network, which is a lightweight convolutional neural network, to generate the transformation states. Secondly, an adversarial module is designed in our work, it ensures that noise information is added as much as possible while keeping the labeling and structure of the dataset intact. With this adversarial module, we are able to promote the performance of the network and generate more proper and harder training samples. Furthermore, we use a candidate pool to store harder samples for further selection to improve the performance of the model. Our system pays more balanced attention to the features of vehicles. Extensive experiments show that our method significantly boosts the performance of V-ReID on the VeRi-776, VehicleID and VERI-Wild datasets. • A data augmentation method is designed that integrates a deep learning framework. • Local-region perspective transformation is optimized with an adversarial model. • More balanced attention is paid to the overall features of vehicles. • Data augmentation and the recognition model are jointly optimized. • Amplified samples are more efficient due to the automatic learning process. • A candidate pool stores the augmented images through the dynamic learning process. • Datasets are augmented by lifting the difficulty rather than the quantity. • The structure of the original dataset is reserved including the size and labels. [ABSTRACT FROM AUTHOR]

Internet of Things (IoT) is widely applied in modern power systems, which could establish the intelligent power grid systems and obtain considerable social and economic benefits. IoT plays an important role in power grid safety production, user interaction, and information collection. However, existing methods cannot address problems of IoT devices accurately and quickly, such as fault detection. Aiming at the shortcomings of current power IoT equipment fault detection methods, this paper proposes a multi-spectral image fusion based on deep learning to detect fault points of power IoT equipment. The deep convolutional neural network is trained by simulating the image of the power device. The results show that the multi-spectral image descriptor based on deep learning presented in this paper shows very high accuracy in block matching, and the effect of image fusion is remarkable. This indicates that the proposed method can accurately integrate multi-spectral images of power equipment, helping to locate fault points quickly and accurately. [ABSTRACT FROM AUTHOR]

Convolution neural networks based methods can derive deep features from training images. However, one challenge is that the dimension of the extracted image features increases dramatically with more network layers. To solve this problem, this paper focuses on the study of dimension reduction. After using deep learning to extract image features, the PCA algorithm is used to achieve dimension reduction. Specifically, we first leverage deep convolutional neural network to extract image features. Then, we introduce and leverage PCA algorithm to achieve dimension reduction. Aiming at the problem that it is difficult to process high-dimensional sparse big data based on PCA algorithm. This paper optimizes the PCA algorithm. After image preprocessing, the feasibility of PCA algorithm for dimension reduction of image feature extraction by deep learning is verified by simulation experiments. The efficiency of the proposed algorithm is proved by comparing the performance of PCA algorithm before and after optimization. [ABSTRACT FROM AUTHOR]

Person re-identification(ReID) is an intelligent video surveillance technology that retrieves the same person from different cameras. This task is extremely challenging due to changes in person poses, different camera views, and occlusion. In recent years, person ReID based on deep learning technology has received widespread attention due to the rapid development and excellent performance of deep learning. In this paper, we first divide person ReID based on deep learning approaches into seven types, i.e., fused hand-crafted features deep model, representation learning model, metric learning model, part-based deep model, video-based model, GAN-based model, unsupervised model. Furthermore, we launched a brief overview of the seven types. Then, we introduce some examples of commonly used datasets, compare the performance of some algorithms on image and video datasets in recent years, and analyze the advantages and disadvantages of various methods. Finally, we summarize the possible future research directions of person ReID technology. • Dividing person ReID based on deep learning approaches into seven types. • Introducing some examples of commonly used datasets. • Comparing the performance of some algorithms on image and video datasets in recent years. • Analyzing the advantages and disadvantages of various methods. • Summarizing the possible future research directions of person ReID technology. [ABSTRACT FROM AUTHOR]

• An improved ASM architecture to extract handcrafted and deep features for AIFR. • ASM-CNN is a superior accessible solution for Rank-1 recognition. • It provides stronger performance as a classifier with ASM-CNN over CNN. Scientific efforts have expanded in age-invariant face recognition (AIFR). Matching faces of large age difference is, therefore, a problem, mostly because of a substantial disparity in the appearance of both young and old age. Owing to age, both the appearance and shape of the face are impaired, making recognition of the face the most challenging task. In recent years, AIFR has become a very common and demanding task. The set of feature extraction and classification algorithm is of prime importance in this field. As the numbers of features obtained from the datasets are large, there is a need to introduce a dimensionality reduction method to map high dimensionality feature space to low variance filter to form the final integrated face age model to be used in the classification process. In this paper, we introduced a novel concept of an improved Active Shape Model (ASM) in conjunction with a specially designed 7-layered Convolutional Neural Network (CNN) in order to accomplish a combination of feature extraction and classification in a single unit. The study approach involves conducting extensive experiments to evaluate the proposed system's performance using three standard datasets: FG-NET, LAG, and CACD. The results reveal that the proposed method outperforms state-of-the-art approaches and achieves excellent accuracy in face recognition across age. The maximum accuracies achieved by demonstrated ASM-CNN methodology for FG-NET, LAG, and CACD databases are 95.02%, 91.76 % and 99.4 % respectively. [ABSTRACT FROM AUTHOR]

• Parametric Self-circulation Gating Unit is proposed for deep neural networks. • Self-circulation gating property evaluates capacity of information circulation. • The strategy of weight initialization for PSGU helps the network to converge. Activation functions are of great importance for the performance and training of deep neural networks. High-performance activation function is expected to effectively prevent the gradient from vanishing and help network converge. This paper provides a novel smooth activation function, called Parameterized Self-circulating Gating Unit (PSGU), aiming to train an adaptive activation function to improve the performance of deep networks. Compared with other works, we propose and study the self-circulation gating property of activation function, and analyze its influence on the signal transmission in network by controlling the flow of information. Specifically, we theoretically analyze and propose the initialization based on PSGU, which adequately explores the properties in neighborhood of the origin. Finally, the proposed activation function and initialization are compared with other methods on commonly-used network architectures, the achieved performances of using PSGU alone or combining with our proposed initialization are over par with the state of the art. [ABSTRACT FROM AUTHOR]

Dense depth completion is essential for autonomous driving and robotic navigation. Existing methods focused on attaining higher accuracy of the estimated depth, which comes at the price of increasing complexity and cannot be well applied in a real-time system. In this paper, a coarse-to-fine and lightweight network (S&CNet) is proposed for dense depth completion to reduce the computational complexity with negligible sacrifice on accuracy. A dual-stream attention module (S&C enhancer) is proposed according to a new finding of deep neural network-based depth completion, which can capture both the spatial-wise and channel-wise global-range information of extracted features efficiently. Then it is plugged between the encoder and decoder of the coarse estimation network so as to improve the performance. The experiments on KITTI dataset demonstrate that the proposed approach achieves competitive result with respect to state-of-the-art works but via an almost four times faster speed. The S&C enhancer can also be easily plugged into other existing works to boost their performances significantly with negligible additional computations. • A novel coarse-to-fine lightweight network is proposed for the dense depth completion. • A dual-stream attention module is proposed to enhance the representation power of the network. • It is found that each feature channel generated by the encoder network corresponds to a different depth range. [ABSTRACT FROM AUTHOR]

Existing NR-IIQA (no reference-based inpainted image quality assessment) algorithms assess the quality of an inpainted image via artificially designed unnaturalness expression, which often fail to capture inpainted artifacts. This paper presents a new deep rank learning-based method for NR-IIQA. The model adopts a siamese deep architecture, which takes a pair of inpainted images as input and outputs their rank order. Each branch utilizes a CNN structure to capture the global structure coherence and a patch-wise coherence assessment module (PCAM) to depict the local color and texture consistency in an inpainted image. To train the deep model, we construct a new dataset, which contains thousands of pairs of inpainted images with ground-truth quality ranking labels. Rich ablation studies are conducted to verify the key modules of the proposed architecture. Comparative experimental results demonstrate that our method outperforms existing NR-IIQA metrics in evaluating both inpainted images and inpainting algorithms. • Siamese CNN-based NR-IIQA, automatically ranking two inpainted images. • A patch-wise coherence assessment module to evaluate local color and texture consistency within and around inpainted contents. • A dataset containing thousands of inpainted image pairs with ground-truth ranking labels. [ABSTRACT FROM AUTHOR]

• A novel technique is proposed for recognizing any air-written trajectories without using any pen-starting-lift signal. • A backward time order stroke recognizer is implemented to characterize a writing trajectory in a backward manner. It can effectively remove redundant trajectory data. • A novel hierarchical arrangement technique that presents an air-written English alphabet tier by tier can well handle both the multiplicity and confusion problems. Air-writing is a new human and smart device communication approach, permits users to write inputs in a natural and relentless way. This touch-less way can prevent users from virus infection such as COVID-19. Compared with other methods, air writing is more challenging due to its unique characteristics such as redundant lifting strokes, multiplicity (different writing styles from various users), and confusion (different character types written in air are similar). Without the need of any starting trigger, a novel reverse time-ordered algorithm is proposed in this paper to efficiently filter out unnecessary lifting strokes, and thus simplifies the matching procedure. As to the second and third issues, a tiered arrangement structure is proposed by sampling the air-writing results with various sampling rates to solve the multiplicity and confusion problems. Analyzed with other recently proposed air writing algorithms, the proposed approach reaches satisfactory recognition accuracy (above 94%) without any starting triggers. [ABSTRACT FROM AUTHOR]

Automated person re-identification in a multi-camera surveillance setup is very important for effective tracking and monitoring crowd movement. In this paper, we propose an efficient hierarchical re-identification approach in which color histogram-based comparison is employed to find the closest matches in the gallery set, and next deep feature-based comparison is carried out using the Siamese network. Reduction in search space after the first level of matching helps in improving the accuracy as well as efficiency of prediction by the Siamese network by eliminating dissimilar elements. A silhouette part-based feature extraction scheme is adopted in each level of hierarchy to preserve the relative locations of the different body parts and make the appearance descriptors more discriminating. The proposed approach has been evaluated on five public data sets and also a new data set captured in our laboratory. Results reveal that it outperforms most state-of-the-art approaches in terms of overall accuracy. • Proposing an efficient re-identification method from color and deep Siamese features. • Carrying out silhouette part-based analysis to improve the accuracy of prediction. • Performing experiments and thorough comparison to evaluate its effectiveness. • Constructing a database of 41 subjects and making it publicly available. [ABSTRACT FROM AUTHOR]

At present, the main super-resolution (SR) method based on convolutional neural network (CNN) is to increase the layer number of the network by skip connection so as to improve the nonlinear expression ability of the model. However, the network also becomes difficult to be trained and converge. In order to train a smaller but better performance SR model, this paper constructs a novel image SR network of multiple attention mechanism(MAMSR), which includes channel attention mechanism and spatial attention mechanism. By learning the relationship between the channels of the feature map and the relationship between the pixels in each position of the feature map, the network can enhance the ability of feature expression and make the reconstructed image more close to the real image. Experiments on public datasets show that our network surpasses some current state-of-the-art algorithms in PSNR, SSIM, and visual effects. [ABSTRACT FROM AUTHOR]

Many deep neural networks are built by using stacked convolutional layers of fixed and single size (often 3 × 3) kernels. This paper describes a method for learning the size of convolutional kernels to provide varying size kernels in a single layer. The method utilizes a differentiable, and therefore backpropagation-trainable Gaussian envelope which can grow or shrink in a base grid. Our experiments compared the proposed adaptive layers to ordinary convolution layers in a simple two-layer network, a deeper residual network, and a U-Net architecture. The results in the popular image classification datasets such as MNIST, MNIST-CLUTTERED, CIFAR-10, Fashion, and "Faces in the Wild" showed that the adaptive kernels can provide statistically significant improvements on ordinary convolution kernels. A segmentation experiment in the Oxford-Pets dataset demonstrated that replacing ordinary convolution layers in a U-shaped network with 7 × 7 adaptive layers can improve its learning performance and ability to generalize. • The kernel size hyperparameter of convolutional layers is made adaptive (learned). • Allows kernels of varying size in a single convolutional layer. • Improves the performance of the RESNET model in image classification. • Improves the performance of the UNET model in image segmentation. [ABSTRACT FROM AUTHOR]

In this paper, we aim to develop an automatic system to monitor and evaluate worker's efficiency for smart manufacturing based on human pose tracking and temporal action localization. First, we explore the generative adversarial networks (GANs) to achieve significantly improved estimation of human body joints. Second, we formulate the automated worker efficiency analysis into a temporal action localization problem in which the action video performed by the worker is matched against a reference video performed by a teacher. We extract invariant spatio-temporal features from the human body pose sequences and perform cross-video matching using dynamic time warping. Our proposed human pose estimation method achieves state-of-the-art performance on the benchmark dataset. Our automated work efficiency analysis is able to achieve action localization with an average IoU (intersection over union) score large than 0.9. This represents one of the first systems to provide automated worker efficiency evaluation. • Highly desirable to develop an automated system for worker efficiency evaluation. • Generative adversarial learning improves human pose estimation. • Action-level cross-video DTW matching helps to locate worker's activity. • Establish a method for automated worker efficiency analysis. [ABSTRACT FROM AUTHOR]

This paper presents a study of automatic design of neural network achitectures for skeleton-based action recognition. Specifically, we encode a skeleton-based action instance into a tensor and carefully define a set of operations to build two types of network cells: normal cells and reduction cells. The recently developed DARTS (Differentiable Architecture Search) is adopted to search for an effective network architecture that is built upon the two types of cells. All operations are 2D based in order to reduce the overall computation and search space. Experiments on the challenging NTU RGB+D and Kinectics datasets have verified that most of the networks developed to date for skeleton-based action recognition are likely not compact and efficient. The proposed method provides an approach to search for such a compact network that is able to achieve comparative or even better performance than the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, we propose an end-to-end hierarchical-based multi-mode attention network and adaptive fusion (HMAN-HAF) strategy to learn different-level salient features for re-ID tasks. First, according to each layer's characteristics, a hierarchical multi-mode attention network (HMAN) is designed to adopt different attention models for different-level salient feature learning. Specifically, refined channel-wise attention (CA) is adopted to capture high-level valuable semantic information, an attentive region model (AR) is used to detect salient regions in the low layer, and fused attention (FA) is designed to capture the salient regions of valuable channels in the middle layer. Second, a hierarchical adaptive fusion (HAF) is constructed to fulfill the complementary strengths of different-level salient features. Experimental results demonstrate that the proposed method outperforms the state-of-the-art methods on the following challenging benchmarks: Market-1501, DukeMTMC-reID and CUHK03. [ABSTRACT FROM AUTHOR]

In this paper, we proposed a semi-supervised common representation learning method with GAN-based Asymmetric Transfer Network (GATN) for cross modality retrieval. GATN utilizes the asymmetric pipeline to guarantee the semantic consistency and adopt (Generative Adversarial Network) GAN to fit the distributions of different modalities. Specifically, the common representation learning across modalities includes two stages: (1) the first stage, GATN trains source mapping network to learn the semantic representation of text modality by supervised method; and (2) the second stage, GAN-based unsupervised modality transfer method is proposed to guide the training of target mapping network, which includes generative network (target mapping network) and discriminative network. Experimental results on three widely-used benchmarks show that GATN have achieved better performance comparing with several existing state-of-the-art methods. • Asymmetric mappings can better model the differences in low level features. • Transfer learning mechanism ensures the training efficiency of cross modal retrieval. • Semantic constraint divides samples into several semantically discriminative clusters. • Adversarial learning ensures the alignment of distributions from two modalities. [ABSTRACT FROM AUTHOR]

Single image dehazing is a critical image pre-processing step for many practical vision systems. Most existing dehazing methods solve this problem utilizing various of hand-crafted priors or by supervised training on the synthetic hazy image information (such as haze-free image, transmission map and atmospheric light). However, the assumptions on the hand-crafted priors are easily violated and collecting realistic transmission map and atmospheric light are unpractical. In this paper, we propose a novel weakly supervised network based on the multi-level multi-scale block. The proposed network reduces the constraint on the training data and automatically estimates the transmission map and the atmospheric light as well as the intermediate haze-free image without using any realistic transmission map and atmospheric light as supervision. Moreover, the estimated intermediate haze-free image helps to generate accurate transmission map and atmospheric light by embedding the physical-model, which presents reliable restoration of the final haze-free image. In particular, our network also can be trained on the real-world dataset to fine-tune the model and the fine-tuning operation improves the dehazing performance on the real-world dataset. Quantitative and qualitative experimental results demonstrate the proposed method performs on par with the supervised methods. • We propose a weakly supervised single image dehazing method. • The network can simultaneously estimate the transmission map and atmospheric light. • The network can be fine-tuned to improve the real-world dehazing result. [ABSTRACT FROM AUTHOR]

• The candidate regions for video actions share both the spatial correlations and the strong temporal correlations. • We propose a proposal based self-attention mechanism (PSM) that determines the correlation between candidate regions. • PSM serves a generic module that can be inserted into the different layers. • The results show the promising improvement achieved by PSM over the state-of-the-art solutions. The rapid development of deep learning has prompted the development of video action detection technology. However, the accuracy of current video action detection algorithms can be improved further. Previous work has improved feature extraction by optimizing the network structure. In addition, the features of the candidate regions have been optimized by changing the representation of the regions. Although these methods have achieved promising results, they fail to consider the correlation among different candidate regions, generating uninformative (even redundant) candidate regions, and thus usually decrease the detection performance in practice. To address this problem, in this paper we propose a self-attention mechanism for candidate regions, which can help pursue the most informative regions. We obtain the region correlation by simultaneously determining the spatial and temporal correlation among different candidate regions. In addition, we focus on how to apply the correlation to optimize the original candidate region features and improve video action detection accuracy. The experimental results show the promising improvement achieved by our method over the state-of-the-art solutions. [ABSTRACT FROM AUTHOR]

• Cross-domain representation learning framework incorporating Siamese network with generative adversarial network for SBIR. • Domain migration with generative adversarial learning for transferring sketches to images with rich information. • Siamese network with new designed loss function for similarity measurement in SBIR. Sketch based image retrieval (SBIR), which uses free-hand sketches to search the images containing similar objects/scenes, is attracting more and more attentions as sketches could be got more easily with the development of touch devices. However, this task is difficult as the huge differences between sketches and images. In this paper, we propose a cross-domain representation learning framework to reduce these differences for SBIR. This framework aims to transfer sketches to images with the information learned both in the sketch domain and image domain by the proposed domain migration generative adversarial network (DMGAN). Furthermore, to reduce the representation gap between the generated images and natural images, a similarity learning network (SLN) is also proposed with the new designed loss function incorporating semantic information. Extensive experiments have been done from different aspects, including comparison with state-of-the-art methods. The results show that the proposed DMGAN and SLN really work for SBIR. [ABSTRACT FROM AUTHOR]

• A new animation method is proposed to generate a video animation from still image based on intelligent computing. • The active objects can generate a virtual driving force to drive the passive objects. • An innovative analytical method is designed to generate a virtual driving force for active objects. • Two types of animation driving force was specified in our method: virtual driving force and virtual wind force. Bringing a single still image into reality is a challenging topic in computer animation because the driven and structural information in single still image is inadequate. In this paper, we present an image animating method for enhancing single still image in social media with virtual realistic and animated motions without prior information. We imitate the interaction between the active objects in an image and their neighboring passive objects. The existing actions in the image and the virtual specified force are employed to animate the active objects. Observing that the change between two subsequent motions of the active objects derives a motion tendency, we can calculate a virtual driving force based on the motion tendency. By virtue of the virtual driving force, the stochastic motion texture is used to animate the passive objects. Finally, the convolutional neural network is employed to optimize the virtual motion animations. In this way, the proposed method produces visually natural results while guaranteeing motion harmony between active objects and passive objects. To demonstrate the applicability and rationality of virtual animation driving force, our method generates several animations from still images in Social Media. [ABSTRACT FROM AUTHOR]

• An unsupervised tree-to-sequence neural machine translation method is proposed. • Learning latent trees for improving neural machine translation. • Comparisons between Gumbel Tree-based model and the baselines, RNMT and Transformer. • Analysis on the learned tree structures and attention scores on the structures. Neural machine translation has improved the translation accuracy greatly and received great attention of the machine translation community. Tree-based translation models aim to model the syntactic or semantic relation among long-distance words or phrases in a sentence. However, it faces the difficulties of expensive manual annotation cost and poor automatic annotation accuracy. In this paper, we focus on how to encode a source sentence into a vector in a unsupervised-tree way and then decode it into a target sentence. Our model incorporates Gumbel Tree-LSTM, which can learn how to compose tree structures from plain text without any tree annotation. We evaluate the proposed model on both spoken and news corpora, and show that the performance of our proposed model outperforms the attentional seq2seq model and the Transformer base model. [ABSTRACT FROM AUTHOR]

• Retaining high frequency detail information is important for image denoising. • Removing batch normalization (BN) is beneficial for image denoising. • Image restoration tasks can be finished by a single end-to-end network. Compared with the traditional image denoising method, although the convolutional neural network (CNN) has better denoising performance, there is an important issue that has not been well resolved: the residual image obtained by learning the difference between noisy image and clean image pairs contains abundant image detail information, resulting in the serious loss of detail in the denoised image. In this paper, in order to relearn the lost image detail information, a mathematical model is deducted from a minimization problem and an end-to-end detail retaining CNN (DRCNN) is proposed. Unlike most denoising methods based on CNN, DRCNN is not only focus to image denoising, but also the integrity of high frequency image content. DRCNN needs less parameters and storage space, therefore it has better generalization ability. Moreover, DRCNN can also adapt to different image restoration tasks such as blind image denoising, single image superresolution (SISR), blind deburring and image inpainting. Extensive experiments show that DRCNN has a better effect than some classic and novel methods. [ABSTRACT FROM AUTHOR]

Peer-to-peer (P2P) lending platform plays a significant role in modern financial systems. However, due to improper supervision, credit risk is inevitable. In this paper, we analyze the traditional financial risk and information technology risk of P2P lending platform. In order to evaluate the performance of assessment algorithms, we present a BP neural network-based algorithm for lending risk assessment. To achieve our task, we crawled large-scale lending data for 2015–2019. Logistic regression is used to compare with BP neural network method. Experimental results show that BP neural network-based algorithm outperforms traditional Logistic regression algorithm and the proposed method can effectively reduce investor risk. [ABSTRACT FROM AUTHOR]

Finance service based on big data faces many issues, such as fraud, credit. In this paper, we study the development of financial business model under the big data. We first analyze the impact mechanism of big data finance on customer information protection of commercial banks. Customer information has the characteristics of large amount of information, high value of data and strong destructive data leakage. Then, we propose two solutions towards issues of finance service including face anti-spoofing algorithm and financial risk evaluation. Experiments show the effectiveness of our proposed method in improving the reliability and security of modern big data finance. [ABSTRACT FROM AUTHOR]

Conventional face image generation using generative adversarial networks (GAN) is limited by the quality of generated images since generator and discriminator use the same backpropagation network. In this paper, we discuss algorithms that can improve the quality of generated images, that is, high-quality face image generation. In order to achieve stability of network, we replace MLP with convolutional neural network (CNN) and remove pooling layers. We conduct comprehensive experiments on LFW, CelebA datasets and experimental results show the effectiveness of our proposed method. [ABSTRACT FROM AUTHOR]

This paper proposes an image retrieval algorithm towards massive-scale multimedia data. In order to be consistent with human visual system, we first design a color attention function to describe the important of different image patches. Subsequently, we combine color and texture to construct candidate regions, which will be fed into a deep neural network (DNN) for deep representation extraction. Then, we design a similarity function to calculate the distance among different images, where top-ranking images are considered as the required images. Experimental results show the effectiveness and robustness of our proposed method. [ABSTRACT FROM AUTHOR]

• Convolutional operations suffer from efficiently capturing global information. • Saliency models can benefit from non-local blocks to capture long-range dependencies. • Non-local blocks make FCNs more efficient in extracting global features. • Non-local neural networks help to find salient similar regions far from each other. • Non-local blocks are effective in extracting global appearance and motion clues. Detection of salient objects in image and video is of great importance in many computer vision applications. In spite of the fact that the state of the art in saliency detection for still images has been changed substantially over the last few years, there have been few improvements in video saliency detection. This paper proposes a novel non-local fully convolutional network architecture for capturing global dependencies more efficiently and investigates the use of recently introduced non-local neural networks in video salient object detection. The effect of non-local operations is studied separately on static and dynamic saliency detection in order to exploit both appearance and motion features. A novel deep non-local fully convolutional network architecture is introduced for video salient object detection and tested on two well-known datasets DAVIS and FBMS. The experimental results show that the proposed algorithm outperforms state-of-the-art video saliency detection methods. [ABSTRACT FROM AUTHOR]

• Reinforced learning. • Reduced space and time complexity. • CNN. • Real time Gait identification. This paper evaluates the performance of contemporary gait identification systems. A time, erosion and neural inspired framework (TEN-FE) for gait identification was proposed to augment the performance of gait identification systems. Performance of TEN-FE framework was evaluated using CASIA and OU-ISIR large population dataset. Proposed framework relies on CNN and Reinforcement Learning to restrict the impact of confounding factors like baggage and bulky clothing on the accuracy of gait identification systems. Difference in gait signature due to time was also considered and normalized. The results observed a clear increase in system's performance with minimal complexity and least hardware requirements. [ABSTRACT FROM AUTHOR]

Neural network based methods for fisheye distortion correction are effective and increasingly popular, although training network require a high amount of labeled data. In this paper, we propose an unsupervised fisheye correction network to address the aforementioned issue. During the training process, the predicted parameters are employed to correct strong distortion that exists in the fisheye image and synthesize the corresponding distortion using the original distortion-free image. Thus, the network is constrained with bidirectional loss to obtain more accurate distortion parameters. We calculate the two losses at the image level as opposed to directly minimizing the difference between the predicted and ground truth of distortion parameters. Additionally, we leverage the geometric prior that the distortion distribution depends on the geometric regions of fisheye images and the straight line should be straight in the corrected images. The network focuses more on the geometric prior regions as opposed to equally perceiving the whole image without any attention mechanisms. To generate more appealing corrected results in visual appearance, we introduce a coarse-to-fine inpainting network to fill the hole regions caused by the irreversible mapping function using distortion parameters. Each module of the proposed network is differentiable, and thus the entire framework is completely end-to-end. When compared with the previous supervised methods, our method is more flexible and shows better practical applications for distortion rectification. The experiment results demonstrate that our proposed method outperforms state-of-the-art methods on the correction performance without any labeled distortion parameters. [ABSTRACT FROM AUTHOR]

• A combined net is proposed to improve salient object detection. • An function is constructed to diffuse saliency from borders to salient objects. • Region Net is adopted to generate salient map with sharp object boundary. • Local-Global Net is proposed to provide reliable multi-scale contextual information. In this paper, a new algorithm based on a combined neural network is proposed to improve salient object detection in the complex images. It consists of two main steps. The first step, an objective function which is optimized on a multi-layer graph structure is constructed to diffuse saliency from borders to salient objects, aiming to roughly estimate the location and extent salient objects of an image, meanwhile, color attribute is adopted to rapidly find a set of object-related regions in the image. The second step, establish a combined neural network with Region Net and Local-Global Net. Region Net is adopted to efficiently generate the salient map with the sharp object boundary. Then Local-Global Net based on multi-scale spatial context is proposed to provide strongly reliable multi-scale contextual information, and thus achieves an optimized performance. Experimental results and comparison analysis demonstrate that the proposed algorithm is more effective and superior than most low-level oriented prior methods in terms of precision recall curves, F-measure and mean absolute errors. [ABSTRACT FROM AUTHOR]

In this paper, we present a salient object detection method based on novel graph structure. Given image is segmented into small image regions as basic units, we firstly construct an effective background-based map, each image region's saliency value is determined by its feature contrast with the image boundary. Then, saliency propagation mechanism is used to update all regions' saliency values by introducing a novel graph structure to better exploit the relationship between adjacent image regions. Finally, we propose an optimization method to further highlight salient objects and suppress background noises. Experimental results demonstrate adequately the superiority of proposed approach. [ABSTRACT FROM AUTHOR]

Based on transfer learning, feature maps of deep convolutional neural networks (DCNNs) have been used to predict human visual attention. In this paper, we conduct extensive comparisons to investigate effects of feature maps on the predictions of the human visual attention using a deep features based saliency model framework. The feature maps of seven pretrained DCNNs are investigated using classical and class activation maps approaches. The performances of various saliency implementations are evaluated over four datasets using three metrics. The results demonstrate that deep feature maps of the pretrained DCNNs can be used to create saliency maps for the prediction of human visual attention. The incorporation of multiple levels of blurred and multi-scale feature maps improves the extraction of salient regions. Moreover, DCNNs pretrained using the Places dataset provide more localized objects that can be beneficial to the top-down saliency maps. [ABSTRACT FROM AUTHOR]

Conditional Generative Adversarial Networks (CGANs) have been introduced to generate realistic images from extremely degraded inputs. However, these generative models without prior knowledge of spatial distributions has limited performance to deal with various complex scenes. In this paper, we proposed a image deblurring network based on CGANs to generate ideal images without any blurring assumption. To overcome adversarial insufficiency, an extended classifier with different attribute domains is formulated to replace the original discriminator of CGANs. Inspired by residual learning, a set of skip-connections are cohered to transfer multi-scaled spatial features to the following high-level operations. Furthermore, this adversary architecture is driven by a composite loss that integrates histogram of gradients (HoG) and geodesic distance. In experiments, an uniformed adversarial iteration is circularly applied to improve image degenerations. Extensive results show that the proposed deblurring approach significantly outperforms state-of-the-art methods on both qualitative and quantitative evaluations. [ABSTRACT FROM AUTHOR]