Search

Scene classification and recognition have always been one of the most challenging tasks of scene understanding due to the inherent ambiguity in visual scenes. The core of scene classification and recognition tasks is scene representation. Deep learning advances in computer vision, especially deep CNNs, have significantly improved scene representation in the last decade. Deep convolutional features extracted from deep CNNs provide discriminative representations of the images and are widely used in various computer vision tasks, such as scene classification. Deep convolutional features capture the appearance characteristics of the image and the spatial information about different image regions. Meanwhile, the semantic and context information obtained from high-level concepts about scene images, such as objects and their relationships, can significantly contribute to identifying scene images. Therefore, in this paper, we divide visual scenes into two categories, object-based and layout-based. Object-based scenes are scenes that have scene-specific objects and, based on those objects, can be described and identified. In contrast, the layout-based scenes do not have scene-specific objects and are described and identified based on the appearance and layout of the image. This paper proposes a new neural network model for representing and classifying visual scenes, which we call G-CNN (GNN-CNN). The proposed model includes two modules, feature extraction and feature fusion, and the feature extraction module composes of visual and semantic branches. The visual branch is responsible for extracting deep CNN features from the image, and the semantic branch is responsible for extracting semantic GNN features from the scene graph corresponding to the image. The feature fusion module is a novel two-stream neural network that fuses the CNN and GNN feature vectors to produce a comprehensive representation of the scene image. Finally, a fully-connected classifier classified the obtained comprehensive feature vector into one of the pre-defined categories. The proposed model has been evaluated on three benchmark scene datasets, UIUC Sports, MIT67, and SUN397, and obtained classification accuracy of 99.91%, 96.01%, and 85.32%, respectively. In addition, a new dataset named Scene40, which has been introduced in our previous paper, is also used for further evaluation of the proposed method. The comparison results based on classification accuracy criteria show that the proposed model can outperform the best previous methods on three benchmark scene datasets. [ABSTRACT FROM AUTHOR]

In recent years, single-image super-resolution (SISR) has acquired tremendous progress with the development of deep learning. However, the majority of SISR methods based on deep learning focus on building more complex networks, which inevitably lead to the problems of computational and memory costs. Thus, these methods may fail to be applied in real-world scenarios. To solve this problem, this paper proposes a lightweight convolution network combined with transformer for SISR named as MMSR. Specifically, an efficient convolutional neural network (CNN) based on multi-scale feature fusion is designed for local feature extraction, which is called MFF-CNN. In addition, we propose a simple and efficient multiple attention block (MAB) to further utilize the context information in features. MAB incorporates channel attention and transformer to help network obtain similar features at a long-term dependence, making full use of global information to further refine texture details. Finally, this paper provides comprehensive results for different settings of the entire network. Experimental results on common used datasets demonstrate that the proposed method can achieve better performances at the 2 × , 3 × and 4 × scales than other state-of-the-art lightweight methods. [ABSTRACT FROM AUTHOR]

Automatic facial expression recognition plays a crucial role in computer vision, and pattern recognition. However, most existing deep learning-based facial expression classifiers usually obtain high average accuracy but have poor recognition accuracy for difficult expressions, like fear and disgust. In this paper, we propose a novel end-to-end architecture termed two-stream inter-class variation enhancement network, which learns the high-level semantic features and subtle inter-class variations in a joint fashion. More precisely, the global feature extraction network is used to extract spatial-channel semantic features, and the variations between different expressions are modeled by a distinction-reinforced network. The outputs of these two streams are weighted integrated in the expression classification network. In addition, a class balanced-weighted cross-entropy loss is designed to further improve feature discrimination. Experiment results indicate that the proposed network can significantly improve the recognition of difficult expressions and achieve a satisfactory average recognition accuracy of 73.67% on FER2013, 86.17% on RAFDB, 98.19% on CK+, and 98.85% on Oulu-CASIA, which outperforms the other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

As an advanced image editing technology, image inpainting leaves very weak traces in the tampered image, causing serious security issues, particularly those based on deep learning. In this paper, we propose the global–local feature fusion network (GLFFNet) to locate the image regions tampered by inpainting based on deep learning. GLFFNet consists of a two-stream encoder and a decoder. In the two-stream encoder, a spatial self-attention stream (SSAS) and a noise feature extraction stream (NFES) are designed. By a transformer network, the SSAS extracts global features regarding deep inpainting manipulations. The NFES is constructed by the residual blocks, which are used to learn manipulation features from noise maps produced by filtering the input image. Through a feature fusion layer, the features output by the encoder is fused and then fed into the decoder, where the up-sampling and convolutional operations are employed to derive the confidential map for inpainting manipulation. The proposed network is trained by the designed two-stage loss function. Experimental results show that GLFFNet achieves a high location accuracy for deep inpainting manipulations and effectively resists JPEG compression and additive noise attacks. [ABSTRACT FROM AUTHOR]

Head pose estimation methods without facial key points have emerged as a promising research field. However, there remain several unsolved challenges. For example, the current methods incur a computational cost, require large memory, and are difficult to deploy in practical applications. We propose a lightweight high-precision head pose estimation method based on a dual-stream convolutional neural network for overcoming these issues. The network comprises a dual-stream lightweight backbone network, external attention module, and soft stagewise regression (SSR) module. Dual-stream lightweight backbone network can extract original image features more effectively while keeping low computational overhead. External attention module can enhance the feature map extraction from the backbone network and improve the feature attention. SSR module calculates the probability of the head in each direction and predicts the head pose by regression. Extensive experiments on Annotated Facial Landmarks in the Wild (AFLW2000) and Biwi Kinect Head Pose Database (BIWI) datasets demonstrate that the model proposed in this paper has fewer parameters and lower estimation errors than the state-of-the-art methods in the field of head pose estimation in recent years. [ABSTRACT FROM AUTHOR]

Medical image segmentation has the priori guiding significance for clinical diagnosis and treatment. In the past ten years, a large number of experimental facts have proved the great success of deep convolutional neural networks in various medical image segmentation tasks. However, the convolutional networks seem to focus too much on the local image details, while ignoring the long-range dependence. The Transformer structure can encode long-range dependencies in image and learn high-dimensional image information through the self-attention mechanism. But this structure currently depends on the database scale to give full play to its excellent performance, which limits its application in medical images with limited database size. In this paper, the characteristics of CNNs and Transformer are integrated to propose a dual encoding–decoding structure of the X-shaped network (X-Net). It can serve as a good alternative to the traditional pure convolutional medical image segmentation network. In the encoding phase, the local and global features are simultaneously extracted by two types of encoders, convolutional downsampling, and Transformer and then merged through jump connection. In the decoding phase, a variational auto-encoder branch is added to reconstruct the input image itself in order to weaken the impact of insufficient data. Comparative experiments on three medical image datasets show that X-Net can realize the organic combination of Transformer and CNNs. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) has become a significant area of research in human behavior analysis, human–computer interaction, and pervasive computing. Recently, deep learning (DL)-based methods have been applied successfully to time-series data generated from smartphones and wearable sensors to predict various activities of humans. Even though DL-based approaches performed very well in activity recognition, they are still facing challenges in handling time series data. Several issues persist with time-series data, such as difficulties in feature extraction, heavily biased data, etc. Moreover, most of the HAR approaches rely on manual feature engineering. In this paper, to design a robust classification model for HAR using wearable sensor data, a hybrid of convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) is used. The proposed multibranch CNN-BiLSTM network does automatic feature extraction from the raw sensor data with minimal data pre-processing. The use of CNN and BiLSTM makes the model capable of learning local features as well as long-term dependencies in sequential data. The different filter sizes used in the proposed model can capture various temporal local dependencies and thus helps to improve the feature extraction process. To evaluate the model performance, three benchmark datasets, i.e., WISDM, UCI-HAR, and PAMAP2, are utilized. The proposed model has achieved 96.05%, 96.37%, and 94.29% accuracies on WISDM, UCI-HAR, and PAMAP2 datasets, respectively. The obtained experimental results demonstrate that the proposed model outperforms the other compared approaches. [ABSTRACT FROM AUTHOR]

In view of the multi-scale changes and the influence of light and angle in the image matching process, it is quite difficult to realize intelligent image registration by using convolutional neural network. The existing image matching algorithm has the following problems in the application process: the existing shallow feature extraction model has lost a lot of effective feature information and low recognition accuracy. Meanwhile, the image registration method based on deep learning is not robust and accurate enough. Therefore, an image registration method based on additive edge cosine loss was proposed in this paper. In the twin network architecture, cosine loss was used to convert Euclidean space into angular space, which eliminated the influence of characteristic intensity and improved the accuracy of registration. The matching cost was directly calculated by the included angle of two vectors in the embedded space, where the size of the angle edge could be quantitatively adjusted through parameter m . We further derived a specific m to quantitatively adjust the loss. In addition, anti-rotation attention mechanism was added to the network to enhance the ability of feature information extraction and adjust the position information of feature vectors to reduce the mismatching caused by image rotation. [ABSTRACT FROM AUTHOR]