Search

Lung nodules are abnormal growths and lesions may exist. Both lungs may have nodules. Most lung nodules are harmless (not cancerous/malignant). Pulmonary nodules are rare in lung cancer. X-rays and CT scans identify the lung nodules. Doctors may term the growth a lung spot, coin lesion, or shadow. It is necessary to obtain properly computed tomography (CT) scans of the lungs to get an accurate diagnosis and a good estimate of the severity of lung cancer. This study aims to design and evaluate a deep learning (DL) algorithm for identifying pulmonary nodules (PNs) using the LUNA-16 dataset and examine the prevalence of PNs using DB-Net. The paper states that a new, resource-efficient deep learning architecture is called for, and it has been given the name of DB-NET. When a physician orders a CT scan, they need to employ an accurate and efficient lung nodule segmentation method because they need to detect lung cancer at an early stage. However, segmentation of lung nodules is a difficult task because of the nodules' characteristics on the CT image as well as the nodules' concealed shape, visual quality, and context. The DB-NET model architecture is presented as a resource-efficient deep learning solution for handling the challenge at hand in this paper. Furthermore, it incorporates the Mish nonlinearity function and the mask class weights to improve segmentation effectiveness. In addition to the LUNA-16 dataset, which contained 1200 lung nodules collected during the LUNA-16 test, the LUNA-16 dataset was extensively used to train and assess the proposed model. The DB-NET architecture surpasses the existing U-NET model by a dice coefficient index of 88.89%, and it also achieves a similar level of accuracy to that of human experts. [ABSTRACT FROM AUTHOR]

Multi-exposure fusion (MEF) is widely researched and applied to high dynamic range (HDR) imaging, where one of the most challenging problems is the artifacts caused by the motion of objects between input images. Recently, deep learning methods have been widely applied for HDR imaging with excellent results, showing significant advantages. However, many methods cannot avoid artifacts due to inaccurate alignment before merging HDR images. In this paper, we propose an end-to-end network (C-ED-GMNET) with a channel and spatial attention network, an encoder–decoder network and a gradual merging network for generating artifact-free HDR images in dynamic scenes. The attention module consists of two submodules to identify useful features and exclude harmful components in inputs from both channel and spatial dimensions, respectively. The attention-guided feature maps are sent to the encoders for further feature extraction, and then, the outputs are sent to a gradual merging module consisting of two steps to generate deep features progressively. Besides, the differences between the merged image and the original images are identified by applying global residual learning with all the inputs, and the merged image feature is recovered by the decoder to obtain the final HDR image. Quantitative and qualitative experiments on two public datasets show that the proposed C-ED-GMNET produces better results than existing state-of-the-art methods and significantly reduces the runtime due to the encoders which reduce the amount of computation. [ABSTRACT FROM AUTHOR]

Rice which is a staple food crop in most Asian countries mainly suffers from higher yield loss due to different factors, and one of the common factors that affect rice yield is rice leaf disease. Many countries do not allow the plantation of new rice varieties unless they are disease-resistant. Hence, disease control serves as an important factor for rice production and this can be effectively accomplished by identifying the rice disease at an early stage which is required for timely pesticide application and disease control. The rice leaf diseases are mainly inspected by the farmers with their naked eye which is error-prone, time-consuming, and often leads to huge yield loss if predicted wrong. This process is effectively handled by different researchers via the use of computer vision and machine learning techniques. To overcome the drawbacks associated with manual processing and low recognition accuracy of different statistical and machine learning techniques, this paper presents a novel concept that uses 3D 2D CNN for feature extraction and an improved backtracking search (IBS) algorithm-optimized deep generative adversarial network (DGAN) for classification. The 3D2D deep convolutional neural network (DCNN) is formed by integrating the 3D fast-learning block with a 2DCNN for extracting the rice disease features such as lesions and shape. The IBS algorithm-optimized GAN architecture is used for rice disease classification and the IBS algorithm is mainly adopted to overcome the instability and overfitting issues that arise in the DGAN. The proposed IBS-optimized GAN architecture offers improved accuracy of 98.7% which is relatively higher than the existing techniques such as extreme gradient boosting (XGBoost), transfer learning techniques, and support vector machine (SVM). [ABSTRACT FROM AUTHOR]

Intelligent recognition of traffic signs and markings is an important component of autonomous driving and intelligent transportation systems. It is also an important theoretical basis for autonomous driving path planning. To address the low accuracy and poor real-time performance of the traffic signs and markings detection in complex and multivariate scenes, a lightweight convolutional neural network recognition method in multiple interference scenes is proposed. Firstly, based on Gamma correction and contrast limited histogram equalization algorithm, the traffic signs and markings image under the low illumination condition is enhanced adaptively. Then, the MobileNet-V2 fuse together with DeepLab-V3 + algorithm to segment traffic signs and markings in multiple scenes. Finally, an identification model is established to realize the adaptive recognition of traffic signs and markings, which are based on Lightweight Convolutional Neural Networks (Lw-CNN). Besides, the recognition method proposed in this paper is verified by the CSUST Chinese Traffic Sign Detection Benchmark (CCTSDB). The experimental results show that the Mean Intersection Over Union (MIOU) of the MobileNet-V2 and DeepLab-V3 + algorithm reaches 83.07%, it increased by 25.8% than before image enhancement. The recognition accuracy of algorithm based on lightweight convolutional neural network is 99.92%, higher than the algorithms of MobileNet-V2 and VGG16. [ABSTRACT FROM AUTHOR]

Unlike 2D images that are represented in regular grids, 3D point clouds are irregular and unordered, hence directly applying convolution neural networks (CNNs) to process point clouds is quite challenging. In this paper, we propose a novel deep neural network named AKNet to achieve point cloud semantic segmentation. The key to our AKNet is the attentive kernel convolution (AKConv), which is a deformed convolution operation for perceiving sufficient local context of 3D scenes. AKConv first constructs the Basic Weight Units that are robust to point's ordering. Then, for capturing the more distinctive local features, the convolution kernels of AKConv are associated with Attentive Weight Units through the self-attentive function acting on Basic Weight Units. Furthermore, 3D point clouds provide richer geometric shape information, which is helpful to recognize objects. However, inputting only raw point features to the convolution function could cause geometric information loss. Thus, we utilize augmented features as input of AKConv. Besides, to preserve the semantic information from the encoding to decoding layers, we introduce the backward encoding (BE) mechanism by utilizing higher-layer semantic features. We conduct experiments on three large-scale point clouds datasets. The experimental results demonstrate that our AKNet outperforms state-of-the-art (SOTA) networks. [ABSTRACT FROM AUTHOR]

In recent years, single-image super-resolution (SISR) has acquired tremendous progress with the development of deep learning. However, the majority of SISR methods based on deep learning focus on building more complex networks, which inevitably lead to the problems of computational and memory costs. Thus, these methods may fail to be applied in real-world scenarios. To solve this problem, this paper proposes a lightweight convolution network combined with transformer for SISR named as MMSR. Specifically, an efficient convolutional neural network (CNN) based on multi-scale feature fusion is designed for local feature extraction, which is called MFF-CNN. In addition, we propose a simple and efficient multiple attention block (MAB) to further utilize the context information in features. MAB incorporates channel attention and transformer to help network obtain similar features at a long-term dependence, making full use of global information to further refine texture details. Finally, this paper provides comprehensive results for different settings of the entire network. Experimental results on common used datasets demonstrate that the proposed method can achieve better performances at the 2 × , 3 × and 4 × scales than other state-of-the-art lightweight methods. [ABSTRACT FROM AUTHOR]

Handwritten character recognition (HCR) is a challenging task because of high intra-class dissimilarity and high inter-class similarity among the character images. Therefore, the features that are used to represent the character images should be such that they maximize the inter-class variations and minimize intra-class variations. In this paper, a multilevel approach for recognition of Meitei Mayek handwritten characters using fusion of features strategy is presented. The proposed methodology has two levels of recognition. The first level employs a simple CNN. A method to identify the structurally similar character classes based on the probability values of the softmax function is also proposed. The fused feature set is employed in the second-level recognition to distinguish between these character classes, thereby enhancing the discriminative power of the fused feature set by reducing the search space in the second level. The fused feature set is a combination of traditional handcrafted feature descriptors and deep features learned by CNN. The fused feature set is fed to an SVM for second-level recognition. The experimental results show a superior performance of the proposed methodology in identifying the structurally similar character classes. This achieves an overall recognition accuracy of 97.08%, which sets the benchmark on the concerned Meitei Mayek handwritten character dataset. The methodology also shows significant results on MNIST, DIDA and CArDIS datasets. [ABSTRACT FROM AUTHOR]

Face recognition in visual surveillance systems is important for various applications to identify individuals who are behaving defiantly at the time of an event or for investigation purposes. Despite the dramatic improvements in facial recognition technology in recent years, it is difficult to recognize faces from surveillance feeds due to the presence of multiple people of different scales and orientations. This paper solves the task of low-resolution face recognition by combining exemplary techniques for extracting distinct features. This research utilizes the attributes learned by customized and learnable filters and injected in the training process to better match them with human brain functionality. The Gabor transform aims to convolve a facial image using a range of Gabor filter coefficients at various scales and orientations, resulting in scale and rotation invariant features. The tailored architecture with residual stream aims to enhance functional representation and prevent the gradient of the prediction engine from affecting the backbone network functional map. Experimental analysis is performed on the SCface and TinyFace databases and is reported with an accuracy of 89.21% on the SCface database and 56.68% on the TinyFace database. [ABSTRACT FROM AUTHOR]

With the worldwide spread of the COVID-19 pandemic, the demand for medical syringes has increased dramatically. Scale defect, one of the most common defects on syringes, has become a major barrier to boosting syringe production. Existing methods for scale defect detection suffer from large volumes of data requirements and the inability to handle diverse and uncertain defects. In this paper, we propose a robust scale defects detection method with only negative samples and favorable detection performance to solve this problem. Different from conventional methods that work in a batch-mode defects detection manner, we propose to locate the defects on syringes with a two-stage framework, which consists of two components, that is, the scale extraction network and the scale defect discriminator. Concretely, the SeNet is first built to utilize the convolutional neural network to extract the main structure of the scale. After that, the scale defect discriminator is designed to detect and label the scale defects. To evaluate the performance of our method, we conduct experiments on one real-world syringe dataset. The competitive results, that is, 99.7% on F1, prove the effectiveness of our method. [ABSTRACT FROM AUTHOR]

Automatic and accurate polyp segmentation is significant for diagnosis and treatment of colorectal cancer. This is a challenging task due to the polyp's shape and size diversity. Recently, various deep convolutional neural networks have been developed for polyp segmentation. However, most state-of-the-art methods have suffered from a poor performance in the segmentation of smaller, flat, or noisy polyp objects. In the paper, we propose a novel Deep Context Attention Network (DCANet) for accurate polyp segmentation based on an encoder–decoder framework. ResNet34 is adopted as the encoder, and five functional modules are introduced to improve the performance. Specifically, the improved local context attention module (LCAM) and global context module (GCM) are exploited to efficiently extract the local multi-scale and global multi-receptive-field context information, respectively. Then, an enhanced feature fusion module (FFM) is devised to effectively select and aggregate context features through spatial-channel attention. Finally, equipped with elaborately designed multi-attention modules (MAM), new decoder and supervision blocks are developed to accurately predict polyp regions via powerful channel-spatial-channel attention. Extensive experiments are conducted on the Kvasir-SEG and EndoScene benchmark datasets. The results demonstrate that the proposed network achieves superior performance compared to other state-of-the-art models. The source code will be available at https://github.com/ZAKAUDD/DCANet. [ABSTRACT FROM AUTHOR]

Automatic facial expression recognition plays a crucial role in computer vision, and pattern recognition. However, most existing deep learning-based facial expression classifiers usually obtain high average accuracy but have poor recognition accuracy for difficult expressions, like fear and disgust. In this paper, we propose a novel end-to-end architecture termed two-stream inter-class variation enhancement network, which learns the high-level semantic features and subtle inter-class variations in a joint fashion. More precisely, the global feature extraction network is used to extract spatial-channel semantic features, and the variations between different expressions are modeled by a distinction-reinforced network. The outputs of these two streams are weighted integrated in the expression classification network. In addition, a class balanced-weighted cross-entropy loss is designed to further improve feature discrimination. Experiment results indicate that the proposed network can significantly improve the recognition of difficult expressions and achieve a satisfactory average recognition accuracy of 73.67% on FER2013, 86.17% on RAFDB, 98.19% on CK+, and 98.85% on Oulu-CASIA, which outperforms the other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

As an advanced image editing technology, image inpainting leaves very weak traces in the tampered image, causing serious security issues, particularly those based on deep learning. In this paper, we propose the global–local feature fusion network (GLFFNet) to locate the image regions tampered by inpainting based on deep learning. GLFFNet consists of a two-stream encoder and a decoder. In the two-stream encoder, a spatial self-attention stream (SSAS) and a noise feature extraction stream (NFES) are designed. By a transformer network, the SSAS extracts global features regarding deep inpainting manipulations. The NFES is constructed by the residual blocks, which are used to learn manipulation features from noise maps produced by filtering the input image. Through a feature fusion layer, the features output by the encoder is fused and then fed into the decoder, where the up-sampling and convolutional operations are employed to derive the confidential map for inpainting manipulation. The proposed network is trained by the designed two-stage loss function. Experimental results show that GLFFNet achieves a high location accuracy for deep inpainting manipulations and effectively resists JPEG compression and additive noise attacks. [ABSTRACT FROM AUTHOR]

Head pose estimation methods without facial key points have emerged as a promising research field. However, there remain several unsolved challenges. For example, the current methods incur a computational cost, require large memory, and are difficult to deploy in practical applications. We propose a lightweight high-precision head pose estimation method based on a dual-stream convolutional neural network for overcoming these issues. The network comprises a dual-stream lightweight backbone network, external attention module, and soft stagewise regression (SSR) module. Dual-stream lightweight backbone network can extract original image features more effectively while keeping low computational overhead. External attention module can enhance the feature map extraction from the backbone network and improve the feature attention. SSR module calculates the probability of the head in each direction and predicts the head pose by regression. Extensive experiments on Annotated Facial Landmarks in the Wild (AFLW2000) and Biwi Kinect Head Pose Database (BIWI) datasets demonstrate that the model proposed in this paper has fewer parameters and lower estimation errors than the state-of-the-art methods in the field of head pose estimation in recent years. [ABSTRACT FROM AUTHOR]

Medical image segmentation has the priori guiding significance for clinical diagnosis and treatment. In the past ten years, a large number of experimental facts have proved the great success of deep convolutional neural networks in various medical image segmentation tasks. However, the convolutional networks seem to focus too much on the local image details, while ignoring the long-range dependence. The Transformer structure can encode long-range dependencies in image and learn high-dimensional image information through the self-attention mechanism. But this structure currently depends on the database scale to give full play to its excellent performance, which limits its application in medical images with limited database size. In this paper, the characteristics of CNNs and Transformer are integrated to propose a dual encoding–decoding structure of the X-shaped network (X-Net). It can serve as a good alternative to the traditional pure convolutional medical image segmentation network. In the encoding phase, the local and global features are simultaneously extracted by two types of encoders, convolutional downsampling, and Transformer and then merged through jump connection. In the decoding phase, a variational auto-encoder branch is added to reconstruct the input image itself in order to weaken the impact of insufficient data. Comparative experiments on three medical image datasets show that X-Net can realize the organic combination of Transformer and CNNs. [ABSTRACT FROM AUTHOR]

Traditional object recognition algorithms cannot meet the requirements of object recognition accuracy in the actual warehousing and logistics field. In recent years, the rapid development of the deep learning theory has provided a technical approach for solving the above problems, and a number of object recognition algorithms has been proposed based on deep learning, which have been promoted and applied. However, deep learning has the following problems in the application process of object recognition: First, the nonlinear modeling ability of the activation function in the deep learning model is poor; second, the deep learning model has a large number of repeated pooling operations during which information is lost. In view of these shortcomings, this paper proposes multiple-parameter exponential linear units with uniform and learnable parameter forms and introduces two learned parameters in the exponential linear unit (ELU), enabling it to represent piecewise linear and exponential nonlinear functions. Therefore, the ELU has good nonlinear modeling capabilities. At the same time, to improve the problem of losing information in the large number of repeated pooling operations, this paper proposes a new global convolutional neural network structure. This network structure makes full use of the local and global information of different layer feature maps in the network. It can reduce the problem of losing feature information in the large number of pooling operations. Based on the above ideas, this paper suggests an object recognition algorithm based on the optimized nonlinear activation function-global convolutional neural network. Experiments were carried out on the CIFAR100 dataset and the ImageNet dataset using the object recognition algorithm proposed in this paper. The results show that the object recognition method suggested in this paper not only has a better recognition accuracy than traditional machine learning and other deep learning models but also has a good stability and robustness. [ABSTRACT FROM AUTHOR]

Scene recognition is challenging due to the complicated spatial arrangement and varied object distribution inside the scene images. Deep learning methods, especially convolutional neural networks (CNNs), have boosted the performance of scene recognition significantly because of the multistage global feature learning ability. However, the CNNs ignore the local semantics and diverse relationships among local regions, which play an important role in scene recognition. In this paper, a region-based adaptive association learning framework is proposed to tackle these challenges. This framework contains two sub-networks to separately extract the features of semantic distribution and contextual arrangement, and deep fusion networks to fuse such features in a joint and boosting fashion. Firstly, we simplify the current CNN structure so that our feature extractor can maintain high-level spatial region representation ability. Then, we propose a novel regional correlation learning architecture to explore the arbitrary direction dependence of the region and the four directions' long-range strip context dependence of the region in different spatial dimensions. Moreover, we design different attention strategies which include context gating, recurrent attention, and gated attention to further refine the extracted regional relationships and highlight local semantic information. Finally, the collaborative relationship features are deeply fused and then fed into a classification layer so that the whole framework is optimized as a whole. Extensive experimental evaluations on both ocean scene and land scene datasets coming from some different fields show that the proposed method achieves better accurate prediction on scene recognition than other state-of-the-art models. [ABSTRACT FROM AUTHOR]

Early detection and the classification of cancer in diagnosed patients can improve the prognosis and improve patient outcomes. In the field of histopathology, the assessment of the disease state is based on the morphological characteristics and spatial distribution of the nuclei in the tissue images. Therefore, the purpose of this research is to propose an automatic histopathological images nuclei segmentation method to accurately predict the boundaries of overlapping and multi-size nuclei. Based on the traditional U-Net, we proposed a convolutional neural network (CNN) that includes iterative attention feature fusion (iAFF) and residual modules for overlapping and multi-size nuclei segmentation task, which is essential and challenging for the development of computer-aided diagnosis (CAD) systems. We extensively evaluate this method on the TNBC and TCGA datasets and the experimental results show that our method can obtain better segmentation performance than the most advanced deep learning models. The proposed method has three advantages in the task of nuclei segmentation. First of all, the iAFF module used in the skip connection fully combines the global channel and the local context and overcomes the semantic and scale inconsistency between the input features. Second, the residual module in the decoder further integrates context information. Third, the method proposed in this paper will not increase too much computational overhead on U-Net, but the effect is significantly improved. Therefore, compared with traditional CNN, multi-level feature fusion network (MFFNet) can reduce redundancy and effectively improve the performance of the model without greatly increasing the network parameters. [ABSTRACT FROM AUTHOR]

Point cloud is usually used to construct the surface shape of three-dimensional geometric objects. Due to the disorder and irregularity of the point cloud, it is still a challenge to fully acquire the semantic features of the point cloud. With the development of graph neural network and graph convolution neural network, researchers are integrating point cloud and graph structure to better represent the semantic features of the point cloud. In this paper, we propose a novel graph convolutional neural network that integrates distance and direction (DDGCN), which constructs a dynamic neighborhood graph by obtaining the similarity matrix of the point cloud, and then uses several multi-layer perceptrons to obtain the local features of the point cloud. For the sake of making the intra-classes in the point cloud data more compact and the spacing between classes larger than the intra-class spacing, we propose a new loss function combined with center loss. The proposed DDGCN has been tested on ModelNet40 dataset, ShapeNet Part dataset and S3DIS dataset, and has achieved state-of-the-art performance in both classification and segmentation tasks. [ABSTRACT FROM AUTHOR]

In this paper, we propose a multi-scale convolution adaptive fusion super-resolution reconstruction network. Firstly, the input is passed through three convolution kernels of different sizes, and then the results are added and fused. Then, after pooling and full connection, the output results of the convolution layer with different sizes are weighted and added by the Softmax weighting mechanism to get the fusion feature map. Since the weights of the different branches with different convolution kernel sizes can be adaptively changed with the input information, the SR reconstruction is effectively improved. The detailed comparative experiments on the public datasets show that the SR reconstruction effect of our model is better than that of some state-of-the-art networks in objective criteria PSNR, SSIM, and subjective visual effect. [ABSTRACT FROM AUTHOR]

Human activity recognition (HAR) has become a significant area of research in human behavior analysis, human–computer interaction, and pervasive computing. Recently, deep learning (DL)-based methods have been applied successfully to time-series data generated from smartphones and wearable sensors to predict various activities of humans. Even though DL-based approaches performed very well in activity recognition, they are still facing challenges in handling time series data. Several issues persist with time-series data, such as difficulties in feature extraction, heavily biased data, etc. Moreover, most of the HAR approaches rely on manual feature engineering. In this paper, to design a robust classification model for HAR using wearable sensor data, a hybrid of convolutional neural network (CNN) and bidirectional long short-term memory (BiLSTM) is used. The proposed multibranch CNN-BiLSTM network does automatic feature extraction from the raw sensor data with minimal data pre-processing. The use of CNN and BiLSTM makes the model capable of learning local features as well as long-term dependencies in sequential data. The different filter sizes used in the proposed model can capture various temporal local dependencies and thus helps to improve the feature extraction process. To evaluate the model performance, three benchmark datasets, i.e., WISDM, UCI-HAR, and PAMAP2, are utilized. The proposed model has achieved 96.05%, 96.37%, and 94.29% accuracies on WISDM, UCI-HAR, and PAMAP2 datasets, respectively. The obtained experimental results demonstrate that the proposed model outperforms the other compared approaches. [ABSTRACT FROM AUTHOR]

Handwriting recognition is an important area in pattern recognition. For many years, Tai Le has been widely used in Southwest China and Southeast Asia, which makes it of great interest for recognition research. The characteristics of the highly similar characters in Tai Le, such as its large proportion of similar characters and the randomness of its writing, bring great challenges to the task of recognition. In this paper, a method based on ensemble deep learning for offline handwritten Tai Le characters is proposed. First, the handwritten Tai Le character dataset SDH2019.2 was constructed and preprocessed. Then, an ensemble deep convolutional neural network (EDCNN) model was constructed by using a stacking strategy. Thirty deep neural network (DNN) and logistic regression algorithms were integrated into a strong Tai Le classifier by stacking. Experiments showed that the proposed model is competitive with the base DNN model and other ensemble models. The results indicate that the performance of Tai Le recognition by the stacking ensemble-based deep neural network model is high, with an accuracy of 98.85%. Additionally, its precision, recall and F1-score of 98.87%, 98.85% and 98.85%, respectively, are superior to those of other classic neural network models. To verify the general applicability of EDCNN, its effectiveness was also verified by recognizing MNIST handwritten digits and Devanagari handwritten characters. [ABSTRACT FROM AUTHOR]

In this paper, a lightweighted Intensive Feature Extrication Deep Network (ExtriDeNet) is proposed for precise hand gesture recognition (HGR). ExtriDeNet primarily consists of two blocks: Intensive Feature Fusion Block (IFFB) and Intensive Feature Assimilation Block (IFAB). IFFB incorporates two different scaled filters 3 × 3 and 5 × 5 to capture contextual features of hands, while IFAB is designed by embedding influential features of IFFB with two extreme minute and high-level feature responses from two receptive fields generated by employing 1 × 1 and 7 × 7 sized filters, respectively. The combination of multiscaled filters enriches the network with the most significant features and enhances the learnability of the network. Thus, the proposed ExtriDeNet efficiently defines the distinctive features of different hand gesture classes and achieves high performance as compared to state-of-the-art HGR approaches. The performance of the proposed network is evaluated on the standard datasets: MUGD, Finger Spelling, OUHands, NUS-I, NUS-II and HGR1 for both subject-dependent and subject-independent scheme. [ABSTRACT FROM AUTHOR]

Single image rain streak removal is a well-explored topic in the field of computer vision. The de-raining problem is modeled as an image decomposition task where a rainy image is decomposed into rain-free background image and rain streek map. Unlike most of the existing de-raining methods, this paper attempts to decompose the rainy image in the frequency domain. The idea is inspired by pseudo-periodic characteristics of the noise signal (here the rain streaks) which leave some traces in the frequency domain, and the same can be utilized to predict the noise signal. In this paper, a deep learning-based rain streak prediction model is proposed which learns in discrete Fourier transform Oppenheim and Schafer (Discrete-TimeSignal Processing, Prentice Hall, Upper Saddle River, 1989) domain. To the best of our knowledge, this is the first approach where compressed domain coefficients are directly used as input to a deep convolutional neural network. The proposed model has been tested on publicly available synthetic datasets Fu et al. (in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https://doi.org/10.1109/CVPR.2017.186, Yang et al. (in: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017. https://doi.org/10.1109/CVPR.2017.183), Yeh et al. (in: 2015 IEEE International Conference on Consumer Electronics-Taiwan, 2015. https://doi.org/10.1109/ICCE-TW.2015.7216999) and results are found to be comparable with the state of the art methods in the spatial domain. The presented analysis and study have an obvious indication to extend transform domain input to train the deep learning architecture especially image de-noising like problems. [ABSTRACT FROM AUTHOR]

Visual object tracking has become one of the most active research topics in computer vision, which has been growing in commercial development as well as academic research. Many visual trackers have been proposed in the last two decades. Recent studies of computer vision for dynamic scenes include motion detection, object classification, environment modeling, tracking of moving objects, understanding of object behaviors, object identification, and data fusion from multiple sensors. This paper provides an in-depth overview of recent object tracking research. Object tracking tasks in realistic scenario often face challenging problems such as camera motion, occlusion, illumination effect, clutter, and similar appearance. A variety of tracker techniques have been published, which combine multiple techniques to solve multiple visual tracking sub-problems. This paper also reviews the latest research trend in object tracking based on convolutional neural networks, which is receiving growing attention. Finally, the paper discusses the future challenges and research directions for the object tracking problems that still need extensive studies in coming years. [ABSTRACT FROM AUTHOR]

Glaucoma disease is affecting a large community worldwide. It gradually affects the optic nerve and may cause partial or complete vision loss. Glaucoma happens due to an increase in the fluid pressure inside the optic nerve, which is also known as intraocular pressure (IOP). Therefore, it is essential to detect it in the early stage to prevent blindness. Recently, deep neural networks have been applied to analyse medical imagery. This paper proposes a framework for glaucoma detection using the deep convolution neural network. In this framework, a preprocessing step uses the CLAHE to enhance the local contrast. Further, we have utilized two segmentation models (EfficientNet + U-Net) for segmenting the optic cup and disc mask from retinal fundus images. Moreover, the CDR ratio is computed from the segmented optic cup and disc masks. The framework detects whether the inputted image is glaucoma infected or not based on the CDR ratio. The accuracy of the proposed framework is compared to various baseline models. A qualitative and quantitative assessment has been done on various benchmark datasets (DRISHTI-GS1 and RIM-ONE). The experimental outcomes illustrate that the proposed framework outperformed the other state-of-the-art methods for glaucoma detection in the retinal fundus image. [ABSTRACT FROM AUTHOR]

Intelligent video surveillance systems have been used recently for automatic monitoring of human interactions. Although they play a significant role in reducing security concerns, there are many challenges for distinguishing between normal and abnormal behaviors such as crowded environments and camera viewpoint. In this paper, we propose a novel deep violence detection framework based on the specific features derived from handcrafted methods. These features are related to appearance, speed of movement, and representative image and fed to a convolutional neural network (CNN) as spatial, temporal, and spatiotemporal streams. The spatial stream trained the network with each frame in the video to learn environment patterns. The temporal stream contained three consecutive frames to learn motion patterns of violent behavior with a modified differential magnitude of optical flow. Moreover, in spatio-temporal stream, we introduced a discriminative feature with a novel differential motion energy image to represent violent actions more interpretable. This approach covers different aspects of violent behavior by fusing the results of these streams. The proposed CNN network is trained with violence-labeled and normal-labeled frames of 3 Hockey, Movie, and ViF datasets which comprised both crowded and uncrowded situations. The experimental results showed that the proposed deep violence detection approach outperformed state-of-the-art works in terms of accuracy and processing time. [ABSTRACT FROM AUTHOR]

Human skin detection, which is one of the important pre-processing phases, has a wide range of applications such as face tracking, skin diseases, video surveillance, web content filtering, and so on. Skin detection is a challenging problem because skin color can vary dramatically in its appearance due to many factors such as illumination conditions, pose variations, race, aging, and complex background. Several methods dealing with skin detection assume that skin pixels can be extracted from background colors according to some thresholding rules related to a specific color model. Nevertheless, it is a complex task to recognize skin pixels under the challenging factors aforementioned. In the recent era, the success of deep convolutional neural network (CNN) has strongly influenced the field of computer vision. However, we could find only a few researches that apply deep learning methods to deal with the skin detection problem. This paper presents a novel approach based on CNN for skin detection. Extensive experiments show that the proposed approach exceeds the best result for other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In view of the multi-scale changes and the influence of light and angle in the image matching process, it is quite difficult to realize intelligent image registration by using convolutional neural network. The existing image matching algorithm has the following problems in the application process: the existing shallow feature extraction model has lost a lot of effective feature information and low recognition accuracy. Meanwhile, the image registration method based on deep learning is not robust and accurate enough. Therefore, an image registration method based on additive edge cosine loss was proposed in this paper. In the twin network architecture, cosine loss was used to convert Euclidean space into angular space, which eliminated the influence of characteristic intensity and improved the accuracy of registration. The matching cost was directly calculated by the included angle of two vectors in the embedded space, where the size of the angle edge could be quantitatively adjusted through parameter m . We further derived a specific m to quantitatively adjust the loss. In addition, anti-rotation attention mechanism was added to the network to enhance the ability of feature information extraction and adjust the position information of feature vectors to reduce the mismatching caused by image rotation. [ABSTRACT FROM AUTHOR]

Visual tracking is the process of estimating the position of an object in a video sequence and plays a very important role in the field of autonomous video processing. Recent work renders that the trackers developed using deep learning techniques such as the convolutional neural network (CNN) exhibits outstanding performances in terms of accuracy and robustness as compared to other state-of-the-art trackers. Multi-domain convolutional neural network (MDNet) is a deep tracker which uses the CNN for estimating the target in each frame of the video sequence. The majority of the tracking challenges could be very easily handled by the MDNet tracker due to its offline training and online tracking features. The offline training stage helps in capturing the target representations into the shared layers of the CNN, while the online tracking uses a large number of probable random samples of bboxes (bounding boxes) around the previous target for estimating the target in the current frame. Once the target is estimated, a process of fine-tuning is performed which will update the weights of shared layers of CNN. The large number of random samples used for target estimation and the huge number of random training samples generated for the fine-tuning during the online stage makes tracking by MDNet computationally complex and slow. The major contribution of this paper is to suggest using guided samples to the input of the CNN rather than random samples. Moreover, it generates a lesser number of highly efficient training samples for the fine-tuning which helps in decreasing the computational complexity of the tracker by half without much compromise on the performance and thus improves the speed of the tracking process. An extensive evaluation has been performed on the proposed Guided MDNet with different datasets like ALOV300++, OTB and VOT, and its performances are measured in terms of metrics like F-score, one-pass evaluation, robustness and accuracy. [ABSTRACT FROM AUTHOR]

Automatic recognition of facial expressions in the wild is a challenging problem and has drawn a lot of attention from the computer vision and pattern recognition community. Since their emergence, the deep learning techniques have proved their efficacy in facial expression recognition (FER) tasks. However, these techniques are parameter intensive, and thus, could not be deployed on resource-constrained embedded platforms for real-world applications. To mitigate these limitations of the deep learning inspired FER systems, in this paper, we present an efficient dual integrated convolution neural network (DICNN) model for the recognition of facial expressions in the wild in real-time, running on an embedded platform. The designed DICNN model with just 1.08M parameters and 5.40 MB memory storage size achieves optimal performance by maintaining a proper balance between recognition accuracy and computational efficiency. We evaluated the DICNN model on four FER benchmark datasets (FER2013, FERPlus, RAF-DB, and CKPlus) using different performance evaluation metrics, namely the recognition accuracy, precision, recall, and F1-score. Finally, to provide a portable solution with high throughput inference, we optimized the designed DICNN model using TensorRT SDK and deployed it on an Nvidia Xavier embedded platform. Comparative analysis results with the other state-of-the-art methods revealed the effectiveness of the designed FER system, which achieved competitive accuracy with multi-fold improvement in the execution speed. [ABSTRACT FROM AUTHOR]

Face recognition is diversely used in modern biometric and security applications. Most of the current face recognition techniques show good results in a constrained environment. However, these techniques face many problems in real-world scenarios such as low-quality images, temporal variations and facial disguises creating variations in facial features. The reason for these deteriorating results is the employment of handcrafted features having weak generalization capabilities and neglecting the complexities associated with domain adaption in case of deep learning models. In this paper, we have studied the efficacy of deep learning methods incorporating simple noise-based data augmentation for disguise invariant face recognition (DIFR). The proposed method detects face in an image using Viola Jones face detector and classifies it using a pre-trained Convolutional Neural Network (CNN) fine-tuned for DIFR. During transfer learning, a pre-trained CNN learns generalized disguise-invariant features from facial images of several subjects to correctly identify them under varying facial disguises. We have compared four different pre-trained 2D CNNs, each with different number of learning parameters, based on their classification accuracy and execution time for selecting a suitable model for DIFR. Comprehensive experiments and comparative analysis have been conducted on six challenging facial disguise datasets. Resnet-18 gives the best trade-off between accuracy and efficiency, by achieving an average accuracy of 98.19% with an average execution time of 0.32 seconds. The promising results achieved in these experiments reflect the efficiency of the proposed method and outperforms the existing methods in all aspects. [ABSTRACT FROM AUTHOR]

The walking manner of a person, also known as gait, is a unique behavioral biometric trait. Existing methods for gait recognition predominantly utilize traditional machine learning. However, the performance of gait recognition can deteriorate under challenging conditions including environmental occlusion, bulky clothing, and different viewing angles. To provide an effective solution to gait recognition under these conditions, this paper proposes a novel deep learning architecture using Graph Convolutional Neural Network (GCNN) that incorporates residual connections for gait recognition from videos. The optimized feature map of the proposed GCNN architecture exhibits the invariant property to viewing angle and subject's clothing. The residual connection is used to capture both spatial and temporal features of a gait sequence. The kinematic dependency extracted from shallower network layer is propagated to deeper layer using residual connection-based GCNN architecture. The proposed method is validated on CASIA-B gait dataset and outperforms all recent state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Human action recognition is still a challenging topic in the computer vision field that has attracted a large number of researchers. It has a significant importance in varieties of applications such as intelligent video surveillance, sports analysis, and human–computer interaction. Recent works attempt to exploit the progress in deep learning architecture to learn spatial and temporal features from action video. However, it remains unclear how to combine spatial and temporal information with convolutional neural network. In this paper, we propose a novel human action recognition method by fusing spatial and temporal features learned from a simple unsupervised convolutional neural network called principal component analysis network (PCANet) in combination with bag-of-features (BoF) and vector of locally aggregated descriptors (VLAD) encoding schemes. Firstly, both spatial and temporal features are learned via PCANet using a subset of frames and temporal templates for each video, while their dimensionality is reduced using whitening transformation (WT). The temporal templates are calculated using short-time motion energy images (ST-MEI) based on frame differencing. Then, the encoding scheme is applied to represent the final dual spatiotemporal PCANet features by feature fusion. Finally, the support vector machine (SVM) classifier is exploited for action recognition. Extensive experiments have been performed on two popular datasets, namely KTH and UCF sports, to evaluate the performance of proposed method. Our experimental results using leave-one-out evaluation strategy demonstrate that the proposed method presents satisfactory and comparable results on both datasets. [ABSTRACT FROM AUTHOR]

Current saliency methods require to learn large-scale regional features using small convolutional kernels, which is not possible with a simple feed-forward network. Some methods solve this problem by using segmentation into superpixels, while others downscale the image through the network and rescale it back to its original size. The objective of this paper is to show that saliency convolutional neural networks (CNN) can be improved by using a Green's function convolution (GFC) to extrapolate edges features into salient regions. The GFC acts as a gradient integrator, allowing to produce saliency features by filling thin edges directly inside the CNN. Hence, we propose the gradient integration and sum (GIS) layer that combines the edges features with the saliency features. Using the HED and DSS architecture, we demonstrated that adding a GIS layer near the network's output allows to reduce the sensitivity to the parameter initialization, to reduce the overfitting and to improve the repeatability of the training. By simply adding a GIS layer to the state-of-the-art DSS model, there is an absolute increase of 1.6% for the F-measure on the DUT-OMRON dataset, with only 10 ms of additional computation time. The GIS layer further allows the network to perform significantly better in the case of highly noisy images or low-brightness images. In fact, we observed an F-measure improvement of 5.2% when noise was added to the dataset and 2.8% when the brightness was reduced. Since the GIS layer is model agnostic, it can be implemented into different fully convolutional networks. Further, we showed that it outperforms the denseCRF post-processing method and is 40 times faster. A major contribution of the current work is the first implementation of Green's function convolution inside a neural network, which allows the network, via very minor architectural changes and no additional parameters, to operate in the feature domain and in the gradient domain at the same time, thus improving the regional representation via edge filling. [ABSTRACT FROM AUTHOR]

In recent years, convolutional neural networks have proven to be a highly efficient approach for face recognition. In this paper, we develop such a framework to learn a robust face verification in an unconstrained environment using aggressive data augmentation. Our objective is to learn a deep face representation from large-scale data with massive noisy and occluded face. Besides, we add an adaptive fusion of softmax loss and center loss as supervision signals, which are helpful to improve the performance and to conduct the final classification. The experiment results show that the suggested system achieves comparable performances with other state-of-the-art methods on the Labeled Faces in the Wild and YouTube face verification tasks. [ABSTRACT FROM AUTHOR]

Pupil localization is one of the most critical and essential requirements for eye gaze estimation and eye movement tracking. Because pupil images contain monotonous and uncomplicated information, the dataset uses a single class of labels to describe the image content, and using convolutional neural networks can quickly and accurately identify the pupil position on the input image. On low-resolution images, traditional methods encounter issues of low accuracy and cumbersome design steps. A lightweight pupil localization algorithm is proposed in this paper, utilizing a convolutional neural network (CNN) with additional training samples. The experimental results demonstrate the algorithm’s significant effectiveness in identifying the pupil position within the training set, with the accuracy of pupil position in the test set reaching 97.78%. This provides an evidence of the algorithm’s feasibility for accurately localizing pupils in low-resolution images. [ABSTRACT FROM AUTHOR]

3-D point cloud registration algorithms have been commonly studied and effectively applied to object pose estimation. Due to limited field of view of a 3-D camera, only partial point cloud of the observed object can be grabbed at each frame. Thus, the registration problem between partial and full point clouds remains challenging due to missing data and arbitrary pose matching. This paper proposes a seemingly novel partial-to-full registration network (PFRNet) based on establishing point-wise correspondences with a full range of uncertainty. Specifically, an effective descriptor is developed to generate distance histograms capturing systematically geometric information for each point to release difficulty in training when considering a full range of uncertainty. Then, a compensation network is proposed to adjust the histogram descriptor extracted from the partial point cloud by learning differences caused by missing data. Next, these two descriptors are input to a shared local feature extractor to generate per-point learned features. Besides, in order to establish corresponding point pairs, a deep network is applied to estimate the outlier and annealing parameters. Finally, the proposed architecture adopts a differentiable singular value decomposition module to output the rigid transformation. Experimental results show that our PFRNet achieves high precision, outperforming baseline methods while maintaining fast estimation on both synthetic ModelNet40 and realistic S3DIS data sets. [ABSTRACT FROM AUTHOR]

With the advent of deep learning, multi-modal data have been of great interest. One of the multi-modal tasks which can be included in the computer vision domain is visual question answering (VQA). In VQA, a question and an image are entered into the model and the model tries to answer the question according to the image. To the best of our knowledge, the current techniques look at the image and only give one answer to the question asked. However, in some situations, there are several answers to the asked question. In this paper, we address this problem and define a new domain in the task of VQA as well as a new computationally efficient approach to cope with multiple-answer VQA. In this approach, we use a sliding window in an efficient manner to examine the answer to the question in different parts of the image. Due to the fact that so far no proper dataset is available for multiple-answer VQA, we provide a new dataset for evaluating our proposed model. The experiments express that our model uses 94% less operation than other models, making it very suitable for real-time applications. [ABSTRACT FROM AUTHOR]

Calligraphy occupies a distinguished position in Chinese traditional culture. For long-term preservation, many calligraphy works have been carved on stones or woods in the form of relief. In this paper, we present a novel solution that enables fast modeling of Chinese calligraphy relief from 2D handwriting image, which benefits from the advances of deep learning. We first construct a relief dataset composed of diverse types of calligraphy fonts and then design a convolutional neural network for height predictions. Through the trained network, one can quickly generate homogeneous type, inhomogeneous type or hybrid style of reliefs. The advantage over previous methods is that our method does not require parameter tuning and is fast in generating calligraphy reliefs from different resolution of inputs. A number of experiments and comparisons prove the effectiveness of our method. [ABSTRACT FROM AUTHOR]

A person's visual aesthetics is an emerging behavioral biometric. Visual aesthetics can be defined as a person's principles pertaining to their sense of beauty or fondness. Utilizing a person's preference to certain images as discriminatory features forms the basis of person identification from visual aesthetics. This paper proposes a novel three-stage framework based on the convolutional neural network, AestheticNet, for the extraction of high-level features and identification of individuals from visual aesthetics. The rank-1 accuracy of 97.73% and rank-5 accuracy of 99.85% are achieved on the publicly available benchmark dataset, which outperforms all state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Single image dehazing, which is the process of removing haze from a single input image, is an important task in computer vision. This task is extremely challenging because it is massively ill-posed. In this paper, we propose a novel end-to-end deep residual convolutional dehazing network (DRCDN) based on convolutional neural networks for single image dehazing, which consists of two subnetworks: one network is used for recovering a coarse clear image, and the other network is used to refine the result. The DRCDN firstly predicts the coarse clear image via a context aggregation subnetwork, which can capture global structure information. Subsequently, it adopts a novel hierarchical convolutional neural network to further refine the details of the clean image by integrating the local context information. The DRCDN is directly trained using complete images and the corresponding ground-truth haze-free images. Experimental results on synthetic datasets and natural hazy images demonstrate that the proposed method performs favorably against the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

We present a new approach for 3D human pose estimation from a single image. State-of-the-art methods for 3D pose estimation have focused on predicting a full-body pose of a single person and have not given enough attention to the challenges in application: incompleteness of body pose and existence of multiple persons in image. In this paper, we introduce depth maps to solve these problems. Our approach predicts the depths of human pose over all spatial grids, which supports 3D poses estimation for incomplete or full bodies of multiple persons. The proposed depth maps encode depths of limbs rather than joints. They are more informative and reversibly convertible to depths of joints. The unified network is trained end to end using mixed 2D and 3D annotated samples. The experiments reveal that our algorithm achieves the state of the art on Human3.6M, the largest publicly available 3D pose estimation benchmark. Moreover, qualitative results have been reported to demonstrate the effectiveness of our approach for 3D pose estimation for incomplete human bodies and multiple persons. [ABSTRACT FROM AUTHOR]

In this paper, we propose a cancelable multi-biometric face recognition method that uses multiple convolutional neural networks (CNNs) to extract deep features from different facial regions. We also propose a new CNN architecture that exploits batch normalization, depth concatenation and a residual learning framework. The proposed method adopts a region-based technique in which face, eyes, nose and mouth regions are detected from the original face images. Multiple CNNs are used to extract deep features from each region, and then, a fusion network combines these features. Moreover, to provide user's privacy and increase the system resistance against spoof attacks, a cancelable biometric technique using bio-convolving encryption is performed on the final facial descriptor. Our experiments on the FERET, LFW and PaSC datasets show excellent and competitive results compared to state-of-the-art methods in terms of recognition accuracy, specificity, precision, recall and fscore. [ABSTRACT FROM AUTHOR]

Interest in automatic crowd behaviour analysis has grown considerably in the last few years. Crowd behaviour analysis has become an integral part all over the world for ensuring peaceful event organizations and minimum casualties in the places of public and religious interests. Traditionally, the area of crowd analysis was computed using handcrafted features. However, the real-world images and videos consist of nonlinearity that must be used efficiently for gaining accuracies in the results. As in many other computer vision areas, deep learning-based methods have taken giant strides for obtaining state-of-the-art performance in crowd behaviour analysis. This paper presents a comprehensive survey of current convolution neural network (CNN)-based methods for crowd behaviour analysis. We have also surveyed popular software tools for CNN in the recent years. This survey presents detailed attributes of CNN with special emphasis on optimization methods that have been utilized in CNN-based methods. It also reviews fundamental and innovative methodologies, both conventional and latest methods of CNN, reported in the last few years. We introduce a taxonomy that summarizes important aspects of the CNN for approaching crowd behaviour analysis. Details of the proposed architectures, crowd analysis needs and their respective datasets are reviewed. In addition, we summarize and discuss the main works proposed so far with particular interest on CNNs on how they treat the temporal dimension of data, their highlighting features and opportunities and challenges for future research. To the best of our knowledge, this is a unique survey for crowd behaviour analysis using the CNN. We hope that this survey would become a reference in this ever-evolving field of research. [ABSTRACT FROM AUTHOR]

In this issue, we first thank the reviewers for their hard work in handling papers during the year 2020 for the Visual Computer. Regular papers The following 6 papers have been submitted as regular papers to the Visual Computer Naoki Kita et al., titled: "Computational design of polyomino puzzles" (also presented at CGI2020 conference). SI: Learning representation from visual data We are happy to present 5 papers for the special issue "Learning representation from visual data" that has been handled by 4 guest editors who are: Dr. Wenming Zheng, Southeast University, China (lead Editor and Associate Editor of the Visual Computer). [Extracted from the article]

Motivated by the recent success of deep networks in providing effective and abstract image representations, in this paper, a multi-layer architecture called the multi-layer local energy patterns (ML-LEP) is proposed for texture representation and classification. The proposed approach follows a multi-layer convolutional neural network paradigm and is built upon the single-layer local energy pattern (LEP) approach, a statistical histogram-based method for texture representation. An important aspect of the proposed multi-layer method compared to other deep convolutional architectures is bypassing the computationally expensive learning stage using fixed filters. As such, the proposed training-free network circumvents the need for large data for learning system parameters. An extensive investigation is carried out to determine the merits of different nonlinear operators in the proposed architecture. For this purpose, different nonlinearities including an energy-based nonlinearity, the absolute operator as well as the rectifier functions are extensively investigated and compared against each other. Extensive experiments conducted on three challenging databases of KTH-TIPS, KTH-TIPS2-a and the UIUC indicate that the extension of the LEP method to the multi-layer LEP is effective and leads to better performance. Moreover, the proposed ML-LEP approach is compared to several other well-known descriptors in the field, achieving the best reported performance on the KTH-TIPS and the KTH-TIPS2-a databases despite being training-free. [ABSTRACT FROM AUTHOR]