Search

Malaria microscopy, microscopic examination of stained blood slides to detect parasite Plasmodium, is considered to be a gold-standard for detecting life-threatening disease malaria. Detecting the plasmodium parasite requires a skilled examiner and may take up to 10 to 15 minutes to completely go through the whole slide. Due to a lack of skilled medical professionals in the underdeveloped or resource deficient regions, many cases go misdiagnosed; resulting in unavoidable complications and/or undue medication. We propose to complement the medical professionals by creating a deep learning-based method to automatically detect (localize) the plasmodium parasites in the photograph of stained film. To handle the unbalanced nature of the dataset, we adopt a two-stage approach. Where the first stage is trained to detect blood cells and classify them into just healthy or infected. The second stage is trained to classify each detected cell further into the life-cycle stage. To facilitate the research in machine learning-based malaria microscopy, we introduce a new large scale microscopic image malaria dataset. Thirty-eight thousand cells are tagged from the 345 microscopic images of different Giemsa-stained slides of blood samples. Extensive experimentation is performed using different CNN backbones including VGG, DenseNet, and ResNet on this dataset. Our experiments and analysis reveal that the two-stage approach works better than the one-stage approach for malaria detection. To ensure the usability of our approach, we have also developed a mobile app that will be used by local hospitals for investigation and educational purposes. The dataset, its annotations, and implementation codes will be released upon publication of the paper.

In recent years, deep learning has been widely used in breast cancer diagnosis, and many high-performance models have emerged. However, most of the existing deep learning models are mainly based on static breast ultrasound (US) images. In actual diagnostic process, contrast-enhanced ultrasound (CEUS) is a commonly used technique by radiologists. Compared with static breast US images, CEUS videos can provide more detailed blood supply information of tumors, and therefore can help radiologists make a more accurate diagnosis. In this paper, we propose a novel diagnosis model based on CEUS videos. The backbone of the model is a 3D convolutional neural network. More specifically, we notice that radiologists generally follow two specific patterns when browsing CEUS videos. One pattern is that they focus on specific time slots, and the other is that they pay attention to the differences between the CEUS frames and the corresponding US images. To incorporate these two patterns into our deep learning model, we design a domain-knowledge-guided temporal attention module and a channel attention module. We validate our model on our Breast-CEUS dataset composed of 221 cases. The result shows that our model can achieve a sensitivity of 97.2% and an accuracy of 86.3%. In particular, the incorporation of domain knowledge leads to a 3.5% improvement in sensitivity and a 6.0% improvement in specificity. Finally, we also prove the validity of two domain knowledge modules in the 3D convolutional neural network (C3D) and the 3D ResNet (R3D).

The classification of benign and malignant parotid tumors is very crucial for the selection of surgical methods and their prognoses. The wide application of deep learning technology in the field of medical imaging also provides new ideas for the computer-aided diagnosis of parotid gland tumors. In addition, because the pathological types of parotid gland tumors are very complicated and the computed tomography (CT) images of benign and malignant patients are also very similar, some clinicians may misjudge tumors due to a lack of experience, which affects the effect of surgical treatment and prognosis. Therefore, this research proposes using deep learning methods to solve this problem. This study uses the four classic pretraining models of VGG16, InceptionV3, ResNet and DenseNet to classify parotid CT images using transfer learning methods and uses an improved convolutional neural network (CNN) model to classify parotid CT images. The experimental results show that the improved CNN model achieves an accuracy of 97.78%, and its classification performance is better than those of the other four transfer learning methods. It can effectively diagnose benign and malignant parotid tumors and improve the diagnostic accuracy.

Rear-end collision prediction has gained an increasing attention for safety improvement in smart cities. It is urgent to design efficient warning strategies for rear-end collisions which is one of the main causes of traffic accidents. The existing researches have been conducted to predict the collisions. Learning-based methods are proposed to solve this complicated issue, which the traditional methods are difficult to solve. However, due to some limitations in terms of the feature extraction and prediction performance, back-propagation learning methods are facing challenge. In this paper, we proposed a novel Rear-end Collision Prediction Mechanism with deep learning method (RCPM), in which a convolutional neural network model is established. In RCPM, the dataset is smoothed and expanded based on genetic theory to alleviate the class imbalance problem. The preprocessed dataset is divided into training and testing sets as the input to train our convolutional neural network model. The experimental results show that compared with the Honda, Berkeley and multi-layer perception neural-network-based algorithms, RCPM effectively improves performance to predict rear-end collisions.

Grain number is crucial for analysis of yield components and assessment of effects of cultivation measures. The grain number per spike and thousand-grain weight can be measured by counting grains manually, but it is time-consuming, tedious and error-prone. Previous image processing algorithms cannot work well with different backgrounds and different sizes. This study used deep learning methods to resolve the limitations of traditional image processing algorithms. Wheat grain image datasets were collected in the scenarios of three varieties, six background and two image acquisition devices with different heights, angles and grain numbers, 1 748 images in total. All images were processed through color space conversion, image flipping and rotation. The grain was manually annotated, and the datasets were divided into training set, validation set and test set. We used the TensorFlow framework to construct the Faster Region-based Convolutional Neural Network Model. Using the transfer learning method, we optimized the wheat grain detection and enumeration model. The total loss of the model was less than 0.5 and the mean average precision was 0.91. Compared with previous grain counting algorithms, the grain counting error rate of this model was less than 3% and the running time was less than 2 s. The model can be effectively applied under a variety of backgrounds, image sizes, grain sizes, shooting angles, and shooting heights, as well as different levels of grain crowding. It constitutes an effective detection and enumeration tool for wheat grain. This study provides a reference for further grain testing and enumeration applications.

Monitoring pig drinking has been a topic of interest to pig researchers and producers for many years. However, challenges still remain due to the fact that pigs like to play with drinkers in nursery environments and that the drinking pig is often touching others. These factors negatively influence the performance of camera-based pig drinking detection algorithms. The aim of this study is to investigate a deep learning method based on convolutional neural network (CNN) and long short-term memory (LSTM) to classify drinking and drinker-playing. In the experiment, two pens of pigs were video recorded for 3 days. In video from the first pen, 5400 2 s drinking episodes and 5400 2 s drinker-playing episodes were generated with 80% of these data being allocated as training set and the remaining 20% as validation set. In video from the second pen, 12,000 2 s drinking and drinker-playing episodes were generated as a test set. Firstly, the CNN architecture ResNet50 was used to extract spatial features. These features were input into LSTM framework to extract spatial–temporal features. Through the fully connected layer, the prediction function Softmax was finally used to classify these drinking and drinker-playing episodes. In the test set, the classification accuracy in the body and head regions of interest was 87.2% and 92.5%, respectively. The results indicate that the proposed method can be used to classify pigs’ drinking and drinker-playing. These classification results have potential to improve the accuracy of pig drinking detection and help farmers to estimate pig welfare.

In recent years, automatic sleep staging methods have achieved competitive performance using electroencephalography (EEG) signals. However, the acquisition of EEG signals is cumbersome and inconvenient. Therefore, we propose a novel sleep staging approach using electrooculogram (EOG) signals, which are more convenient to acquire than the EEG. A two-scale convolutional neural network first extracts epoch-wise temporary-equivalent features from raw EOG signals. A recurrent neural network then captures the long-term sequential information. The proposed method was validated on 101 full-night sleep data from two open-access databases, the montreal archive of sleep studies and Sleep-EDF, achieving an overall accuracy of 81.2 and 76.3%, respectively. The results are comparable to those models trained with EEG signals. In addition, comparisons with six state-of-the-art methods further demonstrate the effectiveness of the proposed approach. Overall, this study provides a new avenue for sleep monitoring.

Mastering the molecular mechanism of breast cancer (BC) can provide an in-depth understanding of BC pathology. This study explored existing technologies for diagnosing BC, such as mammography, ultrasound, magnetic resonance imaging (MRI), computed tomography (CT), and positron emission tomography (PET) and summarized the disadvantages of the existing cancer diagnosis. The purpose of this article is to use gene expression profiles of The Cancer Genome Atlas (TCGA) and Gene Expression Omnibus (GEO) to classify BC samples and normal samples. The method proposed in this article triumphs over some of the shortcomings of traditional diagnostic methods and can conduct BC diagnosis more rapidly with high sensitivity and have no radiation. This study first selected the genes most relevant to cancer through weighted gene co-expression network analysis (WGCNA) and differential expression analysis (DEA). Then it used the protein–protein interaction (PPI) network to screen 23 hub genes. Finally, it used the support vector machine (SVM), decision tree (DT), Bayesian network (BN), artificial neural network (ANN), convolutional neural network CNN-LeNet and CNN-AlexNet to process the expression levels of 23 hub genes. For gene expression profiles, the ANN model has the best performance in the classification of cancer samples. The ten-time average accuracy is 97.36% (±0.34%), the F1 value is 0.8535 (±0.0260), the sensitivity is 98.32% (±0.32%), the specificity is 89.59% (±3.53%) and the AUC is 0.99. In summary, this method effectively classifies cancer samples and normal samples and provides reasonable new ideas for the early diagnosis of cancer in the future.

Protein fold recognition is critical for studying the structures and functions of proteins. The existing protein fold recognition approaches failed to efficiently calculate the pairwise sequence similarity scores of the proteins in the same fold sharing low sequence similarities. Furthermore, the existing feature vectorization strategies are not able to measure the global relationships among proteins from different protein folds. In this article, we proposed a new computational predictor called DeepSVM-fold for protein fold recognition by introducing a new feature vector based on the pairwise sequence similarity scores calculated from the fold-specific features extracted by deep learning networks. The feature vectors are then fed into a support vector machine to construct the predictor. Experimental results on the benchmark dataset (LE) show that DeepSVM-fold obviously outperforms all the other competing methods.

In stochastic gradient descent (SGD) and its variants, the optimized gradient estimators may be as expensive to compute as the true gradient in many scenarios. This paper introduces a calibrated stochastic gradient descent (CSGD) algorithm for deep neural network optimization. A theorem is developed to prove that an unbiased estimator for the network variables can be obtained in a probabilistic way based on the Lipschitz hypothesis. Our work is significantly distinct from existing gradient optimization methods, by providing a theoretical framework for unbiased variable estimation in the deep learning paradigm to optimize the model parameter calculation. In particular, we develop a generic gradient calibration layer which can be easily used to build convolutional neural networks (CNNs). Experimental results demonstrate that CNNs with our CSGD optimization scheme can improve the stateof-the-art performance for natural image classification, digit recognition, ImageNet object classification, and object detection tasks. This work opens new research directions for developing more efficient SGD updates and analyzing the backpropagation algorithm.

This paper proposes a recognition method that combines deep learning with traditional hidden Markov model (HMM) with the aim of improving the recognition accuracy of interaction. First, to construct the classification model, the optimized ALexNet convolutional neural network is used to extract the behavior features, followed by the extraction of features that are used to train the long short-term memory (LSTM) network using the Softmax method. Finally, the particle swarm optimization algorithm is used to fuse the classification results with the traditional HMM classification results so that a hybrid classification model is established to obtain the final behavior recognition result. By conducting experiments on the UT-interaction dataset (six types of interaction behavior), the experimental results show that the hybrid model has higher recognition accuracy than other classical methods.

Cumin (Cuminum cyminum) and fennel (Foeniculum vulgare) are widely used seasonings and play a very important role in industries such as breeding, cosmetics, winemaking, drug discovery, and nano-synthetic materials. However, studies have shown that cumin and fennel from different regions not only differ greatly in the content of lipids, phenols and proteins but also the substances contained in their essential oils are also different. Therefore, realizing precise identification of cumin and fennel from different regions will greatly help in quality control, market fraud and production industrialization. In this experiment, cumin and fennel samples were collected from each region, a total of 480 NIR spectra were collected. We used deep learning and traditional machine learning algorithms combined with near infrared (NIR) spectroscopy to identify their origin. To obtain the model with the best generalization performance and classification accuracy, we used principal component analysis (PCA) to reduce spectral data dimensionality after Rubberband baseline correction, and then established classification models including quadratic discriminant analysis based on PCA (PCA-QDA) and multilayer perceptron based on PCA (PCA-MLP). We also directly input the spectral data after baseline correction into convolutional neural networks (CNN) and generative adversarial networks (GAN). The experimental results show that GAN is more accurate than the PCA-QDA, PCA-MLP and CNN models, and the classification accuracy reached 100%. In the cumin and fennel classification experiment in the same region, the four models achieve great classification results from three regions under the condition that all model parameters remain unchanged. The experimental results show that when the training data are limited and the dimension is high, the model obtained by GAN using competitive learning has more generalization ability and higher classification accuracy. It also provides a new method for solving the problem of limited training data in food research and medical diagnosis in the future.

Deep learning has emerged as a revolutionary technology for protein residue‐residue contact prediction since the 2012 CASP10 competition. Considerable advancements in the predictive power of the deep learning‐based contact predictions have been achieved since then. However, little effort has been put into interpreting the black‐box deep learning methods. Algorithms that can interpret the relationship between predicted contact maps and the internal mechanism of the deep learning architectures are needed to explore the essential components of contact inference and improve their explainability. In this study, we present an attention‐based convolutional neural network for protein contact prediction, which consists of two attention mechanism‐based modules: sequence attention and regional attention. Our benchmark results on the CASP13 free‐modeling targets demonstrate that the two attention modules added on top of existing typical deep learning models exhibit a complementary effect that contributes to prediction improvements. More importantly, the inclusion of the attention mechanism provides interpretable patterns that contain useful insights into the key fold‐determining residues in proteins. We expect the attention‐based model can provide a reliable and practically interpretable technique that helps break the current bottlenecks in explaining deep neural networks for contact prediction. The source code of our method is available at https://github.com/jianlin-cheng/InterpretContactMap.

Deep learning based methods, such as Convolution Neural Network (CNN), have demonstrated their efficiency in hyperspectral image (HSI) classification. These methods can automatically learn spectral-spatial discriminative features within local patches. However, for each pixel in an HSI, it is not only related to its nearby pixels but also has connections to pixels far away from itself. Therefore, to incorporate the long-range contextual information, a deep fully convolutional network (FCN) with an efficient non-local module, named ENL-FCN, is proposed for HSI classification. In the proposed framework, a deep FCN considers an entire HSI as input and extracts spectral-spatial information in a local receptive field. The efficient non-local module is embedded in the network as a learning unit to capture the long-range contextual information. Different from the traditional non-local neural networks, the long-range contextual information is extracted in a specially designed criss-cross path for computation efficiency. Furthermore, by using a recurrent operation, each pixel's response is aggregated from all pixels of HSI. The benefits of our proposed ENL-FCN are threefold: 1) the long-range contextual information is incorporated effectively, 2) the efficient module can be freely embedded in a deep neural network in a plug-and-play fashion, and 3) it has much fewer learning parameters and requires less computational resources. The experiments conducted on three popular HSI datasets demonstrate that the proposed method achieves state-of-the-art classification performance with lower computational cost in comparison with several leading deep neural networks for HSI.

Autonomous detection of structural defect from images is a promising, but also challenging task to replace manual inspection. With the development of deep learning algorithms, several studies have adopted deep convolutional neural networks (CNN) or fully convolutional networks (FCN) to detect cracks in pixel-level. However, a fundamental property of cracks, that they are rotation invariant, has never been exploited. Although the rotation-invariant property can be implicitly learned by data augmentation, the network needs more parameters to learn features of different orientations and thus tend to overfit the training data. In this study, a rotation-invariant FCN called ARF-Crack is proposed that utilizes the rotation-invariant property of cracks explicitly. The architecture of a state-of-the-art FCN called DeepCrack for pixel-level crack detection is adopted and revised where active rotating filters (ARFs) are used to encode the rotation-invariant property into the network. The proposed ARF-Crack is evaluated on several benchmark datasets including concrete cracks, pavement cracks and corrosion images. The experimental results show that the proposed ARF-Crack requires less number of network parameters and achieves the highest average precision values for all the benchmark datasets compared to other approaches. The proposed ARF-Crack has the potential of detecting other rotation-invariant defects.

Deep learning has achieved great success in areas such as computer vision and natural language processing. In the past, some work used convolutional networks to process EEG signals and reached or exceeded traditional machine learning methods. We propose a novel network structure and call it QNet. It contains a newly designed attention module: 3D-AM, which is used to learn the attention weights of EEG channels, time points, and feature maps. It provides a way to automatically learn the electrode and time selection. QNet uses a dual branch structure to fuse bilinear vectors for classification. It performs four, three, and two classes on the EEG Motor Movement/Imagery Dataset. The average cross-validation accuracy of 65.82%, 74.75%, and 82.88% was obtained, which are 7.24%, 4.93%, and 2.45% outperforms than the state-of-the-art, respectively. The article also visualizes the attention weights learned by QNet and shows its possible application for electrode channel selection.

Short-term traffic speed prediction is a promising research topic in intelligent transportation systems (ITSs), which also plays an important role in the real-time decision-making of traffic control and guidance systems. However, the urban traffic speed has strong temporal, spatial correlation and the characteristic of complex nonlinearity and randomness, which makes it challenging to accurately and efficiently forecast short-term traffic speeds. We investigate the relevant literature and found that although most methods can achieve good prediction performance with the complete sample data, when there is a certain missing rate in the database, it is difficult to maintain accuracy with these methods. Recent studies have shown that deep learning methods, especially long short-term memory (LSTM) models, have good results in short-term traffic flow prediction. Furthermore, the attention mechanism can properly assign weights to distinguish the importance of traffic time sequences, thereby further improving the computational efficiency of the prediction model. Therefore, we propose a framework for short-term traffic speed prediction, including data preprocessing module and short-term traffic prediction module. In the data preprocessing module, the missing traffic data are repaired to provide a complete dataset for subsequent prediction. In the prediction module, a combined deep learning method that is an attention-based LSTM (ATT-LSTM) model for predicting short-term traffic speed on urban roads is proposed. The proposed framework was applied to the urban road network in Nanshan District, Shenzhen, Guangdong Province, China, with a 30-day traffic speed dataset (floating car data) used as the experimental sample. Results show that the proposed method outperforms other deep learning algorithms (such as recurrent neural network (RNN) and convolutional neural network (CNN)) in terms of both calculating efficiency and prediction accuracy. The attention mechanism can significantly reduce the error of the LSTM model (up to 12.4%) and improves the prediction performance.

As providing objects that pigs prefer can reduce the occurrence of tail-biting and aggression and consequently improve animal welfare, automatic recognition of pigs’ engagement with different objects can have practical value. Therefore, aim of this study was to develop a computer vision based approach that utilised a recurrent neural network-based deep learning algorithm to recognise pig enrichment engagement (EE) behaviours and preliminarily determine the preference to objects. Two pig pens were studied. 1 day of video was recorded in pen 1, which generated 2400 1 s EE and 2400 1 s non-EE episodes. 80% of these data was randomly selected as training set and the remaining 20% as validation set. Moreover, 4 days of video were recorded and used as the test set in pen 2. Firstly, the HSV (Hue, Saturation, Value) colour space-based tracking algorithm was developed to locate object region of interest. Secondly, the convolutional neural network (CNN) architecture InceptionV3 was used to extract spatial features from each frame. These features were input into the long short-term memory (LSTM) framework to extract spatial-temporal features from each episode. Through the fully connected layer, the prediction function Softmax was finally used to classify these episodes as EE or non-EE behaviour. In the validation set, the proposed algorithm could recognise EE with blue ball, golden ball and wooden beam with an accuracy of 95.2%, 95.4% and 97.3%, respectively. By shortening the radius of the region of interest into a half of the average length of pig body, the corresponding accuracy could be improved into 96.9%, 97.1% and 97.9%, respectively. In the test set, the proposed algorithm could recognise EE with each of these 3 objects with an accuracy of 96.5%, 96.8% and 97.6%, respectively. The proportion of EE with each of these 3 objects was 75.8%, 6.0% and 18.2%, respectively. These results indicate that the proposed method can be used to recognise EE behaviours of pigs, and halving the radius of the region of interest can improve the recognition accuracy of EE behaviours. Moreover, the preference of pigs to objects based on EE duration were preliminarily determined as blue ball > wooden beam > golden ball. The obtained duration of EE behaviours can help farmers to evaluate the enrichment used and thereby to increase the health and welfare of the pigs in their care. Furthermore, the proposed algorithm has reference value for the classification of the behaviours with similar motion patterns.

Convolutional neural network (CNN) based segmentation methods provide an efficient and automated way for clinicians to assess the structure and function of the heart in cardiac MR images. While CNNs can generally perform the segmentation tasks with high accuracy when training and test images come from the same domain (e.g. same scanner or site), their performance often degrades dramatically on images from different scanners or clinical sites. We propose a simple yet effective way for improving the network generalization ability by carefully designing data normalization and augmentation strategies to accommodate common scenarios in multi-site, multi-scanner clinical imaging data sets. We demonstrate that a neural network trained on a single-site single-scanner dataset from the UK Biobank can be successfully applied to segmenting cardiac MR images across different sites and different scanners without substantial loss of accuracy. Specifically, the method was trained on a large set of 3,975 subjects from the UK Biobank. It was then directly tested on 600 different subjects from the UK Biobank for intra-domain testing and two other sets for cross-domain testing: the ACDC dataset (100 subjects, 1 site, 2 scanners) and the BSCMR-AS dataset (599 subjects, 6 sites, 9 scanners). The proposed method produces promising segmentation results on the UK Biobank test set which are comparable to previously reported values in the literature, while also performing well on cross-domain test sets, achieving a mean Dice metric of 0.90 for the left ventricle, 0.81 for the myocardium and 0.82 for the right ventricle on the ACDC dataset; and 0.89 for the left ventricle, 0.83 for the myocardium on the BSCMR-AS dataset. The proposed method offers a potential solution to improve CNN-based model generalizability for the cross-scanner and cross-site cardiac MR image segmentation task., 15 pages, 8 figures

Cumin and fennel are two types of food raw materials that have similar appearances but great differences in efficacy. Therefore, the efficient identification of these two types of food is very important to the related food processing industry and the development of food detection technology. In this study, a deep learning algorithm was designed to process Fourier NIR spectral data for the classification of cumin and fennel. The method was based on the difference in the signal intensity in NIR spectra between cumin and fennel, and the identification was performed by combining the improved multiscale fusion convolutional neural network (MCNN) and the bidirectional long-short-term memory network (BILSTM) based on NIR spectra. The classification accuracies of the MCNN and BILSTM models were 100% and 98.57%, respectively. The models have high sensitivity and fast analysis speed, which directly promote the development of real-time identification technology of cumin and fennel and verify the feasibility of deep learning algorithms in the field of food detection.

Recognizing actions according to video features is an important problem in a wide scope of applications. In this paper, we propose a temporal scale-invariant deep learning framework for action recognition, which is robust to the change of action speed. Specifically, a video is firstly split into several sub-action clips and a keyframe is selected from each sub-action clip. The spatial and motion features of the keyframe are extracted separately by two Convolutional Neural Networks (CNN) and combined in the convolutional fusion layer for learning the relationship between the features. Then, Long Short Term Memory (LSTM) networks are applied to the fused features to formulate long-term temporal clues. Finally, the action prediction scores of the LSTM network are combined by linear weighted summation. Extensive experiments are conducted on two popular and challenging benchmarks, namely, the UCF-101 and the HMDB51 Human Actions. On both benchmarks, our framework achieves superior results over the state-of-the-art methods by 93.7% on UCF-101 and 69.5% on HMDB51, respectively.

Gait, as an essential feature reflecting human health status, has attracted extensive attention in research. Automatic pathological gait identification can contribute to diseases diagnosis and intervention. In this paper, an unobtrusive sensing technology with deep learning methods to discriminate healthy and pathological gaits is proposed. Two accelerometers are mounted on the left and right lower limbs to acquire the motion signals. Based on these signals, three Neural Networks, namely, BPNN (Back Propagation Neural Network), LSTM (Long Short Term Memory) and CNN (Convolutional Neural Networks) are proposed for classifying the gaits. Experimental results exhibit that the accuracy of the proposed method can reach 86%, 81%, and 93% on a database of 15 participants while using BPNN, LSTM, CNN, respectively. With the strong ability of spatial-temporal signal analysis, CNN outperforms the other two neural networks and provides a favorable result. The proposed method can be extended to an automated gait classification tool, which can be used in the diagnosis and identification of pathological gaits.

While intrinsic data structure in subspace provides useful information for visual recognition, it has not yet been well studied in deep feature learning for action recognition. In this paper, we introduce a new spatio-temporal manifold network (STMN) that leverages data manifold structures to regularize deep action feature learning, aiming at simultaneously minimizing the intra-class variations of learned deep features and alleviating the over-fitting problem. To this end, the manifold prior is imposed from the top layer of a convolutional neural network (CNN) and propagated across convolutional layers during forward–backward propagation. The observed correspondence of manifold structures in the data space and feature space validates that the manifold priori can be transferred across the CNN layers. The STMN theoretically recasts the problem of transferring the data structure prior into the deep learning architectures as a projection over the manifold via an embedding method, which can be easily solved by an alternating direction method of multipliers and backward propagation (ADMM-BP) algorithm. The STMN is generic in the sense that it can be plugged into various backbone architectures to learn more discriminative representation for action recognition. The extensive experimental results show that our method achieves comparable or even better performance compared with the state-of-the-art approaches on four benchmark datasets.

Protein fold recognition is one of the most critical tasks to explore the structures and functions of the proteins based on their primary sequence information. The existing protein fold recognition approaches rely on features reflecting the characteristics of protein folds. However, the feature extraction methods are still the bottleneck of the performance improvement of these methods. In this paper, we proposed two new feature extraction methods called MotifCNN and MotifDCNN to extract more discriminative fold-specific features based on structural motif kernels to construct the motif-based convolutional neural networks (CNNs). The pairwise sequence similarity scores calculated based on fold-specific features are then fed into support vector machines to construct the predictor for fold recognition, and a predictor called MotifCNN-fold has been proposed. Experimental results on the benchmark dataset showed that MotifCNN-fold obviously outperformed all the other competing methods. In particular, the fold-specific features extracted by MotifCNN and MotifDCNN are more discriminative than the fold-specific features extracted by other deep learning techniques, indicating that incorporating the structural motifs into the CNN is able to capture the characteristics of protein folds.

A variety of environmental analysis applications have been advanced by the use of satellite remote sensing. Smoke detection based on satellite imagery is imperative for wildfire detection and monitoring. However, the commonly used smoke detection methods mainly focus on smoke discrimination from a few specific classes, which reduces their applicability in different regions of various classes. To this end, in this paper, we present a new large-scale satellite imagery smoke detection benchmark based on Moderate Resolution Imaging Spectroradiometer (MODIS) data, namely USTC_SmokeRS, consisting of 6225 satellite images from six classes (i.e., cloud, dust, haze, land, seaside, and smoke) and covering various areas/regions over the world. To build a baseline for smoke detection in satellite imagery, we evaluate several state-of-the-art deep learning-based image classification models. Moreover, we propose a new convolution neural network (CNN) model, SmokeNet, which incorporates spatial and channel-wise attention in CNN to enhance feature representation for scene classification. The experimental results of our method using different proportions (16%, 32%, 48%, and 64%) of training images reveal that our model outperforms other approaches with higher accuracy and Kappa coefficient. Specifically, the proposed SmokeNet model trained with 64% training images achieves the best accuracy of 92.75% and Kappa coefficient of 0.9130. The model trained with 16% training images can also improve the classification accuracy and Kappa coefficient by at least 4.99% and 0.06, respectively, over the state-of-the-art models.

Convolutional neural network (CNN) receives extensive attention in hyperspectral image classification. While hyper-spectral images contain abundant spectral information but lack spatial information, which usually contributes to poor classification results. In this paper, a novel classification framework called information fusion based CNN (IF-CNN) is proposed to compensate for the shortcomings of hyper-spectral images. The proposed method merges hyperspectral images with abundant spectral information and LiDAR images with rich spatial information as the input of classification framework. Furthermore, the framework consists of two convolutional neural networks: one-dimensional CNN for extracting spectral features, and two-dimensional CNN for extracting spatial correlation features. Experimental results demonstrate that the proposed method achieves excellent performance compared with some existing methods.

Newton’s rings are the fringe patterns of quadratic phase, the curvature radius of optical components can be obtained from the coefficients of quadratic phase. Usually, the coordinate transformation method has been used to the curvature radius, however, the first step of the algorithm is to find the center of the circular fringes. In recent years, deep learning, especially the deep convolutional neural networks (CNNs), has achieved remarkable successes in object detection task. In this work, an new approach based on the Faster region-based convolutional neural network (Faster R-CNN) is proposed to estimate the rings’ center. Once the rings’ center has been detected, the squared distance from each pixel to the rings’ center is calculated, the two-dimensional pattern is transformed into a one-dimensional signal by coordinate transformation, fast Fourier transform of the spectrum reveals the periodicity of the one-dimensional fringe profile, thus enabling the calculation of the unknown surface curvature radius. The effectiveness of this method is demonstrated by the simulation and actual images.

Due to the complexity of the biological factors that may influence the binding of transcription factors to DNA sequences, prediction of the potential binding sites remains a difficult task in computational biology. The attention mechanism in deep learning has shown its capability to learn from input features with long-range dependencies. Until now, no study has applied this mechanism in deep neural network models with input data from massively parallel sequencing. In this study, we aim to build a model for binding site prediction with the combination of attention mechanism and traditional deep learning techniques, including convolutional neural networks and recurrent neural networks. The performance of our methods is evaluated on the ENCODE-DREAM in vivo Transcription Factor Binding Site Prediction Challenge datasets.The benchmark shows that our implementation with attention mechanism (called DeepGRN) improves the performance of the deep learning models. Our model achieves better performance in at least 9 of 13 targets than any of the methods participated in the DREAM challenge. Visualization of the attention weights extracted from the trained models reveals how those weights shift when binding signal peaks move along the genomic sequence, which can interpret how the predictions are made. Case studies show that the attention mechanism helps to extract useful features by focusing on regions that are critical to successful prediction while ignoring the irrelevant signals from the input.

While significant work has been conducted to perform source camera model identification for images, little work has been done specifically for video camera model identification. This is problematic because different forensic traces may be left in digital images and videos captured by the same camera. As our experiments in this paper will show, a system trained to perform camera model identification for images yields unacceptably low performance when given video frames from the same cameras. To overcome this problem, new systems for identifying a videos source must be developed. In this paper, we propose a deep learning based system for determining the source camera model that captured a digital video. To do this, we use a convolutional neural network to produce camera model identification scores for small patches taken from video frames. These patches are chosen by a patch selection system that obtains patches from several appropriate frames temporally distributed throughout the video. Forensic information obtained by the CNN is provided to a fusion system, which combines it to produce a single, more accurate identification result. Through a series of experiments, we evaluate several system design choices and show that our system can achieve 95.9% video camera model identification accuracy.

With the advancement of water conservancy informatization based on Internet of Things (IoT), the hydrological data are increasingly enriched. As a result, more and more algorithms and methods relying on deep learning are introduced in the flood forecasting. Considering the ability of deep learning on complex features extraction, we proposed a flood process forecasting model based on Convolution Neural Network (CNN) with two-dimension (2D) convolutional operation. At first, we imported the rainfall spatial–temporal features by gridding the Xixian basin. After that, we processed the data from Digital Elevation Model (DEM) as the geographical feature and employed the historical streamflow process of Xixian basin as the trend feature. Next, extensive experiments were implemented to determine the optimal hyper-parameters of the proposed CNN flood process forecasting model. Numerical results show that our proposed model demonstrated a better accuracy for predicting the flood peak and arrival occasion, with a 24-hour and 36-hour lead time respectively.

Background To evaluate the Fourier transform infrared spectroscopy (FT-IR) combined with deep learning models to allow for quick diagnosis of abnormal thyroid function. Materials and methods Serum samples of 199 patients with abnormal thyroid function and 183 healthy patients were collected by infrared spectroscopy data and combined with different decibel noise for data expansion. The data were directly imported into three deep models: multilayer perceptron (MLP), a long-short-term memory network (LSTM), and a convolutional neural network (CNN), and 10-fold cross-validation was used to evaluate the performance of the model. Results The accuracy rates of the three models using the original data were 91.3 %, 88.6 % and 89.3 %, and the accuracy rates of the three models after data enhancement were 92.7 %, 93.6 % and 95.1 %. Conclusion The results of this study indicated that the use of large sample serum infrared spectroscopy data combined with deep learning algorithms is a promising method for the diagnosis of abnormal thyroid function.

Monitoring the feeding behaviour of pigs and measuring their feeding time can help farmers to evaluate the pig health and welfare. The aim of this study was to develop a video-based deep learning algorithm to recognise feeding behaviour of nursery pigs and determine the feeding time of pigs on an individual level. In the experiment, two pens of pigs were video recorded for 3 days. In the video of feeding region of pen 1, 96,000 1 s feeding episodes and 96,000 1 s non-feeding episodes were generated. 70% of these data was randomly selected as training set and the remaining 30% as validation set. In the video of feeding region of pen 2, all the labelled 24,656 1 s feeding episodes and 61,744 1 s non-feeding episodes were used as test set. Firstly, the Convolutional Neural Network (CNN) architecture Xception was used to extract spatial features. These features were input into Long Short-term Memory (LSTM) framework to extract spatial-temporal features. Through the fully connected layer, the prediction function Softmax was finally used to classify these 1 s episodes as feeding or non-feeding. An image processing algorithm based on maximum entropy segmentation, HSV (Hue, Saturation and Value) colour space transformation and template matching was proposed to calculate the circularity of the head, the ratio of the head to the feeding sub-region, the accumulated pixels of the head motion, and the distance from the head to the number on pig back in order to determine the identity and feeding time of each pig. In the test set, the proposed algorithm could recognise feeding behaviour with an accuracy of 98.4%, a sensitivity of 98.8%, specificity of 98.3% and precision of 95.9%, and could correctly recognise 98.5% of feeding time of individual pigs from the total 45944 s feeding time of 8 pigs. The results indicate that the proposed method can be used to recognise feeding behavior of pigs and determine feeding time of each pig.

In this study, serum samples from 199 patients with thyroid dysfunction and 183 people with normal thyroid function were collected by Raman spectroscopy, and the data were dimensions-reduced by PCA. The reduced data were input into a multi-feature fusion convolutional neural network (MCNN), the improved AlexNet, VGGNet, GoogLeNet and ResNet, Support Vector Machine (SVM) and Decision Tree (DT) for classification, and the results of the seven classification algorithms were compared. Their classification accuracy are 94.01 %, 91.91 %, 90.34 %, 93.46 %, 92.42 %, 82.78 % and 80.89 %, respectively. The results of this study indicate that the combination of serum Raman spectra and MCNN has a good diagnostic effect for identifying thyroid dysfunction, and it is feasible to improve the classic deep learning models for Raman spectrum classification.

Arrhythmia is an abnormal heartbeat rhythm, and its prevalence increases with age. An electrocardiogram (ECG) is a standard tool for detecting cardiac activity. However, because of the low amplitude, complexity, and non-linearity of the ECG signal, it is difficult to manually perform a rapid and accurate classification. Therefore, an automatic system that can identify different abnormal heartbeats from a large amount of ECG data should be developed for use in the healthcare field. This study proposed an approach based on deep learning that combined convolutional neural networks (CNNs) and long short-term memory networks (LSTM) to automatically identify six types of ECG signals: normal (N) sinus rhythm segments, atrial fibrillation (AFIB), ventricular bigeminy (B), pacing rhythm (P), atrial flutter (AFL), and sinus bradycardia (SBR). The proposed network applied a multi-input structure to process 10 s ECG signal segments and corresponding RR intervals from the MIT-BIH arrhythmia database. With a five-fold cross-validation strategy, this network achieved 99.32 % accuracy. Then, the diversity of the subjects was increased in the training data by supplementing database, improving the previous network model. The method was validated using two additional databases, which are independent of the training database of the network. For the new N and AFIB in additional databases, the proposed method achieved an average accuracy of 97.15 %. The results showed that the proposed model had robust generalization performance and could be used as an auxiliary tool to help clinicians diagnose arrhythmia after training with a larger database.

Aggression is considered as a major animal welfare problem in commercial pig farming. The aim of this study is to develop a deep learning method based on convolutional neural network (CNN) and long short-term memory (LSTM) to recognise aggressive episodes of pigs. Compared to previous studies of pig behaviours based on deep learning, this study directly process video episodes rather than individual frames. In the experiment, nursery pigs (8/pen) were mixed for 3 days and then 8 h of video was recorded in each day. From these videos, 600 aggressive 2 s-episodes were manually selected and then augmented into 2400 episodes by using horizontal, vertical and diagonal mirroring. From the videos, 2400 non-aggressive 2 s-episodes were also manually selected. 80% of the data were randomly allocated as training set and the remaining 20% as validation set. Firstly, the CNN architecture VGG-16 was used to extract spatial features. These features were then input into LSTM framework to further extract temporal features. Through fully connected layer, the prediction function Softmax was finally used to determine if the current episode is aggression or non-aggression. Using the proposed method, aggressive episodes could be recognised with an accuracy of 97.2%. This result indicates that this method can be used to recognise aggressive episodes of pigs. Additionally, this paper further investigates the validity of this method under the conditions of skipping frames and reducing the episode length. The results show that a frame skipping approach whereby 30 fps is reduced into 15 fps within each 2 s-episode can improve the accuracy into 98.4% and halve the total running time.

Segmentation of the left atrium (LA) is crucial for assessing its anatomy in both pre-operative atrial fibrillation (AF) ablation planning and post-operative follow-up studies. In this paper, we present a fully automated framework for left atrial segmentation in gadolinium-enhanced magnetic resonance images (GE-MRI) based on deep learning. We propose a fully convolutional neural network and explore the benefits of multi-task learning for performing both atrial segmentation and pre/post ablation classification. Our results show that, by sharing features between related tasks, the network can gain additional anatomical information and achieve more accurate atrial segmentation, leading to a mean Dice score of 0.901 on a test set of 20 3D MRI images. Code of our proposed algorithm is available at https://github.com/cherise215/atria_segmentation_2018/., STACOM 2018 Workshop, MICCAI 2018

In steerable filters, a filter of arbitrary orientation can be generated by a linear combination of a set of “basis filters.” Steerable properties dominate the design of the traditional filters, e.g., Gabor filters and endow features the capability of handling spatial transformations. However, such properties have not yet been well explored in the deep convolutional neural networks (DCNNs). In this paper, we develop a new deep model, namely, Gabor convolutional networks (GCNs or Gabor CNNs), with Gabor filters incorporated into DCNNs such that the robustness of learned features against the orientation and scale changes can be reinforced. By manipulating the basic element of DCNNs, i.e., the convolution operator, based on Gabor filters, GCNs can be easily implemented and are readily compatible with any popular deep learning architecture. We carry out extensive experiments to demonstrate the promising performance of our GCNs framework, and the results show its superiority in recognizing objects, especially when the scale and rotation changes take place frequently. Moreover, the proposed GCNs have much fewer network parameters to be learned and can effectively reduce the training complexity of the network, leading to a more compact deep learning model while still maintaining a high feature representation capacity. The source code can be found at https://github.com/bczhangbczhang .

Regularly inspecting the components inside nuclear power plants is necessary to ensure their safe performance. Current inspection practices, however, are time consuming, tedious, and subjective that need technicians to watch videos and manually annotate cracks. While an autonomous inspection approach is desirable, state-of-the-art crack detection algorithms cannot perform well since the cracks on nuclear power plant reactors are typically very tiny with low contrast. The existence of scratches, welds, and grind marks on the surface makes the autonomous detection even more challenging. This study introduces a new framework that consists of a convolutional neural network (CNN) based on deep learning, a spatiotemporal registration process, and a Nave Bayes data fusion scheme based on statistics. The proposed framework is evaluated using several inspection videos and achieves 98.3% hit rate against 0.1 false positives per frame where the hit rate is much higher than other state-of-the-art algorithms.

Steerable properties dominate the design of traditional filters, e.g., Gabor filters, and endow features the capability of dealing with spatial transformations. However, such excellent properties have not been well explored in the popular deep convolutional neural networks (DCNNs). In this paper, we propose a new deep model, termed Gabor Convolutional Networks (GCNs or Gabor CNNs), which incorporates Gabor filters into DCNNs to enhance the resistance of deep learned features to the orientation and scale changes. By only manipulating the basic element of DCNNs based on Gabor filters, i.e., the convolution operator, GCNs can be easily implemented and are compatible with any popular deep learning architecture. Experimental results demonstrate the super capability of our algorithm in recognizing objects, where the scale and rotation changes occur frequently. The proposed GCNs have much fewer learnable network parameters, and thus is easier to train with an endtoend pipeline. The source code will be here 1.

The automated building detection in aerial images is a fundamental problem encountered in aerial and satellite images analysis. Recently, thanks to the advances in feature descriptions, Region-based CNN model (R-CNN) for object detection is receiving an increasing attention. Despite the excellent performance in object detection, it is problematic to directly leverage the features of R-CNN model for building detection in single aerial image. As we know, the single aerial image is in vertical view and the buildings possess significant directional feature. However, in R-CNN model, direction of the building is ignored and the detection results are represented by horizontal rectangles. For this reason, the detection results with horizontal rectangle cannot describe the building precisely. To address this problem, in this paper, we proposed a novel model with a key feature related to orientation, namely, Oriented R-CNN (OR-CNN). Our contributions are mainly in the following two aspects: 1) Introducing a new oriented layer network for detecting the rotation angle of building on the basis of the successful VGG-net R-CNN model; 2) the oriented rectangle is proposed to leverage the powerful R-CNN for remote-sensing building detection. In experiments, we establish a complete and bran-new data set for training our oriented R-CNN model and comprehensively evaluate the proposed method on a publicly available building detection data set. We demonstrate State-of-the-art results compared with the previous baseline methods.

In recent years, deep learning is widely used for hyperspectral image (HSI) classification, among them, convolutional neural network (CNN) is most popular. In this paper, we propose a method for hyperspectral data classification by multi-channel convolutional neural network (MC-CNN). In this framework, one dimensional CNN (1D-CNN) is mainly used to extract the spectral feature of hyperspectral images, two dimension CNN (2D-CNN) is mainly used to extract the spatial feature of hyperspectral images, three-dimensional CNN (3D-CNN) is mainly used to extract part of the spatial and spectral information. And then these features are merged and pull into the full connection layer. At last, using neural network classifiers like logistic regression, we can eventually get class labels for each pixel. For comparison and validation, we compare the proposed MC-CNN algorithm with the other three deep learning algorithms. Experimental results show that our MC-CNN-based algorithm outperforms these state-of-the-art algorithms. Showcasing the MC-CNN framework has huge potential for accurate hyperspectral data classification.

Deep learning has been demonstrated to achieve excellent results for image classification and object detection. However, the impact of deep learning on video analysis (e.g. action detection and recognition) has been limited due to complexity of video data and lack of annotations. Previous convolutional neural networks (CNN) based video action detection approaches usually consist of two major steps: frame-level action proposal detection and association of proposals across frames. Also, these methods employ two-stream CNN framework to handle spatial and temporal feature separately. In this paper, we propose an end-to-end deep network called Tube Convolutional Neural Network (T-CNN) for action detection in videos. The proposed architecture is a unified network that is able to recognize and localize action based on 3D convolution features. A video is first divided into equal length clips and for each clip a set of tube proposals are generated next based on 3D Convolutional Network (ConvNet) features. Finally, the tube proposals of different clips are linked together employing network flow and spatio-temporal action detection is performed using these linked video proposals. Extensive experiments on several video datasets demonstrate the superior performance of T-CNN for classifying and localizing actions in both trimmed and untrimmed videos compared to state-of-the-arts.

Deep learning models, especially the convolutional neural networks (CNNs), are very active in hyperspectral remote sensing image classification. In order to better apply the CNN model to hyperspectral classification, we propose a CNN model based on Fletcher&ndash, Reeves algorithm (F&ndash, R CNN), which uses the Fletcher&ndash, Reeves (F&ndash, R) algorithm for gradient updating to optimize the convergence performance of the model in classification. In view of the fact that there are fewer optional training samples in practical applications, we further propose a method of increasing the number of samples by adding a certain degree of perturbed samples, which can also test the anti-interference ability of classification methods. Furthermore, we analyze the anti-interference and convergence performance of the proposed model in terms of different training sample data sets, different batch training sample numbers and iteration time. In this paper, we describe the experimental process in detail and comprehensively evaluate the proposed model based on the classification of CHRIS hyperspectral imagery covering coastal wetlands, and further evaluate it on a commonly used hyperspectral image benchmark dataset. The experimental results show that the accuracy of the two models after increasing training samples and adjusting the number of batch training samples is improved. When the number of batch training samples is continuously increased to 350, the classification accuracy of the proposed method can still be maintained above 80.7%, which is 2.9% higher than the traditional one. And its time consumption is less than that of the traditional one while ensuring classification accuracy. It can be concluded that the proposed method has anti-interference ability and outperforms the traditional CNN in terms of batch computing adaptability and convergence speed.