Search

Seismic fault detection is indispensable for exploring reservoirs of hydrocarbons, and a considerable amount of research has thus been devoted to it. With the rapid development of deep learning in recent years, researchers have begun using convolutional neural networks (CNNs) to identify seismic faults. However, it remains challenging for geophysical interpreters to quickly train an efficient model on a limited number of samples for fault interpretation. In this letter, we propose a workflow that uses a CNN-based method of semantic segmentation to interpret faults by using a small training set. It requires only the selection of some 2-D seismic sections from seismic volume data for interpretation and labeling to be trained to predict faults in the entire area considered. We simplified the VGG16 model to reduce training time and modified it to improve its performance. We used convolution layers instead of fully connected layers at the end of the network to implement the end-to-end classification of seismic images, and adopted dilation convolution to increase the receptive field and hybrid dilation convolution to avoid problems in it. We also applied the atrous spatial pyramid pooling (ASPP) module to further enhance the effects of segmentation. The results were subjected to postprocessing for refinement. The promising performance of the proposed method was verified on a set of real seismic data.

In recent years, deep convolutional neural networks (DCNNs) have made significant progress in cloud detection tasks, and the detection accuracy has been greatly improved. However, most existing CNN-based models have high computational complexity, which limits their practical application, especially for spaceborne optical remote sensing. In addition, most of the methods cannot make adaptive adjustments based on the structural information of the clouds, and blurred boundaries often occur in the detection results. In order to address these problems, this article proposes a lightweight network (DABNet) to achieve high-accuracy detection of complex clouds, not only a clearer boundary but also lower false-alarm rate. Specifically, a deformable context feature pyramid module is proposed to improve the adaptive modeling capability of multiscale features. Besides, a boundary-weighted loss function is designed to direct the network to focus on cloud boundary information and optimize the relevant detection results. The proposed method has been validated on two data sets: the public GF-1 WFV benchmark and our self-built GF-2 cloud detection data set with higher spatial resolution. The experimental results exhibit that DABNet achieves state-of-the-art performance while only using 4.12M parameters and 8.29G multiadds.

Recent progress on remote sensing (RS) scene classification is substantial, benefiting mostly from the explosive development of convolutional neural networks (CNNs). However, different from the natural images in which the objects occupy most of the space, objects in RS images are usually small and separated. Therefore, there is still a large room for improvement of the vanilla CNNs that extract global image-level features for RS scene classification, ignoring local object-level features. In this article, we propose a novel RS scene classification method via enhanced feature pyramid network (EFPN) with deep semantic embedding (DSE). Our proposed framework extracts multiscale multilevel features using an EFPN. Then, to leverage the complementary advantages of the multilevel and multiscale features, we design a DSE module to generate discriminative features. Third, a feature fusion module, called two-branch deep feature fusion (TDFF), is introduced to aggregate the features at different levels in an effective way. Our method produces state-of-the-art results on two widely used RS scene classification benchmarks, with better effectiveness and accuracy than the existing algorithms. Beyond that, we conduct an exhaustive analysis on the role of each module in the proposed architecture, and the experimental results further verify the merits of the proposed method.

The current target tracking and detection algorithms often have mistakes and omissions when the target is occluded or small. To overcome the defects, this paper integrates bi-directional feature pyramid network (BiFPN) into cascade region-based convolutional neural network (R-CNN) for live object tracking and detection. Specifically, the BiFPN structure was utilized to connect between scales and fuse weighted features more efficiently, thereby enhancing the network’s feature extraction ability, and improving the detection effect on occluded and small targets. The proposed method, i.e., Cascade R-CNN fused with BiFPN, was compared with target detection algorithms like Cascade R-CNN and single shot detection (SSD) on a video frame dataset of wild animals. Our method achieved a mean average precision (mAP) of 91%, higher than that of SSD and Cascade R-CNN. Besides, it only took 0.42s for our method to detect each image, i.e., the real-time detection was realized. Experimental results prove that the proposed live object tracking and detection model, i.e., Cascade R-CNN fused with BiFPN, can adapt well to the complex detection environment, and achieve an excellent detection effect.

An increasing number of change detection models are designed with different convolutional neural network (CNNs). However, the mechanism for designing network layers that can effectively extract robust features for different scenes remains unclear. Thus, novel networks with fluid pyramid integration network (FPIN) to detect changes are proposed in this study. Specifically, we first extract multi-scale deep features from feature extraction network. Higher layers extract semantic features robust to illumination and camera pose variations, while lower layers extract texture features that generate clear boundaries and changes with small scales. To employ the advantages of the features extracted from different layers, FPIN progressively fuses higher and lower layer features in a hierarchical order. We propose three different change detection networks based on FPIN and validate them on three publicly available change detection benchmark datasets. Experimental results showed that the proposed networks are better than state-of-the-art change detection methods.

Convolutional neural networks (CNNs) have facilitated impressive improvements in the semantic segmentation of very high-resolution (VHR) remote sensing images. The success of semantic segmentation depends on an effective receptive field (RF) large enough to cover the entire object. Popular methods to enlarge the effective RF include dilated filters, subsampling operations, and stacking layers. Unfortunately, the methods are inefficient or able to cause grid artifacts. Moreover, although the object sizes vary greatly in remote sensing images, the size of the RF cannot reach a compromise between small and large objects. To tackle these problems, we propose adaptive effective receptive convolution (AERFC) for VHR remote sensing images. AERFC adaptively controls the sampling location of convolution and automatically adjusts the effective RF without significantly increasing the parameter number and computational cost. Thus, AERFC reduces the training difficulty, decreases overfitting risk, and reserves details in VHR images. AERFC is also integrated with spatial pyramid pooling (SPP) to aggregate diverse multiscale features for exploring contextual information. Experimental results of the quantitative and qualitative evaluation over four benchmark data sets show that AERFC outperforms state-of-the-art methods.

Unmanned aerial vehicles, or drones, are poised to solve many problems associated with data collection in complex urban environments. Drones are easy to deploy, have a great ability to move and explore the environment, and are relatively cheaper than other data collection methods. This study investigated the use of Cascade Region-based convolutional neural network (R-CNN) networks to enable automatic vehicle counting and tracking in aerial video streams.The presented technique combines feature pyramid networks and a Cascade R-CNN architecture to enable accurate detection and classification of vehicles.The paper discusses the implementation and evaluation of the detection and tracking techniques and highlights their advantages when they are used to collect traffic data.

Convolutional Neural Networks (CNNs) have played an important role in saliency detection. How to detect a salient object as a whole is a key issue. However, most existing learning-based methods are not accurate enough to detect salient objects in complex scenes, such as easily overlooked small salient areas in a whole salient object, which is called scale imbalance problem in this paper. To address this issue, Scale Balance Network (SBN) based on fully convolutional network is proposed to accurately recognize and comprehensively detect salient objects. Firstly, to detect more small salient areas, a specially designed backbone instead of common backbone is adopted in this paper, which can capture larger resolution with more spatial features in deeper layers. Secondly, we present a novel progressive pyramid mechanism named Connective Feature Pyramid Module (CFPM), aiming to make the network focus on the balance between the large salient areas and the small ones. Finally, we present an Edge Enhancement Architecture with Various Kernels (EEAVK) to locate the saliency maps and refine the boundary features. Experimental results on five benchmark datasets show that the proposed SBN method achieves consistently superior performance in comparison with other state-of-the-art ones under different evaluation metrics.

High spatial resolution (HSR) imagery scene classification has become a hot research topic in remote sensing. Scene classification method based on the handcrafted features, such as the bag-of-visual-words (BoVW) model, describes an image by extracting local features of the scene and mapping them to the dictionary space, but usually uses a shallow structure and loses the spatial distribution characteristics of the scene. The method based on deep learning extracts hierarchical features to describe the scene, which can maintain the spatial position information well. However, deep features in different levels have scale recognition restrictions for multi-scale ground objects, and cannot understand complex scenes well. In this paper, the multi-level convolutional pyramid semantic fusion (MCPSF) framework is proposed for HSR imagery scene classification. Differing from previous scene classification methods, which integrate the feature of different levels directly, of which the fusion features have large differences in both sparsity and eigenvalue magnitude, MCPSF integrates multi-level semantic features extracted by BoVW model and convolutional neural network (CNN) model. In MCPSF, two convolution pyramid feature expression strategies are proposed to enhance the ability of capturing multi-scale land objects, i.e., local and convolutional pyramid based BoVW (LCPB) model and local and convolutional pyramid based pooling-stretched (LCPP) model. The effectiveness of the proposed method is verified on 21-class UC Merced (UCM) dataset and 30-class Aerial Image Dataset (AID). The framework was also transferred toa case study of scene annotation in Wuhan. The proposed framework significantly improves the performance when compared with other state-of-the-art methods.

Disease is one of the main factors affecting crop growth. How to reflect the external morphological features of the disease and completely retain the color and texture information of the disease area is one of the key research issues for crop disease segmentation. Meanwhile, aiming at the problem of low segmentation accuracy with traditional convolution neural network-based methods in the crop disease leaf image, this paper proposes a spatial pyramid-oriented encoder-decoder cascade convolution neural network for crop disease leaf segmentation. The network consists of a region disease detection network and a region disease segmentation network. Region disease detection network is a kind of network combining cascade convolution neural network with spatial pyramid. This method connects the three-level convolution neural network model, where the structure of the three-level neural network model varies from simple to complex. Different crop disease leaf features are extracted from the different neural network levels. And images are screened to complete the detection of crop disease leaf. What’s more, a space pyramid pooling layer is added to each network level. This pooling strategy does not require fixed size input, which increases the size selection of input model. The region segmentation network is established based on the Encoder-Decoder structure. The multi-scale convolution kernel is used to improve the local receptive field of the original convolution kernel and accurately segment the crop disease leaf area. Finally, we conduct experiments on the crop disease leaf images under different conditions, the results show that the proposed method has higher segmentation accuracy. In terms of Precision, Correct segmentation, over-segmentation and under-segmentation indexes, etc., the average values of proposed method are more than 90%. The average dice similarity coefficient is over 95% under different background. Moreover, it can meticulously reflect the external morphological features of the crop disease leaf and relatively better retain the color and texture information.

The convolutional neural network (CNN)-based multi-focus image fusion methods which learn the focus map from the source images have greatly enhanced fusion performance compared with the traditional methods. However, these methods have not yet reached a satisfactory fusion result, since the convolution operation pays too much attention on the local region and generating the focus map as a local classification (classify each pixel into focus or de-focus classes) problem. In this article, a global-feature encoding U-Net (GEU-Net) is proposed for multi-focus image fusion. In the proposed GEU-Net, the U-Net network is employed for treating the generation of focus map as a global two-class segmentation task, which segments the focused and defocused regions from a global view. For improving the global feature encoding capabilities of U-Net, the global feature pyramid extraction module (GFPE) and global attention connection upsample module (GACU) are introduced to effectively extract and utilize the global semantic and edge information. The perceptual loss is added to the loss function, and a large-scale dataset is constructed for boosting the performance of GEU-Net. Experimental results show that the proposed GEU-Net can achieve superior fusion performance than some state-of-the-art methods in both human visual quality, objective assessment and network complexity.

Automatic object detection is a fundamental but challenging problem in very high-resolution (VHR) satellite images’ interpretation. Recently, context-based feature pyramid detection architecture has been proposed, which generates high-quality fusion feature levels for multiscale object detection and significantly improves the accuracy of traditional detection frameworks. However, in feature pyramid architecture, small objects are easily lost in deep feature levels, and the context cues will be weakened simultaneously. Moreover, each feature level in the pyramids is simply constructed from single-level layers of the backbone, which are not representative enough for multiscale object detection. In this letter, an end-to-end multiscale convolutional neural network (MSCNN) is proposed, which is based on a unified multiscale backbone named EssNet for extracting features of diverse-scale objects in HRS images. The proposed EssNet backbone maintains the resolution of deep feature levels, which improves the capability of multiscale objects’ feature expression. Meanwhile, a dilated bottleneck structure is introduced into the backbone, which generates high-quality semantic features and improves the prediction capability of multiscale objects. Finally, the whole network can be optimized end-to-end by minimizing a multitask loss. Experiments on publicly available NWPU VHR-10 benchmark and demonstrate that the proposed method outperforms several state-of-the-art detection approaches.

The performance of state-of-the-art object skeleton detection (OSD) methods have been greatly boosted by Convolutional Neural Networks (CNNs). However, the most existing CNN-based OSD methods rely on a ‘skip-layer’ structure where low-level and high-level features are combined to gather multi-level contextual information. Unfortunately, as shallow features tend to be noisy and lack semantic knowledge, they will cause errors and inaccuracy. Therefore, in order to improve the accuracy of object skeleton detection, we propose a novel network architecture, the Multi-Scale Bidirectional Fully Convolutional Network (MSB-FCN), to better gather and enhance multi-scale high-level contextual information. The advantage is that only deep features are used to construct multi-scale feature representations along with a bidirectional structure for better capturing contextual knowledge. This enables the proposed MSB-FCN to learn semantic-level information from different sub-regions. Moreover, we introduce dense connections into the bidirectional structure to ensure that the learning process at each scale can directly encode information from all other scales. An attention pyramid is also integrated into our MSB-FCN to dynamically control information propagation and reduce unreliable features. Extensive experiments on various benchmarks demonstrate that the proposed MSB-FCN achieves significant improvements over the state-of-the-art algorithms.

Small fasteners are widely used in a freight train. However, due to the complex working environment, the missing of fasteners always happens, causing traffic accidents and property loss. Thus, detection of the missing of small fasteners is essential for the safety of freight trains. This article takes three critical missing small fasteners as examples, pin-missing, bolt-missing, and rivet-missing defects, and proposes a two-stage small defect detection model. First, a copy–pasting method is used to augment the defects in the training image, then the ResNet-101 backbone is adopted to extract defect features, and a feature pyramid network is used to construct a feature pyramid. After obtaining defect features, an improved region proposal network is introduced to generate defect proposals. Finally, a fully convolutional neural network is trained to operate pixelwise segmentation. The experimental results on manually collected fastener defect data set show that our model possesses high precision and efficiency on the inspection of pin-missing, bolt-missing, and rivet-missing defects. Besides, the proposed model could be used for similar small defects detection.

The segmentation effect of welding zone is vulnerable to image quality and target shape, which brings difficulty for the radiographic inspection of water-cooled pipe. A boundary-aware semantic segmentation algorithm is proposed to improve the precision and robustness of the welding zone inspection system. First, a convolutional neural network and atrous spatial pyramid pooling are employed to generate the feature map of the input image. Second, a boundary distance field (BDF) regression module is designed to predict the distance fields of the inner and outer boundaries, which can reflect the continuous boundary information of welding zone. Then, spatial attention weight is calculated according to the regressed boundary distance fields, and the calculated weight is fused into the feature map to enhance the significance of welding zone. Finally, the segmentation mask is predicted based on the fused feature map. Experiments indicate that the guidance of boundary information in the semantic segmentation algorithm can effectively reduce the interference of nonweld zones. The segmentation accuracy of the welding zone is 93.70%, and average detection time of a single image is less than 11 ms.

Object detection is one of the core tasks in computer vision. Object detection algorithms often have difficulty detecting objects with diverse scales, especially those with smaller scales. To cope with this issue, Lin et al. proposed feature pyramid networks (FPNs), which aim for a feature pyramid with higher semantic content at every scale level. The FPN consists of a bottom-up pyramid and a top-down pyramid. The bottom-up pyramid is induced by a convolutional neural network as its layers of feature maps. The top-down pyramid is formed by progressive up-sampling of a highly semantic yet low-resolution feature map at the top of the bottom-up pyramid. At each up-sampling step, feature maps of the bottom-up pyramid are fused with the top-down pyramid to produce highly semantic yet high-resolution feature maps in the top-down pyramid. Despite significant improvement, the FPN still misses small-scale objects. To further improve the detection of small-scale objects, this paper proposes scale adaptive feature pyramid networks (SAFPNs). The SAFPN employs weights chosen adaptively to each input image in fusing feature maps of the bottom-up pyramid and top-down pyramid. Scale adaptive weights are computed by using a scale attention module built into the feature map fusion computation. The scale attention module is trained end-to-end to adapt to the scale of objects contained in images of the training dataset. Experimental evaluation, using both the 2-stage detector faster R-CNN and 1-stage detector RetinaNet, demonstrated the proposed approach’s effectiveness.

Object detectors have improved in recent years, obtaining better results and faster inference time. However, small object detection is still a problem that has not yet a definitive solution. The autonomous weapons detection on Closed-circuit television (CCTV) has been studied recently, being extremely useful in the field of security, counter-terrorism, and risk mitigation. This article presents a new dataset obtained from a real CCTV installed in a university and the generation of synthetic images, to which Faster R-CNN was applied using Feature Pyramid Network with ResNet-50 resulting in a weapon detection model able to be used in quasi real-time CCTV (90 ms of inference time with an NVIDIA GeForce GTX-1080Ti card) improving the state of the art on weapon detection in a two stages training. In this work, an exhaustive experimental study of the detector with these datasets was performed, showing the impact of synthetic datasets on the training of weapons detection systems, as well as the main limitations that these systems present nowadays. The generated synthetic dataset and the real CCTV dataset are available to the whole research community.

Recovering the 3D shape of the surgical site is crucial for multiple computer-assisted interventions. Stereo endoscopes can be used to compute 3D depth but computational stereo is a challenging, no...

PURPOSE Ultrasound image segmentation is a challenging task due to a low signal-to-noise ratio and poor image quality. Although several approaches based on the convolutional neural network (CNN) have been applied to ultrasound image segmentation, they have weak generalization ability. We propose an end-to-end, multiple-channel and atrous CNN designed to extract a greater amount of semantic information for segmentation of ultrasound images. METHOD A multiple-channel and atrous convolution network is developed, referred to as MA-Net. Similar to U-Net, MA-Net is based on an encoder-decoder architecture and includes five modules: the encoder, atrous convolution, pyramid pooling, decoder, and residual skip pathway modules. In the encoder module, we aim to capture more information with multiple-channel convolution and use large kernel convolution instead of small filters in each convolution operation. In the last layer, atrous convolution and pyramid pooling are used to extract multi-scale features. The architecture of the decoder is similar to that of the encoder module, except that up-sampling is used instead of down-sampling. Furthermore, the residual skip pathway module connects the subnetworks of the encoder and decoder to optimize learning from the deeper layer and improve the accuracy of segmentation. During the learning process, we adopt multi-task learning to enhance segmentation performance. Five types of datasets are used in our experiments. Because the original training data are limited, we apply data augmentation (e.g., horizontal and vertical flipping, random rotations, and random scaling) to our training data. We use the Dice score, precision, recall, Hausdorff distance (HD), average symmetric surface distance (ASD), and root mean square symmetric surface distance (RMSD) as the metrics for segmentation evaluation. Meanwhile, Friedman test was performed as the nonparametric statistical analysis to evaluate the algorithms. RESULTS For the datasets of brachia plexus (BP), fetal head, and lymph node segmentations, MA-Net achieved average Dice scores of 0.776, 0.973, and 0.858, respectively; with average precisions of 0.787, 0.968, and 0.854, respectively; average recalls of 0.788, 0.978, and 0.885, respectively; average HDs (mm) of 13.591, 10.924, and 19.245, respectively; average ASDs (mm) of 4.822, 4.152, and 4.312, respectively; and average RMSDs (mm) of 4.979, 4.161, and 4.930, respectively. Compared with U-Net, U-Net++, M-Net, and Dilated U-Net, the average performance of the MA-Net increased by approximately 5.68%, 2.85%, 6.59%, 36.03%, 23.64%, and 31.71% for Dice, precision, recall, HD, ASD, and RMSD, respectively. Moreover, we verified the generalization of MA-Net segmentation to lower grade brain glioma MRI and lung CT images. In addition, the MA-Net achieved the highest mean rank in the Friedman test. CONCLUSION The proposed MA-Net accurately segments ultrasound images with high generalization, and therefore, it offers a useful tool for diagnostic application in ultrasound images.

This paper presents a novel deep learning model to aggregate the attentional dilated features for salient object detection by exploring the complementary information between the global and local context in a convolutional neural network. There are two technical contributions to our network design. First, we develop an attentional dense atrous (dilated) spatial pyramid pooling (AD-ASPP) module to selectively use the local saliency cues captured by dilated convolutions with a small rate and the global saliency cues captured by dilated convolutions with a large rate. Second, taking the feature pyramid network as the backbone, we develop an aggregation network to integrate the refined features by formulating two consecutive chains of residual learning based modules: one chain from deep to shallow layers while another chain from shallow to deep layers. We evaluate our network on seven widely-used saliency detection benchmarks by comparing it against 21 state-of-the-art methods. Experimental results show that our network outperforms others on all the seven benchmark datasets.

Accurate and automatic segmentation of medical images is a crucial step for clinical diagnosis and analysis. The convolutional neural network (CNN) approaches based on the U-shape structure have achieved remarkable performances in many different medical image segmentation tasks. However, the context information extraction capability of single stage is insufficient in this structure, due to the problems such as imbalanced class and blurred boundary. In this paper, we propose a novel Context Pyramid Fusion Network (named CPFNet) by combining two pyramidal modules to fuse global/multi-scale context information. Based on the U-shape structure, we first design multiple global pyramid guidance (GPG) modules between the encoder and the decoder, aiming at providing different levels of global context information for the decoder by reconstructing skip-connection. We further design a scale-aware pyramid fusion (SAPF) module to dynamically fuse multi-scale context information in high-level features. These two pyramidal modules can exploit and fuse rich context information progressively. Experimental results show that our proposed method is very competitive with other state-of-the-art methods on four different challenging tasks, including skin lesion segmentation, retinal linear lesion segmentation, multi-class segmentation of thoracic organs at risk and multi-class segmentation of retinal edema lesions.

Semantic scene segmentation has become a key application in computer vision and is an essential part of intelligent transportation systems for complete scene understanding of the surrounding environment. While several methods based on deep fully Convolutional Neural Network (CNN) have been emerging, there are two main challenges: (i) They mainly focus on improvement of the accuracy than efficiency. (ii) They assume structured driving environment like in USA and Europe. While most of the current works focus on the well structured driving environment, we focus our research on India Driving Dataset (IDD) which contains data from unstructured traffic scenario. In this paper, we propose modifications in the DeepLabV3+ framework by using lower atrous rates in Atrous Spatial Pyramid Pooling (ASPP) module for dense traffic prediction. We propose to use dilated Xception network as the backbone for feature extraction. A lightweight segmentation framework is also presented by exploring the effectiveness of MobileNetV2 architecture, which achieves competitively high accuracy and is much smaller than other state-of-art architectures. The performance is evaluated in terms of mean Intersection over Union (mIoU) on 26 fine grained classes of IDD. Our proposed model with 24 M parameters achieves 68.41 mIoU on test set and efficient mobile model achieves mIoU of 61.6 by reducing the parameters to 2.2 M only.

Road segmentation from remote sensing images is a critical task in many applications. In recent years, various approaches, particularly deep learning-based methods, have been proposed for accurate road segmentation. However, most existing road segmentation methods always obtain unsatisfactory results (e.g., heterogeneous pixels) due to the complex backgrounds and view occlusions of buildings and trees around a road; consequently, road segmentation remains a challenging problem. In this study, we propose a novel global context based dilated convolutional neural network (GC-DCNN) to address the aforementioned problem. The structure of GC-DCNN is similar to that of UNet. In particular, building the encoder of GC-DCNN with three residual dilated blocks is suggested to further enlarge the effective receptive field and learn additional discriminative features. Thereafter, a pyramid pooling module is used to capture the multiscale global context features and fuse them to achieve stronger feature representation. The decoder network upsamples the fused features to the same size as the input image, combining the high-resolution features with the contracting path of the encoder network. Moreover, the dice coefficient loss is adopted as the loss function. This function differs from those in most previous studies but is more suitable for road segmentation. Extensive experimental results on two benchmark datasets compared with several baseline models demonstrate the superiority of the proposed GC-DCNN algorithm.

Low-to-high hierarchical convolutional features can significantly improve edge detection. This paper proposes a feature decoder-based algorithm that employs a Feature Decoder Network (FDN) to extract more information within limited Convolutional Neural Network (CNN) features. Previous studies applied convolutional elements by weight fusion, but we measure a feature decoder as a pyramid by qualifying convolutional layers. The feature decoder fuses CNN features of adjacent layers to judge the edge and non-edge pixels, which can learn the relationship and distinction between low-level edge hierarchies and high-level semantic hierarchies. Furthermore, we use Gaussian blur labels to train the network to optimize network convergence and training. From the experimental results, our proposed algorithm performs better on the BSDS500 (average accuracy (AP) of 0.865) and NYUD (OIS F-measure of 0.775) datasets compared to the state-of-the-art algorithms, including RCF.

This article reports the development of a pure deep learning method for segmenting concrete cracks in images. The objectives are to achieve the real-time performance while effectively negating a wide range of various complex backgrounds and crack-like features. To achieve the goals, an original convolutional neural network is proposed. The model consists of standard convolutions, densely connected separable convolution modules, a modified atrous spatial pyramid pooling module, and a decoder module. The semantic damage detection network (SDDNet) is trained on a manually created crack dataset, and the trained network records the mean intersection-over-union of 0.846 on the test set. Each test image is analyzed, and the representative segmentation results are presented. The results show that the SDDNet segments cracks effectively unless the features are too faint. The proposed model is also compared with the most recent models, which show that it returns better evaluation metrics even though its number of parameters is 88 times less than in the compared models. In addition, the model processes in real-time (36 FPS) images at 1025 × 512 pixels, which is 46 times faster than in a recent work.

Traditional deep convolutional neural networks for fully automatic brain tumor segmentation have two problems: spatial information loss caused by both the repeated pooling/striding and weak ability of multi-scale lesion processing. To overcome the first problem, we use a 3D atrous-convolution with a single stride to replace pooling/striding and build the backbone for feature learning. For the second problem, a 3D atrous-convolution feature pyramid is designed and added to the end of the backbone. By integrating with contextual features, this structure improves the discriminating ability of the overall model to segment tumors with various sizes. Finally, a 3D fully connected Conditional Random Field is constructed as a post-processing step for the network’s output to obtain structural segmentation of both the appearance and spatial consistency. Abundant ablation experiments carried on Magnetic Resonance Imaging datasets demonstrate that lossless feature computation and multi-scale information fusion driven by our method are feasible to address the above problems. Compared with the state-of-the-art methods on the public benchmarks, our method achieves competitive performance and can be efficiently implemented into the clinical medical application.

Object detection could be recognized as an essential part of the research to scenarios such as automatic driving and pedestrian detection, etc. Among multiple types of target objects, the identification of small-scale objects faces significant challenges. We would introduce a new feature pyramid framework called Dual Attention based Feature Pyramid Network (DAFPN), which is designed to avoid predicament about multi-scale object recognition. In DAFPN, the attention mechanism is introduced by calculating the top-down pathway and lateral pathway, where the spatial attention, as well as channel attention, would participate, respectively, such that the pyramidal feature maps can be generated with enhanced spatial and channel interdependencies, which bring more semantical information for the feature pyramid. Using the COCO data set, which consists of a considerable quantity of small-scale objects, the experiments are implemented. The analysis results verify the optimized performance of DAFPN compared with the original Feature Pyramid Network (FPN) specifically for the identification on a small scale. The proposed DAFPN is promising for object detection in an era full of intelligent machines that need to detect multi-scale objects.

Purpose Automated segmentation could improve the efficiency of modeling-based pelvic organ prolapse (POP) evaluations. However, segmentation performance is limited by the blurry soft tissue boundaries. In this study, we aimed to present a hybrid solution for uterus, rectum, bladder, and levator ani muscle segmentation by combining a convolutional neural network (CNN) and a level set method. Methods We used 24 sagittal pelvic floor magnetic resonance (MR) series from six anterior vaginal prolapse and six posterior vaginal prolapse subjects (a total 528 MR images). The stress MR images were performed both at rest and at maximal Valsalva. We assigned 264 images for training, 132 images for validation, and 132 images for testing. A CNN was designed by introducing a multi-resolution features pyramid module (MRFP) into an encoder-decoder model. Depth separable convolution and pretraining were used to improve model convergence. Multiclass cross entropy loss and multiclass Dice loss were used for model training. The dice similarity coefficient (DSC) and average surface distance (ASD) were used for evaluating the segmentation results. To prove the effectiveness of our model, we compared it with advanced segmentation methods including Deeplabv3+, U-Net, and FCN-8s. The ablation study was designed to quantify the contributions of MRFP, the encoder network, and pretraining. Besides, we investigated the working mechanism of MRFP in the segmentation network by comparing our model with three of its variants. Finally, the level set method was used to improve the CNN model further. Results Dice loss showed better segmentation performance than multiclass cross entropy loss. MRFP was efficacious for different encoder networks. With MRFP, U-Net and U-Net-X (X represents Xception encoder network) have improved the DSC, on average by 6.8 and 5.3 points. Compared with different CNN models, our model achieved the highest average DSC of 65.6 points and the lowest average ASD of 2.9 mm. With the level set method, the DSC of our model improved to 69.4 points. Conclusions MRFP proved to be effective in addressing the blurry soft tissue boundary problem on pelvic floor MR images. A hybrid solution based on CNN and level set method was presented for pelvic organ segmentation both at rest and at maximal Valsalva; with this method, we achieved state-of-the-art results.

Surface defect detection has received increased attention in relation to the product quality and industry safety. This paper develops an image pyramid convolution neural network (IPCNN) model to detect surface defects in images. The IPCNN is an improvement of the Mask rcnn model. It combines image pyramid and deep convolution neural network to extract pyramid features for defect detections. First, it constructs an image pyramid. Then, images from the pyramid are processed with a deep residual neural network to extract features. Extracted features are then fused and processed with a feature pyramid network to generate pyramid features. A region proposal network then operates on the pyramid features to generate defect bounding boxes and make classifications. Finally, a fully convolutional neural network generates defect masks in the detected bounding boxes. The IPCNN was evaluated on the publicly available COCO dataset and improved the Mask rcnn by approximately one point. Additionally, it generated high precision and recall values on inspecting the oil leak defect on freight train with a small dataset. The comparison with state-of-the-art methods on freight train defect detection showed the promising results of IPCNN. The efficiency of the IPCNN is 0.243 seconds per frame.

In view of the small number of categories and the relatively little amount of labeled data, it is challenging to apply the fusion of deep convolution features directly to remote sensing images. To address this issue, we propose a pyramid multi-subset feature fusion method, which can effectively fuse the deep features extracted from different pre-trained convolutional neural networks and integrate the global and local information of the deep features, thereby obtaining stronger discriminative and low-dimensional features. By introducing the idea of weighting the difference between different categories, the weight discriminant correlation analysis method is designed to make it pay more attention to those categories that are not easy to distinguish. In order to mine global and local feature information, the pyramid method is employed to divide feature fusion into several layers. Each layer divides the features into several subsets and then performs feature fusion on the corresponding feature subsets, and the number of subsets from top to bottom gradually increases. Feature fusion at the top of the pyramid obtains a global representation, while feature fusion at the bottom obtains a local detail representation. Our experiment results on three public remote sensing image data sets demonstrate that the proposed multi-deep features fusion method produces improvements over other state-of-the-art deep learning methods.

Text detection in natural scene images is a key prerequisite for computer vision tasks such as image search, blind navigation, autopilot, and multi-language translation. Existing text detection methods only detect partial region of large-scale texts and are difficult to detect small-scale texts. Aiming at this problem, an anchor-free multi-orientation text detection method is proposed. Firstly, Feature Pyramid Network (FPN) is used to combine the multiple feature layers of Convolutional Neural Network (CNN) to predict the geometric properties of text, which can be used to expand the receptive field of each pixel and thus help to detect more large-scale texts. Secondly, a new loss function independent of the scale of text is designed, which enables the pixels in the small-scale text to have a larger calculation weight, thereby facilitating the detection of small-scale texts. Finally, the results of pixel-level semantic segmentation are used to filter obviously unreasonable candidate text boxes, and at the same time improve the accuracy and recall rate of text detection. The experimental results on ICDAR 2015 and MSRA-TD500 prove the good performance of our method.

Classification is one of the most important task in hyperspectral image processing. In the last few decades, several classification techniques have been introduced. However, most of them could not efficiently extract features from hyperspectral images (HSI). A novel deep learning framework is proposed in this paper which efficiently utilises convolutional neural network (CNN) and spatial pyramid pooling (SPP) for extracting both the spectral–spatial features for classification. The proposed hybrid framework uses principal component analysis (PCA), 3D-CNN, 2D-CNN and SPP. The proposed CNN-based model is applied on three benchmark hyperspectral datasets, and subsequently the performance is compared with state-of-the-art methods in the same field. The obtained results reveal the superiority of the proposed model in effectively classifying HSI.

Deep convolutional neural networks (CNNs) have been widely applied to saliency object detection (SOD) with promising performance. However, the coarse feature resolutions due to a series of pooling and convolutional operations in CNNs considerably decrease the original image resolution, resulting in losing spatial details and fine structures, especially on salient object boundaries. To address this issue, we present a recurrent reverse attention based residual learning network for SOD. We first construct a pair of low-and high-level integrated feature representations by aggregating two groups of low-and high-level feature maps, respectively, among which we design a novel joint feature pyramid pooling module to increase the high-level feature resolution. Afterwards, we progressively learn to refine the residual between each side-output saliency prediction and the ground-truth in a cascaded fashion by alternatively using the high-level and low-level integrated features. Each residual learning module contains a recurrent reverse attention module that focuses on the residual region outside the current predicted salient regions, guiding the network to learn the missing salient parts and details. Finally, we develop a simple yet effective boundary detection loss to compensate the detailed texture loss. Extensive evaluations on six popular SOD benchmark datasets demonstrate remarkable performance boosting of our proposed approach compared with state-of-the-art methods. Especially, our approach runs in real-time with a speed of 32 fps.

Gliomas are characterized by high morbidity and high mortality in primary tumors. The identification of glioma type is helpful for radiologists to facilitate correct medical judgments and better prognosis for patients. In order to avoid harm to patients caused by a biopsy, radiologists attempt to classify Magnetic Resonance Images(MRI) using deep learning methods. In the present paper, we propose a deep learning convolutional neural network ResNet based on the pyramid dilated convolution for Gliomas classification. The pyramid dilated convolution is integrated into the bottom of Resnet to increase the receptive field of the original network and improve the classification accuracy. After adding the pyramid dilated convolution model, the receptive field of the original network underlying convolution was improved. A clinical dataset is used to test the pyramid dilated convolution ResNet neural network model proposed in this paper. The experimental results demonstrate that the proposed method can effectively improve glioma classification performance.

Recently, the deep convolutional neural network has brought great improvements in object detection. However, the balance between high accuracy and high speed has always been a challenging task in multiclass object detection for large-scale remote sensing imagery. One-stage methods are more widely used because of their high efficiency but are limited by their performances on small object detection. In this article, we propose a unified framework called feature-merged single-shot detection (FMSSD) network, which aggregates the context information both in multiple scales and the same scale feature maps. First, our network leverages the atrous spatial feature pyramid (ASFP) module to fuse the context information in multiscale features by using feature pyramid and multiple atrous rates. Second, we propose a novel area-weighted loss function to pay more attention to small objects, while the replaced original loss treats all objects equally. We believe that small objects should be given more weight than large objects because they lose more information during training. Specifically, a monotonic decreasing function about the area is designed to add weights on the loss function. Extensive experiments on the DOTA data set and NWPU VHR-10 data set demonstrate that our method achieves state-of-the-art detection accuracy with high efficiency. We also build a new large-scale data set called AIR-OBJ data set from Google Earth and show the detection results of small objects, which validates the effectiveness on large-scale remote sensing imagery.

With exploiting contextual information over large image regions in an efficient way, the deep convolutional neural network has shown an impressive performance for single-image super-resolution (SR). In this paper, we propose a new deep convolutional network by cascading multiple well-designed inception-residual blocks within the deep Laplacian pyramid framework to progressively restore the missing high-frequency details in the low-resolution images. By optimizing our network structure, the trainable depth of our proposed network gains a significant improvement, which in turn improves super-resolving accuracy. However, the saturation and degradation of training accuracy remains a critical problem. With regard to this, we propose an effective two-stage training strategy, in which we first use the images downsampled from the ground-truth high-resolution (HR) images to pretrain the inception-residual blocks on each pyramid level with an extremely high learning rate enabled by gradient clipping, and then the original ground-truth HR images are used to fine-tune all the pretrained inception-residual blocks for obtaining our final SR models. Furthermore, we present a new loss function operating in both image space and local rank space to optimize our network for exploiting the contextual information among different output components. Extensive experiments on benchmark data sets validate that the proposed method outperforms the existing state-of-the-art SR methods in terms of the objective evaluation as well as the visual quality.

In this letter, a pyramid wavelet convolutional neural network for audio super resolution is presented. Since the audio signal is non‐stationary, previous convolutional neural network based approaches may fail in capturing the details, these method usually focus on the global approximation error and thus produce over smooth results. To cope with this issue, it is suggested to predict the wavelet coefficients of the audio signal, and reconstruct the signal from these coefficients stage by stage rather. The prediction errors of the wavelet coefficients are included to the loss function to force the model to capture the detail components. Experimental results show that the approach, training on the VCTK public dataset, achieves more appealing results than state‐of‐the‐art methods.

Maxillary sinus segmentation plays an important role in the choice of therapeutic strategies for nasal disease and treatment monitoring. Difficulties in traditional approaches deal with extremely heterogeneous intensity caused by lesions, abnormal anatomy structures, and blurring boundaries of cavity. 2D and 3D deep convolutional neural networks have grown popular in medical image segmentation due to utilization of large labeled datasets to learn discriminative features. However, for 3D segmentation in medical images, 2D networks are not competent in extracting more significant spacial features, and 3D ones suffer from unbearable burden of computation, which results in great challenges to maxillary sinus segmentation. In this paper, we propose a deep neural network with an end-to-end manner to generalize a fully automatic 3D segmentation. At first, our proposed model serves a symmetrical encoder-decoder architecture for multitask of bounding box estimation and in-region 3D segmentation, which cannot reduce excessive computation requirements but eliminate false positives remarkably, promoting 3D segmentation applied in 3D convolutional neural networks. In addition, an overestimation strategy is presented to avoid overfitting phenomena in conventional multitask networks. Meanwhile, we introduce residual dense blocks to increase the depth of the proposed network and attention excitation mechanism to improve the performance of bounding box estimation, both of which bring little influence to computation cost. Especially, the structure of multilevel feature fusion in the pyramid network strengthens the ability of identification to global and local discriminative features in foreground and background achieving more advanced segmentation results. At last, to address problems of blurring boundary and class imbalance in medical images, a hybrid loss function is designed for multiple tasks. To illustrate the strength of our proposed model, we evaluated it against the state-of-the-art methods. Our model performed better significantly with an average Dice 0.947±0.031, VOE 10.23±5.29, and ASD 2.86±2.11, respectively, which denotes a promising technique with strong robust in practice.

Determining the authorship of a painting image is a challenging task because paintings of an artist may not have a unique style and various artists may have similar painting styles. In this paper, we present a new approach to categorize digital painting images based on artist. We construct a multi-scale pyramid from a painting image to consider both globally and locally the information contained in one image. For each layer, we train a Convolutional Neural Network (CNN) model to determine the class label. To build the relationship within local image patches, we employ Markov random fields (MRFs) by optimizing the Gibbs energy function defined by (1) the data term measuring the compatibility of labeling with given data, and (2) the smoothness term penalizing assignments that label neighboring patches differently. A new fusion scheme is proposed to aggregate patch-level classification results. The proposed CNN-MRF method is validated using two challenging painting image datasets. Experimental results show that the proposed method is effective and achieves state-of-the-art performance.

Automatic image annotation aims to assign relevant keywords to images and has become a research focus. Although many techniques have been proposed to solve this problem in the last decade, giving promissing performance on standard datasets, we propose a novel automatic image annotation technique in this paper. Our method uses a label transfer mechanism to automatically recommend those promising tags to each image by using the category information of images. As image representation is one of the key technique in image annotation, we use sparse coding based spatial pyramid matching and deep convolutional neural networks to model image features. And metric learning technique is further used to combine these features to achieve more effective image representation in this paper. Experimental results illustrate that the proposed method get similar or better results than the state-of-the-art methods on three standard image datasets.

Aiming at the problem that it is difficult for traffic monitoring videos to detect multi-scale vehicle targets, especially small vehicle targets in complex scenarios, a codec-based vehicle detection algorithm is proposed. This algorithm is based on YOLOv3. In order to solve the multi-scale vehicle target detection problem, a new multi-level feature pyramid structure added with the codec module to detect vehicle targets of different scales. The experimental results on the KITTI dataset and UA-DETRAC dataset confirm that the algorithm in this paper has achieved good detection results for vehicle targets in various environments and at various scales in the surveillance video, especially for small vehicle targets, which can better meet the actual application demand.

Traditional image processing operators often provide some control parameters to tweak the final results. Recently, different convolutional neural networks have been used to approximate or improve these operators. However, in those methods, one single model can only handle one operator of a specific parameter value and does not support parameter tuning. In this paper, we propose a new plugin module, “Adaptive Filterbank Pyramid” , which can be inserted into a backbone network to support multiple operators and continuous parameter tuning. Our module explicitly represents one operator with one filterbank pyramid. To generate the results of a specific operator, the corresponding filterbank pyramid is convolved with the intermediate feature pyramid produced by the backbone network. The weights of the filterbank pyramid are directly regressed by another sub-network, which is jointly trained with the backbone network and adapted to the input parameter, thus enabling continuous parameter tuning. We applied the proposed module for a large variety of image processing tasks, including image smoothing, image denoising, image deblocking, image enhancement and neural style transfer. Experiments show that our method is generalized to different types of image processing tasks and different backbone network structures. Compared to the single-operator-single-parameter baseline, our method can produce comparable results but is significantly more efficient in both training and testing.

Recently, deep learning based video super-resolution (SR) methods combine the convolutional neural networks (CNN) with motion compensation to estimate a high-resolution (HR) video from its low-resolution (LR) counterpart. However, most previous methods conduct downscaling motion estimation to handle large motions, which can lead to detrimental effects on the accuracy of motion estimation due to the reduction of spatial resolution. Besides, these methods usually treat different types of intermediate features equally, which lack flexibility to emphasize meaningful information for revealing the high-frequency details. In this paper, to solve above issues, we propose a deep dual attention network (DDAN), including a motion compensation network (MCNet) and a SR reconstruction network (ReconNet), to fully exploit the spatio-temporal informative features for accurate video SR. The MCNet progressively learns the optical flow representations to synthesize the motion information across adjacent frames in a pyramid fashion. To decrease the mis-registration errors caused by the optical flow based motion compensation, we extract the detail components of original LR neighboring frames as complementary information for accurate feature extraction. In the ReconNet, we implement dual attention mechanisms on a residual unit and form a residual attention unit to focus on the intermediate informative features for high-frequency details recovery. Extensive experimental results on numerous datasets demonstrate the proposed method can effectively achieve superior performance in terms of quantitative and qualitative assessments compared with state-of-the-art methods.

Low spatial resolution is a well-known problem for depth maps captured by low-cost consumer depth cameras. Depth map super-resolution (SR) can be used to enhance the resolution and improve the quality of depth maps. In this paper, we propose a recumbent Y network (RYNet) to integrate the depth information and intensity information for depth map SR. Specifically, we introduce two weight-shared encoders to respectively learn multi-scale depth and intensity features, and a single decoder to gradually fuse depth information and intensity information for reconstruction. We also design a residual channel attention based atrous spatial pyramid pooling structure to further enrich the feature's scale diversity and exploit the correlations between multi-scale feature channels. Furthermore, the violations of co-occurrence assumption between depth discontinuities and intensity edges will generate texture-transfer and depth-bleeding artifacts. Thus, we propose a spatial attention mechanism to mitigate the artifacts by adaptively learning the spatial relevance between intensity features and depth features and reweighting the intensity features before fusion. Experimental results demonstrate the superiority of the proposed RYNet over several state-of-the-art depth map SR methods.

Shadow detection is one of the most challenging issues in computer vision. Inspired by the great success of the convolutional neural network (CNN) for the problem of image restoration, learned features have been widely adopted for shadow detection. However, most existing methods still suffer from ambiguities driven by black-colored objects, which are not actually shaded, as well as the background clutter. In this letter, we propose the attentive feedback feature pyramid network (AFFPN) for shadow detection in a single image. The key idea of the proposed method is to extract shadow-relevant features based on multiple feedback modules, which are defined in the feature pyramid network. Specifically, attentive features extracted from each level of the encoder are progressively refined via connections between feedback modules from high-level to low-level layers for learning properties of shadow more accurately. Experimental results on benchmark datasets show that the proposed method is effective for shadow detection under complicated real-world environments. The code and model are publicly available at: https://github.com/JinheeKIM94/AFFPN_release .

Accurate and effective object detection in remote sensing images plays an extremely important role in marine transport, environmental monitoring and military operations. Due to the powerful ability of feature representation, region-based convolutional neural networks (RCNNs) have been widely used in this field, which firstly generate candidate regions through extracted feature maps and then classify and locate objects. However, most of existing methods generally use traditional backbone networks to extract feature maps with a decreased spatial resolution because of the continuous down-sampling, which will weaken the information detected from small objects. Besides, sliding windows strategy is employed in these methods to generate fixed anchors with a preset scale on feature maps, which is inappropriate for multi-scale object detection in remote sensing images. To solve the above problems, a novel and effective object detection framework named DetNet-FPN (Feature Pyramid Network) is proposed in this paper, in which a feature pyramid with strong feature representation is created by combining feature maps of different spatial resolution, at the same time, the resolution of feature maps is maintained by involving dilation convolutions. Furthermore, to match the proposed backbone, the GA (Guided Anchoring)-RPN strategy is improved for adaptive anchor generation, this strategy simultaneously predicts the locations where the center of objects are likely to exist as well as the scales and aspect ratios at different locations. Extensive experiments and comprehensive evaluations demonstrate the effectiveness of the proposed framework on DOTA and NWPU VHR-10 datasets.

Image rain removal aims to separate the background image from the rainy image. During the past three years, the image rain removal with deep convolutional neural networks has achieved impressive performance. However, how to reach tradeoff between high de-raining performance and low model parameters is still a challenge. To address the issue, the paper is devoted to exploring a novel method based on wavelet deep recursive pyramid convolution residual network (WDRPRN), in which discrete wavelet transform is embedded to decompose the rainy image in different frequency domains, and the deep recursive pyramid convolution residual network (DRPRN) can well predict the residual coefficients between rainy image and clean image. In addition, compared with other neural networks, the DRPRN adopts recursive model that can cost fewer parameters. Plentiful of experiments on synthetic and real-world datasets show that the proposed method is significantly superior to the recent state-of-the-art algorithms.

The semantic segmentation of remote sensing (RS) image is a hot research field. With the development of deep learning, the semantic segmentation based on a full convolution neural network greatly improves the segmentation accuracy. The amount of information on the RS image is very large, but the sample size is extremely uneven. Therefore, even the common network can segment RS images to a certain extent, but the segmentation accuracy can still be greatly improved. The common neural network deepens the network to improve the classification accuracy, but it has a lot of loss to the target spatial features and scale features, and the existing common feature fusion methods can only solve some problems. A segmentation network is built to solve the above problems very well. The network employs the InceptionV-4 network as the backbone and improves it. We modify the network structure and introduce the changed Atrous Spatial Pyramid Pooling module to extract the multi-scale features of the target from different training stages. Without losing the depth of the network, using Inception blocks to strengthen the width of the network can obtain more abstract features. At the same time, the backbone network is used for semantic fusion of the context, it can retain more spatial features, then an effective decoder network is designed. Finally, evaluate our model on the ISPRS 2D Semantic Labeling Contest Potsdam and Inria Aerial Image Labeling Dataset. The results show that the network has very superior performance, reaching 89.62% IOU score and 94.49% F1 score on the Potsdam dataset, and the IOU score on the Inria dataset has been greatly improved.