Search

It is a typical problem in the field of object detection to simultaneously detect objects with large scale variation in one image. Recently proposed state-of-the-art object detectors generally learn pyramidal feature representation to deal with the scale variation, which has been proved effective via various feature pyramid networks. However, the majority of the feature pyramid networks based on heuristic feature fusion strategies may be suboptimal, as excess human guidance will restrict the self-learning of deep neural networks. An adaptive feature pyramid is bound to provide a significant performance boost. In this paper, we propose a novel feature pyramid network named CATFPN that consists of Scale-Wise Feature Concatenation (SWFC) module and Global Context (GC) block. The SWFC module evenly distributes semantic features for each feature layer and the GC block introduces a self-attention mechanism. As a feature pyramid network, the CATFPN can be applied to any detector based on multi-scale features. We adopt the CATFPN in typical RetinaNet and Faster R-CNN detector models, without bells and whistles, achieving 1.1% AP and 0.7% AP improvements over FPN on the MS COCO benchmark, respectively. Our competitive performance reported on the test-dev subset of COCO achieves 42.3% AP.

Sketches, as a kind of probabilistic data structures, have been considered as the most promising solution for network measurement in recent years. Most sketches do not work well for skewed network traffic. To address this problem, we propose a family of sketch frameworks, namely the Pyramid family. The first member of our Pyramid family is the S-Pyramid framework, which includes two techniques: counter-pair sharing for high accuracy, and word acceleration for fast speed. The second member of our Pyramid family is the Mini-Pyramid framework, which projects the S-Pyramid framework into one counter, bringing more flexibility in application while keeping the accuracy. To demonstrate the generality of our Pyramid family, we apply both frameworks to sketches of CM, CU, Count, and Augmented. To demonstrate the flexibility of the Mini-Pyramid framework, we further apply Mini-Pyramid to SBF and the On-Off sketch. The experimental results show that, the S-Pyramid framework can reduce the ARE by up to 7.12 times compared with the original sketches, while improving the throughput by up to 2.37 times; the Mini-Pyramid framework can reduce the ARE by up to 29.2 times, at the cost of 21.3% lower throughput on average.

Both siamese network and correlation filter (CF)-based trackers have exhibited superior performance by formulating tracking as a similarity measure problem, where a similarity map is learned by the correlation between a target template and a region of interest (ROI) with a cosine window. Nevertheless, this window function is usually fixed for various targets and not changed, undergoing significant noise variations during tracking, which easily makes model drift. In this article, we focus on the study of a noise-aware (NA) framework for robust visual tracking. To this end, the impact of various window functions is first investigated in visual tracking. We identify that the low signal-to-noise ratio (SNR) of windowed ROIs makes the above trackers degenerate. At the prediction phase, a novel NA window customized for visual tracking is introduced to improve the SNR of windowed ROIs by adaptively suppressing the variable noise according to the observation of similarity maps. In addition, to further optimize the SNR of windowed pyramid ROIs for scale estimation, we propose to use the particle filter to dynamically sample several windowed ROIs with more favorable signals in temporal domains instead of this pyramid ROIs extracted in spatial domains. Extensive experiments on the popular OTB-2013, OTB-50, OTB-2015, VOT2017, TC128, UAV123, UAV123@10fps, UAV20L, and LaSOT datasets show that our NA framework can be extended to many siamese and CF trackers and our variants obtain superior performance than baseline trackers with a modest impact on efficiency.

When a detection model that has been well-trained on a set of classes faces new classes, incremental learning is always necessary to adapt the model to detect the new classes. In most scenarios, it is required to preserve the learned knowledge of the old classes during incremental learning rather than reusing the training data from the old classes. Since the objects in remote sensing images often appear in various sizes, arbitrary directions, and dense distribution, it further makes incremental learning-based object detection more difficult. In this article, a new architecture for incremental object detection is proposed based on feature pyramid and knowledge distillation. Especially, by means of a feature pyramid network (FPN), the objects with various scales are detected in the different layers of the feature pyramid. Motivated by Learning without Forgetting (LwF), a new branch is expended in the last layer of FPN, and knowledge distillation is applied to the outputs of the old branch to maintain the old learning capability for the old classes. Multitask learning is adopted to jointly optimize the losses from two branches. Experiments on two widely used remote sensing data sets show our promising performance compared with state-of-the-art incremental object detection methods.

As a significant and fundamental task in the remote sensing field, object detection has received increasing attention and research studies. However, geospatial object detection is still a challenge owing to the dramatic variation in object scales, intraclass differences, and interclass similarity from multiscale and multiclass objects. To deal with these problems, an end-to-end feature-reflowing pyramid network (FRPNet) is proposed in this letter. FRPNet has two advantages that contribute to improve object detection accuracy. First, we embed a nonlocal block into the backbone in order to get the relevancy between different regions of the geospatial image for obtaining discriminative features. Furthermore, a feature-reflowing pyramid structure is proposed to generate high-quality feature presentation for each scale through fusing fine-grained features from the adjacent lower level, which improves the detection capability for multiscale and multiclass objects. Experiments on a public remote sensing data set DIOR illustrate that FRPNet can significantly improve the performance when compared to several state-of-the-art detection approaches in terms of mean average precision (mAP).

Change detection (CD) is one of the most important topics in the field of remote sensing. In this letter, we propose an effective satellite images CD network named DifUnet++. As the presentation of explicit difference is more conducive to extract change features, we design a differential pyramid of two input images as the input of Unet++. Considering the scale diversity of changed regions in remote sensing images, a multiply side-outs fusion strategy is adopted to predict the detection results of different scales. Furthermore, a learning upsampling method is utilized to refine the details of CD. The proposed architecture is evaluated on two public satellite image CD data sets. The experimental results show that our method performs much better than state-of-the-art methods.

Object detection (OD) is an important task of computer vision and has been widely used in many fields, including remote sensing (RS). However, the complex scenes, large-scale variation, and dense instances of RS bring huge challenges to OD. To meet these challenges, a novel Nonlocal-aware Pyramid and Multiscale Multitask Refinement Detector (NPMMR-Det) is proposed. Specifically, nonlocal-aware pyramid attention (NP-Attention) is designed for guiding a neural network model to focus more on efficient features and suppress background noise. Then a multiscale refinement feature pyramid network (MSR-FPN) is proposed to fuse the multiscale context features extracted by the NP-Attention guided neural network and adjust the optimal receptive field. In order to use these features more effectively, a multitask refinement head called MTR-Head, with offset sharing and a modulation mechanism, is developed to refine the feature misalignment between the localization task and the classification task. Extensive experiments performed on two public RS data sets demonstrate that the proposed NPMMR-Det achieves competitive performance compared with state-of-the-art methods.

Unlike object detection in natural images that usually achieved great success, remote sensing imagery has its own challenges to detect and localize multiclass objects, such as large-scale change, uncertain direction, and high density. The context information of the objects is very worthwhile for solving these challenges in remote sensing images. In this letter, we propose a context-driven detection network (CDD-Net) to improve the accuracy of multiclass object detection in remote sensing images. For capturing the local neighboring objects and features, a local context feature network (LCFN) is proposed to learn the local context of the region of interest. Meanwhile, a hybrid attention pyramid network (HAPN) is designed, which can steer the focus to more valuable features. The HAPN inserts a squeeze and excitation block (SEB) and three asymmetric convolution blocks (ACBs) in the feature pyramid network (FPN). The experimental results over the DOTA-v1.5 data set demonstrate that the proposed CDD-Net yields promising results.

Seismic fault detection is indispensable for exploring reservoirs of hydrocarbons, and a considerable amount of research has thus been devoted to it. With the rapid development of deep learning in recent years, researchers have begun using convolutional neural networks (CNNs) to identify seismic faults. However, it remains challenging for geophysical interpreters to quickly train an efficient model on a limited number of samples for fault interpretation. In this letter, we propose a workflow that uses a CNN-based method of semantic segmentation to interpret faults by using a small training set. It requires only the selection of some 2-D seismic sections from seismic volume data for interpretation and labeling to be trained to predict faults in the entire area considered. We simplified the VGG16 model to reduce training time and modified it to improve its performance. We used convolution layers instead of fully connected layers at the end of the network to implement the end-to-end classification of seismic images, and adopted dilation convolution to increase the receptive field and hybrid dilation convolution to avoid problems in it. We also applied the atrous spatial pyramid pooling (ASPP) module to further enhance the effects of segmentation. The results were subjected to postprocessing for refinement. The promising performance of the proposed method was verified on a set of real seismic data.

In recent years, deep convolutional neural networks (DCNNs) have made significant progress in cloud detection tasks, and the detection accuracy has been greatly improved. However, most existing CNN-based models have high computational complexity, which limits their practical application, especially for spaceborne optical remote sensing. In addition, most of the methods cannot make adaptive adjustments based on the structural information of the clouds, and blurred boundaries often occur in the detection results. In order to address these problems, this article proposes a lightweight network (DABNet) to achieve high-accuracy detection of complex clouds, not only a clearer boundary but also lower false-alarm rate. Specifically, a deformable context feature pyramid module is proposed to improve the adaptive modeling capability of multiscale features. Besides, a boundary-weighted loss function is designed to direct the network to focus on cloud boundary information and optimize the relevant detection results. The proposed method has been validated on two data sets: the public GF-1 WFV benchmark and our self-built GF-2 cloud detection data set with higher spatial resolution. The experimental results exhibit that DABNet achieves state-of-the-art performance while only using 4.12M parameters and 8.29G multiadds.

Deep learning for 3-D point cloud perception has been a very active research topic in recent years. A current trend is toward the combination of the semantically strong and the fine-grained information from different scales of intermediate representations to boost network generalization power and robustness against scale variation. One prominent challenge is how to effectively conduct the allocation of multiple scales of information. In this letter, we propose a module, named adaptive pyramid context fusion (APCF), to adaptively capture scales of contextual information from a multiscale feature pyramid for the point cloud. The APCF module reweights and aggregates the features from different levels in the feature pyramid via a softmax attention strategy. The allocation of information is adaptively conducted level by level from bottom to up first and then from top to bottom. To ensure both effectiveness and efficiency, we propose a multiscale context-aware network APCF-Net through applying our proposed APCF to the PointConv architecture. Experiments demonstrate that APCF-Net surpasses its vanilla counterpart by a large margin both in effectiveness and efficiency. Especially, APCF-Net outperforms state-of-the-art approaches on 3-D object classification and semantic segmentation task, with the overall accuracy of 93.3% on ModelNet40 and mIoU of 63.1% on ScanNet V2 online test.

This letter presents a new feature pyramid network (FPN) called the gated ladder-shaped FPN (GLFPN) to construct more representative feature pyramids for detecting objects of different sizes in optical remote sensing images. We first use convolution and concatenation operations to fuse three base features extracted by a ResNet backbone. We then obtain multilevel features from these base features. Finally, we use a selective gate to fuse features from multiple levels with equivalent sizes. To evaluate the effectiveness of the proposed GLFPN, we integrate it into the RetinaNet architecture by replacing the conventional FPN. The experimental results on two optical remote sensing image data sets show that the proposed method outperforms the methods compared in this letter.

Discriminative correlation filters (DCF)-based trackers have been increasingly applied to visual tracking due to their high precision while running at high frame rates. However, most recent DCF-based methods solely concentrate on learning the correlation filter with spatial information and thus do not have sufficient descriptive power to discriminate the target from the background in the complex circumstances, such as full occlusion (OCC) and rapid target variation. In this article, we introduce a novel tracking framework that exploits the relationship between the target and its spatiotemporal context to improve tracking accuracy and robustness. Especially, we present our spatiotemporal context model in a hierarchical way, where each layer of the context pyramid is a spatial correlation filter learned from different temporal instances. For gaining an accurate spatiotemporal model, we propose an optimization fusion approach that can adaptively and efficiently learn the effect of each hierarchical layer and exploit these multiple temporal levels of correlation filters for visual tracking. Moreover, an adaptive model update strategy for correlation filters is introduced into the framework to dynamically select proper hierarchical layers, which boosts the temporal diversity of the target appearance, while radically reduces the number of model parameters and guarantees the real-time performance of the tracking method. The experimental results show that, with conventional handcrafted features, our tracker achieves the best success rates among available state-of-the-art trackers with handcrafted features, and provides state-of-the-art performance comparable to those of deep-learning-based trackers on OTB-2013, OTB-2015, VOT-2016, and UAV-20L benchmarks but runs significantly faster than deep trackers.

Depth of field is one of the critical reasons to limit the richness of image information. Usually, in a scene with multiple targets, when the distance between each target and the lens is different, the clear scene image can be get within a certain distance range. This situation restricts the further image processing, such as semantic segmentation, object recognition and 3D reconstruction. Multi‐focus image fusion uses two or more images focused on different targets to fuse scene information, which can solve this problem to a great extent. In general, two or more multi‐focus images can cover almost all near/far targets. The fusion of more than two multi‐focus images can be accomplished by cascading the fusion results of the previous two images and the next image to be processed many times. Therefore, the paper focus on the fusion of two multi‐focus images. Inspired by this, new Pseudo‐Siamese neural network with several residual atrous convolution pyramids with multi‐level perception ability to perceive the multi‐level features and consistency relations of multi‐focus image pairs is proposed, and multi‐layer residual blocks are used to fuse the extracted features. In this process, the residual of the groundtruth and the generated image will be learned. Finally, a fully focused image without blur will be generated. After several ablation experiments and comparison experiments with other methods, the results show that the performance of the method proposed in this paper is state‐of‐the‐art, and overall better than other methods, which are advanced.

The road extraction from high resolution remote sensing image is of great importance in a variety of applications. Recently, the abundant deep convolutional neural networks are proposed for road extraction task. However, the existing approaches lack suitable strategy to utilize multiple views road features for road extraction, which fails to extract road with smooth appearance and accurate boundary under complex scenes. To address this problem, the authors propose a novel deep residual and pyramid pooling network (DRPPNet) for extracting road regions from high resolution remote sensing image. The DRPPNet consists of three parts: deep residual network (DResNet), pyramid pooling module (PPM) and deep decoder (DD). Specially, the DResNet uses several residual blocks to extract deep road features from input images, which can enhance learning ability of DRPPNet and avoid gradient vanish. Then, PPM is proposed to fuse road features from multiple views and it aims to address disadvantage of single view feature. Finally, the DD is used to recover size of feature maps to input size. Extensive experiments on two challenging road datasets demonstrate that proposed method outperforms the state‐of‐the‐art methods greatly on performance of road extraction task.

The excavated areas formed by human activities often indicate the destruction of the ecological environment and topography, so it is of great significance to carry out dynamic monitoring. For the c...

Recent progress on remote sensing (RS) scene classification is substantial, benefiting mostly from the explosive development of convolutional neural networks (CNNs). However, different from the natural images in which the objects occupy most of the space, objects in RS images are usually small and separated. Therefore, there is still a large room for improvement of the vanilla CNNs that extract global image-level features for RS scene classification, ignoring local object-level features. In this article, we propose a novel RS scene classification method via enhanced feature pyramid network (EFPN) with deep semantic embedding (DSE). Our proposed framework extracts multiscale multilevel features using an EFPN. Then, to leverage the complementary advantages of the multilevel and multiscale features, we design a DSE module to generate discriminative features. Third, a feature fusion module, called two-branch deep feature fusion (TDFF), is introduced to aggregate the features at different levels in an effective way. Our method produces state-of-the-art results on two widely used RS scene classification benchmarks, with better effectiveness and accuracy than the existing algorithms. Beyond that, we conduct an exhaustive analysis on the role of each module in the proposed architecture, and the experimental results further verify the merits of the proposed method.

The current target tracking and detection algorithms often have mistakes and omissions when the target is occluded or small. To overcome the defects, this paper integrates bi-directional feature pyramid network (BiFPN) into cascade region-based convolutional neural network (R-CNN) for live object tracking and detection. Specifically, the BiFPN structure was utilized to connect between scales and fuse weighted features more efficiently, thereby enhancing the network’s feature extraction ability, and improving the detection effect on occluded and small targets. The proposed method, i.e., Cascade R-CNN fused with BiFPN, was compared with target detection algorithms like Cascade R-CNN and single shot detection (SSD) on a video frame dataset of wild animals. Our method achieved a mean average precision (mAP) of 91%, higher than that of SSD and Cascade R-CNN. Besides, it only took 0.42s for our method to detect each image, i.e., the real-time detection was realized. Experimental results prove that the proposed live object tracking and detection model, i.e., Cascade R-CNN fused with BiFPN, can adapt well to the complex detection environment, and achieve an excellent detection effect.

Purpose This study aims to investigate the effect of porous material (clay pots) and it is facing on the productivity performance of a pyramid type solar still. The clay pots are placed in the basin facing up and facing down. The numbers of clay pots considered were 9 and 25, and its performance was compared with normal (0 clay pots) solar still. Design/methodology/approach The pyramid solar water distillation system has been designed, fabricated and tested under the actual environmental conditions of Kanchikacherla (16.6834 0N, 80.3904 0E), Andhra Pradesh, India. The solar still is used to produce the fresh water and hot water simultaneously from the brackish (i.e. containing dissolved salts) feed water for domestic applications. From open literature, it was established that the rate of evaporation is higher when the flowing water is held for a longer duration on the black color absorber plate, thereby leading to an increase in productivity of freshwater. Therefore, the pyramid solar still has been tested for smooth absorber plate and the absorber plate with porous heat storage material. Findings The porous material increases the production rate of freshwater compared to a base plate. However, the pyramid still with clay pots has higher productivity at a lower temperature because of the porosity effect. Originality/value The total dissolved solids, electrical conductivity and pH of the distilled water and the saline water have also been measured and compared.

In the field of automatic driving, identifying vehicles and pedestrians is the starting point of other automatic driving techniques. Using the information collected by the camera to detect traffic targets is particularly important. The main bottleneck of traffic object detection is due to the same category of targets, which may have different scales. For example, the pixel‐level of cars may range from 30 to 300 px, which will cause instability of positioning and classification. In this paper, a multi‐dimension feature pyramid is constructed in order to solve the multi‐scale problem. The feature pyramid is built by developing a U‐shaped module and using a cascade‐method. In order to verify the effectiveness of the U‐shaped module, we also designed a new one‐shot detector U‐FPNDet. The model first extracts the basic feature map by using the basic network and constructs the multi‐dimension feature pyramid. Next, a pyramid pooling module is used to get more context information from the scene. Finally, the detection network is run on each level of the pyramid to obtain the final result by NMS. By using this method, a state‐of‐the‐art performance is achieved on both detection and classification on commonly used benchmarks.

A photoelectric autocollimator with high accuracy and extended measurement range based on the quadrangular pyramid is proposed, and the corresponding algorithms are also deduced. A new image processing algorithm has been proposed to improve the accuracy, and the corresponding errors are also estimated, the error does not exceed half a pixel when the distance between the marks more than two radii. The experimental results have verified that the measurement range of the proposed two-dimensional (2D) quadrangular pyramid photoelectric autocollimator can be increased $$\sqrt{2}$$ times than that of the flat mirror photoelectric autocollimator from 10′ to 15′. The accuracy is better than 1″ when the deflection is less than 15′.

Although deep learning has been widely used for dense crowd counting, it still faces two challenges. Firstly, the popular network models are sensitive to scale variance of human head, human occlusions, and complex background due to repeated utilization of vanilla convolution kernels. Secondly, the vanilla feature fusion often depends on summation or concatenation, which ignores the correlation of different features leading to information redundancy and low robustness to background noise. To address these issues, a multi‐scale feature pyramid network (MFP‐Net) for dense crowd counting is proposed in this paper. The proposed MFP‐Net makes two contributions. Firstly, the feature pyramid fusion module is designed that adopts rich convolutions with different depths and scales, not only to expand the receptive field, but also to improve the inference speed of models by using parallel group convolution. Secondly, a feature attention‐aware module is added in the feature fusion stage. The module can achieve local and global information fusion by capturing the importance of the spatial and channel domains to improve model robustness. The proposed MFP‐Net is evaluated on five publicly available datasets, and experiments show that the MFP‐Net not only provides better crowd counting results than comparative models, but also requires fewer parameters.

The retinal vessel is the only microvascular network that can be directly and non‐invasively observed in humans. Cardiovascular and cerebrovascular diseases, such as diabetes, hypertension, can lead to structural changes of the retinal microvascular network. Therefore, it is of great significance to study effective retinal vessel segmentation methods and assist doctors in early diagnoses with quantitative results for vascular networks. In this study, we propose a novel convolutional neural network named feature pyramid U‐Net (FPU‐Net) that extracts multiscale representations by constructing two feature pyramids both on the encoder and the decoder of U‐Net. In this representation, objects features with different size like micro‐vessels and pathology will be fused for better vessel segmentation. The experimental results show that compared with state‐of‐the‐art methods, FPU‐Net is superior in terms of accuracy, sensitivity, F1‐score, and area under the curve and capable of stronger domain generalisation across different datasets.

Vehicle detection in aerial images is an important and challenging task in the field of remote sensing. Recently, deep learning technologies have yielded superior performance for object detection in remote sensing images. However, the detection results of the existing methods are horizontal bounding boxes that ignore vehicle orientations, thereby having limited applicability in scenes with dense vehicles or clutter backgrounds. In this article, we propose a one-stage, anchor-free detection approach to detect arbitrarily oriented vehicles in high-resolution aerial images. The vehicle detection task is transformed into a multitask learning problem by directly predicting high-level vehicle features via a fully convolutional network. That is, a classification subtask is created to look for vehicle central points and three regression subtasks are created to predict vehicle orientations, scales, and offsets of vehicle central points. First, coarse and fine feature maps outputted from different stages of a residual network are concatenated together by a feature pyramid fusion strategy. Upon the concatenated features, four convolutional layers are attached in parallel to predict high-level vehicle features. During training, task uncertainty learned from the training data is used to weight loss function in the multitask learning setting. For inferencing, oriented bounding boxes are generated using the predicted vehicle features, and oriented nonmaximum suppression (NMS) postprocessing is used to reduce redundant results. Experiments on two public aerial image data sets have shown the effectiveness of the proposed approach.

Pedestrian attributes recognition is an important issue in computer vision and has a special role in the field of video surveillance. The previous methods presented to solve this issue are mainly based on multi‐label end‐to‐end deep neural networks. These methods neglect to apply attributes for defining local feature areas and they suffer from the problems of the bounding box presence. A new framework for jointly human semantic parsing and pedestrian attribute recognition to achieve effective attribute recognition is proposed. By extracting human parts via semantic parsing, both semantic and spatial information can be explored with eliminating of background. The framework also uses multi‐scale features to employ rich details and contextual information through proposed attribute recognition‐bidirectional feature pyramid network. For baseline network that has a significant impact on the performance, EfficientNet‐B3 is selected as a baseline network from The EfficientNet family which provides an appropriate trade‐off between the three factors of CNNs scaling (depth/width/resolution). Finally, the proposed framework is tested on datasets PETA, RAP and PA‐100k. Experimental results show that our method has superior performance in both mean accuracy and instance‐based metrics compared to state‐of‐the‐art results.

Convolutional neural networks (CNNs) have facilitated impressive improvements in the semantic segmentation of very high-resolution (VHR) remote sensing images. The success of semantic segmentation depends on an effective receptive field (RF) large enough to cover the entire object. Popular methods to enlarge the effective RF include dilated filters, subsampling operations, and stacking layers. Unfortunately, the methods are inefficient or able to cause grid artifacts. Moreover, although the object sizes vary greatly in remote sensing images, the size of the RF cannot reach a compromise between small and large objects. To tackle these problems, we propose adaptive effective receptive convolution (AERFC) for VHR remote sensing images. AERFC adaptively controls the sampling location of convolution and automatically adjusts the effective RF without significantly increasing the parameter number and computational cost. Thus, AERFC reduces the training difficulty, decreases overfitting risk, and reserves details in VHR images. AERFC is also integrated with spatial pyramid pooling (SPP) to aggregate diverse multiscale features for exploring contextual information. Experimental results of the quantitative and qualitative evaluation over four benchmark data sets show that AERFC outperforms state-of-the-art methods.

Parkinson's disease (PD) is currently an ongoing challenge in daily clinical medicine. To reduce diagnosis time and arduousness and even assess PD levels, a temporal pyramid attention‐based spatiotemporal (PAST) fusion model for diagnosis of PD is produced by using gait data from ground reaction forces. This model is innovative in two aspects. First, by using the temporal pyramid attention module, multiscale temporal attention is obtained from raw sequences. Second, 1D convolutional neural network and bidirectional long short‐term memory layers are used together to learn spatial fusion features from multiple channels in the spatial domain to obtain multichannel, multiscale fusion features. Experiments are performed on the PhysioBank data set, and the results show that the proposed PAST model outperforms other state‐of‐the‐art methods on classification results. This model can assist in the diagnosis and treatment of PD by using gait data.

Single image de‐raining based on convolutional neural network (CNN) has made considerable progress in recent years. However, usually the de‐rained result has dark artifacts and image textures tend to be over‐smoothed. In this paper, a pyramid non‐local enhanced residual dense network is proposed to reduce such distortion. Firstly, the down‐sampled images are input into the Laplacian pyramid, which can extract the overall and partial texture clues, and subsequently a set of images of different scales are produced. Secondly, these images are fed into a non‐local enhanced residual dense block, which can not only capture long‐distance dependencies of feature maps, but also fully utilizes the hierarchical features in every dense block, leading to high accuracy of rain streaks extraction and better preservation of image edge detail. Finally, the de‐rained image is gradually restored by Gaussian reconstruction pyramid. Experimental results on both synthetic data and real‐world data show that the artifacts distortion is obviously reduced by the proposed network. And the quality of de‐rained image is significantly improved compared with the state‐of‐the‐art methods.

For the time being, there are many groundbreaking object detection frameworks used in natural scene images. These algorithms have good detection performance on the data sets of open natural scenes. However, applying these frameworks to remote sensing images directly is not very effective. The existing deep-learning-based object detection algorithms still face some challenges when dealing with remote sensing images because these images usually contain a number of targets with large variations of object sizes as well as interclass similarity. Aiming at the challenges of object detection in optical remote sensing images, we propose an end-to-end cross-scale feature fusion (CSFF) framework, which can effectively improve the object detection accuracy. Specifically, we first use a feature pyramid network (FPN) to obtain multilevel feature maps and then insert a squeeze and excitation (SE) block into the top layer to model the relationship between different feature channels. Next, we use the CSFF module to obtain powerful and discriminative multilevel feature representations. Finally, we implement our work in the framework of Faster region-based CNN (R-CNN). In the experiment, we evaluate our method on a publicly available large-scale data set, named DIOR, and obtain an improvement of 3.0% measured in terms of mAP compared with Faster R-CNN with FPN.

With great significance in military and civilian applications, detecting indistinguishable small objects in wide-scale remote sensing images is still a challenging topic. In this letter, we propose a specially optimized one-stage network (SOON) focusing on extracting spatial information of high-resolution images by understanding and analyzing the combination of feature and semantic information of small objects. The SOON model consists of feature enhancement, multiscale detection, and feature fusion. The first part is implemented by constructing a receptive field enhancement (RFE) module and incorporating it into the network’s specific parts where the information of small objects mainly exists. The second part is achieved by four detectors with different sensitivities, which access to the fused and enhanced features to enable the network to make full use of features in different scales. The third part consolidates the high-level and low-level features by adopting upsampling, concatenation, and convolution operations to build a feature pyramid structure, which explicitly yields strong feature representation and semantic information. In addition, we introduce the soft-nonmaximum suppression to preserve accurate bounding boxes in the postprocessing stage for densely arranged objects. Note that the split and merge strategy and the multiscale training strategy are employed. Extensive experiments and thorough analysis are performed on the NorthWestern Polytechnical University Very-High-Resolution (NWPU VHR)-10-v2 data set and the airplane, car and ship (ACS) data set as compared with several state-of-the-art methods. The satisfactory performance in experiments verifies the effectiveness of the design and optimization.

Automatic facial expression recognition, which has many applications such as drivers, patients, and criminals' emotions recognition, is a challenging task. This is due to the variety of individuals and facial expression variability in different conditions, for instance, gender, race, colour and changing illumination. In addition, there are many regions in a face image such as forehead, mouth, eyes, eyebrows, nose, cheeks and chin, and extracting features of all these regions are expensive in terms of computational time. Each of the six basic emotions of anger, disgust, fear, happiness, sadness and surprise affect some regions more than the other regions. The goal of this study is to evaluate the performance of enhanced local binary pattern, pyramid histogram of oriented gradients feature‐extraction algorithms and their combination in terms of recognition accuracy, feature vector length and computational time on one, two and three combined regions of a face image. Our experimental results show that the combination of both feature‐extraction algorithms yields an average recognition accuracy of 95.33% using three regions, that is, the mouth, nose and eyes on Cohn–Kanade dataset. Besides, the mouth region is the most important part in terms of accuracy in comparison to eyes, nose and combination of both eyes and nose regions.

A prerequisite for solving many agricultural problems is to accurately estimate the area affected by crop disasters and its severity rating. In this letter, we propose a pipeline to segment the drought area and distinguish the severity rating of the maize on RGB images accessed by an unmanned aerial vehicle (UAV) through a semantic segmentation method based on deep learning. First, the ground truth is created through expert evaluation and visual interpretation with the aid of the Normalized Difference Vegetation Index (NDVI). The neural network structure that was used is based on U-Net. Some structural and parameter improvements on U-net were made using SE-ResNeXt-50 as the backbone with the atrous spatial pyramid pooling (ASPP) module. By using RGB images as the input of the neural network for training, the final trained network can work on RGB images captured by a consumer UAV. The experimental results showed that our pipeline achieved an F1-score of 0.9034 and a Jaccard index of 0.8287 on the test set.

Proposal-free instance segmentation methods mainly generate instance-agnostic semantic segmentation labels and instance-aware features to group pixels into different object instances. However, previous methods mostly employ separate modules for these two sub-tasks and require multiple passes for inference. In addition to the lack of efficiency, previous methods also failed to perform as well as proposal-based approaches. To this end, this work proposes a single-shot proposal-free instance segmentation method that requires only one single pass for prediction. Our method is based on learning an affinity pyramid, which computes the probability that two pixels belong to the same instance in a hierarchical manner. Moreover, incorporating with the learned affinity pyramid, a novel cascaded graph partition (CGP) module is presented to fuse the two predictions and segment instances efficiently. As an additional contribution, we conduct an experiment to demonstrate the benefits of proposal-free methods in capturing detailed structures from finely annotated training examples. Our approach is evaluated on the Cityscapes and COCO datasets and achieves state-of-the-art performance.

Disease is one of the main factors affecting crop growth. How to reflect the external morphological features of the disease and completely retain the color and texture information of the disease area is one of the key research issues for crop disease segmentation. Meanwhile, aiming at the problem of low segmentation accuracy with traditional convolution neural network-based methods in the crop disease leaf image, this paper proposes a spatial pyramid-oriented encoder-decoder cascade convolution neural network for crop disease leaf segmentation. The network consists of a region disease detection network and a region disease segmentation network. Region disease detection network is a kind of network combining cascade convolution neural network with spatial pyramid. This method connects the three-level convolution neural network model, where the structure of the three-level neural network model varies from simple to complex. Different crop disease leaf features are extracted from the different neural network levels. And images are screened to complete the detection of crop disease leaf. What’s more, a space pyramid pooling layer is added to each network level. This pooling strategy does not require fixed size input, which increases the size selection of input model. The region segmentation network is established based on the Encoder-Decoder structure. The multi-scale convolution kernel is used to improve the local receptive field of the original convolution kernel and accurately segment the crop disease leaf area. Finally, we conduct experiments on the crop disease leaf images under different conditions, the results show that the proposed method has higher segmentation accuracy. In terms of Precision, Correct segmentation, over-segmentation and under-segmentation indexes, etc., the average values of proposed method are more than 90%. The average dice similarity coefficient is over 95% under different background. Moreover, it can meticulously reflect the external morphological features of the crop disease leaf and relatively better retain the color and texture information.

Automatic surface defect inspection (ASDI) is a crucial and challenging problem in industry because it affects the quality and efficiency of production greatly. Deep learning-based methods achieve promising improvements for surface detection, but they rely on a massive training data set that is impractical in industry. In this study, we propose a generic method that works effectively even on the small size training data set. Especially, we introduce a ResMask generative adversarial network (GAN) framework that is a residual GAN to expand the insufficient defect data sets. Meanwhile, the existing inspection data sets are so much easier to detect than the data sets in the real industrial scenarios that new industrial surface defect data sets containing more diverse and challenging images are established. Then, a coarse-to-fine module (CFM) that consists of a coarse detection subnetwork and a fine segmentation subnetwork is proposed for the needs of fast detection for high-resolution images. In the coarse detection stage, spatial pyramid pooling (SPP) is utilized to increase the receptive field of the network, reduce the false detection rate, and determine the approximate location of defects. In the fine segmentation stage, the receptive field of the network is enlarged by atrous SPP (ASPP), and skip links that incorporate low-level with high-level features achieve pixelwise precision on defect segmentation. Finally, our algorithm has achieved state-of-the-art results in DAGM, HR, and WB data sets (0.859, 0.761, and 0.805, respectively) according to MIoU. It achieves an average processing time of 44.4 ms on the test images with a resolution of $512 \,\,\times \,\, 512$ and 131.6 ms for $2048 \times 2048$ images. On the DAGM data set, the detection accuracy of mean intersection over union (MIoU) reaches 0.859.

Automatic object detection is a fundamental but challenging problem in very high-resolution (VHR) satellite images’ interpretation. Recently, context-based feature pyramid detection architecture has been proposed, which generates high-quality fusion feature levels for multiscale object detection and significantly improves the accuracy of traditional detection frameworks. However, in feature pyramid architecture, small objects are easily lost in deep feature levels, and the context cues will be weakened simultaneously. Moreover, each feature level in the pyramids is simply constructed from single-level layers of the backbone, which are not representative enough for multiscale object detection. In this letter, an end-to-end multiscale convolutional neural network (MSCNN) is proposed, which is based on a unified multiscale backbone named EssNet for extracting features of diverse-scale objects in HRS images. The proposed EssNet backbone maintains the resolution of deep feature levels, which improves the capability of multiscale objects’ feature expression. Meanwhile, a dilated bottleneck structure is introduced into the backbone, which generates high-quality semantic features and improves the prediction capability of multiscale objects. Finally, the whole network can be optimized end-to-end by minimizing a multitask loss. Experiments on publicly available NWPU VHR-10 benchmark and demonstrate that the proposed method outperforms several state-of-the-art detection approaches.

Accurate and efficient extraction of road marking plays an important role in road transportation engineering, automotive vision, and automatic driving. In this article, we proposed a dense feature pyramid network (DFPN)-based deep learning model, by considering the particularity and complexity of road marking. The DFPN concatenated its shallow feature channels with deep feature channels so that the shallow feature maps with high resolution and abundant image details can utilize the deep features. Thus, the DFPN can learn hierarchical deep detailed features. The designed deep learning model was trained end to end for road marking instance extraction with mobile laser scanning (MLS) point clouds. Then, we introduced the focal loss function into the optimization of deep learning model in road marking segmentation part, to pay more attention to the hard-classified samples with a large extent of background. In the experiments, our method can achieve better results than state-of-the-art methods on instance segmentation of road markings, which illustrated the advantage of the proposed method.

Scene text detection methods have achieved significant progresses. However, stack-omnidirectional text dilemma, under-segmentation of very close text words, and over-segmentation of arbitrary-shape long text lines, are still main challenges. Motivated by these problems, we proposed a two stage method called omnidirectional pyramid mask proposal text detector (OPMP). OPMP removes anchor mechanism that requires heuristic non-maximum suppress processing. Instead, it uses an effective pyramid lengthwise and sidewise residual sequence modeling method to produce arbitrary-shape proposals. To accurately extract the features of text shape, OPMP enhances the backbone layers by a multiple arbitrary-shape fitting mechanism. Finally, a multi-grain text classification module is proposed, which reclassifies each text region robustly. Comprehensive ablation studies demonstrate the effectiveness of each proposed component. In addition, experiments on various benchmarks, including ICDAR2015, MLT, MSRA-TD500, CTW1500, and Total-text, show that our method outperforms previous state-of-the-art methods.

Small fasteners are widely used in a freight train. However, due to the complex working environment, the missing of fasteners always happens, causing traffic accidents and property loss. Thus, detection of the missing of small fasteners is essential for the safety of freight trains. This article takes three critical missing small fasteners as examples, pin-missing, bolt-missing, and rivet-missing defects, and proposes a two-stage small defect detection model. First, a copy–pasting method is used to augment the defects in the training image, then the ResNet-101 backbone is adopted to extract defect features, and a feature pyramid network is used to construct a feature pyramid. After obtaining defect features, an improved region proposal network is introduced to generate defect proposals. Finally, a fully convolutional neural network is trained to operate pixelwise segmentation. The experimental results on manually collected fastener defect data set show that our model possesses high precision and efficiency on the inspection of pin-missing, bolt-missing, and rivet-missing defects. Besides, the proposed model could be used for similar small defects detection.

In recent years, the detection of distinctive objects in stereoscopic 3D images has drawn increasing attention. Unlike 2D salient object detection, salient object detection in stereoscopic 3D images is highly challenging. Hence, we propose a novel Deep Convolutional Residual Autoencoder (DCRA) for end-to-end salient object detection in stereoscopic 3D images. The core trainable architecture of the salient object detection model employs raw stereoscopic 3D images as the inputs and their corresponding ground truth saliency masks as the labels. A convolutional residual module is applied to both the encoder and the decoder as a basic building block in the DCRA, and long-range skip connections are employed to bypass the equal-sized feature maps between the encoder and the decoder. To explore the complex relationships and exploit the complementarity between RGB (photometric) and depth (geometric) information, multiple feature map fusion modules are constructed. These modules integrate texture and structure information between the RGB and depth branches of the encoder and fuse their features over several multiscale layers. Finally, to efficiently optimize DCRA parameters, a supervision pyramid based on boundary loss and background prior loss is adopted, which employs supervised learning over the multiscale layers in the decoder to prevent vanishing gradients and accelerate the training at the fusion stage. We compare the proposed DCRA with state-of-the-art methods on two challenging benchmark datasets. The results of these experiments demonstrate that our proposed DCRA performs favorably against the comparison models.

This paper introduces a novel bi-directional con-volutional framework to cope with the large-variance scale problem in scene text detection. Due to the lack of scale normalization in recent CNN-based methods, text instances with large-variance scale are activated inconsistently in feature maps, which makes it hard for CNN-based methods to accurately locate multi-size text instances. Thus, we propose the relationship network (R-Net) that maps multi-scale convolutional features to a scale-invariant space to obtain consistent activation of multi-size text instances. Firstly, we implement an FPN-like backbone with a Spatial Relationship Module (SPM) to extract multi-scale features with powerful spatial semantics. Then, a Scale Relationship Module (SRM) constructed on feature pyramid propagates contextual scale information in sequential features through a bi-directional convolutional operation. SRM supplements the multi-scale information in different feature maps to obtain consistent activation of multi-size text instances. Compared with previous approaches, R-Net effectively handles the large-variance scale problem without complicated post processing and complex hand-crafted hyperparameter setting. Extensive experiments conducted on several benchmarks verify that our R-Net obtains state-of-the-art performance on both accuracy and efficiency. More specifically, R-Net achieves an F-measure of 85.6% at 21.4 frames/s and an F-measure of 81.7% at 11.8 frames/s for ICDAR 2015 and MSRA-TD500 datasets respectively, which is the latest SOTA. The code is available on https://github.com/wangyuxin87/R-Net .

Surface defect segmentation of no-service rail is important for its quality assessment. There are several challenges of uneven illumination, complex background, and difficulty of sample collection for no-service rail surface defects (NRSDs). In this article, we propose an acquisition scheme with two lamp light and color scan line charge-coupled device (CCD) to alleviate uneven illumination. Then, a multiple context information segmentation network is proposed to improve NRSD segmentation. The network makes full use of context information based on dense block, pyramid pooling module, and multi-information integration. Besides, the attention mechanism is applied to optimize extracted information by filtering noise. For the problem of real sample shortage, we propose to utilize artificial samples to train the network. And an NRSD data set NRSD-MN is built with artificial NRSDs and natural NRSDs. Experimental results show that our method is feasible and has a good segmentation effect on artificial and natural NRSDs.

The segmentation effect of welding zone is vulnerable to image quality and target shape, which brings difficulty for the radiographic inspection of water-cooled pipe. A boundary-aware semantic segmentation algorithm is proposed to improve the precision and robustness of the welding zone inspection system. First, a convolutional neural network and atrous spatial pyramid pooling are employed to generate the feature map of the input image. Second, a boundary distance field (BDF) regression module is designed to predict the distance fields of the inner and outer boundaries, which can reflect the continuous boundary information of welding zone. Then, spatial attention weight is calculated according to the regressed boundary distance fields, and the calculated weight is fused into the feature map to enhance the significance of welding zone. Finally, the segmentation mask is predicted based on the fused feature map. Experiments indicate that the guidance of boundary information in the semantic segmentation algorithm can effectively reduce the interference of nonweld zones. The segmentation accuracy of the welding zone is 93.70%, and average detection time of a single image is less than 11 ms.

In recent decades, automatic diagnosis using machine-learning techniques have been the focus of research. Mycobacterium Tuberculosis (TB) is a deadly disease that has plagued most developing countries presents a problem that can be tackled by automatic diagnosis. The World Health Organization (WHO) set years 2030 and 2035 as milestones for a significant reduction in new infections and deaths although lack of well-trained professionals and insufficient or fragile public health systems (in developing countries) are just some of the major factors that have slowed the eradication of the TB endemic. Deep convolutional neural networks (DCNNs) have demonstrated remarkable results across problem domains dealing with grid-like data (i.e., images and videos). Traditionally, a methodology for detecting TB is through radiology combined with previous success DCNN have achieved in image classification makes them the perfect candidate to classify Chest X-Ray (CXR) images. In this study, we propose three types of DCNN trained using two public datasets and another new set which we collected from Konya Education and Research Hospital, Konya, Turkey. Also, the DCNN architectures were integrated with an extra layer called Spatial Pyramid Pooling (SPP) a methodology that equips convolutional neural networks with the ability for robust feature pooling by using spatial bins. The result indicates the potential for an automated system to diagnose tuberculosis with accuracies above a radiologist professional.

Surface defect detection is a critical task in industrial production process. Nowadays, there are lots of detection methods based on computer vision and have been successfully applied in industry, they also achieved good results. However, achieving full automation of surface defect detection remains a challenge, due to the complexity of surface defect, in intraclass. While the defects between interclass contain similar parts, there are large differences in appearance of the defects. To address these issues, this article proposes a pyramid feature fusion and global context attention network for pixel-wise detection of surface defect, called PGA-Net. In the framework, the multiscale features are extracted at first from backbone network. Then the pyramid feature fusion module is used to fuse these features into five resolutions through some efficient dense skip connections. Finally, the global context attention module is applied to the fusion feature maps of adjacent resolution, which allows effective information propagate from low-resolution fusion feature maps to high-resolution fusion ones. In addition, the boundary refinement block is added to the framework to refine the boundary of defect and improve the result of the prediction. The final prediction is the fusion of the five resolutions fusion feature maps. The results of evaluation on four real-world defect datasets demonstrate that the proposed method outperforms the state-of-the-art methods on mean intersection of union and mean pixel accuracy (NEU-Seg: 82.15%, DAGM 2007: 74.78%, MT_defect: 71.31%, Road_defect: 79.54%).

Visual surface defect inspection for metal part has become a rapidly developing research field within the last decade. But due to the variances of defect shapes and scales, the inspection of tiny and irregular shape defects has posed challenges on the robustness of the inspection model. In this context, a deep learning method based on the deformable convolution and concatenate feature pyramid (CFP) neural networks is proposed to improve the inspection. We design a deformable convolution layer in the neural networks as an attention mechanism to adaptively extract the features of defect shape and location, which enhances the inspection of the defects with large shape variances. We also merge the multiple hierarchical features collected from different deformable convolution layers by the CFP, which improves the inspection of tiny defects. The results show that the proposed method has a better generalization ability than traditional convolution neural networks.