Search

High spatial resolution (HSR) imagery scene classification has become a hot research topic in remote sensing. Scene classification method based on the handcrafted features, such as the bag-of-visual-words (BoVW) model, describes an image by extracting local features of the scene and mapping them to the dictionary space, but usually uses a shallow structure and loses the spatial distribution characteristics of the scene. The method based on deep learning extracts hierarchical features to describe the scene, which can maintain the spatial position information well. However, deep features in different levels have scale recognition restrictions for multi-scale ground objects, and cannot understand complex scenes well. In this paper, the multi-level convolutional pyramid semantic fusion (MCPSF) framework is proposed for HSR imagery scene classification. Differing from previous scene classification methods, which integrate the feature of different levels directly, of which the fusion features have large differences in both sparsity and eigenvalue magnitude, MCPSF integrates multi-level semantic features extracted by BoVW model and convolutional neural network (CNN) model. In MCPSF, two convolution pyramid feature expression strategies are proposed to enhance the ability of capturing multi-scale land objects, i.e., local and convolutional pyramid based BoVW (LCPB) model and local and convolutional pyramid based pooling-stretched (LCPP) model. The effectiveness of the proposed method is verified on 21-class UC Merced (UCM) dataset and 30-class Aerial Image Dataset (AID). The framework was also transferred toa case study of scene annotation in Wuhan. The proposed framework significantly improves the performance when compared with other state-of-the-art methods.

Semantic segmentation is a task that covers most of the perception needs of intelligent vehicles in an unified way. Recent studies witnessed that attention mechanisms achieve impressive performance in computer vision task. Current attention mechanisms based segmentation methods differ with each other in position and form of the attention mechanism, and perform differently in practice. This paper firstly introduces the effectiveness of multi-scale context features and attention mechanisms in segmentation tasks. We find that multi-scale and channel attention can play a vital role in constructing effective context features. Based on this analysis, this paper proposes an efficient attention pyramid network (EAPNet) for semantic segmentation. Specifically, to efficient handle the problem of segmenting objects at multiple scales, we design efficient channel attention pyramid (ECAP) which employ atrous convolution with channel attention in cascade or in parallel to capture multi-scale context by using multiple atrous rates. Furthermore, we propose a residual attention fusion block (RAFB), whose purpose is to simultaneously focus on meaningful low-level feature maps and spatial location information. At the same time, we will explore different channel attention modules and spatial attention modules, and describe their impact on network performance. We empirically evaluate our EAPNet on two semantic segmentation datasets, including PASCAL VOC 2012 and Cityscapes datasets. Experimental results show that without MS COCO pre-training and any post-processing, EAPNet achieved 81.7% mIoU on the PASCAL VOC 2012 validation set. With deeplabv3+ as the benchmark, EAPNet improve the model performance of more than 1.50% mIoU.

Disease is one of the main factors affecting crop growth. How to reflect the external morphological features of the disease and completely retain the color and texture information of the disease area is one of the key research issues for crop disease segmentation. Meanwhile, aiming at the problem of low segmentation accuracy with traditional convolution neural network-based methods in the crop disease leaf image, this paper proposes a spatial pyramid-oriented encoder-decoder cascade convolution neural network for crop disease leaf segmentation. The network consists of a region disease detection network and a region disease segmentation network. Region disease detection network is a kind of network combining cascade convolution neural network with spatial pyramid. This method connects the three-level convolution neural network model, where the structure of the three-level neural network model varies from simple to complex. Different crop disease leaf features are extracted from the different neural network levels. And images are screened to complete the detection of crop disease leaf. What’s more, a space pyramid pooling layer is added to each network level. This pooling strategy does not require fixed size input, which increases the size selection of input model. The region segmentation network is established based on the Encoder-Decoder structure. The multi-scale convolution kernel is used to improve the local receptive field of the original convolution kernel and accurately segment the crop disease leaf area. Finally, we conduct experiments on the crop disease leaf images under different conditions, the results show that the proposed method has higher segmentation accuracy. In terms of Precision, Correct segmentation, over-segmentation and under-segmentation indexes, etc., the average values of proposed method are more than 90%. The average dice similarity coefficient is over 95% under different background. Moreover, it can meticulously reflect the external morphological features of the crop disease leaf and relatively better retain the color and texture information.

Fruit detection plays a vital role in harvesting robot platforms. However, complicated environment attributes such as illumination variation, occlusion, have made fruit detection a challenging task. A robust YOLOMuskmelon model that is accurate and fast was proposed to solve detection difficulties. The YOLOMuskmelon model incorporated ReLU activated ResNet43 backbone with new 2,3,4,3,2 residual block arrangement, spatial pyramid pooling(SPP), complete Intersection over Union (CIoU) loss, feature pyramid network(FPN), and distance Intersection over Union-Non Maximum Suppression(DIoU-NMS) to improve detection performance. The obtained average precision (AP) results of YOLOMuskmelon at 89.6% is greater than YOLOv3 at 82.3%, YOLOResNet50 at 85.5%, but less than YOLOv4 at 91.6%. However, the detection speed of YOLOMuskmelon at 96.3 frame per second(fps) outperformed YOLOv3 at 56.6fps, YOLOv4 at 54.1fps and YOLOResNet50 at 71.2fps. Meanwhile, the YOLOMuskmelon which is 56.1% faster than YOLOv4 model showed a better generalization and real-time fruit harvesting robots prospect.

Medical segmentation is an important but challenging task with applications in standardized report generation, remote medicine and reducing medical exam costs by assisting experts. In this paper, we exploit time sequence information using a novel spatio-temporal recurrent deep learning network to automatically segment the thyroid gland in ultrasound cineclips. We train a DeepLabv3+ based convolutional LSTM model in four stages to perform semantic segmentation by exploiting spatial context from ultrasound cineclips. The backbone DeepLabv3+ model is replicated six times and the output layers are replaced with convolutional LSTM layers in an atrous spatial pyramid pooling configuration. Our proposed model achieves mean intersection over union scores of 0.427 for cysts, 0.533 for nodules and 0.739 for thyroid. We demonstrate the potential application of convolutional LSTM models for thyroid ultrasound segmentation.

We address the problem of depth estimation from a single monocular image in the paper. Depth estimation from a single image is an ill-posed and inherently ambiguous problem. In the paper, we propose an encoder-decoder structure with the feature pyramid to predict the depth map from a single RGB image. More specifically, the feature pyramid is used to detect objects of different scales in the image. The encoder structure aims to extract the most representative information from the original image through a series of convolution operations and to reduce the resolution of the input image. We adopt Res2-50 as the encoder to extract important features. The decoder section uses a novel upsampling structure to improve the output resolution. Then, we also propose a novel loss function that adds gradient loss and surface normal loss to the depth loss, which can predict not only the global depth but also the depth of fuzzy edges and small objects. Additionally, we use Adam as our optimization function to optimize our network and speed up convergence. Our extensive experimental evaluation proves the efficiency and effectiveness of the method, which is competitive with previous methods on the Make3D dataset and outperforms state-of-the-art methods on the NYU Depth v2 dataset.

Expression recognition in the wild is easily distorted by nonfrontal and asymmetry faces. In nonfrontal faces, some areas are compressed and distorted. Even after frontalization, these compressed areas may still be blurred and distort expression recognition. Additionally, asymmetrical expressions are common on half or local face areas and produce incorrect expression features. Therefore, this paper presents a half-face frontalization and pyramid Fourier frequency conversion method. Despite the location, range and intensity of incorrect expressions in nonfrontal faces being unknown, according to discrete Fourier transform, it can be proven that the frequency band of the correct expression is much larger than that of incorrect expression on the same face. This can be taken advantage of by pyramid frequency conversion, which is designed based on Fourier frequency conversion. It can adjust incorrect expression frequency in multiscales to take them out off the band-pass of the convolution operations of deep learning and be eliminated completely, whereas correct expression information is reserved. Thus, expressions can be recognized effectively.

Judging imaging quality is an important part of the maintenance of visual intelligent monitoring systems for electrical power scenes. However, accurate and efficient identification of possible abnormalities in imaging quality remains challenging. This paper proposes a novel multi-region two-stream deep architecture to improve judging abnormalities. The proposed architecture incorporates two-stream scheme and multi-region strategy to identify relevant information and explore hidden details. More specifically, in addition to color and intensity in the original images, the two-stream scheme uses high-frequency structure information from gradient images to enhance its performance. The multi-region strategy employs spatial pyramid random cropping and region fusion to handle locally non-uniform changes among categories: spatial pyramid random cropping characterizes images at different spatial pyramid levels, while region fusion focuses attention on cropped regions relevant to quality perception by using adaptive learning weights in a fully connected layer. In this way, the proposed strategy guides the framework to adequately and adaptively explore the discriminative regions hidden in the input images, and provides an end-to-end learning procedure. Experimental results demonstrate its strong performance for judging abnormalities, and the proposed method can be easily extended to the entire surveillance system.

Multi-view based 3D reconstruction aims to obtain 3D structure information of objects in space through two-dimensional images. In this paper, we propose a new multi-view stereo network that can robustly reconstruct the scene. To enhance the feature representation ability of Point-MVSNet, a pyramid attention module is introduced. Specifically, we exploit the attention mechanism for the multi-scale feature pyramid to capture larger receptive fields and richer information. Instead of constructing a feature pyramid as the input, results of the pyramid attention module at different scales are directly used for the next layer. The network eventually generates a high-quality depth estimation for 3D reconstruction from sparse to dense by an iterative refinement schedule. Experiments have been performed to evaluate 3D reconstruction quality by comparison with existing state-of-the-art methods on the DTU dataset. The experimental results indicate our method performs the best in overall quality compared with previous methods, proving the effectiveness of our method. In the end, we use the data collected by mobile devices to implement 3D reconstruction with a combination of traditional and learning-based methods, providing ideas for the 3D reconstruction technology on mobile devices.

Traditional deep learning detection network has poor effect on the detection of infrared dim and small targets on the sea in the case of interference or bad weather. In this paper, an improved dim and small infrared ship detection network based on Haar wavelet is proposed. The HaarConv module is designed based on the high-frequency features obtained by Haar wavelet decomposition, which further increases the feature extraction ability of the backbone network for small targets. Meanwhile, the HaarUp-HaarDown module is designed by using Haar forward and inverse transform, replacing the up-sampling layer of the feature pyramid network and the down-sampling layer of backbone network to retain smaller target features. Furthermore, the pseudo-label-based method enables the network to conduct semi-supervised learning, which reduces the labeling cost and improves detection accuracy while expanding the amount of training data. The above method is applied to the YOLOv5-s lightweight network and 11278 infrared images (3352 labeled) of dim and small ships are collected as a dataset. The results show that the introduction of semi-supervised training method effectively expands the training dataset, and the mAP@.5:.95 increases by 23.5%. The proposed Haar wavelet improvement method can effectively improve the detection accuracy of dim and small infrared ship targets by more than 2%, and the number of parameters increases by only about 0.02M. Compared with existing methods, the proposed method reaches the state-of-the-art result and has good generalization performance.

Enhancement of low-light images is a challenging task due to the impact of low brightness, low contrast, and high noise. The inability to collect natural labeled data intensifies this problem further. Many researchers have attempted to solve this problem using learning-based approaches; however, most models ignore the impact of noise in low-lit images. In this paper, an encoder-decoder architecture, made up of separable convolution layers that solve the issues encountered in low-light image enhancement, is proposed. The architecture is trained end-to-end on a custom low-light image dataset (LID), comprising both clean and noisy images. We introduce a unique multi-context feature extraction module (MC-FEM) where the input first passes through a feature pyramid of dilated separable convolutions for hierarchical-context feature extraction followed by separable convolutions for feature compression. The model is optimized using a novel three-part loss function that focuses on high-level contextual features, structural similarity, and patch-wise local information. We conducted several ablation studies to determine the optimal model for low-light image enhancement under noisy and noiseless conditions. We have used performance metrics like peak-signal-to-noise ratio, structural similarity index matrix, visual information fidelity, and average brightness to demonstrate the superiority of the proposed work against the state-of-the-art algorithms. Qualitative results presented in this paper prove the strength and suitability of our model for real-time applications.

Nowadays, pyramid schemes have caused extremely negative effects on people’s lives and seriously damaged the social economy. With the rapid development of network and communication technology, people’s direct or indirect social interaction is more frequent, which makes the phenomenon of pyramid schemes more serious. Therefore, it is necessary to study transmission mechanisms and transmission rules of pyramid schemes. In order to study the influence of government management and social interaction topology on the spreading of pyramid schemes, a novel SHIPR (susceptible-hesitator-involved-punished-resister) pyramid scheme spreading model is proposed on scale-free networks. The spreading dynamics of pyramid schemes are analyzed in detail by mean-field theory. Then, the basic reproduction number $R_{0}$ and equilibria are got. Theoretical analysis shows that the basic reproduction number $R_{0}$ has a great correlation with government crackdown intensity for involved individuals, the coverage rate of government anti-pyramid scheme publicity for susceptible individuals and hesitator, and social interaction topology. Furthermore, the local asymptotic stability of fraud-elimination equilibrium is analyzed based on the Routh-Hurwitz criterion, the global asymptotic stability of the fraud-elimination equilibrium is discussed by the Lyapunov function, the global attractivity of fraud-prevailing equilibrium is proved in detail by comparison principle. Finally, numerical simulations verify the theoretical analysis results.

Although person re-identification has made great progress, unsupervised cross-domain adaptive person re-identification is still a challenging problem. With no labeled data in target domain, the performance may have a significant drop. In this paper, we propose an unsupervised cross-domain adaptive person re-identification framework based on horizontal pyramid similarity learning (UHPS). Firstly, horizontal pyramid features are extracted by dividing the deep feature maps into different number of partial feature bins. These feature bins with diverse scales can incorporate not only the global information but also local information in different spatial scales, making the framework more robust in complex environment. Then, horizontal pyramid similarity learning is proposed with the mechanism of fusing together the internal similarity of the target domain and the similarity between the source domain and target domain. Finally, the unsupervised clustering algorithm DBSCAN embeded with the horizontal pyramid similarity is employed to select training data in the target domain and estimate the pseudo labels in each training iteration, for the purpose of adapting the framework to the target domain. The results on Market1501 and DukeMTMC-reID confirm that the proposed framework can adapt to the target domain effectively and outperforms the state-of-the-art unsupervised cross domain person re-identification approaches.

Object detection has always been a challenging task in the field of computer vision due to complex background, large scale variation and many small objects, which are especially pronounced for remote sensing imagery. In recent years, object detection in remote sensing with the development of deep learning has also made great breakthroughs. At present, almost all state-of-the-art object detectors rely on pre-defined anchor boxes for remote sensing imagery. However, too many anchor boxes will introduce a large number of hyper-parameters, which not only increase the memory footprint, but also increase the computational redundancy of the detection model. In contrast, we propose an anchor-free single-stage detector for remote sensing imagery object detection, avoiding a large number of hyper-parameters related to the anchor box, which usually affect the performance of the detection model. Specially, considering the large-scale differences in the objects and the characteristics of small objects in remote sensing imagery, we design a dense path aggregation feature pyramid network (DPAFPN), which can make full use of the high-level semantic information and low-level location information in remote sensing imagery, and to a certain extent, avoid information loss during shallow feature transfer. In our experiments, extensive experiments on two public remote sensing datasets DOTA, NWPU VHR-10 were conducted. The experimental results demonstrate that our detector has good performance and is meaningful for object detection in remote sensing imagery.

Aiming at the problem that it is difficult for traffic monitoring videos to detect multi-scale vehicle targets, especially small vehicle targets in complex scenarios, a codec-based vehicle detection algorithm is proposed. This algorithm is based on YOLOv3. In order to solve the multi-scale vehicle target detection problem, a new multi-level feature pyramid structure added with the codec module to detect vehicle targets of different scales. The experimental results on the KITTI dataset and UA-DETRAC dataset confirm that the algorithm in this paper has achieved good detection results for vehicle targets in various environments and at various scales in the surveillance video, especially for small vehicle targets, which can better meet the actual application demand.

This study proposes a novel face detector called DEFace that focuses on the challenging tasks of face detection to cope with a small size that is under 12 pixels and occlusions due to a mask or human body parts. This study proposed the extended feature pyramid network (FPN) module to detect small faces by expanding the range of P layer, and the network by adding a receptive context module (RCM) after each predicted feature head from the top-down pathway in the FPN architecture to enhance the feature discriminability and the robustness. Based on the FPN principle, the combination between the low- and high-resolutions are beneficial for object detection with different object sizes. Furthermore, with assistance from the RCM, the proposed method can use a broad range of context information especially for small faces. To evaluate the performance of the proposed method, various public face datasets are used such as the WIDER Face dataset, the face detection dataset and benchmark (FDDB), and the masked faces (MAFA) dataset, which consist of challenging samples such as small face regions and occlusions by hair or other people. The results indicate that DEFace can detect the face region more accurately in comparison to the other state-of-the-art methods while maintaining the processing time.

This work aims to establish visual correspondences between a pair of images depicting objects of the same semantic category. It encounters many challenges such as non-overlapping of scenes or objects, background clutter, and large intra-class variation. Existing methods handle this task with handcrafted features, which cannot effectively fit the correlations between non-overlapping images. Besides, additional training or information may be implemented into the learned features. In this paper, we propose a novel approach for semantic correspondence, which is based on deep feature representation, geometric and semantic associations between intra-class objects, and hierarchical matching selection according to the convolutional feature pyramid. Firstly, we construct the initial correspondence by developing a sparse feature matching model on the coarsest feature level, which enforces the nearest-neighbor searching under semantic and geometric consistency constraints. Further, a narrowing strategy is proposed and employed from the coarsest to the finest feature level, which hierarchically refine and optimize the correspondence. The results illustrate that this approach achieves competitive performance on the public datasets for semantic correspondence.

One of the major challenges in video object detection is drastic scale changes of objects due to camera motion. In this paper, we propose a two-path Convolutional Long Short-Term Memory (convLSTM) pyramid network designed to extract and convey multi-scale temporal contextual information in order to handle object scale changes efficiently. The proposed two-path convLSTM pyramid consists of a stack of multi-input convLSTM modules. It is updated in top-down and bottom-up pathways so that the temporal contextual information for small-to-large and large-to-small scale changes is exploited. The proposed multi-input convLSTM module uses two input feature maps of different resolutions to store and exchange temporal contextual information of different scales between neighboring convLSTM modules. The outputs of the proposed convLSTM pyramid network constitute a feature pyramid where each feature map contains multi-scale temporal contextual information from earlier frames. The proposed convLSTM pyramid can be combined with various still-image object detectors to improve the performance of video object detection. Experimental results on ImageNet VID dataset show that the proposed method achieves state-of-the-art performance and can handle scale changes efficiently in video object detection.

Salient object detection aims to obtain the most attractive objects from the input images, which severs as a pre-processing step for many image processing tasks. This paper presents a novel deep neural network design for salient object detection by formulating a pyramid spatial context module, PSC module for short, to capture the spatial context information at multiple scales. To achieve this, we first adopt convolutional operations with different dilated rates to generate the feature maps with different respective fields, and then use the two-round recurrent translations to explore multiple types of spatial context features on these feature maps. By further inserting this module in a deep network, namely PSCNet, we are able to optimize the network in an end-to-end manner for salient object detection. We evaluate the proposed method on six public benchmark datasets by comparing it with 25 salient object detection methods. The experimental results demonstrate that our PSCNet performs favorably against all the other methods.

Aiming at the problem of weakly supervised learning in traditional Chinese painting image classification, a novel multi-instance learning algorithm based on Long and Short-Term Memory neural network with attention mechanism (ALSTM-MIL) is proposed. Firstly, by using the Pyramid Overlapping Grid Division (POGP), a multi-instance modeling scheme is designed to convert Chinese painting images into multi-instance bag, thereby transforming the problem of Chinese painting image classification into a MIL problem. Secondly, an efficient sequence generator is designed. It selects discriminative instances from the positive bags, construct a discriminative instance set (DIS), and convert multi-instance bags into equal-length ordered sequences. Thirdly, an LSTM network model with an attention mechanism is designed to perform semantic analysis on multi-instance bags to obtain their memory coding features, and then combined with the Softmax classifier to achieve semantic classification of traditional Chinese painting images. Experimental results on the Chinese painting (CP) image set show that the LSTM network built on the visual feature set is feasible, and the performance of the proposed MIL algorithm is also superior to other classification algorithms.

For real-time object detectors, accuracy and efficiency are two important considerations. In this paper, we propose a lightweight anchor-free detector, MAOD, to better balance efficiency and accuracy. Our object detector contains three components: an efficient backbone network (MobileDet), a lightweight feature pyramid structure (L-FPN) and an anchor-free per-pixel prediction method. MobileDet and L-FPN provide more accurate and faster multi-scale feature extraction. Our anchor-free per-pixel prediction method achieves efficient classification and location regression tasks. On the benchmark MS-COCO dataset, MAOD achieves 46.1% AP at the speed of 68 FPS with the input size $512\times512$ . When the input size is $800\times800$ , MAOD achieves 47.1% AP at the speed of 43 FPS. The fast version of MAOD ( $320\times320$ input size) can run at 91 FPS with 43.3% AP. Compared with other state-of-the-art object detectors, our detector has similar accuracy while maintaining extremely fast inference speed. MAOD achieves an optimal efficiency-accuracy tradeoff.

Low spatial resolution is a well-known problem for depth maps captured by low-cost consumer depth cameras. Depth map super-resolution (SR) can be used to enhance the resolution and improve the quality of depth maps. In this paper, we propose a recumbent Y network (RYNet) to integrate the depth information and intensity information for depth map SR. Specifically, we introduce two weight-shared encoders to respectively learn multi-scale depth and intensity features, and a single decoder to gradually fuse depth information and intensity information for reconstruction. We also design a residual channel attention based atrous spatial pyramid pooling structure to further enrich the feature's scale diversity and exploit the correlations between multi-scale feature channels. Furthermore, the violations of co-occurrence assumption between depth discontinuities and intensity edges will generate texture-transfer and depth-bleeding artifacts. Thus, we propose a spatial attention mechanism to mitigate the artifacts by adaptively learning the spatial relevance between intensity features and depth features and reweighting the intensity features before fusion. Experimental results demonstrate the superiority of the proposed RYNet over several state-of-the-art depth map SR methods.

The algorithm in this paper is called semantic information supplementary pyramid network(SIS-net). We choose Generative Adversarial Network (GAN) as its fundamental model. SIS-net's generator imitates the feature pyramid network (FPN) structure to recycle features spanning across multiple receptive scales to restore a sharp image. However, to solve the problem caused by the phenomenon of semantic dilution in the FPN network, we have innovatively designed a semantic information supplement (SIS) mechanism. SIS mechanism contains two essential components: semantic information storage box (info-box) and feature fusion expanding. In the process of feature fusion expanding, the semantic information features coming from the info-box is supplemented to make greater use of detailed clues. In addition, SIS-net uses the intermediate layer path to extract image features in a single time to obtain a multi-scale effect. The running speed of SIS-net has obvious advantages over other algorithms, and can basically complete real-time deblurring tasks. Extensive experiments show that our SIS-net achieves both qualitative and quantitative improvements against state-of-the-art methods. The code is available at https://github.com/yimingliu123/SIS-NET.

Facial Expression Recognition (FER) is a challenging task that improves natural human-computer interaction. This paper focuses on automatic FER on a single in-the-wild (ITW) image. ITW images suffer real problems of pose, direction, and input resolution. In this study, we propose a pyramid with super-resolution (PSR) network architecture to solve the ITW FER task. We also introduce a prior distribution label smoothing (PDLS) loss function that applies the additional prior knowledge of the confusion about each expression in the FER task. Experiments on the three most popular ITW FER datasets showed that our approach outperforms all the state-of-the-art methods.

Simultaneous Localization and Mapping (SLAM) plays an important role in the computer vision and robotic field. The traditional SLAM framework adopts a strong static world assumption for convenience of analysis. It is very essential to know how to deal with the dynamic environment in the entire industry with widespread attention. Faced with these challenges, researchers consider introducing semantic information to collaboratively solve dynamic objects in the scene. So, in this paper, we proposed a PSPNet-SLAM: Pyramid Scene Parsing Network SLAM, which integrated the Semantic thread of pyramid structure and geometric threads of reverse ant colony search strategy into ORB-SLAM2. In the proposed system, a pyramid-structured PSPNet was used for semantic thread to segment dynamic objects in combination with context information. In the geometric thread, we proposed a OCMulti-View Geometry thread. On the one hand, the optimal error compensation homography matrix was designed to improve the accuracy of dynamic point detection. On the other hand, we came up with a reverse ant colony collection strategy to enhance the real-time performance of the system and reduce its time consumption during the detection of dynamic objects. We have evaluated our SLAM in public data sheets and real-time world and compared it with ORB-SLAM2, DynaSLAM. Many improvements have been achieved in this system including location accuracy in high-dynamic scenarios, which also outperformed the other four state-of-the-art SLAM systems coping with the dynamic environments. The real-time performance has been delivered, compared with the geometric thread of the excellent DynaSALM system.

Accurate and effective object detection in remote sensing images plays an extremely important role in marine transport, environmental monitoring and military operations. Due to the powerful ability of feature representation, region-based convolutional neural networks (RCNNs) have been widely used in this field, which firstly generate candidate regions through extracted feature maps and then classify and locate objects. However, most of existing methods generally use traditional backbone networks to extract feature maps with a decreased spatial resolution because of the continuous down-sampling, which will weaken the information detected from small objects. Besides, sliding windows strategy is employed in these methods to generate fixed anchors with a preset scale on feature maps, which is inappropriate for multi-scale object detection in remote sensing images. To solve the above problems, a novel and effective object detection framework named DetNet-FPN (Feature Pyramid Network) is proposed in this paper, in which a feature pyramid with strong feature representation is created by combining feature maps of different spatial resolution, at the same time, the resolution of feature maps is maintained by involving dilation convolutions. Furthermore, to match the proposed backbone, the GA (Guided Anchoring)-RPN strategy is improved for adaptive anchor generation, this strategy simultaneously predicts the locations where the center of objects are likely to exist as well as the scales and aspect ratios at different locations. Extensive experiments and comprehensive evaluations demonstrate the effectiveness of the proposed framework on DOTA and NWPU VHR-10 datasets.

Small object is one of the primary challenges in the field of object detection, which is notably pronounced to the detection in the images from Unmanned Aerial Vehicles (UAV). Existing detectors based on deep-learning methods usually apply the feature extraction networks with a large down-sampling factor to obtain higher-level features. However, such big stride tends to make the feature information of small objects become the little point or even vanish in the low-resolution feature maps due to the limitation of pixels. Therefore, a novel structure called Multi-branch Parallel Feature Pyramid Networks (MPFPN) is proposed in this article, which aims to extract more abundant feature information of the objects with a small size. Specifically, the parallel branch is designed to recover the features that missed in the deeper layers. Meanwhile, a supervised spatial attention module (SSAM) is applied to weaken the impact of background noise inference and focus object information. Furthermore, we adopt cascade architecture in the Fast R-CNN stage for a more powerful localization capability. Experiments on the public drone-based datasets named VisDrone-DET demonstrate that our method achieves competitive performance compared with other state-of-the-art detection algorithms.

The spectral computed tomography (CT) based on photon counting detectors can collect the incident photons with different energy ranges. However, due to the low photon counts in narrow energy bin and the unhomogeneous response problem of detector cells, there are severe noise and ring artifacts in reconstructed spectral CT images. We proposed an image denoising and ring artifacts removal method via improved Fully Convolutional Pyramid Residual Network (FCPRN). In our study, we scanned a mouse specimen with spectral CT based on photon counting detector, and reconstructed mouse CT images as data set. Then we use the data set to train our network for image denoising and ring artifacts removal. Experimental results demonstrated that the proposed method could reduce noise and suppress ring artifacts of spectral CT images concurrently in different energy ranges. And the performance of the FCPRN is better than that of some networks for CT image denoising.

The main challenge for scene parsing arises when complex scenes with highly diverse objects are encountered. The objects not only differ in scale and appearance but also in semantics. Previous works focus on encoding the multi-scale contextual information (via pooling or atrous convolutions) generally on top of compact high-level features (i.e., at a single stage). In this work, we argue that a rich set of cues exist at multiple stages of the network, encapsulating low, mid and high-level scene details. Therefore, an optimal scene parsing model must aggregate multi-scale context at all three levels of the feature hierarchy; a capability that lacks in state-of-the-art scene parsing models. To address this limitation, we introduce a novel architecture with three new blocks that systematically aggregate low, mid and high tier features. The heart of our approach is a high-level feature aggregation module that augments sparsely connected atrous convolution with dense local and layer-wise connections to avoid gridding artifacts. Besides, we employ a novel feature pyramid augmentation and semantic refinement unit to generate low- and mid-level features that are mixed with high-level features at the decoder. We extensively evaluate our proposed approach on the large-scale Cityscapes and ADE2K benchmarks. Our approach surpasses many latest models on both datasets, achieving mean intersection-over-union (mIoU) scores of 80.5% and 44.0% on Cityscapes and ADE20K, respectively.

Image rain removal aims to separate the background image from the rainy image. During the past three years, the image rain removal with deep convolutional neural networks has achieved impressive performance. However, how to reach tradeoff between high de-raining performance and low model parameters is still a challenge. To address the issue, the paper is devoted to exploring a novel method based on wavelet deep recursive pyramid convolution residual network (WDRPRN), in which discrete wavelet transform is embedded to decompose the rainy image in different frequency domains, and the deep recursive pyramid convolution residual network (DRPRN) can well predict the residual coefficients between rainy image and clean image. In addition, compared with other neural networks, the DRPRN adopts recursive model that can cost fewer parameters. Plentiful of experiments on synthetic and real-world datasets show that the proposed method is significantly superior to the recent state-of-the-art algorithms.

The semantic segmentation of remote sensing (RS) image is a hot research field. With the development of deep learning, the semantic segmentation based on a full convolution neural network greatly improves the segmentation accuracy. The amount of information on the RS image is very large, but the sample size is extremely uneven. Therefore, even the common network can segment RS images to a certain extent, but the segmentation accuracy can still be greatly improved. The common neural network deepens the network to improve the classification accuracy, but it has a lot of loss to the target spatial features and scale features, and the existing common feature fusion methods can only solve some problems. A segmentation network is built to solve the above problems very well. The network employs the InceptionV-4 network as the backbone and improves it. We modify the network structure and introduce the changed Atrous Spatial Pyramid Pooling module to extract the multi-scale features of the target from different training stages. Without losing the depth of the network, using Inception blocks to strengthen the width of the network can obtain more abstract features. At the same time, the backbone network is used for semantic fusion of the context, it can retain more spatial features, then an effective decoder network is designed. Finally, evaluate our model on the ISPRS 2D Semantic Labeling Contest Potsdam and Inria Aerial Image Labeling Dataset. The results show that the network has very superior performance, reaching 89.62% IOU score and 94.49% F1 score on the Potsdam dataset, and the IOU score on the Inria dataset has been greatly improved.

Facial landmark detection can be applied in various facial analysis tasks. It is a challenging problem due to the various poses and high real-time requirements. To balance accuracy and computational efficiency, we propose an efficient coarse-to-fine network by combining heatmap regression and lightweight coordinate regression to detect facial landmarks. The heatmap regression network branch employs an efficient spatial pyramid and attention mechanism to regress to a better quality heatmap. The lightweight coordinate regression network branch introduces the local region perceptron to make the small network focus on the region of interest. Our method develops the advantages of heatmap regression and coordinate regression, which can improve landmark detection. Moreover, we propose a novel two-stage FTS loss function that can further effectively solve the outlier problem in facial landmark detection. The comprehensive experiments demonstrate the effectiveness of our approach.

Finding and segmenting objects in noisy low-contrast in situ underwater plankton images is challenging because of the difficulty of separating potential plankton objects from the complex background and numerous and diverse other particles. In the present study, a dynamic downscaling model was developed to rapidly extract complete and clean regions of interest (ROIs) from images with highly variable content and quality. The original image was downscaled, and dynamic segmentation was performed in a scale pyramid space to ensure the integrity of weak targets based on local two-dimensional (2D) entropy parameters. Subsequently, a series of local thresholds and clustering gradients was examined iteratively for ROI selection. The performance of the local 2D entropy parameters relative to water turbidity (in a scattering medium and with high background noise) and image size was examined. To suppress the background and increase the sharpness of potential targets, a sharpness descriptor and gradient clustering were employed. The method was compared with the currently commonly used local threshold-based Sauvola segmentation using the same set of images. The results showed that the proposed method improves the ROI extraction accuracy and reduces oversegmentation for in situ underwater plankton images. It was concluded that the proposed method is a fast and robust segmentation technique and could facilitate the deployment of in situ plankton imaging systems for process-based research and routine plankton monitoring.

Ship detection of synthetic aperture radar (SAR) images is one of the research hotspots in the field of marine surveillance. Fusing salient features to detection network can effectively improve the precision of ship detection. However, how to effectively fuse the salient features of SAR images is still a difficult task. In this paper, to improve the ship detection precision, we design a novel one-stage ship detection network to fuse salient features and deep convolutional neural network (CNN) features. Firstly, a saliency map extraction algorithm is proposed. The algorithm is applied to generate saliency map by using multi-scale pyramid features and frequency domain features. Secondly, the backbone of the ship detection network contains a two-stream network. The upper-stream network uses the original SAR image as input to extract multi-scale deep CNN features. The lower-stream network uses the corresponding saliency map as input to acquire multi-scale salient features. Thirdly, for integrating the salient features to deep CNN features, a novel salient feature fusion method is designed. Finally, an improved bi-directional feature pyramid network is applied to the ship detection network for reducing the computational complexity and network parameters. The proposed methods are evaluated on the public ship detection dataset and the experimental results shows that it can make a significant improvement in the precision of SAR image ship detection.

Locating the tiny insulator defect object with complex backgrounds in high-resolution aerial images is a challenging task. In this paper, we propose a novel method which cascades detection and segmentation networks to identify the defect from the global and local two levels: (1) The improved Faster R-CNN is carried out to capture both defects and insulators in the entire image. ResNeXt-101 is adopted as the feature extraction network so as to fully extract features, and Feature Pyramid Network (FPN) is built to enhance the ability of detecting small targets. In addition, the Online Hard Example Mining (OHEM) training strategy is applied to solve the imbalance problem of positive and negative samples. (2) All the detected insulators are extracted and fed into the improved U-Net network to futher inspect at pixel level, we utilize the pre-trained ResNeXt-50 as the encoder of U-Net, incorporate an attention module, Spatial and Channel Squeeze & Excitation Block (SCSE), into the decoding path to highlight the meaningful information. A hybrid loss which merges binary cross entropy (BCE) loss and dice coefficient loss is designed to train our network for figuring out the class imbalance issue. The missed detection can be greatly reduced with the combination of two modified network, which makes comprehensive use of the original map information and local information. On the test set of actual images, the insulator defect recognition precision and recall of the cascade network is 91.9% and 95.7%, exhibiting strong robustness and accuracy.

To detect the shape of a small magnetic target in the shallow underground layer, this article proposes a recognition method based on Mask-RCNN. Firstly, using COMSOL software and MATLAB software to establish the database of magnetic targets model under different shapes and orientations, which greatly enriched the diversity of the training data set. Then, the ${G}_{\mathrm {zz}}$ component of the magnetic gradient tensor matrix is selected to highlight the shape features of the magnetic target, and the contour image is generated. The experimental data set is created by using the deep learning annotation tool Labelme. Finally, Resnet101 is used as the backbone network and feature pyramid network (FPN) structure is used to extract features. The regional recommendation network (RPN) is trained end-to-end to create regional recommendations for each feature map. The detection results of 200 test images show that the average detection accuracy of the method is 97%, and the recall rate is 94%. The simulation results show that the recognition accuracy and robustness of the method are improved.

Effectively detecting road objects in various environments would significantly improve driving safety for autonomous vehicles. However, small objects, low illumination, and blurred outline in images strongly limit the performance of current road object detection methods. To solve these problems, this paper proposed a novel deep learning anchor-free approach based on CenterNet. The atrous spatial pyramid pooling (ASPP) was used to extract features from multiple scales to improve the detection performance while not increasing the computational cost and the number of parameters. The space to depth algorithm was then adopted in our proposed approach to optimize the traditional downsampling process. A large-scale naturalistic driving dataset (BDD100K) was used to examined the effectiveness of our proposed approach. The experimental results show that our proposed approach can effectively improve the detection performance on small objects in various traffic situations.

Although excessive proposals using traditional sliding-window methods or prevailing anchor-based techniques have been proposed to deal with deep learning-based pedestrian detection, it is still a promising yet challenging problem. In this paper, we propose a precise, flexible and thoroughly anchor-free, as well as proposal-free framework named Pedestrian-as-Points Network (PP-Net) for pedestrian detection. Specifically, we model a pedestrian as a single point, i.e., the center point of the instance, and predict the pedestrian scale at each detected center point. In order to achieve higher accuracy, we build a pyramid-like structure based on the backbone as a feature extractor to aggregate multi-level information. In addition, we construct a deep guidance module (DGM) at the top of the backbone, so that the higher-level information can be captured in the process of building a feature pyramid network (FPN) to avoid the dilution of high-level information on the top-down pathway. We further design a feature fusion unit (FFU) to fuse the fine-level features well with the coarse-level semantic information from the top-down pathway. With the only post-processing non-maximum suppression (NMS), we achieve better performance than many state-of-the-arts methods on the challenging pedestrian detection datasets.

Spatial pyramid matching (SPM) is one of the widely used methods to incorporate spatial information into the image representation. Despite its effectiveness, the traditional SPM is not rotation invariant. A rotation invariant SPM has been proposed in the literature but it has many limitations regarding the effectiveness. In this paper, we investigate how to make SPM robust to rotation by addressing those limitations. In an SPM framework, an image is divided into an increasing number of partitions at different pyramid levels. In this paper, our main focus is on how to partition images in such a way that the resulting structure can deal with image-level rotations. To do that, we investigate three concentric ring partitioning schemes. Apart from image partitioning, another important component of the SPM framework is a weight function. To apportion the contribution of each pyramid level to the final matching between two images, the weight function is needed. In this paper, we propose a new weight function which is suitable for the rotation-invariant SPM structure. Experiments based on image classification and retrieval are performed on five image databases. The detailed result analysis shows that we are successful in enhancing the effectiveness of SPM for image classification and retrieval.

In order to solve the problem of low recognition rate and low real-time performance of vehicle detection in complex road environment, a data-driven forward vehicle detection algorithm based on improved tiny-YOLOv3 is proposed. Based on tiny-YOLOv3, the context feature information is combined to increase the two scale detections of tiny-YOLOv3 to three. The spatial pyramid pooling (SPP) module is added to increase the number of feature channels to improve the network feature extraction ability. According to the dense arrangement of vehicles on the horizontal axis in the road image ahead, we change the grid size of tiny-YOLOv3 and increase the number of candidate boxes on the horizontal axis. In addition, combined with the characteristics of the vehicle size in the road image ahead, K-means clustering method is used to select the appropriate number and size of target candidate boxes. We obtain the optimal detection model by multi-scale training of the improved network. The experimental results show that the average accuracy of the improved algorithm on the KITTI datasets is 91.03%, which is 7.12% higher than that of tiny-YOLOv3. And the detection speed of improved network is 144 frames/s, which meets the real-time requirements.

Flies carry pathogens that endanger the health of humans and animals. The color and shape of the fly species are very similar, which is difficult to recognize. This paper proposes a fly species recognition method based on improved RetinaNet and convolutional block attention module (CBAM). Firstly, the proposed method used ResNeXt101 as a feature extraction network, and the improved CBAM called Stochastic-CBAM was added. Then, we built a multi-scale feature pyramid through an improved feature pyramid network (FPN) and integrated multi-level feature information. Finally, the small full convolutional network (FCN) was used as the classification subnet and the bounding box regression subnet. The Kullback-Leibler (KL) loss replaced smooth L1 loss as a bounding box regression loss function for learning bounding box regression and positioning uncertainty at the same time. We experimentally compared the proposed method with other the state-of-the-art methods on the established dataset. Experimental results showed that the mean Average Precision (mAP) of this method reached 90.38%, which was better than the state-of-the-art methods. The average time to recognize a single image was 0.131s. This method has a good detection effect on the fly species recognition.

Defect detection is an essential requirement for quality control in the production of printed circuit boards (PCBs) manufacturing. The traditional defect detection methods have various drawbacks, such as strongly depending on a carefully designed template, highly computational cost, and noise-susceptibility, which pose a significant challenge in a production environment. In this paper, a deep learning-based image detection method for PCB defect detection is proposed. This method builds a new network based on Faster RCNN. We use a ResNet50 with Feature Pyramid Networks as the backbone for feature extraction, to better detect small defects on the PCB. Secondly, we use GARPN to predict more accurate anchors and merge the residual units of ShuffleNetV2. The experimental results show that this method is more suitable for use in production than other PCB defect detection methods. We have also tested in other PCB defects dataset, and experiments have shown that this method is equally valid.

Traffic congestion due to vehicular accidents seriously affects normal travel, and accurate and effective mitigating measures and methods must be studied. To resolve traffic accident compensation problems quickly, a vehicle-damage-detection segmentation algorithm based on transfer learning and an improved mask regional convolutional neural network (Mask RCNN) is proposed in this paper. The experiment first collects car damage pictures for preprocessing and uses Labelme to make data set labels, which are divided into training sets and test sets. The residual network (ResNet) is optimized, and feature extraction is performed in combination with Feature Pyramid Network (FPN). Then, the proportion and threshold of the Anchor in the region proposal network (RPN) are adjusted. The spatial information of the feature map is preserved by bilinear interpolation in ROIAlign, and different weights are introduced in the loss function for different-scale targets. Finally, the results of self-made dedicated dataset training and testing show that the improved Mask RCNN has better Average Precision (AP) value, detection accuracy and masking accuracy, and improves the efficiency of solving traffic accident compensation problems.

Ultrasound imaging technology plays an important role to assist doctors in diagnosing thyroid nodules. The tissue structure around the thyroid is very complex, which makes it difficult to segment and extract the ultrasound image of thyroid nodules accurately. For address this problem, this paper proposes a model algorithm for thyroid nodule ultrasound image segmentation using ASPP fusion features. First, spatial pyramid pooling and depthwise separable convolution are combined in order to solve the problem that the size of the mapping feature will change in the process of better capturing the context information. Besides, Atrous Spatial Pyramid Pooling (ASPP) is proposed to achieve the purpose of processing input image channel and spatial information separately. In order to appropriately reduce the dimension and size of feature images, a $1\times 1$ convolution operation is performed before each convolution calculation, and the model size is optimized. In the decoding stage, decoder module appropriately adjusts the feature map with a relatively low resolution previously from decoder module, and sets the output channel number of two convolutions to the same value. All features have the same dimension by adjustment, and features can be fused by element-wise summation. Finally, Dice Similarity Coefficient (DSC), Prevent Match (PM) and Correspondence Patio (CR) are used as evaluation criteria to compare with other model algorithms. The experimental results show that the proposed model can significantly improve the segmentation effect of ultrasound images for thyroid nodules compared with traditional models.

Modern approaches for semantic segmentation usually concatenate the feature map of the last convolutional layer and multi-scale features to form the final feature representation, which can achieve the more accurate classification of target pixels for the input image. However, the feature information of the last layer is not complete and refined so that there is a performance bottleneck in the concatenation between the final feature map and multi-scale feature representations. To solve this problem, we propose the Feature Recombining Network to get more refined and precise features for Semantic Segmentation. Our network is composed of Feature Recombining Module and Modified Pyramid Pooling Module. The two modules can extract more detailed and representative features through the feature recombination and acquire richer context information than the previous module respectively. Experiments show that our modules are effective to improve the segmentation precision and the Modified Pyramid Pooling Module is also superior to the previous module. Based on our proposed network, we achieve the performance of 51.9% mIoU on Pascal Context dataset and 44.75% mIoU on ADE20K dataset.

One-stage object detectors are simple and efficient; however, they cannot extract sufficient object features due to simplistic structures. At the same time, the classification score cannot reflect the actual positioning of the candidate box. Therefore, it is not accurate to use classification score only as the candidate box position score in non-maximum suppression (NMS) stage. These two shortcomings degrade the detection accuracy. In this paper, a novel feature pyramid architecture named refined feature pyramid network (ReFPN) is introduced to obtain better object features. ReFPN designs a refined module which is parallel with feature pyramid network (FPN) to extract the semantic features of objects, and then the extraction of features are used to optimize the features of FPN by summation. In addition, we design the refined center-ness (RCenter-ness) branch that predicts the position score of each point on the feature map to improve the localization accuracy. The predicted position score is multiplied by the classification score to obtain the final position score that has a stronger correlation with localization accuracy. The final position score is inputted to the subsequent NMS, which improves localization accuracy. The proposed method in this paper is named ReFPN-FCOS. The sufficient experiments on COCO2017 datasets demonstrate the effectiveness of ReFPN-FCOS on improving classification accuracy and localization accuracy. The average precisions of this method achieve 1.1% and 1.3 % higher than those of FCOS, when using ResNet50 and ResNet101 as backbone respectively. Code download link: https://github.com/xjl-le/mmdete.

To solve the complex background problems (e.g. Noise interference, object overlap, and different illumination) that affect the classification performance on plant leaf images, this paper proposes an instance segmentation and classification method for plant leaf images based on IFPN SNMS CFFI-Mask R-CNN (ISC-MRCNN) and ACPSOSVM-Dual Channels Convolutional Neural Network (APS-DCCNN). To obtain the foreground of plant leaf images, the lateral connection structure of the feature map pyramid in ISC-MRCNN fuses the feature maps of different depths, so that the network learns more detailed features. Then, the Soft Non-Maximum Suppression Algorithm is employed to improve the detection performance of overlapping objects. Next, the pooling method of integrating the continuous function can reduce the precision loss during the alignment of the mapping between the feature map and the original image. Finally, by constructing a mask filter layer, complex backgrounds are masked. To distinguish the similarity between plant leaf images, APS-DCCNN is used to classify the foreground images. In this process, the Support Vector Machine is used to replace softmax and then an Adaptive Chaotic Particle Swarm Algorithm is employed to optimize it. The experimental results show that compared with Mask R-CNN, the average precision of ISC-MRCNN has increased by 1.89% under different thresholds. The proposed method is suitable for the object detection and instance segmentation problems with complex background. Besides, compared with traditional CNN, the average precision of the classification results obtained by APS-DCCNN has improved by 1.59%. This has shown that the proposed method is suitable for the classification of plant leaves.

A deep convolutional neural network has been widely used in image semantic segmentation in recent years, its deployment on mobile terminals, however is limited by its high computational costs. Given the slow inference speed and large memory usage of deep convolutional neural networks, we propose a lightweight and densely connected pyramid network (LDPNet) for real-time semantic segmentation. Firstly, a densely connected atrous pyramid (DCAP) module is constructed in the encoding process to extract multi-scale context information for forwarding propagation, strengthen the reuse of features, and offset the spatial information lost in the down-sampling process of the feature map. Secondly, a cross-fusion (CF) module embedded in each other during the decoding process is proposed, which uses high-level semantic features to effectively guide the fusion of low-level spatial details while strengthening context information. Our network is tested on two complex urban road scene data sets. Among them, experiments on the Cityscapes data set show that our structure has 87 frames per second (FPS) on a single NVIDIA GTX1080Ti GPU. The Mean Intersection over Union (mIoU) reaches 71.1%, and the parameter is only 0.8M. Compared with the existing similar networks, the new system achieves a state-of-the-art trade-off between efficiency and accuracy.

Recently, optical flow guided video saliency detection methods have achieved high performance. However, the computation cost of optical flow is usually expensive, which limits the applications of these methods in time-critical scenarios. In this article, we propose an end-to-end cross complementary network (CCNet) based on fully convolutional network for video saliency detection. The CCNet consists of two effective components: single-image representation enhancement (SRE) module and spatiotemporal information learning (STIL) module. The SRE module provides robust saliency feature learning for a single image through a pyramid pooling module followed by a lightweight channel attention module. As an effective alternative operation of optical flow to extract spatiotemporal information, the STIL introduces a spatiotemporal information fusion module and a video correlation filter to learn the spatiotemporal information, the inner collaborative and interactive information between consecutive input groups. In addition to enhancing the feature representation of a single image, the combination of SRE and STIL can learn the spatiotemporal information and the correlation between consecutive images well. Extensive experimental results demonstrate the effectiveness of our method in comparison with 14 state-of-the-art approaches.