Search

It is a typical problem in the field of object detection to simultaneously detect objects with large scale variation in one image. Recently proposed state-of-the-art object detectors generally learn pyramidal feature representation to deal with the scale variation, which has been proved effective via various feature pyramid networks. However, the majority of the feature pyramid networks based on heuristic feature fusion strategies may be suboptimal, as excess human guidance will restrict the self-learning of deep neural networks. An adaptive feature pyramid is bound to provide a significant performance boost. In this paper, we propose a novel feature pyramid network named CATFPN that consists of Scale-Wise Feature Concatenation (SWFC) module and Global Context (GC) block. The SWFC module evenly distributes semantic features for each feature layer and the GC block introduces a self-attention mechanism. As a feature pyramid network, the CATFPN can be applied to any detector based on multi-scale features. We adopt the CATFPN in typical RetinaNet and Faster R-CNN detector models, without bells and whistles, achieving 1.1% AP and 0.7% AP improvements over FPN on the MS COCO benchmark, respectively. Our competitive performance reported on the test-dev subset of COCO achieves 42.3% AP.

When a detection model that has been well-trained on a set of classes faces new classes, incremental learning is always necessary to adapt the model to detect the new classes. In most scenarios, it is required to preserve the learned knowledge of the old classes during incremental learning rather than reusing the training data from the old classes. Since the objects in remote sensing images often appear in various sizes, arbitrary directions, and dense distribution, it further makes incremental learning-based object detection more difficult. In this article, a new architecture for incremental object detection is proposed based on feature pyramid and knowledge distillation. Especially, by means of a feature pyramid network (FPN), the objects with various scales are detected in the different layers of the feature pyramid. Motivated by Learning without Forgetting (LwF), a new branch is expended in the last layer of FPN, and knowledge distillation is applied to the outputs of the old branch to maintain the old learning capability for the old classes. Multitask learning is adopted to jointly optimize the losses from two branches. Experiments on two widely used remote sensing data sets show our promising performance compared with state-of-the-art incremental object detection methods.

As a significant and fundamental task in the remote sensing field, object detection has received increasing attention and research studies. However, geospatial object detection is still a challenge owing to the dramatic variation in object scales, intraclass differences, and interclass similarity from multiscale and multiclass objects. To deal with these problems, an end-to-end feature-reflowing pyramid network (FRPNet) is proposed in this letter. FRPNet has two advantages that contribute to improve object detection accuracy. First, we embed a nonlocal block into the backbone in order to get the relevancy between different regions of the geospatial image for obtaining discriminative features. Furthermore, a feature-reflowing pyramid structure is proposed to generate high-quality feature presentation for each scale through fusing fine-grained features from the adjacent lower level, which improves the detection capability for multiscale and multiclass objects. Experiments on a public remote sensing data set DIOR illustrate that FRPNet can significantly improve the performance when compared to several state-of-the-art detection approaches in terms of mean average precision (mAP).

Object detection (OD) is an important task of computer vision and has been widely used in many fields, including remote sensing (RS). However, the complex scenes, large-scale variation, and dense instances of RS bring huge challenges to OD. To meet these challenges, a novel Nonlocal-aware Pyramid and Multiscale Multitask Refinement Detector (NPMMR-Det) is proposed. Specifically, nonlocal-aware pyramid attention (NP-Attention) is designed for guiding a neural network model to focus more on efficient features and suppress background noise. Then a multiscale refinement feature pyramid network (MSR-FPN) is proposed to fuse the multiscale context features extracted by the NP-Attention guided neural network and adjust the optimal receptive field. In order to use these features more effectively, a multitask refinement head called MTR-Head, with offset sharing and a modulation mechanism, is developed to refine the feature misalignment between the localization task and the classification task. Extensive experiments performed on two public RS data sets demonstrate that the proposed NPMMR-Det achieves competitive performance compared with state-of-the-art methods.

Unlike object detection in natural images that usually achieved great success, remote sensing imagery has its own challenges to detect and localize multiclass objects, such as large-scale change, uncertain direction, and high density. The context information of the objects is very worthwhile for solving these challenges in remote sensing images. In this letter, we propose a context-driven detection network (CDD-Net) to improve the accuracy of multiclass object detection in remote sensing images. For capturing the local neighboring objects and features, a local context feature network (LCFN) is proposed to learn the local context of the region of interest. Meanwhile, a hybrid attention pyramid network (HAPN) is designed, which can steer the focus to more valuable features. The HAPN inserts a squeeze and excitation block (SEB) and three asymmetric convolution blocks (ACBs) in the feature pyramid network (FPN). The experimental results over the DOTA-v1.5 data set demonstrate that the proposed CDD-Net yields promising results.

In recent years, deep convolutional neural networks (DCNNs) have made significant progress in cloud detection tasks, and the detection accuracy has been greatly improved. However, most existing CNN-based models have high computational complexity, which limits their practical application, especially for spaceborne optical remote sensing. In addition, most of the methods cannot make adaptive adjustments based on the structural information of the clouds, and blurred boundaries often occur in the detection results. In order to address these problems, this article proposes a lightweight network (DABNet) to achieve high-accuracy detection of complex clouds, not only a clearer boundary but also lower false-alarm rate. Specifically, a deformable context feature pyramid module is proposed to improve the adaptive modeling capability of multiscale features. Besides, a boundary-weighted loss function is designed to direct the network to focus on cloud boundary information and optimize the relevant detection results. The proposed method has been validated on two data sets: the public GF-1 WFV benchmark and our self-built GF-2 cloud detection data set with higher spatial resolution. The experimental results exhibit that DABNet achieves state-of-the-art performance while only using 4.12M parameters and 8.29G multiadds.

This letter presents a new feature pyramid network (FPN) called the gated ladder-shaped FPN (GLFPN) to construct more representative feature pyramids for detecting objects of different sizes in optical remote sensing images. We first use convolution and concatenation operations to fuse three base features extracted by a ResNet backbone. We then obtain multilevel features from these base features. Finally, we use a selective gate to fuse features from multiple levels with equivalent sizes. To evaluate the effectiveness of the proposed GLFPN, we integrate it into the RetinaNet architecture by replacing the conventional FPN. The experimental results on two optical remote sensing image data sets show that the proposed method outperforms the methods compared in this letter.

Modern autonomous vehicles are required to perform various visual perception tasks for scene construction and motion decision. The multiobject tracking and instance segmentation (MOTS) are the main tasks since they directly influence the steering and braking of the car. Implementing both tasks using a multitask learning neural network presents significant challenges in performance and complexity. Current work on MOTS devotes to improve the precision of the network with a two-stage tracking by detection model, which is difficult to satisfy the real-time requirement of autonomous vehicles. In this article, a real-time multitask network named YolTrack based on one-stage instance segmentation model is proposed to perform the MOTS task, achieving an inference speed of 29.5 frames per second (fps) with slight accuracy and precision drop. The YolTrack uses ShuffleNet V2 with feature pyramid network (FPN) as a backbone, from which two decoders are extended to generate instance segments and embedding vectors. Segmentation masks are used to improve the tracking performance by performing logic AND operation with feature maps, proving that foreground segmentation plays an important role in object tracking. The different scales of multiple tasks are balanced by the optimized geometric mean loss during the training phase. Experimental results on the KITTI MOTS data set show that YolTrack outperforms other state-of-the-art MOTS architectures in real-time aspect and is appropriate for deployment in autonomous vehicles.

Traffic sign detection has made great progress with the rise of deep learning in recent years. As a result of the complex and changeable traffic environment, detecting small traffic signs in a real-world scene is still a challenging problem. In this paper, a novel group multi-scale attention pyramid network is proposed to address the problem. Specifically, to aggregate the feature at different scales and suppress the messy information in the background, an effective multi-scale attention module is proposed. Furthermore, a feature fusion module, named adaptive pyramid convolution, is further designed, which can drive the network to learn the optimal feature fusion pattern and construct an informative feature pyramid for detecting traffic signs in different sizes. Extensive experimental results on the public traffic sign detection datasets demonstrate the effectiveness and superiority of the proposed method when compared with several state-of-the-art methods.

Recent progress on remote sensing (RS) scene classification is substantial, benefiting mostly from the explosive development of convolutional neural networks (CNNs). However, different from the natural images in which the objects occupy most of the space, objects in RS images are usually small and separated. Therefore, there is still a large room for improvement of the vanilla CNNs that extract global image-level features for RS scene classification, ignoring local object-level features. In this article, we propose a novel RS scene classification method via enhanced feature pyramid network (EFPN) with deep semantic embedding (DSE). Our proposed framework extracts multiscale multilevel features using an EFPN. Then, to leverage the complementary advantages of the multilevel and multiscale features, we design a DSE module to generate discriminative features. Third, a feature fusion module, called two-branch deep feature fusion (TDFF), is introduced to aggregate the features at different levels in an effective way. Our method produces state-of-the-art results on two widely used RS scene classification benchmarks, with better effectiveness and accuracy than the existing algorithms. Beyond that, we conduct an exhaustive analysis on the role of each module in the proposed architecture, and the experimental results further verify the merits of the proposed method.

The current target tracking and detection algorithms often have mistakes and omissions when the target is occluded or small. To overcome the defects, this paper integrates bi-directional feature pyramid network (BiFPN) into cascade region-based convolutional neural network (R-CNN) for live object tracking and detection. Specifically, the BiFPN structure was utilized to connect between scales and fuse weighted features more efficiently, thereby enhancing the network’s feature extraction ability, and improving the detection effect on occluded and small targets. The proposed method, i.e., Cascade R-CNN fused with BiFPN, was compared with target detection algorithms like Cascade R-CNN and single shot detection (SSD) on a video frame dataset of wild animals. Our method achieved a mean average precision (mAP) of 91%, higher than that of SSD and Cascade R-CNN. Besides, it only took 0.42s for our method to detect each image, i.e., the real-time detection was realized. Experimental results prove that the proposed live object tracking and detection model, i.e., Cascade R-CNN fused with BiFPN, can adapt well to the complex detection environment, and achieve an excellent detection effect.

In the field of automatic driving, identifying vehicles and pedestrians is the starting point of other automatic driving techniques. Using the information collected by the camera to detect traffic targets is particularly important. The main bottleneck of traffic object detection is due to the same category of targets, which may have different scales. For example, the pixel‐level of cars may range from 30 to 300 px, which will cause instability of positioning and classification. In this paper, a multi‐dimension feature pyramid is constructed in order to solve the multi‐scale problem. The feature pyramid is built by developing a U‐shaped module and using a cascade‐method. In order to verify the effectiveness of the U‐shaped module, we also designed a new one‐shot detector U‐FPNDet. The model first extracts the basic feature map by using the basic network and constructs the multi‐dimension feature pyramid. Next, a pyramid pooling module is used to get more context information from the scene. Finally, the detection network is run on each level of the pyramid to obtain the final result by NMS. By using this method, a state‐of‐the‐art performance is achieved on both detection and classification on commonly used benchmarks.

Crowd counting is one of the most challenging issues in computer vision community for safety and security through surveillance systems. It has extensive range of applications, such as disaster management, surveillance event detection, intelligence gathering and analysis, public safety control, traffic monitoring, design of public spaces, anomaly detection and military. Early approaches still encounter many issues, like non-uniform density distribution, partial occlusion and discrepancies in scale and point of view. To address the above problems, Feature Pyramid Networks are introduced in deep convolution networks for counting the individuals in the Crowd. The designed network has extracted the features at all resolutions and is constructed rapidly from only one input image. This method achieves out-performance results compared to the well-known networks on three demanding standard crowd counting datasets.

Although deep learning has been widely used for dense crowd counting, it still faces two challenges. Firstly, the popular network models are sensitive to scale variance of human head, human occlusions, and complex background due to repeated utilization of vanilla convolution kernels. Secondly, the vanilla feature fusion often depends on summation or concatenation, which ignores the correlation of different features leading to information redundancy and low robustness to background noise. To address these issues, a multi‐scale feature pyramid network (MFP‐Net) for dense crowd counting is proposed in this paper. The proposed MFP‐Net makes two contributions. Firstly, the feature pyramid fusion module is designed that adopts rich convolutions with different depths and scales, not only to expand the receptive field, but also to improve the inference speed of models by using parallel group convolution. Secondly, a feature attention‐aware module is added in the feature fusion stage. The module can achieve local and global information fusion by capturing the importance of the spatial and channel domains to improve model robustness. The proposed MFP‐Net is evaluated on five publicly available datasets, and experiments show that the MFP‐Net not only provides better crowd counting results than comparative models, but also requires fewer parameters.

In existing state-of-the-art object detectors, feature pyramid networks (FPN) and multiscale feature fusion are still typically used. The traditional FPN fusion strategy is based on the top-down fusion of high-level semantic information. The top-down fusion method generally uses upsampling based on interpolation, which often results in jagged edges, mosaic distortion, and edge blurring. Moreover, in order to improve accuracy, the FPN-based fusion strategy must add multiple top-down components for fusion, which increases computational costs and leads to a poor balance between precision and speed. In this paper, we propose a novel fusion strategy based on a backbone network. We aim to design simple and efficient components for high-quality object detection. Our proposed strategy, bi-directional skip connection FPN (BiSCFPN), consists of three components: a bi-directional skip connection (BiSC), a selective dilated convolution module (SDCM), and sub-pixel convolution (SP). The BiSC aims to enhance semantic information between different feature layers in the backbone network and simultaneously uses the SDCM to improve the receptive fields of differently sized targets in the fusion stage. Finally, SP learns the relationship between the features of upsampling and downsampling images to effectively mitigate the problems caused by the traditional interpolation method. BiSCFPN achieves an average precision of 38.2% in tests with the Microsoft Common Objects in Context (MS COCO) test-dev dataset at a real-time speed of ~ 50 FPS ( 608 × 608 ) using an Nvidia GeForce RTX 2080 Ti graphics card and significantly improves the balance between precision and speed.

The retinal vessel is the only microvascular network that can be directly and non‐invasively observed in humans. Cardiovascular and cerebrovascular diseases, such as diabetes, hypertension, can lead to structural changes of the retinal microvascular network. Therefore, it is of great significance to study effective retinal vessel segmentation methods and assist doctors in early diagnoses with quantitative results for vascular networks. In this study, we propose a novel convolutional neural network named feature pyramid U‐Net (FPU‐Net) that extracts multiscale representations by constructing two feature pyramids both on the encoder and the decoder of U‐Net. In this representation, objects features with different size like micro‐vessels and pathology will be fused for better vessel segmentation. The experimental results show that compared with state‐of‐the‐art methods, FPU‐Net is superior in terms of accuracy, sensitivity, F1‐score, and area under the curve and capable of stronger domain generalisation across different datasets.

Learning-based methods have attracted considerable interest in image dehazing. However, most existing methods are not well adapted to different hazy conditions, especially when dealing with the heavily hazy scene. There is often a significant amount of haze that remains in the images recovered by most methods. To address this issue, we propose an end-to-end Haze Concentration Adaptive Network (HCAN), including a pyramid feature extractor (PFE), a feature enhancement module (FEM), and a multi-scale feature attention module (MSFAM) for image dehazing. Specifically, PFE based on the feature pyramid structure leverages complementary features from different CNN layers to help the clear image prediction. Then, FEM fuses four kinds of images with different haze density (i.e., three recovered images in the FEM with light haze density, and the input hazy image with strong haze condition) to guide the network to adaptively perceive images under different haze conditions. Finally, MSFAM is designed under two principles, multi-scale structure and attention mechanism. It is used to help the network produce a clear image with more details, and ease the network training. Comprehensive experiments demonstrate that the proposed HCAN performs favorably against the state-of-the-art methods in terms of PSNR, SSIM, and visual effect. The results, per-trained models and code are available at https://github.com/TaoWangzj/HCAN .

Vehicle detection in aerial images is an important and challenging task in the field of remote sensing. Recently, deep learning technologies have yielded superior performance for object detection in remote sensing images. However, the detection results of the existing methods are horizontal bounding boxes that ignore vehicle orientations, thereby having limited applicability in scenes with dense vehicles or clutter backgrounds. In this article, we propose a one-stage, anchor-free detection approach to detect arbitrarily oriented vehicles in high-resolution aerial images. The vehicle detection task is transformed into a multitask learning problem by directly predicting high-level vehicle features via a fully convolutional network. That is, a classification subtask is created to look for vehicle central points and three regression subtasks are created to predict vehicle orientations, scales, and offsets of vehicle central points. First, coarse and fine feature maps outputted from different stages of a residual network are concatenated together by a feature pyramid fusion strategy. Upon the concatenated features, four convolutional layers are attached in parallel to predict high-level vehicle features. During training, task uncertainty learned from the training data is used to weight loss function in the multitask learning setting. For inferencing, oriented bounding boxes are generated using the predicted vehicle features, and oriented nonmaximum suppression (NMS) postprocessing is used to reduce redundant results. Experiments on two public aerial image data sets have shown the effectiveness of the proposed approach.

Pedestrian attributes recognition is an important issue in computer vision and has a special role in the field of video surveillance. The previous methods presented to solve this issue are mainly based on multi‐label end‐to‐end deep neural networks. These methods neglect to apply attributes for defining local feature areas and they suffer from the problems of the bounding box presence. A new framework for jointly human semantic parsing and pedestrian attribute recognition to achieve effective attribute recognition is proposed. By extracting human parts via semantic parsing, both semantic and spatial information can be explored with eliminating of background. The framework also uses multi‐scale features to employ rich details and contextual information through proposed attribute recognition‐bidirectional feature pyramid network. For baseline network that has a significant impact on the performance, EfficientNet‐B3 is selected as a baseline network from The EfficientNet family which provides an appropriate trade‐off between the three factors of CNNs scaling (depth/width/resolution). Finally, the proposed framework is tested on datasets PETA, RAP and PA‐100k. Experimental results show that our method has superior performance in both mean accuracy and instance‐based metrics compared to state‐of‐the‐art results.

Unmanned aerial vehicles, or drones, are poised to solve many problems associated with data collection in complex urban environments. Drones are easy to deploy, have a great ability to move and explore the environment, and are relatively cheaper than other data collection methods. This study investigated the use of Cascade Region-based convolutional neural network (R-CNN) networks to enable automatic vehicle counting and tracking in aerial video streams.The presented technique combines feature pyramid networks and a Cascade R-CNN architecture to enable accurate detection and classification of vehicles.The paper discusses the implementation and evaluation of the detection and tracking techniques and highlights their advantages when they are used to collect traffic data.

RoIPool/RoIAlign is an indispensable process for the typical two-stage object detection algorithm, it is used to rescale the object proposal cropped from the feature pyramid to generate a fixed size feature map. However, these cropped feature maps of local receptive fields will heavily lose global context information. To tackle this problem, we propose a novel end-to-end trainable framework, called global context aware (GCA) RCNN, aiming at assisting the neural network in strengthening the spatial correlation between the background and the foreground by fusing global context information. The core component of our GCA framework is a context aware mechanism, in which both global feature pyramid and attention strategies are used for feature extraction and feature refinement, respectively. Specifically, we leverage the dense connection to improve the information flow of the global context at different stages in the top-down process of FPN, and further use the attention mechanism to refine the global context at each level in the feature pyramid. In the end, we also present a lightweight version of our method, which only slightly increases model complexity and computational burden. Experimental results on COCO benchmark dataset demonstrate the significant advantages of our approach.

For the time being, there are many groundbreaking object detection frameworks used in natural scene images. These algorithms have good detection performance on the data sets of open natural scenes. However, applying these frameworks to remote sensing images directly is not very effective. The existing deep-learning-based object detection algorithms still face some challenges when dealing with remote sensing images because these images usually contain a number of targets with large variations of object sizes as well as interclass similarity. Aiming at the challenges of object detection in optical remote sensing images, we propose an end-to-end cross-scale feature fusion (CSFF) framework, which can effectively improve the object detection accuracy. Specifically, we first use a feature pyramid network (FPN) to obtain multilevel feature maps and then insert a squeeze and excitation (SE) block into the top layer to model the relationship between different feature channels. Next, we use the CSFF module to obtain powerful and discriminative multilevel feature representations. Finally, we implement our work in the framework of Faster region-based CNN (R-CNN). In the experiment, we evaluate our method on a publicly available large-scale data set, named DIOR, and obtain an improvement of 3.0% measured in terms of mAP compared with Faster R-CNN with FPN.

With great significance in military and civilian applications, detecting indistinguishable small objects in wide-scale remote sensing images is still a challenging topic. In this letter, we propose a specially optimized one-stage network (SOON) focusing on extracting spatial information of high-resolution images by understanding and analyzing the combination of feature and semantic information of small objects. The SOON model consists of feature enhancement, multiscale detection, and feature fusion. The first part is implemented by constructing a receptive field enhancement (RFE) module and incorporating it into the network’s specific parts where the information of small objects mainly exists. The second part is achieved by four detectors with different sensitivities, which access to the fused and enhanced features to enable the network to make full use of features in different scales. The third part consolidates the high-level and low-level features by adopting upsampling, concatenation, and convolution operations to build a feature pyramid structure, which explicitly yields strong feature representation and semantic information. In addition, we introduce the soft-nonmaximum suppression to preserve accurate bounding boxes in the postprocessing stage for densely arranged objects. Note that the split and merge strategy and the multiscale training strategy are employed. Extensive experiments and thorough analysis are performed on the NorthWestern Polytechnical University Very-High-Resolution (NWPU VHR)-10-v2 data set and the airplane, car and ship (ACS) data set as compared with several state-of-the-art methods. The satisfactory performance in experiments verifies the effectiveness of the design and optimization.

Many deep learning detectors have been proposed to address the scale variation issue in pedestrian detection. However, it is still far from being fully solved because most of these detectors handle the issue by directly fusing multiple extracted features with the weighted average or concatenation. Therefore, this paper designs a novel and effective anchor-free network (namely MPAF-Net) for multi-scale pedestrian detection. Specifically, we first improve Res2Net as our backbone network by designing a novel bottleneck block to make the proposed detector has a strong multi-scale representation ability for pedestrians. Furthermore, based on Conditional Random Fields (CRFs), the multi-scale features obtained from three different feature pyramid levels of backbone network are refined with a designed message passing mechanism. By doing so, the complementary information from the features at other scales can be dynamically passed to enhance the scale-specific feature. Finally, pedestrian detection is simplified as a straightforward center and scale prediction task by performing convolution on the concatenation of refined multi-scale convolution features. This way, the proposed MPAF-Net can be trained in an end-to-end fashion. Experimental results show that MPAF-Net achieves state-of-the-art performance on CityPersons, Caltech, and CrowdHuman benchmarks.

In this paper, we explore an issue that is to detect vehicle headlights from the nighttime traffic surveillance images with highly reflections. In the night, reflections on the road (water) surface, vehicles bodies, or some reflective objects (such as lane markings, traffic signs) will interfere the headlight detection seriously. Although the existing methods have achieved good results, however, most of them failed to detect the headlight when headlights are far from camera. In order to solve the issue, we propose a novel method for vehicle headlights detection. The proposed method makes full use of the brightness and gradient information of the headlights in the night. First, we propose an effective region-of-interest (ROI) segmentation method which is based on multi-scale local saliency detection. The method pre-serve faint or small-sized objects and retain the original shape of the object to the greatest extent. Then, we compute the pyramid histogram of oriented gradients (PHOG) features, which are used to train support vector machine (SVM) classifier. Finally, the extracted bright blocks are classified according to the pre-trained SVM classifier. Experimental results and quantitative evaluations in different scenes demonstrate that our proposed method can achieve a better result compared with previous methods.

Convolutional Neural Networks (CNNs) have played an important role in saliency detection. How to detect a salient object as a whole is a key issue. However, most existing learning-based methods are not accurate enough to detect salient objects in complex scenes, such as easily overlooked small salient areas in a whole salient object, which is called scale imbalance problem in this paper. To address this issue, Scale Balance Network (SBN) based on fully convolutional network is proposed to accurately recognize and comprehensively detect salient objects. Firstly, to detect more small salient areas, a specially designed backbone instead of common backbone is adopted in this paper, which can capture larger resolution with more spatial features in deeper layers. Secondly, we present a novel progressive pyramid mechanism named Connective Feature Pyramid Module (CFPM), aiming to make the network focus on the balance between the large salient areas and the small ones. Finally, we present an Edge Enhancement Architecture with Various Kernels (EEAVK) to locate the saliency maps and refine the boundary features. Experimental results on five benchmark datasets show that the proposed SBN method achieves consistently superior performance in comparison with other state-of-the-art ones under different evaluation metrics.

Feature pyramids have delivered significant improvement in object detection. However, building effective feature pyramids heavily relies on expert knowledge, and also requires strenuous efforts to balance effectiveness and efficiency. Automatic search methods, such as NAS-FPN, automates the design of feature pyramids, but the low search efficiency makes it difficult to apply in a large search space. In this paper, we propose a novel search framework for a feature pyramid network, called AutoDet, which enables to automatic discovery of informative connections between multi-scale features and configure detection architectures with both high efficiency and state-of-the-art performance. In AutoDet, a new search space is specifically designed for feature pyramids in object detectors, which is more general than NAS-FPN. Furthermore, the architecture search process is formulated as a combinatorial optimization problem and solved by a Simulated Annealing-based Network Architecture Search method (SA-NAS). Compared with existing NAS methods, AutoDet ensures a dramatic reduction in search times. For example, our SA-NAS can be up to 30x faster than reinforcement learning-based approaches. Furthermore, AutoDet is compatible with both one-stage and two-stage structures with all kinds of backbone networks. We demonstrate the effectiveness of AutoDet with outperforming single-model results on the COCO dataset. Without pre-training on OpenImages, AutoDet with the ResNet-101 backbone achieves an AP of 39.7 and 47.3 for one-stage and two-stage architectures, respectively, which surpass current state-of-the-art methods.

Recently, object detection has been a vital aspect in the vision community, while scale variation of objects in images or videos usually brings challenge for performance improvement. To combat this problem, conventional paradigms generally adopt image pyramid or Feature Pyramid Network (FPN) to process objects at different scales. However, existing multi-scale deep convolution neural networks mostly set different scales in a heuristic way, which may introduce inconsistency between the region of interest and the semantic scope. In this paper, we propose an innovative paradigm called Consistent Scale Normalization (CSN) to weaken the influence of scale variation for object detection. The proposed CSN can realize a consistent compression for the scale space of objects, in both training and testing phases. Extensive experimental testing is performed on COCO object detection benchmark in comparison with several state-of-the-art methods. In addition to object detection, experiments on instance segmentation and multi-task human pose estimation are also conducted. Furthermore, the CSN paradigm is beneficial to reduce the difficulty of network learning. The results verify the effectiveness and superiority of the CSN paradigm.

High spatial resolution (HSR) imagery scene classification has become a hot research topic in remote sensing. Scene classification method based on the handcrafted features, such as the bag-of-visual-words (BoVW) model, describes an image by extracting local features of the scene and mapping them to the dictionary space, but usually uses a shallow structure and loses the spatial distribution characteristics of the scene. The method based on deep learning extracts hierarchical features to describe the scene, which can maintain the spatial position information well. However, deep features in different levels have scale recognition restrictions for multi-scale ground objects, and cannot understand complex scenes well. In this paper, the multi-level convolutional pyramid semantic fusion (MCPSF) framework is proposed for HSR imagery scene classification. Differing from previous scene classification methods, which integrate the feature of different levels directly, of which the fusion features have large differences in both sparsity and eigenvalue magnitude, MCPSF integrates multi-level semantic features extracted by BoVW model and convolutional neural network (CNN) model. In MCPSF, two convolution pyramid feature expression strategies are proposed to enhance the ability of capturing multi-scale land objects, i.e., local and convolutional pyramid based BoVW (LCPB) model and local and convolutional pyramid based pooling-stretched (LCPP) model. The effectiveness of the proposed method is verified on 21-class UC Merced (UCM) dataset and 30-class Aerial Image Dataset (AID). The framework was also transferred toa case study of scene annotation in Wuhan. The proposed framework significantly improves the performance when compared with other state-of-the-art methods.

Semantic segmentation is a task that covers most of the perception needs of intelligent vehicles in an unified way. Recent studies witnessed that attention mechanisms achieve impressive performance in computer vision task. Current attention mechanisms based segmentation methods differ with each other in position and form of the attention mechanism, and perform differently in practice. This paper firstly introduces the effectiveness of multi-scale context features and attention mechanisms in segmentation tasks. We find that multi-scale and channel attention can play a vital role in constructing effective context features. Based on this analysis, this paper proposes an efficient attention pyramid network (EAPNet) for semantic segmentation. Specifically, to efficient handle the problem of segmenting objects at multiple scales, we design efficient channel attention pyramid (ECAP) which employ atrous convolution with channel attention in cascade or in parallel to capture multi-scale context by using multiple atrous rates. Furthermore, we propose a residual attention fusion block (RAFB), whose purpose is to simultaneously focus on meaningful low-level feature maps and spatial location information. At the same time, we will explore different channel attention modules and spatial attention modules, and describe their impact on network performance. We empirically evaluate our EAPNet on two semantic segmentation datasets, including PASCAL VOC 2012 and Cityscapes datasets. Experimental results show that without MS COCO pre-training and any post-processing, EAPNet achieved 81.7% mIoU on the PASCAL VOC 2012 validation set. With deeplabv3+ as the benchmark, EAPNet improve the model performance of more than 1.50% mIoU.

Fruit detection plays a vital role in harvesting robot platforms. However, complicated environment attributes such as illumination variation, occlusion, have made fruit detection a challenging task. A robust YOLOMuskmelon model that is accurate and fast was proposed to solve detection difficulties. The YOLOMuskmelon model incorporated ReLU activated ResNet43 backbone with new 2,3,4,3,2 residual block arrangement, spatial pyramid pooling(SPP), complete Intersection over Union (CIoU) loss, feature pyramid network(FPN), and distance Intersection over Union-Non Maximum Suppression(DIoU-NMS) to improve detection performance. The obtained average precision (AP) results of YOLOMuskmelon at 89.6% is greater than YOLOv3 at 82.3%, YOLOResNet50 at 85.5%, but less than YOLOv4 at 91.6%. However, the detection speed of YOLOMuskmelon at 96.3 frame per second(fps) outperformed YOLOv3 at 56.6fps, YOLOv4 at 54.1fps and YOLOResNet50 at 71.2fps. Meanwhile, the YOLOMuskmelon which is 56.1% faster than YOLOv4 model showed a better generalization and real-time fruit harvesting robots prospect.

Automatic object detection is a fundamental but challenging problem in very high-resolution (VHR) satellite images’ interpretation. Recently, context-based feature pyramid detection architecture has been proposed, which generates high-quality fusion feature levels for multiscale object detection and significantly improves the accuracy of traditional detection frameworks. However, in feature pyramid architecture, small objects are easily lost in deep feature levels, and the context cues will be weakened simultaneously. Moreover, each feature level in the pyramids is simply constructed from single-level layers of the backbone, which are not representative enough for multiscale object detection. In this letter, an end-to-end multiscale convolutional neural network (MSCNN) is proposed, which is based on a unified multiscale backbone named EssNet for extracting features of diverse-scale objects in HRS images. The proposed EssNet backbone maintains the resolution of deep feature levels, which improves the capability of multiscale objects’ feature expression. Meanwhile, a dilated bottleneck structure is introduced into the backbone, which generates high-quality semantic features and improves the prediction capability of multiscale objects. Finally, the whole network can be optimized end-to-end by minimizing a multitask loss. Experiments on publicly available NWPU VHR-10 benchmark and demonstrate that the proposed method outperforms several state-of-the-art detection approaches.

The performance of state-of-the-art object skeleton detection (OSD) methods have been greatly boosted by Convolutional Neural Networks (CNNs). However, the most existing CNN-based OSD methods rely on a ‘skip-layer’ structure where low-level and high-level features are combined to gather multi-level contextual information. Unfortunately, as shallow features tend to be noisy and lack semantic knowledge, they will cause errors and inaccuracy. Therefore, in order to improve the accuracy of object skeleton detection, we propose a novel network architecture, the Multi-Scale Bidirectional Fully Convolutional Network (MSB-FCN), to better gather and enhance multi-scale high-level contextual information. The advantage is that only deep features are used to construct multi-scale feature representations along with a bidirectional structure for better capturing contextual knowledge. This enables the proposed MSB-FCN to learn semantic-level information from different sub-regions. Moreover, we introduce dense connections into the bidirectional structure to ensure that the learning process at each scale can directly encode information from all other scales. An attention pyramid is also integrated into our MSB-FCN to dynamically control information propagation and reduce unreliable features. Extensive experiments on various benchmarks demonstrate that the proposed MSB-FCN achieves significant improvements over the state-of-the-art algorithms.

We address the problem of depth estimation from a single monocular image in the paper. Depth estimation from a single image is an ill-posed and inherently ambiguous problem. In the paper, we propose an encoder-decoder structure with the feature pyramid to predict the depth map from a single RGB image. More specifically, the feature pyramid is used to detect objects of different scales in the image. The encoder structure aims to extract the most representative information from the original image through a series of convolution operations and to reduce the resolution of the input image. We adopt Res2-50 as the encoder to extract important features. The decoder section uses a novel upsampling structure to improve the output resolution. Then, we also propose a novel loss function that adds gradient loss and surface normal loss to the depth loss, which can predict not only the global depth but also the depth of fuzzy edges and small objects. Additionally, we use Adam as our optimization function to optimize our network and speed up convergence. Our extensive experimental evaluation proves the efficiency and effectiveness of the method, which is competitive with previous methods on the Make3D dataset and outperforms state-of-the-art methods on the NYU Depth v2 dataset.

Monocular depth estimation is an undirected problem, so constructing a network to predict better image depth information is an important research topic. This paper proposes a mixed-scale Unet network (MAPUnet) with a dense atrous pyramid based on the coder-decoder structure widely used in computer vision. We innovatively introduce the Unet++ structure of the image segmentation network for depth estimation. We reset the number of convolutional layers of the network under the framework of the Unet++ network and innovatively connect the decoders densely. Moreover, by choosing the appropriate size of the atrous radius, we form a dense atrous pyramid based on different feature layers to better connect the features in the deep and shallow layers of the network. To verify the effectiveness of the proposed algorithm, we test the network on the KITTI dataset and the NYU Depth V2 dataset. We compare the network with the current state-of-the-art methods. The proposed method has higher accuracy and has steadily improved relative to the threshold of accuracy and root-mean-square error. We also conduct ablation studies, studies targeting the effectiveness of the network framework, and discussions on the convergence time and parameter complexity of the network. We will open-source the code at https://github.com/yang-yi-fan/MAPUnet.

Due to the advantages of real-time detection and improved performance, single-shot detectors have gained great attention recently. To solve the complex scale variations, single-shot detectors make scale-aware predictions based on multiple pyramid layers. Typically, small objects are detected on shallow layers while large objects are detected on deep layers. However, the features in the pyramid are not scale-aware enough, which limits the detection performance. Two common problems in single-shot detectors caused by object scale variations can be observed: (1) false negative problem, i.e. , small objects are easily missed due to the weak features; (2) part-false positive problem, i.e. , the salient part of a large object is sometimes detected as an object. With this observation, a new Neighbor Erasing and Transferring (NET) mechanism is proposed for feature scale-unmixing to explore scale-aware features in this paper. In NET, a Neighbor Erasing Module (NEM) is designed to erase the salient features of large objects and emphasize the features of small objects in shallow layers. A Neighbor Transferring Module (NTM) is introduced to transfer the erased features and highlight large objects in deep layers. With this mechanism, a single-shot network called NETNet is constructed for scale-aware object detection. In addition, we propose to aggregate nearest neighboring pyramid features to enhance our NET. Experiments on MS COCO dataset and UAVDT dataset demonstrate the effectiveness of our method. NETNet obtains 38.5% AP at a speed of 27 FPS and 32.0% AP at a speed of 55 FPS on MS COCO dataset. As a result, NETNet achieves a better trade-off for real-time and accurate object detection.

Accurate and efficient extraction of road marking plays an important role in road transportation engineering, automotive vision, and automatic driving. In this article, we proposed a dense feature pyramid network (DFPN)-based deep learning model, by considering the particularity and complexity of road marking. The DFPN concatenated its shallow feature channels with deep feature channels so that the shallow feature maps with high resolution and abundant image details can utilize the deep features. Thus, the DFPN can learn hierarchical deep detailed features. The designed deep learning model was trained end to end for road marking instance extraction with mobile laser scanning (MLS) point clouds. Then, we introduced the focal loss function into the optimization of deep learning model in road marking segmentation part, to pay more attention to the hard-classified samples with a large extent of background. In the experiments, our method can achieve better results than state-of-the-art methods on instance segmentation of road markings, which illustrated the advantage of the proposed method.

Small fasteners are widely used in a freight train. However, due to the complex working environment, the missing of fasteners always happens, causing traffic accidents and property loss. Thus, detection of the missing of small fasteners is essential for the safety of freight trains. This article takes three critical missing small fasteners as examples, pin-missing, bolt-missing, and rivet-missing defects, and proposes a two-stage small defect detection model. First, a copy–pasting method is used to augment the defects in the training image, then the ResNet-101 backbone is adopted to extract defect features, and a feature pyramid network is used to construct a feature pyramid. After obtaining defect features, an improved region proposal network is introduced to generate defect proposals. Finally, a fully convolutional neural network is trained to operate pixelwise segmentation. The experimental results on manually collected fastener defect data set show that our model possesses high precision and efficiency on the inspection of pin-missing, bolt-missing, and rivet-missing defects. Besides, the proposed model could be used for similar small defects detection.

In recent years, the detection of distinctive objects in stereoscopic 3D images has drawn increasing attention. Unlike 2D salient object detection, salient object detection in stereoscopic 3D images is highly challenging. Hence, we propose a novel Deep Convolutional Residual Autoencoder (DCRA) for end-to-end salient object detection in stereoscopic 3D images. The core trainable architecture of the salient object detection model employs raw stereoscopic 3D images as the inputs and their corresponding ground truth saliency masks as the labels. A convolutional residual module is applied to both the encoder and the decoder as a basic building block in the DCRA, and long-range skip connections are employed to bypass the equal-sized feature maps between the encoder and the decoder. To explore the complex relationships and exploit the complementarity between RGB (photometric) and depth (geometric) information, multiple feature map fusion modules are constructed. These modules integrate texture and structure information between the RGB and depth branches of the encoder and fuse their features over several multiscale layers. Finally, to efficiently optimize DCRA parameters, a supervision pyramid based on boundary loss and background prior loss is adopted, which employs supervised learning over the multiscale layers in the decoder to prevent vanishing gradients and accelerate the training at the fusion stage. We compare the proposed DCRA with state-of-the-art methods on two challenging benchmark datasets. The results of these experiments demonstrate that our proposed DCRA performs favorably against the comparison models.

Multi-view based 3D reconstruction aims to obtain 3D structure information of objects in space through two-dimensional images. In this paper, we propose a new multi-view stereo network that can robustly reconstruct the scene. To enhance the feature representation ability of Point-MVSNet, a pyramid attention module is introduced. Specifically, we exploit the attention mechanism for the multi-scale feature pyramid to capture larger receptive fields and richer information. Instead of constructing a feature pyramid as the input, results of the pyramid attention module at different scales are directly used for the next layer. The network eventually generates a high-quality depth estimation for 3D reconstruction from sparse to dense by an iterative refinement schedule. Experiments have been performed to evaluate 3D reconstruction quality by comparison with existing state-of-the-art methods on the DTU dataset. The experimental results indicate our method performs the best in overall quality compared with previous methods, proving the effectiveness of our method. In the end, we use the data collected by mobile devices to implement 3D reconstruction with a combination of traditional and learning-based methods, providing ideas for the 3D reconstruction technology on mobile devices.

Traditional deep learning detection network has poor effect on the detection of infrared dim and small targets on the sea in the case of interference or bad weather. In this paper, an improved dim and small infrared ship detection network based on Haar wavelet is proposed. The HaarConv module is designed based on the high-frequency features obtained by Haar wavelet decomposition, which further increases the feature extraction ability of the backbone network for small targets. Meanwhile, the HaarUp-HaarDown module is designed by using Haar forward and inverse transform, replacing the up-sampling layer of the feature pyramid network and the down-sampling layer of backbone network to retain smaller target features. Furthermore, the pseudo-label-based method enables the network to conduct semi-supervised learning, which reduces the labeling cost and improves detection accuracy while expanding the amount of training data. The above method is applied to the YOLOv5-s lightweight network and 11278 infrared images (3352 labeled) of dim and small ships are collected as a dataset. The results show that the introduction of semi-supervised training method effectively expands the training dataset, and the mAP@.5:.95 increases by 23.5%. The proposed Haar wavelet improvement method can effectively improve the detection accuracy of dim and small infrared ship targets by more than 2%, and the number of parameters increases by only about 0.02M. Compared with existing methods, the proposed method reaches the state-of-the-art result and has good generalization performance.

Enhancement of low-light images is a challenging task due to the impact of low brightness, low contrast, and high noise. The inability to collect natural labeled data intensifies this problem further. Many researchers have attempted to solve this problem using learning-based approaches; however, most models ignore the impact of noise in low-lit images. In this paper, an encoder-decoder architecture, made up of separable convolution layers that solve the issues encountered in low-light image enhancement, is proposed. The architecture is trained end-to-end on a custom low-light image dataset (LID), comprising both clean and noisy images. We introduce a unique multi-context feature extraction module (MC-FEM) where the input first passes through a feature pyramid of dilated separable convolutions for hierarchical-context feature extraction followed by separable convolutions for feature compression. The model is optimized using a novel three-part loss function that focuses on high-level contextual features, structural similarity, and patch-wise local information. We conducted several ablation studies to determine the optimal model for low-light image enhancement under noisy and noiseless conditions. We have used performance metrics like peak-signal-to-noise ratio, structural similarity index matrix, visual information fidelity, and average brightness to demonstrate the superiority of the proposed work against the state-of-the-art algorithms. Qualitative results presented in this paper prove the strength and suitability of our model for real-time applications.

Object detection is one of the core tasks in computer vision. Object detection algorithms often have difficulty detecting objects with diverse scales, especially those with smaller scales. To cope with this issue, Lin et al. proposed feature pyramid networks (FPNs), which aim for a feature pyramid with higher semantic content at every scale level. The FPN consists of a bottom-up pyramid and a top-down pyramid. The bottom-up pyramid is induced by a convolutional neural network as its layers of feature maps. The top-down pyramid is formed by progressive up-sampling of a highly semantic yet low-resolution feature map at the top of the bottom-up pyramid. At each up-sampling step, feature maps of the bottom-up pyramid are fused with the top-down pyramid to produce highly semantic yet high-resolution feature maps in the top-down pyramid. Despite significant improvement, the FPN still misses small-scale objects. To further improve the detection of small-scale objects, this paper proposes scale adaptive feature pyramid networks (SAFPNs). The SAFPN employs weights chosen adaptively to each input image in fusing feature maps of the bottom-up pyramid and top-down pyramid. Scale adaptive weights are computed by using a scale attention module built into the feature map fusion computation. The scale attention module is trained end-to-end to adapt to the scale of objects contained in images of the training dataset. Experimental evaluation, using both the 2-stage detector faster R-CNN and 1-stage detector RetinaNet, demonstrated the proposed approach’s effectiveness.

Surface defect detection is a critical task in industrial production process. Nowadays, there are lots of detection methods based on computer vision and have been successfully applied in industry, they also achieved good results. However, achieving full automation of surface defect detection remains a challenge, due to the complexity of surface defect, in intraclass. While the defects between interclass contain similar parts, there are large differences in appearance of the defects. To address these issues, this article proposes a pyramid feature fusion and global context attention network for pixel-wise detection of surface defect, called PGA-Net. In the framework, the multiscale features are extracted at first from backbone network. Then the pyramid feature fusion module is used to fuse these features into five resolutions through some efficient dense skip connections. Finally, the global context attention module is applied to the fusion feature maps of adjacent resolution, which allows effective information propagate from low-resolution fusion feature maps to high-resolution fusion ones. In addition, the boundary refinement block is added to the framework to refine the boundary of defect and improve the result of the prediction. The final prediction is the fusion of the five resolutions fusion feature maps. The results of evaluation on four real-world defect datasets demonstrate that the proposed method outperforms the state-of-the-art methods on mean intersection of union and mean pixel accuracy (NEU-Seg: 82.15%, DAGM 2007: 74.78%, MT_defect: 71.31%, Road_defect: 79.54%).