Search

Unmanned Aerial Vehicle (UAV) has seen a dramatic rise in popularity for remote-sensing image acquisition and analysis in recent years. It has brought promising results in low-altitude monitoring tasks that require detailed visual inspections. Semantic segmentation is one of the hot topics in UAV remote sensing image analysis, as its capability to mine contextual semantic information from UAV images is crucial for achieving a fine-grained understanding of scenes. However, in the remote sensing field, recent reviews have not focused on combining "UAV remote sensing" and "semantic segmentation" to summarize the advanced works and future trends. In this study, we focus primarily on describing various recent semantic segmentation methods applied in UAV remote sensing images and summarizing their advantages and limitations. According to the distinction in modeling contextual semantic information, we have categorized and outlined the methods based on graph-based contextual models and deep-learning-based models. Publicly available UAV-based image datasets are also gathered to encourage systematic research on advanced semantic segmentation methods. We provide quantitative results of representative methods on two high-resolution UAV-based image datasets for fair comparisons and discussions in terms of semantic segmentation accuracy and model inference efficiency. Besides, this paper concludes some remaining challenges and future directions in semantic segmentation for UAV remote sensing images and points out that methods based on deep learning will become the future research trend. • Summarize semantic segmentation methods and datasets for UAV remote sensing images. • Provide fair comparisons of representative methods in accuracy and efficiency. • Identify challenges and future directions for UAV image semantic segmentation. • Semantic segmentation methods based on deep learning will be the mainstream. [ABSTRACT FROM AUTHOR]

Feature pyramid network (FPN) is a typical structure in object detection. It can improve the accuracy of detection results by fusing feature information at different resolutions and enhancing the expression ability of different levels of features. Among them, the mismatch between the resolution of feature information and the receptive field and the limited way of feature fusion hinder the full exchange of feature information. To solve the above problems, this paper designs a new structure called an object detection feature pyramid network based on context information and an efficient bidirectional fusion (CB-FPN): (1) Before feature fusion, this study designs a context enhancement module with cross stage partial network (CSPNet) module (CEM-CSP). By using carefully designed dilated convolutions on high-level features, rich context information and receptive fields are obtained to match appropriate feature information. (2) In feature fusion, this study designed a bidirectional efficient feature pyramid network (BE-FPN) module to fuse features efficiently. After adding these two modified architectures to Faster R-CNN with ResNet-50, the average precision (AP) improves from 37.5 to 39.2 on COCO val-2017 data set. In addition, extensive experiments show the effectiveness of our methods on one-stage, two-stage, and anchor-free models. [ABSTRACT FROM AUTHOR]

With the advancement of science and technology, the development and application of unmanned mobile vehicles (UMVs) have emerged as topics of crucial concern in the global industry. The development goals and directions of UMVs vary according to their industrial uses, which include navigation, autonomous driving, and environmental recognition; these uses have become the priority development goals of researchers in various fields. UMVs employ sensors to collect environmental data for environmental analysis and path planning. However, the analysis function of a single sensor is generally affected by natural environmental factors, resulting in poor identification results. Therefore, this study introduces fusion technology that employs heterogeneous sensors in the Ackerman UMV, leveraging the advantages of each sensor to enhance accuracy and stability in environmental detection and identification. This study proposes a fusion technique involving heterogeneous imaging and LiDAR (laser imaging, detection, and ranging) sensors in an Ackerman UMV. A camera is used to obtain real-time images, and YOLOv4-tiny and simple online real-time tracking are then employed to detect the location of objects and conduct object classification and object tracking. LiDAR is simultaneously used to obtain real-time distance information of detected objects. An inertial measurement unit is used to gather odometry information to determine the position of the Ackerman UMV. Static maps are created using simultaneous localization and mapping. When the user commands the Ackerman UMV to move to the target point, the vehicle control center composed of the robot operating system activates the navigation function through the navigation control module. The Ackerman UMV can reach the destination and instantly identify obstacles and pedestrians when in motion. [ABSTRACT FROM AUTHOR]

As a dense instance segmentation problem, rebar counting in a complex environment such as rebar yard and rebar transpotation has received significant attention in both academic and industrial contexts. Traditional counting approaches, such as manual counting and machine vision-based algorithms, are often inefficient or inaccurate since rebars with varied sizes and shapes are stacked overlapping, rebar image is not clear for complex light condition such as dawn, night and strong light, and other environmental noises exist in rebar image; thus, they no longer fulfil the requirements of modern automation. This paper proposes MaskID, an innovative counting method based on deep learning and heuristic strategies. First, an improved version of the Mask region-based convolutional neural network (Mask R-CNN) was designed to obtain the segmentation results through splitting and rescaling so as to capture more detail in a large-scale rebar image. Then, a series of intelligent denoising strategies corresponding to aspect ratio of recognized box, overlapping recognized objects, object distribution and environmental noise, were applied to improve the segmentation results. The performance of the proposed method was evaluated on open-competition and test-platform datasets. The F1-score was found to be over 0.99 on all datasets. The experimental results demonstrate that the proposed method is effective for dense rebar counting and significantly outperforms existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Even though deep reinforcement learning (DRL) has been extensively applied to quadrotor flight control to simplify parameter adjustment, it has some drawbacks in terms of control performance, such as instability and asymmetry. To address these problems, we propose an odd symmetric actor to achieve stable and symmetric control performance, and an even critic to stabilize the training process. Concretely, the bias of neural networks is eliminated, and the absolute value operation is adopted to construct the activation function. Furthermore, we devise a cascade architecture, where each module trained with DRL controls a symmetric subsystem of the quadrotor. Comparative simulations have verified the effectiveness of the proposed control scheme, which shows superiority in dealing with high-dimensional, nonlinear subsystems and disadvantage in dealing with low-dimensional, linear subsystems. [ABSTRACT FROM AUTHOR]

Colorectal polyps are known to be potential precursors to colorectal cancer. Effective polyp segmentation during colonoscopy examinations can help the clinicians accurately locate potential polyp areas, and reduce misdiagnosis and missed diagnosis. Although existing approaches have achieved significant breakthroughs in medical image segmentation, polyp segmentation is still far from being solved. This is mainly due to the following reasons: (1) most of them tend to ignore the feature misalignment issues during the feature aggregation process; and (2) few algorithms explicitly consider the impact of boundary information on the performance of polyp segmentation. To solve the above issues, we formulate a novel neural network for polyp segmentation in endoscopy images. Different from existing approaches, we explore a new paradigm to enhance multi-level feature fusion by introducing the Feature Fusion Module, which leverages the semantic offset field learned to align the multi-level feature maps for resolving the feature misalignment issue. In addition, we design an auxiliary boundary branch to focus on boundary-aware information, and thus boost the performance of boundary prediction. Specifically, the reference boundary map learned through end-to-end optimization can be considered as a complementary feature of the high-level semantics representation, and then integrated into the main branch via the Boundary Embedding Module, which in turn promotes further refinement of the prediction, especially the boundary. Extensive experiments demonstrate that our approach achieves state-of-the-art performance in polyp segmentation. [ABSTRACT FROM AUTHOR]

Currently, an increasing number of convolutional neural networks (CNNs) focus specifically on capturing contextual features (con. feat) to improve performance in semantic segmentation tasks. However, high-level con. feat are biased towards encoding features of large objects, disregard spatial details, and have a limited capacity to discriminate between easily confused classes (e.g., trees and grasses). As a result, we incorporate low-level features (low. feat) and class-specific discriminative features (dis. feat) to boost model performance further, with low. feat helping the model in recovering spatial information and dis. feat effectively reducing class confusion during segmentation. To this end, we propose a novel deep multi-feature learning framework for the semantic segmentation of VHR RSIs, dubbed MFNet. The proposed MFNet adopts a multi-feature learning mechanism to learn more complete features, including con. feat, low. feat, and dis. feat. More specifically, aside from a widely used context aggregation module for capturing con. feat, we additionally append two branches for learning low. feat and dis. feat. One focuses on learning low. feat at a shallow layer in the backbone network through local contrast processing, while the other groups con. feat and then optimizes each class individually to generate dis. feat with better inter-class discriminative capability. Extensive quantitative and qualitative evaluations demonstrate that the proposed MFNet outperforms most state-of-the-art models on the ISPRS Vaihingen and Potsdam datasets. In particular, thanks to the mechanism of multi-feature learning, our model achieves an overall accuracy score of 91.91% on the Potsdam test set with VGG16 as a backbone, performing favorably against advanced models with ResNet101. [ABSTRACT FROM AUTHOR]

Brain tissue segmentation plays a critical role in the diagnosis, treatment, and study of brain diseases. Accurately identifying these boundaries is essential for improving segmentation accuracy. However, distinguishing boundaries between different brain tissues can be challenging, as they often overlap. Existing deep learning methods primarily calculate the overall segmentation results without adequately addressing local regions, leading to error propagation and mis-segmentation along boundaries. In this study, we propose a novel mis-segmentation-focused loss function based on a two-stage nnU-Net framework. Our approach aims to enhance the model's ability to handle ambiguous boundaries and overlapping anatomical structures, thereby achieving more accurate brain tissue segmentation results. Specifically, the first stage targets the identification of mis-segmentation regions using a global loss function, while the second stage involves defining a mis-segmentation loss function to adaptively adjust the model, thus improving its capability to handle ambiguous boundaries and overlapping anatomical structures. Experimental evaluations on two datasets demonstrate that our proposed method outperforms existing approaches both quantitatively and qualitatively. [ABSTRACT FROM AUTHOR]

When a machine tool is used for a long time, its bearing experiences wear and failure due to heat and vibration, resulting in damage to the machine tool. In order to make the machine tool stable for processing, this paper proposes a smart bearing diagnosis system (SBDS), which uses a gradient-weighted class activation mapping (Grad-CAM)-based convolutional neuro-fuzzy network (GC-CNFN) to detect the bearing status of the machine tool. The developed GC-CNFN is composed of a convolutional layer and neuro-fuzzy network. The convolutional layer can automatically extract vibration signal features, which are then classified using the neuro-fuzzy network. Moreover, Grad-CAM is used to analyze the attention of the diagnosis model. To verify the performance of bearing fault classification, the 1D CNN (ODCNN) and improved 1D LeNet-5 (I1DLeNet) were adopted to compare with the proposed GC-CNFN. Experimental results showed that the proposed GC-CNFN required fewer parameters (20K), had a shorter average calculation time (117.7 s), and had a higher prediction accuracy (99.88%) in bearing fault classification. The proposed SBDS can not only accurately classify bearing faults, but also help users understand the current status of the machine tool. [ABSTRACT FROM AUTHOR]

Breast cancer, a common cancer type, is a major health concern in women. Recently, researchers used convolutional neural networks (CNNs) for medical image analysis and demonstrated classification performance for breast cancer diagnosis from within histopathological image datasets. However, the parameter settings of a CNN model are complicated, and using Breast Cancer Histopathological Database data for the classification is time-consuming. To overcome these problems, this study used a uniform experimental design (UED) and optimized the CNN parameters of breast cancer histopathological image classification. In UED, regression analysis was used to optimize the parameters. The experimental results indicated that the proposed method with UED parameter optimization provided 84.41% classification accuracy rate. In conclusion, the proposed method can improve the classification accuracy effectively, with results superior to those of other similar methods. [ABSTRACT FROM AUTHOR]

Face verification models based on centralized training on large face datasets have achieved excellent performance on various test benchmarks. However, due to the increasingly sophisticated privacy protection law, centrally collecting large amount of face images becomes more difficult. We consider learning a face verification model in the federated setting, where each client has access to the face images of only one class and class embeddings cannot be shared to other clients because of data privacy. In this paper, we propose Federated face verification (FedFV), in which server transfers some equivalent class embeddings to clients so that the clients' class embeddings can be separated far away from each other. We show that our proposed method FedFV outperforms the existing approaches in several face verification benchmarks. [ABSTRACT FROM AUTHOR]

Diverse wavefront modulations with multifunctional acoustic devices have been of great interest to physics and engineering communities. However, traditional design methods of multifunctional acoustic devices rely on a deterministic physical model and redundant iterative optimization, resulting in inflexibility and consuming of time. In this work, we present and experimentally implement a deep learning-assisted tunable acoustic metagrating for multifunctional wavefront modulation with 95.2% accuracy and a 105 order of magnitude decrease in computational time compared to a classical optimization method. The presented tunable structure formed by a periodic array of 3C-shaped unit cells excites controllable Willis coupling, exhibiting corresponding asymmetrical scattering patterns. With the support of a deep learning strategy, the optimal configuration between structure parameters and Willis coupling magnitude could be efficiently confirmed, realizing various extraordinary wavefront modulations, including abnormal reflection, perfect beam splitting, and multi-channel energy distribution in arbitrary ratios. The polarizability tensor retrieval method is used to characterize the Willis coupling of different modulation structures, demonstrating the refined abstraction of the deep learning strategy on Willis coupling. Meanwhile, the numerical and experimental results are in good agreement with the desired wavefront modulation, verifying the effectiveness of the proposed method. Our work develops deep learning-assisted multifunctional wavefront modulation with the advantages of high accuracy, efficiency, flexibility, and refined abstraction of a physical mechanism, paving the way for a combination of deep learning and pragmatic multifunctional acoustic applications. [ABSTRACT FROM AUTHOR]

Human identification is an important topic in event detection, person tracking, and public security. There have been numerous methods proposed for human identification, such as face identification, person re-identification, and gait identification. Typically, existing methods predominantly classify a queried image to a specific identity in an image gallery set (I2I). This is seriously limited for the scenario where only a textual description of the query or an attribute gallery set is available in a wide range of video surveillance applications (A2I or I2A). However, very few efforts have been devoted towards modality-free identification, i.e., identifying a query in a gallery set in a scalable way. In this work, we take an initial attempt, and formulate such a novel Modality-Free Human Identification (named MFHI) task as a generic zero-shot learning model in a scalable way. Meanwhile, it is capable of bridging the visual and semantic modalities by learning a discriminative prototype of each identity. In addition, the semantics-guided spatial attention is enforced on visual modality to obtain interpretable representations with both high global category-level and local attribute-level discrimination. Finally, we design and conduct an extensive group of experiments on two common challenging identification tasks, including face identification and person re-identification, demonstrating that our method outperforms a wide variety of state-of-the-art methods on modality-free human identification. [ABSTRACT FROM AUTHOR]

Channel Pruning is one of the most widespread techniques used to compress deep neural networks while maintaining their performances. Currently, a typical pruning algorithm leverages neural architecture search to directly find networks with a configurable width, the key step of which is to identify representative subnet for various pruning ratios by training a supernet. However, current methods mainly follow a serial training strategy to optimize supernet, which is very time-consuming. In this work, we introduce PSE-Net, a novel parallel-subnets estimator for efficient channel pruning. Specifically, we propose a parallel-subnets training algorithm that simulate the forward–backward pass of multiple subnets by droping extraneous features on batch dimension, thus various subnets could be trained in one round. Our proposed algorithm facilitates the efficiency of supernet training and equips the network with the ability to interpolate the accuracy of unsampled subnets, enabling PSE-Net to effectively evaluate and rank the subnets. Over the trained supernet, we develop a prior-distributed-based sampling algorithm to boost the performance of classical evolutionary search. Such algorithm utilizes the prior information of supernet training phase to assist in the search of optimal subnets while tackling the challenge of discovering samples that satisfy resource constraints due to the long-tail distribution of network configuration. Extensive experiments demonstrate PSE-Net outperforms previous state-of-the-art channel pruning methods on the ImageNet dataset while retaining superior supernet training efficiency. For example, under 300M FLOPs constraint, our pruned MobileNetV2 achieves 75.2% Top-1 accuracy on ImageNet dataset, exceeding the original MobileNetV2 by 2.6 units while only cost 30%/16% times than BCNet/AutoAlim. • We propose PSE-Net, a novel parallel-subnets estimator for efficient channel pruning. • The prior-distributed-based sampling algorithm to facilitate subnets search. • 30%/16% total time for estimator training compared with BCNet/AutoSlim. • Exceeds the SOTAs on ImageNet and downstream tasks for a variety of FLOPs budgets. [ABSTRACT FROM AUTHOR]

It is difficult to use supervised machine-learning methods for infrared (IR) and visible (VIS) image fusion (IVF) because of the shortage of ground-truth target fusion images, and image quality and contrast control are rarely considered in existing IVF methods. In this study, we proposed a simple IVF pipeline that converts the IVF problem into a supervised binary classification problem (sharp vs. blur) and uses image enhancement techniques to improve the image quality in three locations in the pipeline. We took a biological vision consistent assumption that the sharp region contains more useful information than the blurred region. A deep binary classifier based on a convolutional neural network (CNN) was designed to compare the sharpness of the infrared region and visible regions. The output score map of the deep classifier was treated as a weight map in the weighted average fusion rule. The proposed deep binary classifier was trained using natural visible images from the MS COCO dataset, rather than images from the IVF domain (called "cross domain training" here). Specifically, our proposed pipeline contains four stages: (1) enhancing the IR and VIS input images by linear transformation and the High-Dynamic-Range Compression (HDRC) method, respectively; (2) inputting the enhanced IR and VIS images to the trained CNN classifier to obtain the weight map; and (3) using a weight map to obtain the weighted average of the enhanced IR and VIS images; and (4) using single scale Retinex (SSR) to enhance the fused image to obtain the final enhanced fusion image. Extensive experimental results on public IVF datasets demonstrate the superior performance of our proposed approach over other state-of-the-art methods in terms of both subjective visual quality and 11 objective metrics. It was demonstrated that the complementary information between the infrared and visible images can be efficiently extracted using our proposed binary classifier, and the fused image quality is significantly improved. The source code is available at https://github.com/eyob12/Deep_infrared_and_visible_image_fusion. [ABSTRACT FROM AUTHOR]

Nowadays, graph representation learning has aroused a lot of research interest, which aims to learn the latent low-dimensional representations of graph nodes, while preserving the graph structure. Based on the local smooth assumption, some existing methods have achieved significant success. However, although the structure information of data has been taken into consideration, these models fail to capture enough connectivity pattern such as high-order connections. To alleviate this issue, we propose a Graph Diffusion Network (GDN) that can dynamically preserve local and global consistency of graph. More specifically, Graph Diffusion Auto-encoder is utilized as the main framework in GDN to nonlinearly maintain global information volume. Different from simple auto-encoders, the forward propagation in our model is conducted through Graph Diffusion System which can guide the random walk of information flow to sense the high-order local relationships on graph. Furthermore, to discover a customized graph structure that reveals the similarities between nodes, the connection relationship between nodes are refined by learned metrics with the preservation of scale-free property. By the dynamically self-refining on the graph structure, it can be promoted towards learning the intrinsic node representations in a progressive way. Experimental results on node classification tasks demonstrate the effectiveness of the proposed GDN model. [ABSTRACT FROM AUTHOR]

• Jointly optimizing the undersampling pattern and reconstruction model. • Optimizing undersampling pattern and probability distribution in a differentiable way. • Formulating the relationship between 2D probability distribution and sampling rate. • The optimized undersampling pattern outperforms the state-of-the-art methods. [Display omitted] With 3D magnetic resonance imaging (MRI), a tradeoff exists between higher image quality and shorter scan time. One way to solve this problem is to reconstruct high-quality MRI images from undersampled k-space. There have been many recent studies exploring effective k-space undersampling patterns and designing MRI reconstruction methods from undersampled k-space, which are two necessary steps. Most studies separately considered these two steps, although in theory, their performance is dependent on each other. In this study, we propose a joint optimization model, trained end-to-end, to simultaneously optimize the undersampling pattern in the Fourier domain and the reconstruction model in the image domain. A 2D probabilistic undersampling layer was designed to optimize the undersampling pattern and probability distribution in a differentiable manner. A 2D inverse Fourier transform layer was implemented to connect the Fourier domain and the image domain during the forward and back propagation. Finally, we discovered an optimized relationship between the probability distribution of the undersampling pattern and its corresponding sampling rate. Further testing was performed using 3D T1-weighted MR images of the brain from the MICCAI 2013 Grand Challenge on Multi-Atlas Labeling dataset and locally acquired brain 3D T1-weighted MR images of healthy volunteers and contrast-enhanced 3D T1-weighted MR images of high-grade glioma patients. The results showed that the recovered MR images using our 2D probabilistic undersampling pattern (with or without the reconstruction network) significantly outperformed those using the existing start-of-the-art undersampling strategies for both qualitative and quantitative comparison, suggesting the advantages and some extent of the generalization of our proposed method. [ABSTRACT FROM AUTHOR]

• We analyze the limitation of the classification head in one-stage detectors, which fills the gap in the literature. • We explain the classifier's limitation by visualizing its representations and analyzing its robustness to the scene context. • The findings give insights to design location-aware multi-dilation module (LAMD) in the classifiers for robust detection. • Experiments on MS COCO across various detectors with different backbones show that our method can achieve higher performance. Recent progress on one-stage detectors focuses on improving the quality of bounding boxes, while they pay less attention to the classification head. In this work, we focus on investigating the influence of the classification head. To understand the behavior of the classifier in one-stage detectors, we resort to the methods of the Explainable deep learning area. We visualize its learned representations via activation maps and analyze its robustness to image scene context. Based on the analysis, we observe that the classifier limits the performance of the detector due to its limited receptive field and the lack of object locations. Then, we design a simple but efficient location-aware multi-dilation module (LAMD) to enhance the weak classifier. We conduct extensive experiments on the COCO benchmark to validate the effectiveness of LAMD. The results suggest that our LAMD can achieve consistent improvements and leads to robust detection across various one-stage detectors with different backbones. [ABSTRACT FROM AUTHOR]

Zero-shot learning (ZSL) has received increasing attention in recent years especially in areas of fine-grained object recognition, retrieval, and image captioning. The key to ZSL is to transfer knowledge from the seen to the unseen classes via auxiliary class attribute vectors. However, the popularly learned projection functions in previous works cannot generalize well since they assume the distribution consistency between seen and unseen domains at sample-level. Besides, the provided non-visual and unique class attributes can significantly degrade the recognition performance in semantic space. In this paper, we propose a simple yet effective convolutional prototype learning (CPL) framework for zero-shot recognition. By assuming distribution consistency at task-level, our CPL is capable of transferring knowledge smoothly to recognize unseen samples. Furthermore, inside each task, discriminative visual prototypes are learned via a distance based training mechanism. Consequently, we can perform recognition in visual space, instead of semantic space. An extensive group of experiments are then carefully designed and presented, demonstrating that CPL obtains more favorable effectiveness, over currently available alternatives under various settings. • A simple yet effective convolutional prototype learning (CPL) framework is proposed for zero-shot recognition task. • CPL can facilitate the effective transferring of the learned knowledge from the source domain to the unseen target domain. • The generated prototypes are more discriminative due to the joint embedding of both the attribute and class level semantic. • Compared with some state-of-the-art ZSL methods, our CPL shows competitive results under various ZSL settings. [ABSTRACT FROM AUTHOR]