Search

Accurate object detection requires correct classification and high-quality localization. Currently, most of the single shot detectors (SSDs) conduct simultaneous classification and regression using a fully convolutional network. Despite high efficiency, this structure has some inappropriate designs for accurate object detection. The first one is the mismatch of bounding box classification, where the classification results of the default bounding boxes are improperly treated as the results of the regressed bounding boxes during the inference. The second one is that only one-time regression is not good enough for high-quality object localization. To solve the problem of classification mismatch, we propose a novel reg-offset-cls (ROC) module including three hierarchical steps: the regression of the default bounding box, the prediction of new feature sampling locations, and the classification of the regressed bounding box with more accurate features. For high-quality localization, we stack two ROC modules together. The input of the second ROC module is the output of the first ROC module. In addition, we inject a feature enhanced (FE) module between two stacked ROC modules to extract more contextual information. The experiments on three different datasets (i.e., MS COCO, PASCAL VOC, and UAVDT) are performed to demonstrate the effectiveness and superiority of our method. Without any bells or whistles, our proposed method outperforms state-of-the-art one-stage methods at a real-time speed. The source code is available at https://github.com/JialeCao001/HSD.

3D reconstruction from single-view images is a long-standing research problem. To date, there are various methods based on point clouds and volumetric representations. In spite of success in 3D models generation, it is quite challenge for these approaches to deal with models with complex topology and fine geometric details. Thanks to the recent advance of deep shape representations,learning the structure and detail representation using deep neural networks is a promising direction. In this paper, we propose a novel approach named STD-Net to reconstruct 3D models utilizing mesh representation that is well suited for characterizing complex structures and geometry details. Our method consists of (1) an auto-encoder network for recovering the structure of an object with bounding box representation from a single-view image; (2) a topology-adaptive GCN for updating vertex position for meshes of complex topology; and (3) a unified mesh deformation block that deforms the structural boxes into structure-aware meshes. Evaluation on ShapeNet shows that STD-Net has better performance than state-of-the-art methods in reconstructing complex structures and fine geometric details.

The austere challenge of visual object tracking is to find the target to be tracked in various noise interference and obtain its accurate bounding box coordinates. Recently, the object tracking technology based on the Siamese network has made great breakthroughs, and more and more Siamese network trackers have been proposed with superior performance. They still have some shortcomings. To this end, a new Multi-Stage visual tracking algorithm with Siamese Anchor-Free Proposal Network (MS-SiamAFPN) is proposed in this paper. The algorithm is a three-stage Siamese network tracker composed of Feature Extraction and Fusion (FEF) sub-network, Classification and Regression (CR) sub-network, Validation and Regression (VR) sub-network in series. Firstly, the Anchor-Free Proposal Network (AFPN) module is designed in the CR stage, which can make full use of positive and negative samples for training while reducing neural network parameters. Secondly, aim to achieve better robustness and recognizability in the VR stage, on the one hand, a novel Feature Purification (FP) module is designed, which can automatically select the important channels, and extract the features of irregular regions on the input fusion features, so as to strengthen the representation ability of image features. On the other hand, the target recognition and position regression are regarded as different processing tasks, and the recognition score and position fine-tuning of candidate targets are obtained by newly designing the Dual-Branch Network (DBN) structure, thereby avoiding feature ambiguity. Due to the synergy of the above these innovations, MS-SiamAFPN has obtained a large performance improvement, and achieved SOTA performance in multiple public dataset benchmarks.

Weakly supervised semantic segmentation is receiving great attention due to its low human annotation cost. In this paper, we aim to tackle bounding box supervised semantic segmentation, i.e., training accurate semantic segmentation models using bounding box annotations as supervision. To this end, we propose Affinity Attention Graph Neural Network ($A^2$GNN). Following previous practices, we first generate pseudo semantic-aware seeds, which are then formed into semantic graphs based on our newly proposed affinity Convolutional Neural Network (CNN). Then the built graphs are input to our $A^2$GNN, in which an affinity attention layer is designed to acquire the short- and long- distance information from soft graph edges to accurately propagate semantic labels from the confident seeds to the unlabeled pixels. However, to guarantee the precision of the seeds, we only adopt a limited number of confident pixel seed labels for $A^2$GNN, which may lead to insufficient supervision for training. To alleviate this issue, we further introduce a new loss function and a consistency-checking mechanism to leverage the bounding box constraint, so that more reliable guidance can be included for the model optimization. Experiments show that our approach achieves new state-of-the-art performances on Pascal VOC 2012 datasets (val: 76.5\%, test: 75.2\%). More importantly, our approach can be readily applied to bounding box supervised instance segmentation task or other weakly supervised semantic segmentation tasks, with state-of-the-art or comparable performance among almot all weakly supervised tasks on PASCAL VOC or COCO dataset. Our source code will be available at https://github.com/zbf1991/A2GNN., Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TAPMI 2021)

Small-object detection has long been a challenge. High-megapixel cameras are used to solve this problem in industries. However, current detectors are inefficient for high-resolution images. In this work, we propose a new module called Pre-Locate Net, which is a plug-and-play structure that can be combined with most popular detectors. We inspire the use of classification ideas to obtain candidate regions in images, greatly reducing the amount of calculation, and thus achieving rapid detection in high-resolution images. Pre-Locate Net mainly includes two parts, candidate region classification and behavior classification. Candidate region classification is used to obtain a candidate region, and behavior classification is used to estimate the scale of an object. Different follow-up processing is adopted according to different scales to balance the variance of the network input. Different from the popular candidate region generation method, we abandon the idea of regression of a bounding box and adopt the concept of classification, so as to realize the prediction of a candidate region in the shallow network. We build a high-resolution dataset of aircraft and landing gears covering complex scenes to verify the effectiveness of our method. Compared to state-of-the-art detectors (e.g., Guided Anchoring, Libra-RCNN, and FASF), our method achieves the best mAP of 94.5 on 1920 × 1080 images at 16.7 FPS.

Existing RGBT tracking methods usually localize a target object with a bounding box, in which the trackers are often affected by the inclusion of background clutter. To address this issue, this article presents a novel algorithm, called noise-robust cross-modal ranking, to suppress background effects in target bounding boxes for RGBT tracking. In particular, we handle the noise interference in cross-modal fusion and seed labels from the following two aspects. First, the soft cross-modality consistency is proposed to allow the sparse inconsistency in fusing different modalities, aiming to take both collaboration and heterogeneity of different modalities into account for more effective fusion. Second, the optimal seed learning is designed to handle label noises of ranking seeds caused by some problems, such as irregular object shape and occlusion. In addition, to deploy the complementarity and maintain the structural information of different features within each modality, we perform an individual ranking for each feature and employ a cross-feature consistency to pursue their collaboration. A unified optimization framework with an efficient convergence speed is developed to solve the proposed model. Extensive experiments demonstrate the effectiveness and efficiency of the proposed approach comparing with state-of-the-art tracking methods on GTOT and RGBT234 benchmark data sets.

Accurate and fast 3D object detection from point clouds is a key task in autonomous driving. Existing one-stage 3D object detection methods can achieve real-time performance, however, they are dominated by anchor-based detectors which are inefficient and require additional post-processing. In this paper, we eliminate anchors and model an object as a single point--the center point of its bounding box. Based on the center point, we propose an anchor-free CenterNet3D network that performs 3D object detection without anchors. Our CenterNet3D uses keypoint estimation to find center points and directly regresses 3D bounding boxes. However, because inherent sparsity of point clouds, 3D object center points are likely to be in empty space which makes it difficult to estimate accurate boundaries. To solve this issue, we propose an extra corner attention module to enforce the CNN backbone to pay more attention to object boundaries. Besides, considering that one-stage detectors suffer from the discordance between the predicted bounding boxes and corresponding classification confidences, we develop an efficient keypoint-sensitive warping operation to align the confidences to the predicted bounding boxes. Our proposed CenterNet3D is non-maximum suppression free which makes it more efficient and simpler. We evaluate CenterNet3D on the widely used KITTI dataset and more challenging nuScenes dataset. Our method outperforms all state-of-the-art anchor-based one-stage methods and has comparable performance to two-stage methods as well. It has an inference speed of 20 FPS and achieves the best speed and accuracy trade-off. Our source code will be released at https://github.com/wangguojun2018/CenterNet3d., 13 pages, 5 figures

3D tracking in point cloud is a core competence of autonomous robots to perceive and forecast the environment. How to initialize bounding box seeds and optimize their position and orientation are very crucial for 3D object tracking in point clouds. Nevertheless, existing methods mainly resort to developing a powerful classifier based on the initial bounding box seeds. In this work, we propose an end-to-end deep supervised descent method (SDM), which seamlessly integrates multiple seeds generation for the initialization of seeds and sequential updates for the estimation of accurate result. Specifically, we start with transforming the SDM iterative process into a trainable recurrent module. It explicitly learns a series of descent directions in the parameter space, to gradually optimize the initial seeds. Moreover, to alleviate drifting of this process, we initialize multiple seeds based on aggregated point sets generated by the deep Hough voting. Besides, a discrimination module is introduced to determine the bounding box with the highest score as the final result. Importantly, a specific multi-task loss is proposed to train our model in an end-to-end way. Experiments on KITTI, PandaSet and Waymo datasets show that our method could achieve significant improvements (up to 11.2\% in success ratio) as compared to state-of-the-art trackers.

Intelligent systems, like driver assistance systems, remain within vehicles and help drivers by providing essential information about traffic, blockage of roads, and possible routes for safe driving. The objective of scene text spotting in a driver assistance system is to localize and recognize scene texts, signs of milestones, traffic panels, and road marks in natural scene images. However, text edges get fainted due to adverse weather conditions, like fog, rain, smog, or poor contrast. This makes the task of spotting more challenging. In this paper, we propose an end-to-end trainable deep neural network, known as Met-MLTS, that can address the issue of spotting multi-oriented text instances in scene images captured in adverse meteorological conditions. It localizes words, predicts script class, and performs word spotting for every rotated bounding box. It is a fast multilingual scene text spotter that utilizes hierarchical spatial context, channel-wise inter-dependencies, and semantic edge supervision to localize and recognize words and predict script class in scene images using smartphones. We explore inter-class interference to reduce the misclassification problem. A light-weight recognition module for multilingual character segmentation, word-level recognition, and script identification is incorporated. We demonstrate the efficacy of our spotting network on resource-constraint devices.

Recently, 3D object detection based on deep learning has achieved impressive performance in complex indoor and outdoor scenes. Among the methods, the two-stage detection method performs the best; however, this method still needs improved accuracy and efficiency, especially for small size objects or autonomous driving scenes. In this paper, we propose an improved 3D object detection method based on a two-stage detector called the Improved Point-Voxel Region Convolutional Neural Network (IPV-RCNN). Our proposed method contains online training for data augmentation, upsampling convolution and k-means clustering for the bounding box to achieve 3D detection tasks from raw point clouds. The evaluation results on the KITTI 3D dataset show that the IPV-RCNN achieved a 96% mAP, which is 3% more accurate than the state-of-the-art detectors.

Autonomous unmanned aerial vehicles (UAVs) are essential for detecting and tracking specific events, such as automatic navigation. The intelligent monitoring of people's social distances in crowds is one of the most significant events caused by the coronavirus. The virus is spreading more quickly among the crowds, and the disease cycle continues in congested areas. Due to the error that occurs when humans monitor their activity, an automated model is required to alert to social distance violations in crowds. As a result, this article proposes a two-step framework based on autonomous UAV videos, including human tracking and deep learning-based recognition of the crowd's social distance. The deep architecture is a modified-fast and lightweight ShuffleNet learning structure. First, the Kalman filter is used to determine the positions of individuals, and then the modified ShuffleNet is used to refine the bounding boxes obtained and determine the social distance. The social distance is calculated using the initial refinement of the bounding box obtained during the tracking step and the scale in frames of the human body. The observed average accuracy, average processing time (APT), and processed frame per second (FPS) for three congestion datasets were 97.5%, 84 milliseconds, and 11.5 FPS, respectively. Real-time decision-making was achieved by reducing the size and resolution of the frames. Additionally, the frames were re-labeled to reduce the computational complexity associated with detecting social distancing. The experimental results demonstrated that the proposed method could operate more quickly and accurately on various resolution frames of UAV videos with difficult conditions. IEEE

Thermal sensors play an important role in intelligent transportation system. This paper studies the problem of RGB and thermal (RGBT) tracking in challenging situations by leveraging multimodal data. A RGBT object tracking method is proposed in correlation filter tracking framework based on short term historical information. Given the initial object bounding box, hierarchical convolutional neural network (CNN) is employed to extract features. The target is tracked for RGB and thermal modalities separately. Then the backward tracking is implemented in the two modalities. The difference between each pair is computed, which is an indicator of the tracking quality in each modality. Considering the temporal continuity of sequence frames, we also incorporate the history data into the weights computation to achieve a robust fusion of different source data. Experiments on three RGBT datasets show the proposed method achieves comparable results to state-of-the-art methods.

In this paper, we explore the challenging 1-to-N map matching problem, which exploits a compact description of map data, to improve the scalability of map matching techniques used by various robot vision tasks. We propose a first method explicitly aimed at fast succinct map matching, which consists only of map-matching subtasks. These tasks include offline map matching attempts to find a compact part-based scene model that effectively explains each map using fewer larger parts. The tasks also include an online map matching attempt to efficiently find correspondence between the part-based maps. Our part-based scene modeling approach is unsupervised and uses common pattern discovery (CPD) between the input and known reference maps. This enables a robot to learn a compact map model without human intervention. We also present a practical implementation that uses the state-of-the-art CPD technique of randomized visual phrases (RVP) with a compact bounding box (BB) based part descriptor, which consists of keypoint and descriptor BBs. The results of our challenging map-matching experiments, which use a publicly available radish dataset, show that the proposed approach achieves successful map matching with significant speedup and a compact description of map data that is tens of times more compact. Although this paper focuses on the standard 2D point-set map and the BB-based part representation, we believe our approach is sufficiently general to be applicable to a broad range of map formats, such as the 3D point cloud map, as well as to general bounding volumes and other compact part representations., Offprint of IROS2013 paper

Evolutionary algorithms can outperform conventional placement algorithms such as simulated annealing, analytical placement as well as manual placement on metrics such as runtime, wirelength, pipelining cost, and clock frequency when mapping FPGA hard block intensive designs such as systolic arrays on Xilinx UltraScale+ FPGAs. For certain hard-block intensive, systolic array accelerator designs, the commercial-grade Xilinx Vivado CAD tool is unable to provide a legal routing solution without tedious manual placement constraints. Instead, we formulate an automatic FPGA placement algorithm for these hard blocks as a multi-objective optimization problem that targets wirelength squared and maximum bounding box size metrics. We build an end-to-end placement and routing flow called RapidLayout using the Xilinx RapidWright framework. RapidLayout runs 5-6$\times$ faster than Vivado with manual constraints and eliminates the weeks-long effort to generate placement constraints manually for the hard blocks. We also perform automated post-placement pipelining of the long wires inside each convolution block to target 650MHz URAM-limited operation. RapidLayout outperforms (1) the simulated annealer in VPR by 33% in runtime, 1.9-2.4$\times$ in wirelength, and 3-4$\times$ in bounding box size, while also (2) beating the analytical placer UTPlaceF by 9.3$\times$ in runtime, 1.8-2.2$\times$ in wirelength, and 2-2.7$\times$ in bounding box size. We employ transfer learning from a base FPGA device to speed-up placement optimization for similar FPGA devices in the UltraScale+ family by 11-14$\times$ than learning the placements from scratch., Comment: 8 pages, 10 figures. Conference: FPL 2020

Textual information appearing on the captured image may contain personal information. In various circumstances, publishing such images in the public domain may create a threat of privacy leak. To avoid these situations, we propose a text concealment method. To accomplish this task, we have used a conditional generator, which is a text region conditioned concealment network. The text regions predicted by the detector network are used as a conditioning criterion for the concealment process. The text region prediction in the form of a word-level bounding box may contain stroke pixels as well as the background pixels. However, to reduce the number of background pixels for proper conditioning of text concealment network, a character level annotation is used for the generator in place of word-level bounding box annotation. It helps to focus more on strokes as compared to the background pixels. A character-level symmetric line representation of text has been proposed to obtain finer level text region prediction as compared to the character-level bounding box. The proposed model is trainable from end-to-end. The text region conditioned generator is trained from the loss of global and local discriminators. The proposed method is validated on public scene text image datasets such as ICDAR 2015, COCO-Text and Synthesis dataset. The proposed architecture shows competitive results as compared to other state-of-the-art approaches.

Many nations and organizations are committing to achieving the goal of `Vision Zero' which aims to bring the number of road deaths close to zero by the year 2050. The core of the strategy is a safe transportation system with optimized vehicles and transportation routes. The industry continues to develop integrated safety systems to make vehicles safer, smarter, and more capable in safety-critical scenarios. Passive safety systems are now focusing on pre-crash deployment of restraint systems to better protect vehicle passengers. Current commonly used bounding box methods for the shape estimation of potential crash partners lack the fidelity required for edge case collision detection and advanced crash modeling of future pre-crash technologies. This has led to the development of novel algorithms for vehicle contour estimation in literature. With this work, we present a framework for assessing and comparing different contour estimation algorithms, including a simple bounding box, oriented bounding box, polynomial fit estimation, complemented convex hull, and three-arc fit. Tests on simulated virtual and experimental LiDAR measurements of a simplified vehicle contour have been conducted to determine performance at varying relative angles and distances. It has been concluded that the convex hull and the three-arc methods are the best performing of the studied algorithms, with each having different strengths. The three-arc algorithm offers higher accuracy estimations at low relative angles and near-mid distances, whereas the convex hull method requires low computation time and can provide accurate estimations even at extreme relative angles and distances.

Recently, the object detection and recognition based applications are widely adopted in various real-time and offline applications. The computer vision based automatic learning schemes have gained huge attraction from researchers due to their significant nature of learning that can significantly improve the detection performance. The advances in deep and convolutional neural networks have improved the efficiency of applications based on recognition and detection. However, enhancing precision, decreasing detection error, and detecting camouflaged items are still regarded as difficult problems. In this work, we concentrated on these problems and presented a model based on Faster-RCNN that utilizes saliency detection, proposal generation and bounding box regression, for better detection along with loss functions. The suggested method is referred to as the saliency driven Faster RCNN model for object detection and recognition using computer vision approach (SGFr-RCNN). The performance of the suggested strategy is assessed using the data sets (PASCAL VOC 2007, PASCAL VOC 2012 & CAMO_UOW) and contrasted with current methods in terms of mean average precision. The comparative research demonstrates the important improvement in the results of the suggested strategy relative to the current methods.

Object detection and location from remote sensing (RS) images is challenging, computationally expensive, and labor intense. Benefiting from research on convolutional neural networks (CNNs), the performance in this field has improved in the recent years. However, object detection methods based on CNNs require a large number of images with annotation information for training. For object location, these annotations must contain bounding boxes. Furthermore, objects in RS images are usually small and densely co-located, leading to a high cost of manual annotation. We tackle the problem of weakly supervised object detection under such conditions, aiming to learn detectors with only image-level annotations, i.e., without bounding box annotations. Based on the fact that the feature maps of a CNN are localizable, we hierarchically fuse the location information from the shallow feature map with the class activation map to obtain accurate object locations. In order to mitigate the loss of small or densely distributed objects, we introduce a divergent activation module and a similarity module into the network. The divergent activation module is used to improve the response strength of the low-response areas in the shallow feature map. Densely distributed objects in RS images, such as aircraft in an airport, often exhibit a certain similarity. The similarity module is used to improve the feature distribution of the shallow feature map and to suppress background noise. Comprehensive experiments on a public dataset and a self-assembled dataset (which we made publicly available) show the superior performance of our method compared to state-of-the-art object detectors.

The extraction of text information from traffic panels is one of the challenging problems in computer vision. Although the past decade has seen a promising shift and important progress in object detection, few works and a limited number of datasets focus specifically on extracting text from traffic signs. To address the lack of data for text detection in traffic panels, especially those with Arabic scripts, this paper introduces a new multilingual and multipurpose dataset named ASAYAR. It consists of three sub-datasets: Arabic-Latin scene text localization, traffic sign detection, and directional symbol detection. The dataset contains 1763 images collected on different Moroccan highways, and annotated manually, using 16 object categories. The fully annotated ASAYAR images contains more than 20000 bounding box objects. The paper also investigates the usability of the dataset, by evaluating the performance of the state-of-the-art algorithms for object and text detection. Experimental results show good detection scores, demonstrating the potential contribution of ASAYAR in the development of methods for text extraction from traffic panels.

Visual grounding (VG) aims to locate the most relevant object or region in an image, based on a natural language query. Generally, it requires the machine to first understand the query, identify the key concepts in the image, and then locate the target object by specifying its bounding box. However, in many real-world visual grounding applications, we have to face with ambiguous queries and images with complicated scene structures. Identifying the target based on highly redundant and correlated information can be very challenging, and often leading to unsatisfactory performance. To tackle this, in this paper, we exploit an attention module for each kind of information to reduce internal redundancies. We then propose an accumulated attention (A-ATT) mechanism to reason among all the attention modules jointly. In this way, the relation among different kinds of information can be explicitly captured. Moreover, to improve the performance and robustness of our VG models, we additionally introduce some noises into the training procedure to bridge the distribution gap between the human-labeled training data and the real-world poor quality data. With this "noised" training strategy, we can further learn a bounding box regressor, which can be used to refine the bounding box of the target object. We evaluate the proposed methods on four popular datasets (namely ReferCOCO, ReferCOCO+, ReferCOCOg, and GuessWhat?!). The experimental results show that our methods significantly outperform all previous works on every dataset in terms of accuracy.

Weakly supervised semantic instance segmentation with only image-level supervision, instead of relying on expensive pixel-wise masks or bounding box annotations, is an important problem to alleviate the data-hungry nature of deep learning. In this article, we tackle this challenging problem by aggregating the image-level information of all training images into a large knowledge graph and exploiting semantic relationships from this graph. Specifically, our effort starts with some generic segment-based object proposals (SOP) without category priors. We propose a multiple instance learning (MIL) framework, which can be trained in an end-to-end manner using training images with image-level labels. For each proposal, this MIL framework can simultaneously compute probability distributions and category-aware semantic features, with which we can formulate a large undirected graph. The category of background is also included in this graph to remove the massive noisy object proposals. An optimal multi-way cut of this graph can thus assign a reliable category label to each proposal. The denoised SOP with assigned category labels can be viewed as pseudo instance segmentation of training images, which are used to train fully supervised models. The proposed approach achieves state-of-the-art performance for both weakly supervised instance segmentation and semantic segmentation. The code is available at https://github.com/yun-liu/LIID.

Recent years have witnessed the prevalence of using the Siamese network for RGB-T tracking because of its remarkable success in RGB object tracking. Despite their faster than real-time speeds, existing RGB-T Siamese trackers suffer from low accuracy and poor robustness, compared to other state-of-the-art RGB-T trackers. To address such issues, a new complementarity- and distractor-aware RGB-T tracker based on Siamese network (referred to as SiamCDA) is developed in this paper. To this end, several modules are presented, where the feature pyramid network (FPN) is incorporated into the Siamese network to capture the cross-level information within unimodal features extracted from the RGB or the thermal images. Next, a complementarity-aware multi-modal feature fusion module (CA-MF) is specially designed to capture the cross-modal information between RGB features and thermal features. In the final bounding box selection phase, a distractor-aware region proposal selection module (DAS) further enhances the robustness of our tracker. On top of the technical modules, we also build a large-scale, diverse synthetic RGB-T tracking dataset, containing more than 4831 pairs of synthetic RGB-T videos and 12K synthetic RGB-T images. Extensive experiments on three RGB-T tracking benchmark datasets demonstrate the outstanding performance of our proposed tracker with a tracking speed over 37 frames per second (FPS).

Instance segmentation needs to locate all instances in an image correctly and segment each instance precisely. Currently, the most dominant methods for instance segmentation take object detection as a pre-task. However, they rely on the accuracy of object detection incredibly. If the pre-task cannot predict an accurate bounding box, the performance of instance segmentation will degenerate. In this paper, we present a novel method for instance segmentation to solve this problem, which is called Segmenting Beyond the Bounding Box (S3B-Net).Our S3B-Net designs a sub-network to help instance segmentation methods based on object detection to segment the part of an instance beyond the bounding box. Specifically, the sub-network first predicts a two-dimensional pixel embedding for each pixel. Then, the Gaussian function is employed to calculate a pixel’s probability belongs to a corresponding instance according to the two-dimensional pixel embedding. Finally, the output of the sub-network combines with the output of instance segmentation based on object detection to generate a more precise instance mask. Our sub-network can easily extend on the existing instance segmentation method based on object detection to segment instance beyond the bounding box. We do our experiments on dominant instance segmentation datasets, such as the COCO dataset and Cityscapes dataset. The results show that our method can achieve 6.8 points gain compared with the baseline Mask R-CNN with ResNet-50-FPN in Cityscapes datasets, and 1.7 points gain with ResNet-101-FPN-DCN in COCO datasets. Our S3B-Net outperforms the previous state-of-the-art instance segmentation method, which proves our method is competitive. The source code of our method will be made available.

The manipulating robots receive much attention by offering better services, where object grasping is still challenging especially under background interferences. In this article, a novel two-stream grasping convolutional neural network (CNN) with simultaneous detection and segmentation is proposed. The proposed method is cascaded by an improved simultaneous detection and segmentation network BlitzNet and a two-stream grasping CNN TsGNet. The improved BlitzNet introduces the channel-based attention mechanism, and achieves an improvement of detection accuracy and segmentation accuracy with the combination of the learning of multitask loss weightings and background suppression. Based on the obtained bounding box and the segmentation mask of the target object, the target object is separated from the background, and the corresponding depth map and grayscale map are sent to TsGNet. By adopting depthwise separable convolution and designed global deconvolution network, TsGNet achieves the best grasp detection with only a small amount of network parameters. This best grasp in the pixel coordinate system is converted to a desired 6-D pose for the robot, which drives the manipulator to execute grasping. The proposed method combines a grasping CNN with simultaneous detection and segmentation to achieve the best grasp with a good adaptability to background. With the Cornell grasping dataset, the image-wise accuracy and object-wise accuracy of the proposed TsGNet are 93.13% and 92.99%, respectively. The effectiveness of the proposed method is verified by the experiments.

Side collisions caused by sudden vehicle cut-ins comprise a significant proportion of traffic accidents. Due to the complex and dynamic nature of traffic environments, the warning algorithms in advanced driving assistant systems (ADAS) often misjudge and misdiagnose risk and omit necessary warnings, because they rely solely on the sensing information of the single vehicle equipped with ADAS and have limited insights from and communication with the surrounding vehicles and traffic environment. To improve the effectiveness of ADAS in cut-in scenarios, this study established a collision warning model in a vehicle-to-vehicle (V2V) communication environment. Firstly, based on the support vector machine-recursive feature elimination (SVM-RFE) lane-change intent-recognition model, the lane-change feasibility and the change rate of the lateral offset, the logical ``and'' was used to establish a lane-change behavior prediction model, and a trajectory prediction model was established based on the long short-term memory (LSTM). Then, based on the proposed comprehensive prediction model for lane-change behavior, the driving trajectory prediction model, and the oriented bounding box (OBB) detection algorithm, a collision warning model was established for a V2V environment. Finally, based on a driving simulation platform and a real-world vehicle test, a cut-in experiment in a V2V environment was designed and implemented. By comparing the warning confusion matrix and warning time, it was found that the proposed cut-in collision warning model is superior to the traditional collision warning model. The results of this study can provide new modeling ideas and a theoretical basis for ADAS to further optimize for a cut-in scenario.

Mobile Laser Scanning (MLS) systems provide highly dense 3D point clouds that enable the acquisition of accurate traffic facilities information for intelligent transportation system. Road guardrails with safety features that can separate traffic and define moving spaces for pedestrians and vehicles face challenges such as diverse guardrail types and continuous slopes in point clouds data. This paper proposes a novel approach for rapidly extracting urban road guardrails from MLS point clouds, combining a proposed multi-level filtering with a modified Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering, and adapting for most types of guardrails and rough slope roads. We develop a multi-level filter to detect the road surface and remove the undesirable points. Through a proposed modified DBSCAN clustering, the guardrails are extracted after a four-step screening, which includes the limits based on the number of points, the fitting error, the bounding box size and the average reflection intensity for each cluster. The proposed method achieves high precisions of 97.2% and 96.4% respectively for the lane-separating guardrails and the anti-fall guardrails on the dataset. Extensive experiments with test dataset captured by a RIEGL VMX-450 MLS, show that our method outperforms the state-of-the-art method to extract 3D guardrails from point clouds.

Building footprint delineation from remote sensing imagery is a basic task in surveying and mapping and geographic information system (GIS), which benefits many engineering applications but requires an enormous amount of human delineation. In this study, we aim to find a way to replace human delineation with automated algorithms. For this purpose, we designed a novel pipeline for the automated production of building vector maps from aerial images, which consists of a bounding-box generation module, a graph convolutional network (GCN)-based polygon prediction module, and an empirical polygon regularization module. First, we introduce the bounding box generation module based on region-based object detection, which along with our overlap cropping strategy is used to generate a bounding box for each building instance. The generated bounding boxes have a horizontal rectangle version and a rotated version, which are used to initialize the next stage of our method. Second, we propose a GCN-based method, which is the core of this study, to conduct the initial accurate building polygon prediction by integrating multiresolution optimization and multilevel loss constraints. Our proposed method, to the authors' best knowledge, is the first of its kind that introduces GCN to building extraction. The final step is to apply a regularization algorithm to translate the predicted polygons into structured and highly accurate building footprints. We validated the proposed method on two large aerial building data sets, WHU data set and Inria data set, where it was shown to significantly outperform other state-of-the-art methods more than 10%. Specifically, with the ground-truth rotated bounding boxes of buildings, our method is able to automatically delineate 91% of the buildings in the WHU data set and with the predicted rotated bounding the percent reached human-level delineation.

Most existing trackers use bounding boxes for object tracking. However, the background contained in the bounding box inevitably decreases the accuracy of the target model, which affects the performance of the tracker and is particularly pronounced for non-rigid objects. To address the above issue, this paper proposes a novel hybrid level set model, which can robustly address the issue of topology changing, occlusions and abrupt motion in non-rigid object tracking by accurately tracking the object contour. In particular, an appearance model is first obtained by repeatedly training and relabeling the initial labeled frame using competing one-class SVMs. Then, by integrating the trained appearance model, an edge detector and image spatial information into the level set model, a new hybrid level set model is presented, which accurately locates the object contour and feeds back to the competing one-class SVMs to update the appearance model of the next frame. In addition, a motion model is defined to predict the accurate location of the object when occlusion and abrupt motion occur in the next frame. Finally, the experimental results on state-of-the-art benchmarks demonstrate the feasibility and effectiveness of the proposed model and the superiority of the proposed method over existing trackers in terms of accuracy and robustness.

Ship detection has broad applications in many areas, including fishery management, maritime rescue and maritime monitoring. Recently, numerous detectors based on deep learning have been carried in ship detection in synthetic aperture radar (SAR) images. However, detecting the inshore ships faces enormous challenges because of the strong scattering interference of the inland area. In order to address such issues, a novel method named strong scattering points network for ship detection is proposed in this paper. First, according to the SAR imaging mechanism, the ships usually appear strong scattering phenomenon in the SAR images. Therefore, the proposed method detects the strong scattering points on the ship, and then aggregates their positions to obtain the ship’s arbitrary orientation box. Second, our method designs an embedding vector to cluster these points as an individual object to regress the oriented bounding box. Third, in order to distinguish the strong scattering points on land, a ship attention module is employed to extract the image texture features and representations of local features. It can suppress the false alarm caused by land interference in the detection process. Furthermore, to demonstrate the effectiveness of the proposed algorithm, this paper introduces a new ship dataset for oriented ship detection named large-scale dataset for ship detection in SAR images (LDSD). Moreover, the public SAR ship detection dataset (SSDD) is utilized to verify the robustness and generalization ability of the detector. The experimental results on two datasets show that our method has a strong anti-interference ability in the inshore background and achieves state-of-the-art detection performance.

We present COVID-CT-Mask-Net model that predicts COVID-19 from CT scans. The model works in two stages: first, it detects the instances of ground glass opacity and consolidation in CT scans, then predicts the condition from the ranked bounding box detections. To develop the solution for the three-class problem (COVID, common pneumonia and control), we used the COVIDx-CT dataset derived from the dataset of CT scans collected by China National Center for Bioinformation. We use about 5% of the training split of COVIDx-CT to train the model, and without any complicated data normalization, balancing and regularization, and training only a small fraction of the model’s parameters, we achieve a 90.80% COVID sensitivity, 91.62% common pneumonia sensitivity and 92.10% normal sensitivity, and an overall accuracy of 91.66% on the test data (21182 images), bringing the ratio of test/train data to 7.06, which implies a very high capacity of the model to generalize to new data. We also establish an important result, that ranked regional predictions (bounding boxes with scores) in Mask R-CNN can be used to make accurate predictions of the image class. The full source code, models and pretrained weights are available on https://github.com/AlexTS1980/COVID-CT-Mask-Net.

Obstacle avoidance for robotic manipulators can be challenging when they operate in unstructured environments. This problem is probed with the sim-to-real (sim2real) deep reinforcement learning, such that a moving policy of the robotic arm is learnt in a simulator and then adapted to the real world. However, the problem of sim2real adaptation is notoriously difficult. To this end, this work proposes (1) a unified representation of obstacles and targets to capture the underlying dynamics of the environment while allowing generalization to unseen goals and (2) a flexible end-to-end model combining the unified representation with the deep reinforcement learning control module that can be trained by interacting with the environment. Such a representation is agnostic to the shape and appearance of the underlying objects, which simplifies and unifies the scene representation in both simulated and real worlds. We implement this idea with a vision-based actor-critic framework by devising a bounding box predictor module. The predictor estimates the 3D bounding boxes of obstacles and targets from the RGB-D input. The features extracted by the predictor are fed into the policy network, and all the modules are jointly trained. This makes the policy learn object-aware scene representation, which leads to a data-efficient learning of the obstacle avoidance policy. Our experiments in simulated environment and the real-world show that the end-to-end model of the unified representation achieves better sim2real adaption and scene generalization than state-of-the-art techniques.

Automatic building extraction from remote sensing imagery is crucial to urban construction and management. To address the main challenges of diverse building scale and appearance, this letter proposes an automatic building instance extraction method based on an improved hybrid task cascade (HTC). Our method consists of three components by obtaining high-resolution representation, defining guided anchor, and forming focal loss to boost the adaptability of automatic building instance extraction. Comprehensive experimental results on WHU aerial building data set demonstrated that compared with the mainstream Mask R-CNN method, our method increased AP and AR in bounding box branch and mask branch by 9.8%-6.5% and 10.7%-8.0% respectively, especially APS and APL in the two branches by 10.1%-6.9% and 3.4%-2.4%, respectively. We evaluated the effectiveness and complexity of these components separately and discussed the universality and practicability of deep learning method in automatic building extraction.

Although the orientation and scale properties of the objects in remote sensing images have been widely considered in the modern deep learning based object detection methods, the spatial distribution property of objects has rarely been investigated. There is a distinct spatial distribution difference between close-range objects and remote sensing objects: the former may exhibit extensive mutual occlusion and overlap, whereas the latter rarely overlap. A current remote sensing object detection algorithm that ignores the spatial distribution difference may unnecessarily apply the massive anchor-based proposal bounding box generation and non-maximum suppression (NMS) operations. In this paper, considering the unique spatial distribution of remote sensing objects, and also the other spatial properties, we propose a novel, compact, and spatial-oriented object detection framework for remote sensing images. The proposed two-stage convolutional neural network (CNN) framework, which we call RSADet (the Remote-sensing Spatial Adaptation DETector), considers the spatial distribution, scale, and orientation/shape varieties of the objects in remote sensing images. In the first stage, each object instance is inferred on the scale-attention boosted CNN heatmaps to generate candidate bounding boxes, instead of using the anchor-based proposal box generation and NMS. In the second stage, deformable convolutions are introduced to adapt to the geometric variations of different object instances and to avoid the impact of complex and changeable backgrounds. A new bounding box confidence (IoU score) prediction branch is introduced as a convenient constraint for eliminating unreliable boxes and improving the performance. Experiments were conducted on a large single-class remote sensing object detection dataset (the Ningbo Pylon dataset) built as part of this study and an open-source extraordinarily large multi-class dataset (the DIOR dataset). Compared with the advanced detectors from both the computer vision and remote sensing communities, the proposed RSADet achieved a state-of-the-art performance on both datasets.

The ``shared head for classification and localization'' (sibling head) has been leading the fashion of the object detection community over the past years. However, the article by Song et al. (2020) found that there is a spatial misalignment between the two object functions of sibling head and proposed an operator called task-aware spatial disentanglement (TSD) to decouple the classification and regression from the spatial dimension by generating two disentangled proposals. But there are two problems in applying TSD to detecting rotated objects. The first problem is that the deformable pooling with the disentangled proposals in TSD is not rotation-invariant. The second problem is that the angle offset is not considered when getting the disentangled proposals. To address the above problems, we propose a rotation-invariant TSD (RITSD) method. The proposed method adopts the rotation-invariant deformable pooling and angle offset learning when getting region-of-interest (RoI) features. Considering the rotated anchor increases the computational burden and training time, we use a three-stage backbone that first utilizes horizontal proposals to obtain rotated proposals and then obtains the predicted rotated bounding boxes. Finally, detailed ablation studies on our collected dataset and HRSC2016 were performed to demonstrate the feasibility and superiority of our proposed method. From the heatmap, we find the features of near bow and stern areas are beneficial for classification and the side areas of ships are good at bounding box regression.

An improved face detection method ground on TinyYOLOv3 algorithm is put forward in this paper in view of the low recognition rate of traditional face detection methods in complex background and the long detection time of existing face detection methods ground on deep learning The main network of TinyYOLOv3 is redesigned to extract more abundant semantic information, which is increasing the number of network layers. The deep separable convolution is used instead of the traditional convolution and the features of different network layers are fused. Meanwhile, the accuracy of face detection is guaranteed, size of this network model is reduced. The CIoU (complete intersection over union) loss is used on improving this original prediction loss of bounding box coordinates. The channel attention mechanism is integrated into the feature extraction network to improve the positioning accuracy and detection accuracy. The network convergence speed is accelerated by hyperparameter optimization and priori box clustering method. The face detection experiments were conducted on the Wider Face data sets. And the experimental results manifest that the proposed algorithm has high recognition accuracy in complex scenes. At the same time, the proposed algorithm is better than others in terms of detection speed and model size, and can meet the requirements of embedded devices.

As an important remote sensing means, synthetic aperture radar (SAR) has many superiorities to other sensors. How to effectively detect and locate ships in SAR images is also a popular field. In previous ship detection research, most algorithms focus on detecting the horizontal bounding box of ship targets, which ignore the rotation angle of each ships. Thus, too much background noise in the horizontal detection results makes them difficult to describe each ship accurately. Inspired by the powerful feature representation ability of convolutional neural networks (CNNs), a novel anchor-free and keypoint-based deep learning method is proposed for oriented ship detection in multiresolution SAR images. Our detector first extracts multilevel features from the input SAR image with a backbone network and feature pyramid network. Next, considering multiscale ships in multiresolution SAR images, we detect different sizes of ships on different levels of feature maps with identical head network structures. In each head network, the classification subnetwork determines each pixel in feature maps as the central pixel of this ship or not, and the regression subnetwork regresses the oriented bounding box for each ship. In the training process, the proposed oriented nonnormalized Gaussian function is used to describe the center point of ship targets, while the nonuniform weighting of the different level loss functions is used to suppress the imbalanced sample distribution. Experimental results on two authoritative SAR-oriented ship detection datasets and two Gaofen-3 images demonstrate the effectiveness and robustness of the proposed methods.

Ship detection in the synthetic aperture radar (SAR) image is of great significance in the fields of military and coastal defense. Most ship detection methods are designed based on the object detection framework, which can only provide the vertices' coordinates of the bounding box covering the ship targets but cannot provide more detailed contour information. Target segmentation can further explore the shape and edge information of the objects, which can be used as a blazing novel means for automatic object detection. In this letter, a 3-D atrous encoder-decoder neural network with global attention modules (GAM-EDNet) is proposed to achieve ship segmentation in SAR images. The encoder-decoder structure with atrous convolution is developed as the network body to fully exploit the structural information of the ship targets with various sizes. To increase the structural information of the single-polarization SAR images, a 3-D image cube is designed as the input of the GAM-EDNet. A global attention module is proposed to further improve the segmentation performance by integrating the high-level semantic features with the low-level location features. Besides, an SAR ship segmentation dataset (SAR-HR4) is built to evaluate the segmentation performance, and the experimental results show that the proposed GAM-EDNet achieves better performance than other state-of-the-art methods.

This study develops a deep learning (DL) model to extract the ship size from Sentinel-1 synthetic aperture radar (SAR) images, named SSENet. We employ a single shot multibox detector (SSD)-based model to generate a rotatable bounding box (RBB) for the ship. We design a deep-neural-network (DNN)-based regression model to estimate the accurate ship size. The hybrid inputs to the DNN-based model include the initial ship size and orientation angle obtained from the RBB and the abstracted features extracted from the input SAR image. We design a custom loss function named mean scaled square error (MSSE) to optimize the DNN-based model. The DNN-based model is concatenated with the SSD-based model to form the integrated SSENet. We employ a subset of the OpenSARShip, a data set dedicated to Sentinel-1 ship interpretation, to train and test SSENet. The training/testing data set includes 1500/390 ship samples. Experiments show that SSENet is capable of extracting the ship size from SAR images end to end. The mean absolute errors (MAEs) are under 0.8 pixels, and their length and width are 7.88 and 2.23 m, respectively. The hybrid input significantly improves the model performance. The MSSE reduces the MAE of length by nearly 1 m and increases the MAE of width by 0.03m compared to the mean square error (MSE) loss function. Compared with the well-performed gradient boosting regression (GBR) model, SSENet reduces the MAE of length by nearly 2 m (18.68%) and that of width by 0.06 m (2.51%). SSENet shows robustness on different training/testing sets.

Ship detection has been an active and vital topic in the field of remote sensing for a decade, but it is still a challenging problem due to the large scale variations, the high aspect ratios, the intensive arrangement, and the background clutter disturbance. In this letter, we propose a locality-aware rotated ship detection (LARSD) framework based on a multi-scale convolutional neural network (CNN) to tackle these issues. The proposed framework applies a UNet-like multi-scale CNN to generate multi-scale feature maps with high-level semantic information in high resolution. Then, a rotated anchor-based regression is applied for directly predicting the probability, the edge distances, and the angle of ships. Finally, a locality-aware score alignment is proposed to fix the mismatch between classification results and location results caused by the independence of each subnet. Furthermore, to enlarge the datasets of ship detection, we build a new high-resolution ship detection (HRSD) dataset, where 2499 images and 9269 instances were collected from Google Earth with different resolutions. Experiments based on public dataset HRSC2016 and our HRSD dataset demonstrate that our detection method achieves state-of-the-art performance., Comment: 5 pages, 8 figures

This paper proposes a novel method for active collision avoidance to protect the human who enters a robot’s workspace in a human-robot collaborative environment. The proposed method uses a somatosensory sensor to monitor the robot’s workspace and detect anyone attempting to enter it. When someone enters the workspace, a Kinect detects and calculates the position of his or her skeleton points in real-time. However, due to the measurement errors and noise of the device, the tracking error increases over time. Therefore, the proposed method applies an improved particle filter (IPF) to accurately estimate the position of the skeleton points. In order to detect the human-robot collision in real-time, the proposed method uses cylinders to establish the bounding box model for human bones and robots and the human-robot collision is replaced by the collision between the cylinders, greatly improving the efficiency of collision detection. Moreover, taking human safety and productivity into account, the robot velocity control is carried out based on the distance between the robot and human. Then, the proposed method uses a rule-based logic system to analyze human motion so that the robot can take appropriate measures to avoid humans. Finally, the dynamic roadmap (DRM) approach plans new paths in real-time to allow robots to bypass humans. By actively avoiding collisions, the proposed method ensures that the robot will never touch the human body. The significant advantage of the proposed method is that it can detect humans in real-time, analyze their behavior and protect humans without any modification to the robot. The proposed method has been tested in practical applications, and the results show that it can successfully guarantee the safety of people entering the robot’s workspace.

Object detection is regarded as a significant research branch of optical remote sensing image analysis. Aimed at the characteristics of remote sensing image targets such as direction diversity, small scales, and dense arrangements, the state-of-the-art detectors used in optical remote sensing images are primarily extended from some general natural detection approaches based on deep convolutional neural networks (DCNNs). Nevertheless, the detectors intended for remote sensing targets are commonly based on an anchor mechanism and perform regression tasks in the Cartesian coordinate system, which requires complex multiangle anchors, inclined nonmaximum suppression (NMS), and so on. In this letter, a high-resolution polar network (HRPNet) is proposed to detect targets in remote sensing images. As an anchor-free and high-resolution detection network, HRPNet converts the detection of the oriented bounding box into the regression of one polar angle and four polar radii at the corresponding pole points in polar coordinates, which dramatically decreases the computational complexities. As revealed by extensive experiments, the proposed HRPNet shows a competitive advantage on the DOTA and HRRSD data sets.

The key to efficient person search is jointly localizing pedestrians and learning discriminative representation for person re-identification (re-ID). Some recently developed models are built with separate detection and re-ID branches on top of shared region feature extraction networks. There are two factors that are detrimental to re-ID feature learning. One is the background information redundancy resulting from the large receptive field of neurons. The other is the body part missing and background clutter caused by inaccurate localization. In this work, a bottom-up fusion (BUF) subnet is proposed to fuse the bounding box features pooled from multiple network stages. With a few parameters introduced, BUF leverages the multi-level features with various sizes of receptive fields to mitigate the background-bias problem. To further suppress the non-pedestrian regions, the newly introduced segmentation head generates a foreground probability map as guidance for the network to focus on the foreground regions. The resulting foreground attention module (FAM) enhances the foreground features. Moreover, for robust feature learning in practical person search, we propose to adaptively smooth the labels of the pedestrian boxes with consideration of the detection quality. Extensive experiments on PRW and CUHK-SYSU validate the effectiveness of the proposals. Our Bottom-Up Foreground-Aware Feature Fusion (BUFF) network with ALS achieves considerable gains over the state-of-the-art on PRW and competitive performance on CUHK-SYSU.

Vehicle detection, the process of identifying vehicles as axis-aligned bounding boxes in still images, is widely used to estimate the range, time-to-collision, and motion of autonomous vehicles (AVs). Bounding boxes, while convenient, are too coarse to adapt well to vehicle shape and pose variations. In this work, we present TBox (Trapezoid & Box), a novel fine-grained representation useful for both localization and recognition that extends the bounding box by restricting the spatial extent of a vehicle to a set of keypoints and indicating semantically significant local areas using subclasses. In contrast to the previous monolithic models, we propose a cascaded anchor-free architecture to estimate the bounding box and TBox. One subnetwork uses a stacked hourglass network to detect each vehicle as a pair of corners without using anchors. Specifically, it learns corner affinity fields, enabling it to perform robust corner grouping. The other subnetwork estimates a TBox as a set of keypoints. This subnetwork utilizes the bounding box results to avoid ambiguous keypoint associations and reuses existing features to reduce the number of parameters. We also propose a multitask learning strategy for training the cascaded model that implicitly integrates the global context with local details, introducing improvements for both tasks. During testing, a refinement algorithm explicitly uses robust local keypoints to correct possible global box errors, ensuring tight geometric representations for nearby critical vehicles. The experiments show that our method outperforms existing anchor-free detectors for vehicle detection and achieves better performance on the TBox task while using a small model.

Voting-based methods (e.g., VoteNet) have achieved promising results for 3D object detection. However, the simple voting operation in VoteNet may lead to less accurate voting results that are far away from the true object centers. In this work, we propose a simple but effective 3D object detection method called Transformer3D-Det (T3D), in which we additionally introduce a transformer based vote refinement module to refine the voting results of VoteNet and can thus significantly improve the 3D object detection performance. Specifically, our T3D framework consists of three modules: a vote generation module, a vote refinement module, and a bounding box generation module. Given an input point cloud, we first utilize the vote generation module to generate multiple coarse vote clusters. Then, the clustered coarse votes will be refined by using our transformer based vote refinement module to produce more accurate and meaningful votes. Finally, the bounding box generation module takes the refined vote clusters as the input and generates the final detection results for the input point cloud. To suppress the effect of the inaccurate votes, we also propose a new non-vote loss function to train our T3D. As a result, our T3D framework can achieve better 3D object detection performance. Comprehensive experiments on two benchmark datasets ScanNetV2 and SUN RGB-D demonstrate the effectiveness of our T3D framework for 3D object detection.