Search

Accurate object detection requires correct classification and high-quality localization. Currently, most of the single shot detectors (SSDs) conduct simultaneous classification and regression using a fully convolutional network. Despite high efficiency, this structure has some inappropriate designs for accurate object detection. The first one is the mismatch of bounding box classification, where the classification results of the default bounding boxes are improperly treated as the results of the regressed bounding boxes during the inference. The second one is that only one-time regression is not good enough for high-quality object localization. To solve the problem of classification mismatch, we propose a novel reg-offset-cls (ROC) module including three hierarchical steps: the regression of the default bounding box, the prediction of new feature sampling locations, and the classification of the regressed bounding box with more accurate features. For high-quality localization, we stack two ROC modules together. The input of the second ROC module is the output of the first ROC module. In addition, we inject a feature enhanced (FE) module between two stacked ROC modules to extract more contextual information. The experiments on three different datasets (i.e., MS COCO, PASCAL VOC, and UAVDT) are performed to demonstrate the effectiveness and superiority of our method. Without any bells or whistles, our proposed method outperforms state-of-the-art one-stage methods at a real-time speed. The source code is available at https://github.com/JialeCao001/HSD.

The austere challenge of visual object tracking is to find the target to be tracked in various noise interference and obtain its accurate bounding box coordinates. Recently, the object tracking technology based on the Siamese network has made great breakthroughs, and more and more Siamese network trackers have been proposed with superior performance. They still have some shortcomings. To this end, a new Multi-Stage visual tracking algorithm with Siamese Anchor-Free Proposal Network (MS-SiamAFPN) is proposed in this paper. The algorithm is a three-stage Siamese network tracker composed of Feature Extraction and Fusion (FEF) sub-network, Classification and Regression (CR) sub-network, Validation and Regression (VR) sub-network in series. Firstly, the Anchor-Free Proposal Network (AFPN) module is designed in the CR stage, which can make full use of positive and negative samples for training while reducing neural network parameters. Secondly, aim to achieve better robustness and recognizability in the VR stage, on the one hand, a novel Feature Purification (FP) module is designed, which can automatically select the important channels, and extract the features of irregular regions on the input fusion features, so as to strengthen the representation ability of image features. On the other hand, the target recognition and position regression are regarded as different processing tasks, and the recognition score and position fine-tuning of candidate targets are obtained by newly designing the Dual-Branch Network (DBN) structure, thereby avoiding feature ambiguity. Due to the synergy of the above these innovations, MS-SiamAFPN has obtained a large performance improvement, and achieved SOTA performance in multiple public dataset benchmarks.

Weakly supervised semantic segmentation is receiving great attention due to its low human annotation cost. In this paper, we aim to tackle bounding box supervised semantic segmentation, i.e., training accurate semantic segmentation models using bounding box annotations as supervision. To this end, we propose Affinity Attention Graph Neural Network ($A^2$GNN). Following previous practices, we first generate pseudo semantic-aware seeds, which are then formed into semantic graphs based on our newly proposed affinity Convolutional Neural Network (CNN). Then the built graphs are input to our $A^2$GNN, in which an affinity attention layer is designed to acquire the short- and long- distance information from soft graph edges to accurately propagate semantic labels from the confident seeds to the unlabeled pixels. However, to guarantee the precision of the seeds, we only adopt a limited number of confident pixel seed labels for $A^2$GNN, which may lead to insufficient supervision for training. To alleviate this issue, we further introduce a new loss function and a consistency-checking mechanism to leverage the bounding box constraint, so that more reliable guidance can be included for the model optimization. Experiments show that our approach achieves new state-of-the-art performances on Pascal VOC 2012 datasets (val: 76.5\%, test: 75.2\%). More importantly, our approach can be readily applied to bounding box supervised instance segmentation task or other weakly supervised semantic segmentation tasks, with state-of-the-art or comparable performance among almot all weakly supervised tasks on PASCAL VOC or COCO dataset. Our source code will be available at https://github.com/zbf1991/A2GNN., Accepted by IEEE Transactions on Pattern Analysis and Machine Intelligence (TAPMI 2021)

Small-object detection has long been a challenge. High-megapixel cameras are used to solve this problem in industries. However, current detectors are inefficient for high-resolution images. In this work, we propose a new module called Pre-Locate Net, which is a plug-and-play structure that can be combined with most popular detectors. We inspire the use of classification ideas to obtain candidate regions in images, greatly reducing the amount of calculation, and thus achieving rapid detection in high-resolution images. Pre-Locate Net mainly includes two parts, candidate region classification and behavior classification. Candidate region classification is used to obtain a candidate region, and behavior classification is used to estimate the scale of an object. Different follow-up processing is adopted according to different scales to balance the variance of the network input. Different from the popular candidate region generation method, we abandon the idea of regression of a bounding box and adopt the concept of classification, so as to realize the prediction of a candidate region in the shallow network. We build a high-resolution dataset of aircraft and landing gears covering complex scenes to verify the effectiveness of our method. Compared to state-of-the-art detectors (e.g., Guided Anchoring, Libra-RCNN, and FASF), our method achieves the best mAP of 94.5 on 1920 × 1080 images at 16.7 FPS.

Accurate and fast 3D object detection from point clouds is a key task in autonomous driving. Existing one-stage 3D object detection methods can achieve real-time performance, however, they are dominated by anchor-based detectors which are inefficient and require additional post-processing. In this paper, we eliminate anchors and model an object as a single point--the center point of its bounding box. Based on the center point, we propose an anchor-free CenterNet3D network that performs 3D object detection without anchors. Our CenterNet3D uses keypoint estimation to find center points and directly regresses 3D bounding boxes. However, because inherent sparsity of point clouds, 3D object center points are likely to be in empty space which makes it difficult to estimate accurate boundaries. To solve this issue, we propose an extra corner attention module to enforce the CNN backbone to pay more attention to object boundaries. Besides, considering that one-stage detectors suffer from the discordance between the predicted bounding boxes and corresponding classification confidences, we develop an efficient keypoint-sensitive warping operation to align the confidences to the predicted bounding boxes. Our proposed CenterNet3D is non-maximum suppression free which makes it more efficient and simpler. We evaluate CenterNet3D on the widely used KITTI dataset and more challenging nuScenes dataset. Our method outperforms all state-of-the-art anchor-based one-stage methods and has comparable performance to two-stage methods as well. It has an inference speed of 20 FPS and achieves the best speed and accuracy trade-off. Our source code will be released at https://github.com/wangguojun2018/CenterNet3d., 13 pages, 5 figures

3D tracking in point cloud is a core competence of autonomous robots to perceive and forecast the environment. How to initialize bounding box seeds and optimize their position and orientation are very crucial for 3D object tracking in point clouds. Nevertheless, existing methods mainly resort to developing a powerful classifier based on the initial bounding box seeds. In this work, we propose an end-to-end deep supervised descent method (SDM), which seamlessly integrates multiple seeds generation for the initialization of seeds and sequential updates for the estimation of accurate result. Specifically, we start with transforming the SDM iterative process into a trainable recurrent module. It explicitly learns a series of descent directions in the parameter space, to gradually optimize the initial seeds. Moreover, to alleviate drifting of this process, we initialize multiple seeds based on aggregated point sets generated by the deep Hough voting. Besides, a discrimination module is introduced to determine the bounding box with the highest score as the final result. Importantly, a specific multi-task loss is proposed to train our model in an end-to-end way. Experiments on KITTI, PandaSet and Waymo datasets show that our method could achieve significant improvements (up to 11.2\% in success ratio) as compared to state-of-the-art trackers.

Intelligent systems, like driver assistance systems, remain within vehicles and help drivers by providing essential information about traffic, blockage of roads, and possible routes for safe driving. The objective of scene text spotting in a driver assistance system is to localize and recognize scene texts, signs of milestones, traffic panels, and road marks in natural scene images. However, text edges get fainted due to adverse weather conditions, like fog, rain, smog, or poor contrast. This makes the task of spotting more challenging. In this paper, we propose an end-to-end trainable deep neural network, known as Met-MLTS, that can address the issue of spotting multi-oriented text instances in scene images captured in adverse meteorological conditions. It localizes words, predicts script class, and performs word spotting for every rotated bounding box. It is a fast multilingual scene text spotter that utilizes hierarchical spatial context, channel-wise inter-dependencies, and semantic edge supervision to localize and recognize words and predict script class in scene images using smartphones. We explore inter-class interference to reduce the misclassification problem. A light-weight recognition module for multilingual character segmentation, word-level recognition, and script identification is incorporated. We demonstrate the efficacy of our spotting network on resource-constraint devices.

Recently, 3D object detection based on deep learning has achieved impressive performance in complex indoor and outdoor scenes. Among the methods, the two-stage detection method performs the best; however, this method still needs improved accuracy and efficiency, especially for small size objects or autonomous driving scenes. In this paper, we propose an improved 3D object detection method based on a two-stage detector called the Improved Point-Voxel Region Convolutional Neural Network (IPV-RCNN). Our proposed method contains online training for data augmentation, upsampling convolution and k-means clustering for the bounding box to achieve 3D detection tasks from raw point clouds. The evaluation results on the KITTI 3D dataset show that the IPV-RCNN achieved a 96% mAP, which is 3% more accurate than the state-of-the-art detectors.

Autonomous unmanned aerial vehicles (UAVs) are essential for detecting and tracking specific events, such as automatic navigation. The intelligent monitoring of people's social distances in crowds is one of the most significant events caused by the coronavirus. The virus is spreading more quickly among the crowds, and the disease cycle continues in congested areas. Due to the error that occurs when humans monitor their activity, an automated model is required to alert to social distance violations in crowds. As a result, this article proposes a two-step framework based on autonomous UAV videos, including human tracking and deep learning-based recognition of the crowd's social distance. The deep architecture is a modified-fast and lightweight ShuffleNet learning structure. First, the Kalman filter is used to determine the positions of individuals, and then the modified ShuffleNet is used to refine the bounding boxes obtained and determine the social distance. The social distance is calculated using the initial refinement of the bounding box obtained during the tracking step and the scale in frames of the human body. The observed average accuracy, average processing time (APT), and processed frame per second (FPS) for three congestion datasets were 97.5%, 84 milliseconds, and 11.5 FPS, respectively. Real-time decision-making was achieved by reducing the size and resolution of the frames. Additionally, the frames were re-labeled to reduce the computational complexity associated with detecting social distancing. The experimental results demonstrated that the proposed method could operate more quickly and accurately on various resolution frames of UAV videos with difficult conditions. IEEE

Thermal sensors play an important role in intelligent transportation system. This paper studies the problem of RGB and thermal (RGBT) tracking in challenging situations by leveraging multimodal data. A RGBT object tracking method is proposed in correlation filter tracking framework based on short term historical information. Given the initial object bounding box, hierarchical convolutional neural network (CNN) is employed to extract features. The target is tracked for RGB and thermal modalities separately. Then the backward tracking is implemented in the two modalities. The difference between each pair is computed, which is an indicator of the tracking quality in each modality. Considering the temporal continuity of sequence frames, we also incorporate the history data into the weights computation to achieve a robust fusion of different source data. Experiments on three RGBT datasets show the proposed method achieves comparable results to state-of-the-art methods.

In this paper, we explore the challenging 1-to-N map matching problem, which exploits a compact description of map data, to improve the scalability of map matching techniques used by various robot vision tasks. We propose a first method explicitly aimed at fast succinct map matching, which consists only of map-matching subtasks. These tasks include offline map matching attempts to find a compact part-based scene model that effectively explains each map using fewer larger parts. The tasks also include an online map matching attempt to efficiently find correspondence between the part-based maps. Our part-based scene modeling approach is unsupervised and uses common pattern discovery (CPD) between the input and known reference maps. This enables a robot to learn a compact map model without human intervention. We also present a practical implementation that uses the state-of-the-art CPD technique of randomized visual phrases (RVP) with a compact bounding box (BB) based part descriptor, which consists of keypoint and descriptor BBs. The results of our challenging map-matching experiments, which use a publicly available radish dataset, show that the proposed approach achieves successful map matching with significant speedup and a compact description of map data that is tens of times more compact. Although this paper focuses on the standard 2D point-set map and the BB-based part representation, we believe our approach is sufficiently general to be applicable to a broad range of map formats, such as the 3D point cloud map, as well as to general bounding volumes and other compact part representations., Offprint of IROS2013 paper

Textual information appearing on the captured image may contain personal information. In various circumstances, publishing such images in the public domain may create a threat of privacy leak. To avoid these situations, we propose a text concealment method. To accomplish this task, we have used a conditional generator, which is a text region conditioned concealment network. The text regions predicted by the detector network are used as a conditioning criterion for the concealment process. The text region prediction in the form of a word-level bounding box may contain stroke pixels as well as the background pixels. However, to reduce the number of background pixels for proper conditioning of text concealment network, a character level annotation is used for the generator in place of word-level bounding box annotation. It helps to focus more on strokes as compared to the background pixels. A character-level symmetric line representation of text has been proposed to obtain finer level text region prediction as compared to the character-level bounding box. The proposed model is trainable from end-to-end. The text region conditioned generator is trained from the loss of global and local discriminators. The proposed method is validated on public scene text image datasets such as ICDAR 2015, COCO-Text and Synthesis dataset. The proposed architecture shows competitive results as compared to other state-of-the-art approaches.

Recently, the object detection and recognition based applications are widely adopted in various real-time and offline applications. The computer vision based automatic learning schemes have gained huge attraction from researchers due to their significant nature of learning that can significantly improve the detection performance. The advances in deep and convolutional neural networks have improved the efficiency of applications based on recognition and detection. However, enhancing precision, decreasing detection error, and detecting camouflaged items are still regarded as difficult problems. In this work, we concentrated on these problems and presented a model based on Faster-RCNN that utilizes saliency detection, proposal generation and bounding box regression, for better detection along with loss functions. The suggested method is referred to as the saliency driven Faster RCNN model for object detection and recognition using computer vision approach (SGFr-RCNN). The performance of the suggested strategy is assessed using the data sets (PASCAL VOC 2007, PASCAL VOC 2012 & CAMO_UOW) and contrasted with current methods in terms of mean average precision. The comparative research demonstrates the important improvement in the results of the suggested strategy relative to the current methods.

Object detection and location from remote sensing (RS) images is challenging, computationally expensive, and labor intense. Benefiting from research on convolutional neural networks (CNNs), the performance in this field has improved in the recent years. However, object detection methods based on CNNs require a large number of images with annotation information for training. For object location, these annotations must contain bounding boxes. Furthermore, objects in RS images are usually small and densely co-located, leading to a high cost of manual annotation. We tackle the problem of weakly supervised object detection under such conditions, aiming to learn detectors with only image-level annotations, i.e., without bounding box annotations. Based on the fact that the feature maps of a CNN are localizable, we hierarchically fuse the location information from the shallow feature map with the class activation map to obtain accurate object locations. In order to mitigate the loss of small or densely distributed objects, we introduce a divergent activation module and a similarity module into the network. The divergent activation module is used to improve the response strength of the low-response areas in the shallow feature map. Densely distributed objects in RS images, such as aircraft in an airport, often exhibit a certain similarity. The similarity module is used to improve the feature distribution of the shallow feature map and to suppress background noise. Comprehensive experiments on a public dataset and a self-assembled dataset (which we made publicly available) show the superior performance of our method compared to state-of-the-art object detectors.

The extraction of text information from traffic panels is one of the challenging problems in computer vision. Although the past decade has seen a promising shift and important progress in object detection, few works and a limited number of datasets focus specifically on extracting text from traffic signs. To address the lack of data for text detection in traffic panels, especially those with Arabic scripts, this paper introduces a new multilingual and multipurpose dataset named ASAYAR. It consists of three sub-datasets: Arabic-Latin scene text localization, traffic sign detection, and directional symbol detection. The dataset contains 1763 images collected on different Moroccan highways, and annotated manually, using 16 object categories. The fully annotated ASAYAR images contains more than 20000 bounding box objects. The paper also investigates the usability of the dataset, by evaluating the performance of the state-of-the-art algorithms for object and text detection. Experimental results show good detection scores, demonstrating the potential contribution of ASAYAR in the development of methods for text extraction from traffic panels.

Visual grounding (VG) aims to locate the most relevant object or region in an image, based on a natural language query. Generally, it requires the machine to first understand the query, identify the key concepts in the image, and then locate the target object by specifying its bounding box. However, in many real-world visual grounding applications, we have to face with ambiguous queries and images with complicated scene structures. Identifying the target based on highly redundant and correlated information can be very challenging, and often leading to unsatisfactory performance. To tackle this, in this paper, we exploit an attention module for each kind of information to reduce internal redundancies. We then propose an accumulated attention (A-ATT) mechanism to reason among all the attention modules jointly. In this way, the relation among different kinds of information can be explicitly captured. Moreover, to improve the performance and robustness of our VG models, we additionally introduce some noises into the training procedure to bridge the distribution gap between the human-labeled training data and the real-world poor quality data. With this "noised" training strategy, we can further learn a bounding box regressor, which can be used to refine the bounding box of the target object. We evaluate the proposed methods on four popular datasets (namely ReferCOCO, ReferCOCO+, ReferCOCOg, and GuessWhat?!). The experimental results show that our methods significantly outperform all previous works on every dataset in terms of accuracy.

Weakly supervised semantic instance segmentation with only image-level supervision, instead of relying on expensive pixel-wise masks or bounding box annotations, is an important problem to alleviate the data-hungry nature of deep learning. In this article, we tackle this challenging problem by aggregating the image-level information of all training images into a large knowledge graph and exploiting semantic relationships from this graph. Specifically, our effort starts with some generic segment-based object proposals (SOP) without category priors. We propose a multiple instance learning (MIL) framework, which can be trained in an end-to-end manner using training images with image-level labels. For each proposal, this MIL framework can simultaneously compute probability distributions and category-aware semantic features, with which we can formulate a large undirected graph. The category of background is also included in this graph to remove the massive noisy object proposals. An optimal multi-way cut of this graph can thus assign a reliable category label to each proposal. The denoised SOP with assigned category labels can be viewed as pseudo instance segmentation of training images, which are used to train fully supervised models. The proposed approach achieves state-of-the-art performance for both weakly supervised instance segmentation and semantic segmentation. The code is available at https://github.com/yun-liu/LIID.

Recent years have witnessed the prevalence of using the Siamese network for RGB-T tracking because of its remarkable success in RGB object tracking. Despite their faster than real-time speeds, existing RGB-T Siamese trackers suffer from low accuracy and poor robustness, compared to other state-of-the-art RGB-T trackers. To address such issues, a new complementarity- and distractor-aware RGB-T tracker based on Siamese network (referred to as SiamCDA) is developed in this paper. To this end, several modules are presented, where the feature pyramid network (FPN) is incorporated into the Siamese network to capture the cross-level information within unimodal features extracted from the RGB or the thermal images. Next, a complementarity-aware multi-modal feature fusion module (CA-MF) is specially designed to capture the cross-modal information between RGB features and thermal features. In the final bounding box selection phase, a distractor-aware region proposal selection module (DAS) further enhances the robustness of our tracker. On top of the technical modules, we also build a large-scale, diverse synthetic RGB-T tracking dataset, containing more than 4831 pairs of synthetic RGB-T videos and 12K synthetic RGB-T images. Extensive experiments on three RGB-T tracking benchmark datasets demonstrate the outstanding performance of our proposed tracker with a tracking speed over 37 frames per second (FPS).

Instance segmentation needs to locate all instances in an image correctly and segment each instance precisely. Currently, the most dominant methods for instance segmentation take object detection as a pre-task. However, they rely on the accuracy of object detection incredibly. If the pre-task cannot predict an accurate bounding box, the performance of instance segmentation will degenerate. In this paper, we present a novel method for instance segmentation to solve this problem, which is called Segmenting Beyond the Bounding Box (S3B-Net).Our S3B-Net designs a sub-network to help instance segmentation methods based on object detection to segment the part of an instance beyond the bounding box. Specifically, the sub-network first predicts a two-dimensional pixel embedding for each pixel. Then, the Gaussian function is employed to calculate a pixel’s probability belongs to a corresponding instance according to the two-dimensional pixel embedding. Finally, the output of the sub-network combines with the output of instance segmentation based on object detection to generate a more precise instance mask. Our sub-network can easily extend on the existing instance segmentation method based on object detection to segment instance beyond the bounding box. We do our experiments on dominant instance segmentation datasets, such as the COCO dataset and Cityscapes dataset. The results show that our method can achieve 6.8 points gain compared with the baseline Mask R-CNN with ResNet-50-FPN in Cityscapes datasets, and 1.7 points gain with ResNet-101-FPN-DCN in COCO datasets. Our S3B-Net outperforms the previous state-of-the-art instance segmentation method, which proves our method is competitive. The source code of our method will be made available.

The manipulating robots receive much attention by offering better services, where object grasping is still challenging especially under background interferences. In this article, a novel two-stream grasping convolutional neural network (CNN) with simultaneous detection and segmentation is proposed. The proposed method is cascaded by an improved simultaneous detection and segmentation network BlitzNet and a two-stream grasping CNN TsGNet. The improved BlitzNet introduces the channel-based attention mechanism, and achieves an improvement of detection accuracy and segmentation accuracy with the combination of the learning of multitask loss weightings and background suppression. Based on the obtained bounding box and the segmentation mask of the target object, the target object is separated from the background, and the corresponding depth map and grayscale map are sent to TsGNet. By adopting depthwise separable convolution and designed global deconvolution network, TsGNet achieves the best grasp detection with only a small amount of network parameters. This best grasp in the pixel coordinate system is converted to a desired 6-D pose for the robot, which drives the manipulator to execute grasping. The proposed method combines a grasping CNN with simultaneous detection and segmentation to achieve the best grasp with a good adaptability to background. With the Cornell grasping dataset, the image-wise accuracy and object-wise accuracy of the proposed TsGNet are 93.13% and 92.99%, respectively. The effectiveness of the proposed method is verified by the experiments.

Mobile Laser Scanning (MLS) systems provide highly dense 3D point clouds that enable the acquisition of accurate traffic facilities information for intelligent transportation system. Road guardrails with safety features that can separate traffic and define moving spaces for pedestrians and vehicles face challenges such as diverse guardrail types and continuous slopes in point clouds data. This paper proposes a novel approach for rapidly extracting urban road guardrails from MLS point clouds, combining a proposed multi-level filtering with a modified Density-Based Spatial Clustering of Applications with Noise (DBSCAN) clustering, and adapting for most types of guardrails and rough slope roads. We develop a multi-level filter to detect the road surface and remove the undesirable points. Through a proposed modified DBSCAN clustering, the guardrails are extracted after a four-step screening, which includes the limits based on the number of points, the fitting error, the bounding box size and the average reflection intensity for each cluster. The proposed method achieves high precisions of 97.2% and 96.4% respectively for the lane-separating guardrails and the anti-fall guardrails on the dataset. Extensive experiments with test dataset captured by a RIEGL VMX-450 MLS, show that our method outperforms the state-of-the-art method to extract 3D guardrails from point clouds.

Building footprint delineation from remote sensing imagery is a basic task in surveying and mapping and geographic information system (GIS), which benefits many engineering applications but requires an enormous amount of human delineation. In this study, we aim to find a way to replace human delineation with automated algorithms. For this purpose, we designed a novel pipeline for the automated production of building vector maps from aerial images, which consists of a bounding-box generation module, a graph convolutional network (GCN)-based polygon prediction module, and an empirical polygon regularization module. First, we introduce the bounding box generation module based on region-based object detection, which along with our overlap cropping strategy is used to generate a bounding box for each building instance. The generated bounding boxes have a horizontal rectangle version and a rotated version, which are used to initialize the next stage of our method. Second, we propose a GCN-based method, which is the core of this study, to conduct the initial accurate building polygon prediction by integrating multiresolution optimization and multilevel loss constraints. Our proposed method, to the authors' best knowledge, is the first of its kind that introduces GCN to building extraction. The final step is to apply a regularization algorithm to translate the predicted polygons into structured and highly accurate building footprints. We validated the proposed method on two large aerial building data sets, WHU data set and Inria data set, where it was shown to significantly outperform other state-of-the-art methods more than 10%. Specifically, with the ground-truth rotated bounding boxes of buildings, our method is able to automatically delineate 91% of the buildings in the WHU data set and with the predicted rotated bounding the percent reached human-level delineation.

Most existing trackers use bounding boxes for object tracking. However, the background contained in the bounding box inevitably decreases the accuracy of the target model, which affects the performance of the tracker and is particularly pronounced for non-rigid objects. To address the above issue, this paper proposes a novel hybrid level set model, which can robustly address the issue of topology changing, occlusions and abrupt motion in non-rigid object tracking by accurately tracking the object contour. In particular, an appearance model is first obtained by repeatedly training and relabeling the initial labeled frame using competing one-class SVMs. Then, by integrating the trained appearance model, an edge detector and image spatial information into the level set model, a new hybrid level set model is presented, which accurately locates the object contour and feeds back to the competing one-class SVMs to update the appearance model of the next frame. In addition, a motion model is defined to predict the accurate location of the object when occlusion and abrupt motion occur in the next frame. Finally, the experimental results on state-of-the-art benchmarks demonstrate the feasibility and effectiveness of the proposed model and the superiority of the proposed method over existing trackers in terms of accuracy and robustness.

Ship detection has broad applications in many areas, including fishery management, maritime rescue and maritime monitoring. Recently, numerous detectors based on deep learning have been carried in ship detection in synthetic aperture radar (SAR) images. However, detecting the inshore ships faces enormous challenges because of the strong scattering interference of the inland area. In order to address such issues, a novel method named strong scattering points network for ship detection is proposed in this paper. First, according to the SAR imaging mechanism, the ships usually appear strong scattering phenomenon in the SAR images. Therefore, the proposed method detects the strong scattering points on the ship, and then aggregates their positions to obtain the ship’s arbitrary orientation box. Second, our method designs an embedding vector to cluster these points as an individual object to regress the oriented bounding box. Third, in order to distinguish the strong scattering points on land, a ship attention module is employed to extract the image texture features and representations of local features. It can suppress the false alarm caused by land interference in the detection process. Furthermore, to demonstrate the effectiveness of the proposed algorithm, this paper introduces a new ship dataset for oriented ship detection named large-scale dataset for ship detection in SAR images (LDSD). Moreover, the public SAR ship detection dataset (SSDD) is utilized to verify the robustness and generalization ability of the detector. The experimental results on two datasets show that our method has a strong anti-interference ability in the inshore background and achieves state-of-the-art detection performance.

We present COVID-CT-Mask-Net model that predicts COVID-19 from CT scans. The model works in two stages: first, it detects the instances of ground glass opacity and consolidation in CT scans, then predicts the condition from the ranked bounding box detections. To develop the solution for the three-class problem (COVID, common pneumonia and control), we used the COVIDx-CT dataset derived from the dataset of CT scans collected by China National Center for Bioinformation. We use about 5% of the training split of COVIDx-CT to train the model, and without any complicated data normalization, balancing and regularization, and training only a small fraction of the model’s parameters, we achieve a 90.80% COVID sensitivity, 91.62% common pneumonia sensitivity and 92.10% normal sensitivity, and an overall accuracy of 91.66% on the test data (21182 images), bringing the ratio of test/train data to 7.06, which implies a very high capacity of the model to generalize to new data. We also establish an important result, that ranked regional predictions (bounding boxes with scores) in Mask R-CNN can be used to make accurate predictions of the image class. The full source code, models and pretrained weights are available on https://github.com/AlexTS1980/COVID-CT-Mask-Net.

Obstacle avoidance for robotic manipulators can be challenging when they operate in unstructured environments. This problem is probed with the sim-to-real (sim2real) deep reinforcement learning, such that a moving policy of the robotic arm is learnt in a simulator and then adapted to the real world. However, the problem of sim2real adaptation is notoriously difficult. To this end, this work proposes (1) a unified representation of obstacles and targets to capture the underlying dynamics of the environment while allowing generalization to unseen goals and (2) a flexible end-to-end model combining the unified representation with the deep reinforcement learning control module that can be trained by interacting with the environment. Such a representation is agnostic to the shape and appearance of the underlying objects, which simplifies and unifies the scene representation in both simulated and real worlds. We implement this idea with a vision-based actor-critic framework by devising a bounding box predictor module. The predictor estimates the 3D bounding boxes of obstacles and targets from the RGB-D input. The features extracted by the predictor are fed into the policy network, and all the modules are jointly trained. This makes the policy learn object-aware scene representation, which leads to a data-efficient learning of the obstacle avoidance policy. Our experiments in simulated environment and the real-world show that the end-to-end model of the unified representation achieves better sim2real adaption and scene generalization than state-of-the-art techniques.

Automatic building extraction from remote sensing imagery is crucial to urban construction and management. To address the main challenges of diverse building scale and appearance, this letter proposes an automatic building instance extraction method based on an improved hybrid task cascade (HTC). Our method consists of three components by obtaining high-resolution representation, defining guided anchor, and forming focal loss to boost the adaptability of automatic building instance extraction. Comprehensive experimental results on WHU aerial building data set demonstrated that compared with the mainstream Mask R-CNN method, our method increased AP and AR in bounding box branch and mask branch by 9.8%-6.5% and 10.7%-8.0% respectively, especially APS and APL in the two branches by 10.1%-6.9% and 3.4%-2.4%, respectively. We evaluated the effectiveness and complexity of these components separately and discussed the universality and practicability of deep learning method in automatic building extraction.

Although the orientation and scale properties of the objects in remote sensing images have been widely considered in the modern deep learning based object detection methods, the spatial distribution property of objects has rarely been investigated. There is a distinct spatial distribution difference between close-range objects and remote sensing objects: the former may exhibit extensive mutual occlusion and overlap, whereas the latter rarely overlap. A current remote sensing object detection algorithm that ignores the spatial distribution difference may unnecessarily apply the massive anchor-based proposal bounding box generation and non-maximum suppression (NMS) operations. In this paper, considering the unique spatial distribution of remote sensing objects, and also the other spatial properties, we propose a novel, compact, and spatial-oriented object detection framework for remote sensing images. The proposed two-stage convolutional neural network (CNN) framework, which we call RSADet (the Remote-sensing Spatial Adaptation DETector), considers the spatial distribution, scale, and orientation/shape varieties of the objects in remote sensing images. In the first stage, each object instance is inferred on the scale-attention boosted CNN heatmaps to generate candidate bounding boxes, instead of using the anchor-based proposal box generation and NMS. In the second stage, deformable convolutions are introduced to adapt to the geometric variations of different object instances and to avoid the impact of complex and changeable backgrounds. A new bounding box confidence (IoU score) prediction branch is introduced as a convenient constraint for eliminating unreliable boxes and improving the performance. Experiments were conducted on a large single-class remote sensing object detection dataset (the Ningbo Pylon dataset) built as part of this study and an open-source extraordinarily large multi-class dataset (the DIOR dataset). Compared with the advanced detectors from both the computer vision and remote sensing communities, the proposed RSADet achieved a state-of-the-art performance on both datasets.

An improved face detection method ground on TinyYOLOv3 algorithm is put forward in this paper in view of the low recognition rate of traditional face detection methods in complex background and the long detection time of existing face detection methods ground on deep learning The main network of TinyYOLOv3 is redesigned to extract more abundant semantic information, which is increasing the number of network layers. The deep separable convolution is used instead of the traditional convolution and the features of different network layers are fused. Meanwhile, the accuracy of face detection is guaranteed, size of this network model is reduced. The CIoU (complete intersection over union) loss is used on improving this original prediction loss of bounding box coordinates. The channel attention mechanism is integrated into the feature extraction network to improve the positioning accuracy and detection accuracy. The network convergence speed is accelerated by hyperparameter optimization and priori box clustering method. The face detection experiments were conducted on the Wider Face data sets. And the experimental results manifest that the proposed algorithm has high recognition accuracy in complex scenes. At the same time, the proposed algorithm is better than others in terms of detection speed and model size, and can meet the requirements of embedded devices.

As an important remote sensing means, synthetic aperture radar (SAR) has many superiorities to other sensors. How to effectively detect and locate ships in SAR images is also a popular field. In previous ship detection research, most algorithms focus on detecting the horizontal bounding box of ship targets, which ignore the rotation angle of each ships. Thus, too much background noise in the horizontal detection results makes them difficult to describe each ship accurately. Inspired by the powerful feature representation ability of convolutional neural networks (CNNs), a novel anchor-free and keypoint-based deep learning method is proposed for oriented ship detection in multiresolution SAR images. Our detector first extracts multilevel features from the input SAR image with a backbone network and feature pyramid network. Next, considering multiscale ships in multiresolution SAR images, we detect different sizes of ships on different levels of feature maps with identical head network structures. In each head network, the classification subnetwork determines each pixel in feature maps as the central pixel of this ship or not, and the regression subnetwork regresses the oriented bounding box for each ship. In the training process, the proposed oriented nonnormalized Gaussian function is used to describe the center point of ship targets, while the nonuniform weighting of the different level loss functions is used to suppress the imbalanced sample distribution. Experimental results on two authoritative SAR-oriented ship detection datasets and two Gaofen-3 images demonstrate the effectiveness and robustness of the proposed methods.

Ship detection in the synthetic aperture radar (SAR) image is of great significance in the fields of military and coastal defense. Most ship detection methods are designed based on the object detection framework, which can only provide the vertices' coordinates of the bounding box covering the ship targets but cannot provide more detailed contour information. Target segmentation can further explore the shape and edge information of the objects, which can be used as a blazing novel means for automatic object detection. In this letter, a 3-D atrous encoder-decoder neural network with global attention modules (GAM-EDNet) is proposed to achieve ship segmentation in SAR images. The encoder-decoder structure with atrous convolution is developed as the network body to fully exploit the structural information of the ship targets with various sizes. To increase the structural information of the single-polarization SAR images, a 3-D image cube is designed as the input of the GAM-EDNet. A global attention module is proposed to further improve the segmentation performance by integrating the high-level semantic features with the low-level location features. Besides, an SAR ship segmentation dataset (SAR-HR4) is built to evaluate the segmentation performance, and the experimental results show that the proposed GAM-EDNet achieves better performance than other state-of-the-art methods.

This paper proposes a novel method for active collision avoidance to protect the human who enters a robot’s workspace in a human-robot collaborative environment. The proposed method uses a somatosensory sensor to monitor the robot’s workspace and detect anyone attempting to enter it. When someone enters the workspace, a Kinect detects and calculates the position of his or her skeleton points in real-time. However, due to the measurement errors and noise of the device, the tracking error increases over time. Therefore, the proposed method applies an improved particle filter (IPF) to accurately estimate the position of the skeleton points. In order to detect the human-robot collision in real-time, the proposed method uses cylinders to establish the bounding box model for human bones and robots and the human-robot collision is replaced by the collision between the cylinders, greatly improving the efficiency of collision detection. Moreover, taking human safety and productivity into account, the robot velocity control is carried out based on the distance between the robot and human. Then, the proposed method uses a rule-based logic system to analyze human motion so that the robot can take appropriate measures to avoid humans. Finally, the dynamic roadmap (DRM) approach plans new paths in real-time to allow robots to bypass humans. By actively avoiding collisions, the proposed method ensures that the robot will never touch the human body. The significant advantage of the proposed method is that it can detect humans in real-time, analyze their behavior and protect humans without any modification to the robot. The proposed method has been tested in practical applications, and the results show that it can successfully guarantee the safety of people entering the robot’s workspace.

The key to efficient person search is jointly localizing pedestrians and learning discriminative representation for person re-identification (re-ID). Some recently developed models are built with separate detection and re-ID branches on top of shared region feature extraction networks. There are two factors that are detrimental to re-ID feature learning. One is the background information redundancy resulting from the large receptive field of neurons. The other is the body part missing and background clutter caused by inaccurate localization. In this work, a bottom-up fusion (BUF) subnet is proposed to fuse the bounding box features pooled from multiple network stages. With a few parameters introduced, BUF leverages the multi-level features with various sizes of receptive fields to mitigate the background-bias problem. To further suppress the non-pedestrian regions, the newly introduced segmentation head generates a foreground probability map as guidance for the network to focus on the foreground regions. The resulting foreground attention module (FAM) enhances the foreground features. Moreover, for robust feature learning in practical person search, we propose to adaptively smooth the labels of the pedestrian boxes with consideration of the detection quality. Extensive experiments on PRW and CUHK-SYSU validate the effectiveness of the proposals. Our Bottom-Up Foreground-Aware Feature Fusion (BUFF) network with ALS achieves considerable gains over the state-of-the-art on PRW and competitive performance on CUHK-SYSU.

The occlusion of the target in tracking usually causing failure that is a serious issue for long-term sequences. Though the siamese network-based trackers obtained considerable performance, it still suffers from the target missing issue when the target is obscured by the same semantic information interferent. To address this issue, a novel occlusion awareness algorithm is proposed, which can both address the occlusion issue and the same semantic information false identification issues. In addition, a novel generative adversarial training and long short term memory (LSTM) based target trajectory prediction algorithm is proposed to predict the possible direction of the target in the following frames. The proposed trajectory prediction algorithm can deal with complicated tracking situations more robustly than the traditional algorithms, e.g. Kalman filter. To further improve the occlusion awareness ability of the proposed algorithm, an occlusion supervision-based training strategy is proposed, which can improve the robustness of the occlusion awareness ability of the proposed occlusion awareness model. In addition, for accurate estimation of the target bounding box, a distance intersection over union (DIOU) loss for regression training is adopted. A comprehensive evaluation is performed on OTB2015, VOT2016, and VOT2018 to evaluate the effectiveness of the proposed algorithm. The experiment results demonstrate that the proposed algorithms perform well and can largely alleviate the tracking failure issue of the siamese network-based tracker caused by occlusion and the same semantic information target identification.

Autonomous vision-based taxi, takeoff, and landing systems are gaining popularity in fixed-wing commercial, military, and unmanned aerial vehicles presenting new technical and regulatory challenges. With ongoing advances in deep learning, machine learning, and big data management, the dangers of takeoff and landing operations have greatly decreased, and with less dependence on simultaneous and complex controllers. However, in a real-world scenario, airport runway detection for takeoff and landing operations is still deemed challenging, due to the vast amount of contextual information, complex backgrounds, and dangerous geographical scenarios. In this work, we propose an end-to-end airport runway detection network for automatic runway detection and localization centered on a two-stage region-based convolutional neural network that could significantly improve navigation efficiency and dependence on complex control systems. The first stage is an enhanced region proposal network with domain-specific constraints added, which automatically learns and extracts multiscale (high, mid, and low-level) features using a deep convolutional neural network and filters low-quality positive samples for robust runway detection. The second stage classifies the candidate regions and determines the precise localization of predictions using an enhanced anchor refinement and regression algorithm with a nonmaximum suppression function for bounding box optimization. Transfer learning and data augmentation techniques are used to propagate relevant domain-specific semantic and context information during the training phase to handle the problem of imbalances in the dataset classes. The results show the effectiveness of our algorithmic contributions as well as the benefits of the specific enhancements, in terms of, improved efficiency and accuracy of runway detection in remote sensing images.

The vehicle information acquired by current video-based traffic surveillance methods is usually onefold. In this paper, a traffic surveillance system for obtaining comprehensive vehicle information, including type, number of axles, 3D bounding box, speed, length, current driving lane, and traffic volume, is proposed based on instance segmentation which is realized by Mask R-CNN. The work mainly contains: 1) An annotated image dataset which contains car, coach, truck, and wheel is established for training Mask R-CNN; 2) The method for identifying the number of axles based on MaskIoU is proposed; 3) The 3D bounding box is generated based on the high-quality vehicle segmentation, which is more accurate than that generated by background subtraction methods; 4) For calculating the vehicle speed and length, the reference points on the road plane are determined by the lane with dashed line and one vanishing point to calculate the homography matrix; 5) In order to keep the obtained vehicle information reliability, the results identified from multi-frames are obtained by means of the tracking method SORT for processing. The proposed system is verified in different scenes. The average recognition accuracies of the vehicle types and number of axles are more than 97% and 88% in three testing videos. The speed errors of more than 90% vehicles are less than 4%. And the ratios of both missing and repetition in vehicle counting are low in general.

Purpose This study aims at exploiting artificial intelligence (AI) for the identification, segmentation and quantification of COVID-19 pulmonary lesions. The limited data availability and the annotation quality are relevant factors in training AI-methods. We investigated the effects of using multiple datasets, heterogeneously populated and annotated according to different criteria. Methods We developed an automated analysis pipeline, the LungQuant system, based on a cascade of two U-nets. The first one (U-net$$_1$$ 1 ) is devoted to the identification of the lung parenchyma; the second one (U-net$$_2$$ 2 ) acts on a bounding box enclosing the segmented lungs to identify the areas affected by COVID-19 lesions. Different public datasets were used to train the U-nets and to evaluate their segmentation performances, which have been quantified in terms of the Dice Similarity Coefficients. The accuracy in predicting the CT-Severity Score (CT-SS) of the LungQuant system has been also evaluated. Results Both the volumetric DSC (vDSC) and the accuracy showed a dependency on the annotation quality of the released data samples. On an independent dataset (COVID-19-CT-Seg), both the vDSC and the surface DSC (sDSC) were measured between the masks predicted by LungQuant system and the reference ones. The vDSC (sDSC) values of 0.95±0.01 and 0.66±0.13 (0.95±0.02 and 0.76±0.18, with 5 mm tolerance) were obtained for the segmentation of lungs and COVID-19 lesions, respectively. The system achieved an accuracy of 90% in CT-SS identification on this benchmark dataset. Conclusion We analysed the impact of using data samples with different annotation criteria in training an AI-based quantification system for pulmonary involvement in COVID-19 pneumonia. In terms of vDSC measures, the U-net segmentation strongly depends on the quality of the lesion annotations. Nevertheless, the CT-SS can be accurately predicted on independent test sets, demonstrating the satisfactory generalization ability of the LungQuant.

In Visual Question Answering (VQA), an attention mechanism has a critical role in specifying the different objects present in an image or tells the machine where to focus by invoking the vivid visuals in an image. However, the current VQA systems uses the region-based or bounding box based image features to learn the attention distribution, which are not capable enough to answer the questions related to foreground or background object present in an image. In this paper, we have proposed a VQA model that uses image features which are effective enough to answer questions related to foreground object and background region. Also, we have used graph neural network to encode the relationship between the image regions and objects in an image. Further, we have generated image captions based on these visual relationship based image representation. Thus, the proposed model uses two attention modules to take advantage of each other’s knowledge, to generate more influential attention modules together with the captions based image representation to extract the features which are capable enough to answer questions related to the foreground object and background region. Finally, the performance of proposed architecture is improved by combining the hybrid simulated annealing-Mantaray Foraging Optimization (SA-MRFO) algorithm, which selects the optimal weight parameter for the proposed model. To estimate the performance of the proposed model, two benchmark datasets are used: VQA 2.0 and VQA-CP v2.

Hyperspectral face recognition plays an important role in remote sensing. However, it faces many challenges such as difficulty in data acquisition, low signal to noise ratio (SNR), and high dimensionality. In this paper, we develop a novel method for hyperspectral face recognition by extracting histogram of oriented features (HOG) and using collaborate representation-based classifier (CRC) to classify unknown face data cubes. To improve overall classification rates, we also implement noise reduction in hyperspectral face data cubes. We also crop the face images by a bounding box and use this bounding box image to classify the testing faces. Experiments show that our new method outperforms several existing methods for both the PolyU-HSFD dataset and the CMU-HSFD dataset for hyperspectral face recognition. The contribution of this paper is the following: We introduce the MNF-based denoising method in this paper, which is new to the best of our knowledge. We also combine it with HOG features and CRC classifier so that better recognition rate can be achieved.

Since the outbreak of the COVID-19 pandemic, computer vision researchers have been working on automatic identification of this disease using radiological images. The results achieved by automatic classification methods far exceed those of human specialists, with sensitivity as high as 100% being reported. However, prestigious radiology societies have stated that the use of this type of imaging alone is not recommended as a diagnostic method. According to some experts the patterns presented in these images are unspecific and subtle, overlapping with other viral pneumonias. This report seeks to evaluate the analysis the robustness and generalizability of different approaches using artificial intelligence, deep learning and computer vision to identify COVID-19 using chest X-rays images. We also seek to alert researchers and reviewers to the issue of "shortcut learning". Recommendations are presented to identify whether COVID-19 automatic classification models are being affected by shortcut learning. Firstly, papers using explainable artificial intelligence methods are reviewed. The results of applying external validation sets are evaluated to determine the generalizability of these methods. Finally, studies that apply traditional computer vision methods to perform the same task are considered. It is evident that using the whole chest X-Ray image or the bounding box of the lungs, the image regions that contribute most to the classification appear outside of the lung region, something that is not likely possible. In addition, although the investigations that evaluated their models on data sets external to the training set, the effectiveness of these models decreased significantly, it may provide a more realistic representation as how the model will perform in the clinic. The results indicate that, so far, the existing models often involve shortcut learning, which makes their use less appropriate in the clinical setting.

We propose a progressive hierarchical analysis model to perceive the collective activities. Compared with previous activity recognition works, it not only recognizes the collective activities, but also perceives the location and the action category of each individual. At first, we perform the person temporal consistency detection procedure for each individual of the collective activities. A person detection network and conditional random field are used to receive the bounding box sequences of the activity participators. Then, we recognize the individual actions using the learned spatial features and the motion features based on LSTM. At last, the combination of the recognized person-level action category vector, the scene context features and the interaction Context features are used to recognize the collective activities. We evaluate the proposed approach on benchmark collective activity datasets. Extensive experiments demonstrate the effects of the progressive hierarchical analysis model.

The field of computer vision is increasingly becoming an active area of research with tremendous efforts being put towards giving computers the capability of sight. As human beings we are able to see, distinguish between different objects based on their unique features and even trace their movements if they are within our view. For computers to really see they also need to have the capability of identifying different objects and equally track them. This paper focuses on that aspect of identifying objects which the user chooses; the object chosen is differentiated from other objects by comparison of pixel characteristics. The chosen object is then to be tracked with a bounding box for ease of identification of the object's location. A real time video feed captured by a web camera is to be utilized and it’s from this environment visible within the camera view that an object is to be selected and tracked. The scope of this paper mainly focuses on the development of a software application that will achieve real time object tracking. The software module will allow the user to identify the object of interest someone wishes to track, while the algorithm employed will enable noise and size filtering for ease of tracking of the object.

We present SSPNet, a novel method for learning the spatiotemporal saliency of a target for visual tracking. State-of-the-art trackers typically track targets by predicting the target state, ie coordinates of a bounding box encompassing the target, from the target candidates sampled around a previous target state. However, they have two limitations: 1) vulnerability to tracking distractors present in a frame and 2) the strong bias of the target state estimation to the initial frame. The proposed method addresses this problem by predicting the spatiotemporal features of the target so-called current and future target saliencies. Given a frame, the current target saliency represents the spatial aspect of the target, whereas the future target saliency depicts how the target will appear in the next frame based on the temporal features. This technique improves tracking accuracy in two ways: we can exploit the similarity between the current and the future target saliencies to detect the distractors. Further, SSPNet provides better prior knowledge about the current target state compared to using only the previous frame, which mitigates the bias to the initial frame and occlusion problem . We show that SSPNet outperforms the state-of-the-art trackers, particularly in challenging sequences.

This paper proposes a 3D nesting algorithm which combines the flower pollination algorithm (FPA) with the oriented bounding box (OBB) collision detection to solve the 3D packing of irregular shaped objects for powder-based additive manufacturing. In the powder-based 3D printing process, stacking multiple models most compactly in the build volume is an important task because no support structure is needed. Given a fixed printing area, the post-nesting build height directly impacts the printing cost and efficiency. To reduce the printing cost, 3D models must be packed as closely as possible and the build height must be minimized. This has been found to be a combinatorial optimization and an NP-hard problem. We propose to use the most recent and effective meta-heuristic optimization algorithm FPA, combined with the collision detection of printed objects set up as the optimization constraint using the OBB tree, to find the near-optimal 3D nesting solution. In FPA optimization, a significant amount of time is spent on collision detection. This paper uses the safety clearance distance to adaptively reduce the OBB tree subdivision, hence significantly reducing the computation of collision detection. As a result, the proposed method finds the model positions and rotations of the global solution, and generates the near-optimal solution in reasonable time. Finally, our method is compared with state-of-the-art commercial softwares. The results show that the proposed method produces lower build height with better efficiency. For real-world complex engine parts (30 different STL models and a total of 192,018 triangles), the computation time is only 160 s, and the build height is 12.4% and 13% better than the results from Netfabb and Magics, respectively.

In this paper, a real-time unmanned aerial vehicles (UAVs) detection framework is proposed for GPU embedded applications. To achieve this, this paper proposed a new modified model based on You Only Look Once (YOLO) to detect multi-UAV in aerial images with a complex background. The proposed CNN architecture uses five inception module and dilated convolution with two different factors to increase detection accuracy of YOLOv3 tiny, while preserving its computational time. To further improve the detection accuracy, Generalized Intersection over Union loss is replaced with the bounding box regression loss in the original YOLOv3 tiny loss function. To obtain higher frame rate, scalable kernel correlation filter (sKCF) algorithm is integrated to the detection model. More specifically, the proposed UAV detection method is used to initialize the sKCF tracker at every $$n\mathrm{th}$$ frame. Thus, the detected UAVs can be detected in intermediate frames with a low memory footprint. The proposed model and compared models are trained and tested on a variety of training and test airborne videos, captured under different outdoor scenarios. The average precision of the proposed model is 0.8265 and achieves 32 FPS performance on Jetson-TX2. The results show that the proposed model which is widened with inception module using dilated convolutions has a very high performance in terms of accuracy and speed even when it is compared with deep CNN architectures such as Darknet 53.

Computational dentistry uses computerized methods and mathematical models for dental image analysis. One of the fundamental problems in computational dentistry is accurate tooth instance segmentation in high-resolution mesh data of intra-oral scans (IOS). This paper presents a new computational model based on deep neural networks, called Mask-MCNet, for end-to-end learning of tooth instance segmentation in 3D point cloud data of IOS. The proposed Mask-MCNet localizes each tooth instance by predicting its 3D bounding box and simultaneously segments the points that belong to each individual tooth instance. The proposed model processes the input raw 3D point cloud in its original spatial resolution without employing a voxelization or down-sampling technique. Such a characteristic preserves the finely detailed context in data like fine curvatures in the border between adjacent teeth and leads to a highly accurate segmentation as required for clinical practice (e.g. orthodontic planning). The experiments show that the Mask-MCNet outperforms state-of-the-art models by achieving 98% Intersection over Union (IoU) score on tooth instance segmentation which is very close to human expert performance.

Nowadays, face detection and head pose estimation have a lot of application such as face recognition, aiding in gaze estimation and modeling attention. For these two tasks, it is usually to design two different models. However, the head pose estimation model often depends on the region of interest (ROI) detected in advance, which means that a serial face detector is needed. Even the lightest face detector will slow down the whole forward inference time and cannot achieve real-time performance when detecting the head pose of multiple people. We can see that both face detection and head pose estimation need face features, so a shared face feature map can be used between them. In this paper, a multi-task learning model is proposed that can solve both problems simultaneously. We directly detect the location of the center point of the bounding box of face; at this location, we calculate the size of the bounding box of face and the head attitude. We evaluate our model’s performance on the AFLW. The proposed model has great competitiveness with the multi-stage face attribute analysis model, and our model can achieve real-time performance.

In order to capture more contextual information, various attention mechanisms are applied to object detectors. However, the spatial interaction in the commonly used attention mechanisms is single scale, and it cannot capture the context information of the objects from the feature maps of different scales, which will lead to the underutilization of the context information. In addition, since the predicted bounding box does not completely fit the shape and pose of the object, it has room for further improvement in the performance. In this paper, we propose a multi-scale global context feature pyramid network to obtain a feature pyramid with richer context information, which is a two-layer lightweight neck structure. Moreover, we extend the regression branch by adding an additional prediction head to predict the corner offsets of the bounding boxes to further refine the bounding boxes, which can effectively improve the accuracy of the predicted bounding boxes. Extensive experiments are conducted on the MS COCO 2017 detection datasets. Without bells and whistles, the proposed methods show an average 2% improvement over the RetinaNet baseline.