Search

Object detection and segmentation are some of the key components of Computer Vision, which have wide ranging real world applications. The current state of the art techniques in computer vision are based on Deep Neural Networks and one of the key challenges is using the state of the art techniques in these fields on novel images, and videos in different environments, and classes. These methods require expensive manual annotations and transfer learning to make them work on domains different from their training data sets. In this thesis, we explore both domain adaptation, and deep learning techniques that don’t necessarily rely on the idea of a class, to help with the annotation of private videos. We implemented the initial idea of domain adaptation for directly annotating objects and followed with using video object segmentation (VOS) tracking methods for propagating annotations. Their application to a novel video acquired in the GURU lab is explored as well as ways to improve their performance.

This paper presents a method for endoscope’s autonomous positioning by a robotic endoscope holder for minimally invasive surgery. The method improves human-robot cooperation in robot-assisted surgery by allowing the endoscope holder to acknowledge the surgeon’s view projection and navigate the camera without manual control. The real-time prediction of next desired camera location is estimated using segmented instrument’s tip locations from endoscope video and surgeon’s attention focus given by tracked virtual reality headset. To tackle the issue of real-time surgical instrument segmentation for more precise instrument tip localization, we propose the YOLOv3 and ResNet Combined Neural Network. The method showed an 86.6% IoU across MICCAI’17 Endovis datasets with 30 frames per second processing speed. The proposed pipeline was implemented in ROS on Ubuntu with visualization running under Windows operating system in Unity3D. The simulation demonstrates the robotic arm, endoscope, and surgical environment visualized in 3D in the virtual reality headset to provide a stable view of the endoscope and improve the surgeon’s perception of the operating environment.

With advances in machine vision systems (e.g., artificial eye, unmanned aerial vehicles, surveillance monitoring) scene semantic recognition (SSR) technology has attracted much attention due to its related applications such as autonomous driving, tourist navigation, intelligent traffic and remote aerial sensing. Although tremendous progress has been made in visual interpretation, several challenges remain (i.e., dynamic backgrounds, occlusion, lack of labeled data, changes in illumination, direction, and size). Therefore, we have proposed a novel SSR framework that intelligently segments the locations of objects, generates a novel Bag of Features, and recognizes scenes via Maximum Entropy. First, denoising and smoothing are applied on scene data. Second, modified Fuzzy C-Means integrates with super-pixels and Random Forest for the segmentation of objects. Third, these segmented objects are used to extract a novel Bag of Features that concatenate different blobs, multiple orientations, Fourier transform and geometrical points over the objects. An Artificial Neural Network recognizes the multiple objects using the different patterns of objects. Finally, labels are estimated via Maximum Entropy model. During experimental evaluation, our proposed system illustrated a remarkable mean accuracy rate of 90.07% over the MSRC dataset and 89.26% over the Caltech 101 for object recognition, and 93.53% over the Pascal-VOC12 dataset for scene recognition, respectively. The proposed system should be applicable to various emerging technologies, such as augmented reality, to represent the real-world environment for military training and engineering design, as well as for entertainment, artificial eyes for visually impaired people and traffic monitoring to avoid congestion or road accidents.

Human motion recognition is playing an increasingly important role in modern society. Direct recognition of complex motions has great limitations, so we usually study basic motions first. The key to the establishment of the basic motion set is how to determine the number of basic motions. Sum of the Squared Errors Distribution (SSED) is therefore proposed to determine the number of clustering classes of Self Organizing Maps (SOM). Secondly, the Weighted Tangent Segmentation (WTS) is also proposed to segment complex motions into simple ones, and the sequence with the time stamp is generated. Finally, Back Propagation with the time stamp (BPTS) is proposed to classify the simple motions according to the basic motion set, and the complex motion is recognized according to the time stamp. The motions of human upper limbs are used to verify the effectiveness of this method. SSED determines that the number of clusters of human upper limb motion is 10. The experimental results show that the correct segmentation rate (CSR) is 98.13%. The recognition rate of basic motions is 98.67% and 99.33% in user independent (UI) and user dependent (UD) experiments, respectively. The recognition rate of complex motions is 96.37%. Experiments on UI and UD recognitions verified the effectiveness of our algorithm. Compared with other recognition algorithms, the complex motion recognition method proposed in this article has better recognition performance.

Extracting meaningful information on objects varying scale and shape is a challenging task while obtaining distinctive features on small to large size objects to enhance overall object segmentation accuracy from 3D point cloud. To handle this challenge, we propose an attention-based multi-scale atrous convolutional neural network (AMSASeg) for object segmentation from 3D point cloud. Specifically, a backbone network consists of three modules: distinctive atrous spatial pyramid pooling (DASPP), FireModule, and FireDeconv. The DASPP utilizes average pooling operations and atrous convolutions with different sizes to aggregate distinctive information on objects at multiple scales. The FireModule and FireDeconv are responsible to efficiently extract general features. Meanwhile, a spatial attention module (SAM) and channel attention module (CAM) aggregate spatial and semantic information on smaller objects from low-level and high-level layers, respectively. Our network enables to encode multi-scale information and extract distinct feature on overall objects to enhance segmentation performance. We evaluate our method on KITTI dataset. Experimental results demonstrate that the proposed network is effective to improve segmentation performance on small to large objects at real-time speed.

Medical ultrasound (US) systems are widely used for the diagnosis of internal tissues. However, there are challenges associated with acquiring and interpreting US images, such as incorrect US probe placement and limited available spatial information. In this study, we expand the capabilities of medical US imaging using a robotic framework with a high level of autonomy. A 3D camera is used to capture the surface of an anthropomorphic phantom as a point cloud, which is then used for path planning and navigation of the US probe. Robotic positioning of the probe is realised using an impedance controller, which maintains stable contact with the surface during US scanning and compensates for uneven and moving surfaces. Robotic US positioning accuracy is measured to be 1.19 +/- 0.76mm. The mean force along US probe z-direction is measured to be 6.11 +/- 1.18N on static surfaces and 6.63 +/- 2.18N on moving surfaces. Overall lowest measured force of 1.58N demonstrates constant probe-to-surface contact during scanning. Acquired US images are used for the 3D reconstruction and multi-modal visualization of the surface and the inner anatomical structures of the phantom. Finally, K-means clustering is used to segment different tissues. Best segmentation accuracy of the jugular vein according to Jaccard similarity coefficient is measured to be 0.89. With such an accuracy, this system could substantially improve autonomous US acquisition and enhance the diagnostic confidence of clinicians.

As more and more fields utilize deep learning, there is an increasing demand to make suitable training data for each field. The existing interactive object segmentation models can easily make the mask label data because these can accurately segment the area of the target object through user interaction. However, it is difficult to accurately segment the target part in the object using the existing models. We propose a method to increase the accuracy of part segmentation by using the proposed interactive object segmentation model trained only with edge images instead of color images. The results evaluated with the PASCAL VOC Part dataset show that the proposed method can accurately segment the target part compared to the existing interactive object segmentation model and the semantic part-segmentation model.

Historically, herbarium specimens have provided users with documented occurrences of plants in specific locations over time. Herbarium collections have therefore been the basis of systematic botany for centuries (Younis et al. 2020). According to the latest summary report based on the data from Index Herbariorum, there are around 3400 active herbaria in the world containing 397 million specimens that are spread across 182 countries (Thiers 2021). Exponential growth in high quality image capturing devices induced by the enormous amount of uncovered collections has further led to rising interest in large scale digitization initiatives across the world (Le Bras et al. 2017). As herbarium specimens are increasingly becoming digitised and accessible in online repositories, an important need has also emerged to develop automated tools to process and enrich these collections to facilitate better access to the preserved archives. This rising number of digitised herbarium sheets provides an opportunity to employ computer-based image processing techniques, such as deep learning, to automatically identify species and higher taxa (Carranza-Rojas and Joly 2018, Carranza-Rojas et al. 2017, Younis et al. 2020) or to extract other useful information from the herbaria sheets, such as detecting handwritten text, color bars, scales and barcodes. The species identification task works well for herbarium sheets that have only one species in a page. However, there are many herbarium books that have multiple species on the same page (as shown in Fig. 1) for which the complexity of the identification problem increases tremendously. It also involves a great deal of time and effort if they are to be enriched manually. In this work, we propose a pipeline that can automatically detect, identify, and enrich plant species in multi-specimen herbaria. The core idea of the pipeline is to detect unique plant species and handwritten text around the plant species and map the text to the correct plant species. As shown in Fig. 2, the proposed pipeline begins with the pre-processing of the images. The images are rotated and aligned such that the longest edge is maintained as its height. In the case of herbarium books, the pages are detected and morphological transformations are performed to reduce occlusions (Thirukokaranam Chandrasekar and Verstockt 2020). A YOLOv3 (You Only Look Once version 3) object detection model (Zhao and Li 2020) is trained from scratch to detect plants and text. The model was trained on a dataset of single species herbarium sheets with a mosaic augmentation technique to extend the plants model to detect multiple species. The first results of the training shows impressive results although it could be further improved with more labelled data. We also plan to train an object segmentation model and contrast its performance with the plant detection model for multi-specimen herbarium sheets. After detecting both the plants and the text, the text will be recognized with a state-of-the-art handwritten text recognition (HTR) model. The recognized text can then be matched with a database of specimens, to identify each detected specimen. Furthermore, additional textual metadata (e.g. date, locality, collector's name, institution) visible on the sheet will be recognized and used to enrich the collection.

In recent years, ship detection in satellite remote sensing images has become an important research topic. Most existing methods detect ships by using a rectangular bounding box but do not perform segmentation down to the pixel level. This paper proposes a ship detection and segmentation method based on an improved Mask R-CNN model. Our proposed method can accurately detect and segment ships at the pixel level. By adding a bottom-up structure to the FPN structure of Mask R-CNN, the path between the lower layers and the topmost layer is shortened, allowing the lower layer features to be more effectively utilized at the top layer. In the bottom-up structure, we use channel-wise attention to assign weights in each channel and use the spatial attention mechanism to assign a corresponding weight at each pixel in the feature maps. This allows the feature maps to respond better to the target’s features. Using our method, the detection and segmentation mAPs increased from 70.6% and 62.0% to 76.1% and 65.8%, respectively.

In the last years, deep learning and reinforcement learning methods have significantly improved mobile robots in such fields as perception, navigation, and planning. But there are still gaps in applying these methods to real robots due to the low computational efficiency of recent neural network architectures and their poor adaptability to robotic experiments’ realities. In this article, we consider an important task in mobile robotics - navigation to an object using an RGB-D camera. We develop a new neural network framework for robot control that is fast and resistant to possible noise in sensors and actuators. We propose an original integration of semantic segmentation, mapping, localization, and reinforcement learning methods to improve the effectiveness of exploring the environment, finding the desired object, and quickly navigating to it. We created a new HISNav dataset based on the Habitat virtual environment, which allowed us to use simulation experiments to pre-train the model and then upload it to a real robot. Our architecture is adapted to work in a real-time environment and fully implements modern trends in this area.

The moving object detection refers to the detection of physical moving objects from a video, which is applied in video surveillance, object recognition, object counting, human-computer interaction, and so on. Moreover, nowadays, real-time moving object detection is used as services in the cloud, edge, and fog computing. However, the existing methods do not meet the trade-off between accuracy and complexity. To address these issues, we present a background subtraction-based moving object detection method, called Fast-D. In this paper, we look at the `non-smoothing color feature' to make the moving object detection more robust in real-time. Each color feature is given equal significance during the classification of a pixel. Background model and threshold are initialized for each pixel. And then, the background model and threshold are updated dynamically when there are changes in the background of the video. Adaptive post-processing is considered to discard salt and pepper noise and fill holes in the detected moving object silhouettes. The evaluation of our proposed method on four complex datasets exhibits the significance.

Addressing the perception problem of texture-less objects that undergo large deformations and movements, this article presents a novel RGB-D learning-free deformable object tracker in combination with a camera position optimisation system for optimal deformable object perception. The approach is based on the discretisation of the object''s visible area through the generation of a supervoxel graph that allows weighting new supervoxel candidates between object states over time. Once a deformation state of the object is determined, supervoxels of its associated graph serve as input for the camera position optimisation problem. Satisfactory results have been obtained in real time with a variety of objects that present different deformation characteristics.

Urban trees are essential elements in outdoor scenes recorded via terrestrial laser scanning. Although considerable interest has been centered on tree detection and reconstruction in recent years, trees cannot be easily extracted from dense and unorganized data because of the complexity and diversity of trees. In this paper, we present a top-down approach for detecting trees from point cloud data acquired for dense urban areas. Appropriate feature subsets are chosen, and then the candidate tree clusters are selected via a binary classification. After distinguishing the 3D points belonging to tree-like objects, individual trees are extracted by spectral clustering. Furthermore, a weighted constraint rule is proposed to refine the individual tree clusters. The methodology is tested on five real-world datasets that include different varieties of trees. The results reveal that most of the individual trees can be correctly detected and extracted. The results are quantitatively evaluated and reveal a global F1 value of approximately 97 percent and a precision of approximately 98 percent. Comparative analysis on the datasets is also provided to prove the effectiveness of our proposed method.

Varying weather conditions, including rainfall and snowfall, are generally regarded as a challenge for computer vision algorithms. One proposed solution to the challenges induced by rain and snowfall is to artificially remove the rain from images or video using rain removal algorithms. It is the promise of these algorithms that the rain-removed image frames will improve the performance of subsequent segmentation and tracking algorithms. However, rain removal algorithms are typically evaluated on their ability to remove synthetic rain on a small subset of images. Currently, their behavior is unknown on real-world videos when integrated with a typical computer vision pipeline. In this paper, we review the existing rain removal algorithms and propose a new dataset that consists of 22 traffic surveillance sequences under a broad variety of weather conditions that all include either rain or snowfall. We propose a new evaluation protocol that evaluates the rain removal algorithms on their ability to improve the performance of subsequent segmentation, instance segmentation, and feature tracking algorithms under rain and snow. If successful, the de-rained frames of a rain removal algorithm should improve segmentation performance and increase the number of accurately tracked features. The results show that a recent single-frame-based rain removal algorithm increases the segmentation performance by 19.7% on our proposed dataset, but it eventually decreases the feature tracking performance and showed mixed results with recent instance segmentation methods. However, the best video-based rain removal algorithm improves the feature tracking accuracy by 7.72%., Published in IEEE Transactions on Intelligent Transportation Systems

The analysis of medical image data currently requires the interpretation of a trained and experienced user. The technological advances in imaging machinery and the understanding of disease onset, as well as medical planning, all favour the need for ever more automatic and robust methods for evaluating the health state of a subject. Here, we concentrate on methods for processing medical image data, as currently provided by existing imaging technologies, in particular the effectiveness of automatic image filtering in order to remove noise and improve the sharpness of distinct objects. The filtering approach is based on a partial differential equation, namely the Perona-Malik anisotropic diffusion equation. The approach adopted for terminating the iterative filtering procedure is based on image quality descriptors. In specific, we observe the rate of change of these to infer the transient effects of the filtering process. The entire pipeline is demonstrated to work effectively on different sets of medical image data, including MRI, CTA and CT, both in individual 2-D images in a stack, as well as treating the complete 3D volumetric dataset.

Monocular depth estimation is an essential technique for tasks like 3D reconstruction. Although many works have emerged in recent years, they can be improved by better utilizing the multi-scale information of the input images, which is proved to be one of the keys in generating high-quality depth estimations. In this paper, we propose a new monocular depth estimation method named HMA-Depth, in which we follow the encoder-decoder scheme and combine several techniques such as skip connections and the atrous spatial pyramid pooling. To obtain more precise local information from the image while keeping a good understanding of the global context, a hierarchical multi-scale attention module is adopted and its outputs are combined to generate the final output that is with both good details and good overall accuracy. Experimental results on two commonly-used datasets prove that HMA-Depth can outperform the existing approaches. Code is available 1 1 https://github.com/saranew/HMADepth., MVA2021,17th International Conference on Machine Vision Applications,July 25–27, 2021, Fully Online

We introduce a model that segments lesions and predicts COVID-19 from chest CT scans through the derivation of an affinity matrix between lesion masks. The novelty of the methodology is based on the computation of the affinity between the lesion masks’ features extracted from the image. First, a batch of vectorized lesion masks is constructed. Then, the model learns the parameters of the affinity matrix that captures the relationship between features in each vector. Finally, the affinity is expressed as a single vector of pre-defined length. Without any complicated data manipulation, class balancing tricks, and using only a fraction of the training data, we achieve a 91.74% COVID-19 sensitivity, 85.35% common pneumonia sensitivity, 97.26% true negative rate and 91.94% F1-score. Ablation studies show that the method can quickly generalize to new datasets. All source code, models and results are publicly available on https://github.com/AlexTS1980/COVID-Affinity-Model.

As humans, we have a remarkable ability of telling objects apart from cluttered backgroundand tracing their contours even with occlusions. This ability has long fascinated computer vision researchers to study the principles and algorithms for object segmentation. Object segmentation has both theoretical and practical interests as it is an essential step towards 3D image understanding and intelligent image editing.To segment an object, we have to recognize it in order to obtain knowledge of what parts should be grouped together. In this thesis, we formulate object segmentation as an image labeling problem in random field models to facilitate integrating top-down recognition knowledge with bottom-up image cues. The integration can be driven by either bottom-up segmentation or top-down recognition. The segmentation-driven process requires object-level segmentation hypotheses drawn from bottom-up cues while the recognition-driven process needs shape and context to be effectively represented. This thesis addresses these issues in a data-driven approach. First, we propose to generate object segmentation proposals from segmentation trees using exemplars. Compared to previous parametric methods, our data-driven method takes advantage of both diversity and informativenessof exemplars and thus produce a compact set of highly plausible proposals. Second, wepropose novel random field models that enjoy joint learning of shape representation andobject segmentation. Different from previous works that use shape representation as prior,our model emphasizes the structured prediction from the recognition model to the shapemodel. This difference ensures the the shape is well preserved in the resulting segmentationmasks with robustness to partial occlusions. Third, we develop a novel nonparametricmethod based on multiscale shape transfer, which in turns forms a higher-order randomfield. Compared to previous works that transfer rigid or deformable masks in image subwindows, our method explores shape masks in multiple granularities and is able to produce high quality segmentations in an efficient way. The last but not least, we develop a novel scene parsing system where small objects are segmented in context. With extensive use of context in multiscale and particular care to the long-tailed label distribution, our systemdemonstrates state-of-the-art results in large-scale problems.

State-of-the-art semantic segmentation methods rely too much on complicated deep networks and thus cannot train efficiently. This paper introduces a novel Circle-U-Net architecture that exceeds the original U-Net on several standards. The proposed model includes circle connect layers, which is the backbone of ResUNet-a architecture. The model possesses a contracting part with residual bottleneck and circle connect layers that capture context and expanding paths, with sampling layers and merging layers for a pixel-wise localization. The results of the experiment show that the proposed Circle-U-Net achieves an improved accuracy of 5.6676%, 2.1587% IoU (Intersection of union, IoU) and can detect 67% classes greater than U-Net, which is better than current results.

Neurodegenerative diseases are a group of largely incurable disorders characterised by the progressive loss of neurons and for which often the molecular mechanisms are poorly understood. To bridge this gap, researchers employ a range of techniques. A very prominent and useful technique adopted across many different fields is imaging and the analysis of histopathological and fluorescent label tissue samples. Although image acquisition has been efficiently automated recently, automated analysis still presents a bottleneck. Although various methods have been developed to automate this task, they tend to make use of single-purpose machine learning models that require extensive training, imposing a significant workload on the experts and introducing variability in the analysis. Moreover, these methods are impractical to audit and adapt, as their internal parameters are difficult to interpret and change. Here, we present a novel unsupervised automated schema for object segmentation of images, exemplified on a dataset of tissue images. Our schema does not require training data, can be fully audited and is based on a series of understandable biological decisions. In order to evaluate and validate our schema, we compared it with a state-of-the-art automated segmentation method for post-mortem tissues of ALS patients.

International audience; The binary partition tree (BPT) allows for the hierarchical representation of images in a multiscale way, by providing a tree of nodes corresponding to image regions. In particular, cuts of a BPT can be interpreted as segmentations of the associated image. Building the BPT of an image then constitutes a relevant preliminary step for optimization-based segmentation methods. A wide literature has been devoted to the construction of BPTs, and their involvement in such segmentation tasks. Comparatively, there exist few works dedicated to evaluate the quality of BPTs, i.e. their ability to allow further segmentation methods to compute good results. We propose such a framework for evaluating the quality of a BPT with respect to the object segmentation problem, i.e. the segmentation of one or several objects from an image. This framework is supervised, since the notion of segmentation quality is not only depending on the application but also on the user's objectives, expressed via the chosen ground-truth and quality metric. We develop two sides within this framework. First, we propose an intrinsic quality analysis, that relies on the structural coherence of the BPT with respect to ground-truth. More precisely, we evaluate to what extent the BPT structure is well-matching such examples, in a set / combinatorial fashion. Second, we propose an extrinsic analysis, by allowing the user to assess the quality of a BPT based on chosen metrics that correspond to the desired properties of the subsequent segmentation. In particular, we evaluate to what extent a BPT can provide good results with respect to such metrics whereas handling the trade-off with the cardinality of the cuts.

Stereo vision utilizes two cameras to acquire two respective images, and then determines the depth map by calculating the disparity between two images. In general, object segmentation and stereo matching are some of the important technologies that are often used in establishing stereo vision systems. In this study, we implement a highly efficient self-organizing map (SOM) neural network hardware accelerator as unsupervised color segmentation for real-time stereo imaging. The stereo imaging system is established by pipelined, hierarchical architecture, which includes an SOM neural network module, a connected component labeling module, and a sum-of-absolute-difference-based stereo matching module. The experiment is conducted on a hardware resources-constrained embedded system. The performance of stereo imaging system is able to achieve 13.8 frames per second of 640 &times, 480 resolution color images.

A four-step approach for image filtering and object segmentation is explored. The key steps are: i) a low-pass digital differentiator is used to compute the image gradient (vector) field; ii) the regularised anisotropic diffusion method is used to filter this vector field; iii) the modified image is reconstructed from the filtered gradient field as a least-squares best fit; iv) object segmentation is performed on the reconstructed image. The advantages of this approach is the easier identification of noise in the gradient field, and consequently the image filtering can be more effective. The least-squares fit allows for non-local adjustment to the image to improve overall smoothness while enhancing object contrast. Importantly one can also ensure that relevant feature boundary locations are preserved, while appearing smoother due to the filtering. The proposed filtering methodology is compared to image filtering applied directly to the image intensity. A set of challenging medical images of mammography exams and confocal microscopy experiments are used as numerical tests.

The segmentation of rock grains on images depicting bulk rock materials is considered. The rocks material images are transformed by selected texture operators, to obtain a set of features describing them. The first order features, second-order features, run-length matrix, grey tone difference matrix, and Laws' energies are used for that purpose. The features are classified using k-nearest neighbours, support vector machines, and artificial neural networks classifiers. The results show that the border of rocks grains can be determined with above 70% accuracy. The multi-texture approach was also investigated, leading to an increase in accuracy to over 77% for early-fusion of features. Attempts were made to reduce feature space dimensionality by manually picking features as well as by use of the principal component analysis. The outcomes showed a significant decrease in accuracy. The obtained results have been visually compared with the ground truth. The observed compliance can be considered satisfactory.

Various types of knowledge and features have been explored for level set-based segmentation. On the ground, the prior knowledge and carefully-designed features perform well to identify the foreground-background contrast, which improves the performance of the segmentation method for complicated and distorted data. However, this is not the case for underwater environments, since the features available on the ground are not suitable for challenging underwater environments. Thus, underwater image segmentation currently lags behind ground-based segmentation. In this paper, novel cues and a suitable model formulation for object segmentation from underwater images are proposed. We consider the special haze effect over underwater images and extract an informative feature (transmission feature) from haze condensation. The saliency feature is also used for underwater object segmentation. Consequently, in our method, the object-background difference can be presented by these features on two levels, i.e., the edge-level transmission and region-level saliency features. These two types of features are integrated into a unified level set formulation to propose a solution that handles the challenging issues in underwater object segmentation. The experimental comparisons of our method with other methods comprehensively demonstrate the satisfactory performance of our method.

A deep convolutional neural network has recently witnessed rapid progress due to the strong feature learning capability. In this paper, we focus on its application in the industrial field and propose a method based on a fully convolutional network (FCN) for detecting defects in tire X-ray images. Owing to the capability of pixel-wise prediction of FCN, the location, and segmentation of defects are completed simultaneously. The network architecture used in the method mainly consists of three phases. The first phase is a traditional deep network, which is used to extract the feature of tire images, and feature maps are obtained at the last layer. By replacing fully connected layers into convolution layers, final feature maps retain sufficient spatial information. By adding up-sampling layers, in the second phase, outputs with the same size as the original image can be generated. After the first two phases, we develop the coarse segmentation results and refine them through fusing multi-scale feature maps. The experimental results show that the proposed method can accurately locate and segment defects in tire images.

The issue of achieving an appropriate segmentation for indoor point cloud scenes remains difficult. Although available methods continue to improve the benchmark performance, more attentions need to be paid to deal with the drawbacks of inaccurate or incomplete segments in division. To push the research to the next level, this work proposes an learning-free algorithm for the segmentation of indoor point clouds which consists of two stages. The first stage extracts edges of RGBD point clouds and applies them in the voxel clustering process to avoid generating supervoxels which are situated across object boundaries. After this pre-segmentation, a two-phase merging procedure is presented in the second part. By conducting region growing on optimized supervoxels, a set of local regions is obtained. Then we propose to define the convexity-concavity of adjacent regions based on the observations of object structures and merge the convexly connected regions to achieve object-level segmentation. This algorithm is straightforward to implement and requires no training data. Experimental results show that it produces supervoxels with plausible boundaries and arrives at better object-level segmentation.

Video segmentation is the foundation of many senior computer vision applications. For robotic vision applications, video object segmentation is facing more difficulties. Since the videos are usually collected from those complicated scenes and the camera perspective may be static for some time, object saliency and motion clues which are widely used in traditional video segmentation approaches could be unreliable. Besides, its online processing ability is of crucial importance. In this paper, we propose an online video object segmentation method, which is organized as a segmentation-by-detection framework. It first effectively segments out meaningful objects in short video frame batches with the help of object detectors, and then the segmentation of different batches can be associated with each other via bi-directional notebook based connecting strategy. Experiments conducted on public available datasets verify the above-mentioned property and show good performance of the proposed method.

Multi-oriented text detection in the wild is a challenging task due to the variations of scales, orientations, illumination, and languages. The traditional anchor mechanism on generic object detection can only generate horizontal proposals, which cannot be applied to detecting multi-oriented text regions. Considering this, in this paper, we propose a novel convolutional regression network (CRN) to localize multi-oriented text in natural images, which consists of two components: region proposal extractor and text locator. To be specific, we first present a hierarchical deconvolution module (HDM), a text-line and geometry segmentation module (TGM) to segment the multi-oriented proposals accurately, both of which are fully convolutional networks. Then, a classification and regression module (CRM) is adopted to process the proposals and obtain the final localization results. The whole framework can be trained in an end-to-end mechanism which is suitable for detecting multi-oriented texts. The extensive experiments are conducted on three mainstream scene-text datasets, and the experimental results evidence the proposed CRN achieves competitive performance.

A tracking-by-segmentation algorithm, which tracks and segments a target object in a video sequence, is proposed in this paper. In the first frame, we segment out the target object in a user-annotated bounding box. Then, we divide subsequent frames into superpixels. We develop a superpixel-wise neural network for tracking-by-segmentation, called TBSNet, which extracts multi-level convolutional features of each superpixel and yields the foreground probability of the superpixel as the output. We train TBSNet in two stages. First, we perform offline training to enable TBSNet to discriminate general objects from the background. Second, during the tracking, we fine-tune TBSNet to distinguish the target object from non-targets and adapt to color change and shape variation of the target object. Finally, we perform conditional random field optimization to improve the segmentation quality further. Experimental results demonstrate that the proposed algorithm outperforms the state-of-the-art trackers on four challenging data sets.

This paper presents a novel distributed object segmentation framework that allows one to extract potentially large coherent objects from digital images. The proposed approach requires minimum user supervision and permits to segment the objects accurately. It works in three steps starting with the user input in form of few mouse clicks on the target object. First, based on user input, the statistical characteristics of the target distributed object are modeled with Gaussian mixture model. This model serves as the primary segmentation of the object. In the second step, the segmentation result is refined by performing connected component analysis to reduce false positives. In the final step the resulting segmentation map is dilated to select the neighboring pixels that are potentially incorrectly classified; this allows us to recast the segmentation as a graph partitioning problem that can be solved using the well-known graph cut technique. Extensive experiments have been carried out on heterogeneous images to test the accuracy of the proposed method for the segmentation of various types of distributed objects. Examples of application of proposed technique in remote sensing to segment roads and rivers from aerial images are also presented. The visual and objective evaluation and comparison with the existing techniques show that the proposed tool can deliver optimal performance when applied to tough object segmentation tasks.

We present a method for generating synthetic ground truth for training segmentation networks for presegmenting point clouds in pose estimation problems. Our method replaces global pose estimation algorithms such as RANSAC which requires manual fine-tuning with a robust CNN, without having to hand-label segmentation masks for the given object. The data is generated by blending cropped images of the objects with arbitrary backgrounds. We test the method in two scenarios, and show that networks trained on the generated data segments the objects with high accuracy, allowing them to be used in a pose estimation pipeline.

Video object segmentation (VOS) is a highly challenging problem since the initial mask, defining the target object, is only given at test-time. The main difficulty is to effectively handle appearance changes and similar background objects, while maintaining accurate segmentation. Most previous approaches fine-tune segmentation networks on the first frame, resulting in impractical frame-rates and risk of overfitting. More recent methods integrate generative target appearance models, but either achieve limited robustness or require large amounts of training data. We propose a novel VOS architecture consisting of two network components. The target appearance model consists of a light-weight module, which is learned during the inference stage using fast optimization techniques to predict a coarse but robust target segmentation. The segmentation model is exclusively trained offline, designed to process the coarse scores into high quality segmentation masks. Our method is fast, easily trainable and remains highly effective in cases of limited training data. We perform extensive experiments on the challenging YouTube-VOS and DAVIS datasets. Our network achieves favorable performance, while operating at higher frame-rates compared to state-of-the-art. Code and trained models are available at https://github.com/andr345/frtm-vos., CVPR 2020. arXiv admin note: substantial text overlap with arXiv:1904.08630

Object recognition has been widely investigated in computer vision for many years. Currently, this process is carried out through neural networks, but there are very few public datasets available with mask and class labels of the objects for the training process in usual applications. In this paper, we address the problem of fast generation of synthetic datasets to train neural models because creating a handcraft labeled dataset with object segmentation is a very tedious and time-consuming task. We propose an efficient method to generate a synthetic labeled dataset that adequately combines background images with foreground segmented objects. The synthetic images can be created automatically with random positioning of the objects or, alternatively, the method can produce realistic images by keeping the realism in the scales and positions of the objects. Then, we employ Mask-RCNN deep learning model, to detect and segment classes of kitchen objects using images. In the experimental evaluation, we study both synthetic datasets, automatic or realistic, and we compare the results. We analyze the performance with the most widely used indexes and check that the realistic synthetic dataset, quickly created through our method, can provide competitive results and accurately classify the different objects.

This is the author version accepted for publication on May 28th, 2019.The final (editor OpenAccess version) can be accessed via https://rdcu.be/bPlwnSupplementary video material available at https://object-removal.telecom-paristech.fr/; International audience; We present a system for the removal of objects from videos. As an input, the system only needs a user to draw a few strokes on the first frame, roughly delimiting the objects to be removed. To the best of our knowledge, this is the first system allowing the semi-automatic removal of objects in videos with complex backgrounds. The key steps of our system are the following: after initialization, segmentation masks are first refined and then automatically propagated through the video. The missing regions are then synthesized using video inpainting techniques. Our system can deal with multiple, possibly crossing objects, with complex motions, and with dynamic textures. This results in a computational tool that can alleviate tedious manual operations for editing high-quality videos.

We propose a technique for coherently co-clustering uncalibrated views of a scene with a contour-based representation. Our work extends the previous framework, an iterative algorithm for segmenting sequences with small variations, where the partition solution space is too restrictive for scenarios where consecutive images present larger variations. To deal with a more flexible scenario, we present three main contributions. First, motion information has been considered both for region adjacency and region similarity. Second, a two-step iterative architecture is proposed to increase the partition solution space. Third, a feasible global optimization that allows to jointly process all the views has been implemented. In addition to the previous contributions, which are based on low-level features, we have also considered introducing higher level features as semantic information in the co-clustering algorithm. We evaluate these techniques on multiview and temporal datasets, showing that they outperform state-of-the-art approaches.

We introduce a generative probabilistic model for segmentation of brain lesions in multi-dimensional images that generalizes the EM segmenter, a common approach for modelling brain images using Gaussian mixtures and a probabilistic tissue atlas that employs expectation-maximization (EM), to estimate the label map for a new image. Our model augments the probabilistic atlas of the healthy tissues with a latent atlas of the lesion. We derive an estimation algorithm with closed-form EM update equations. The method extracts a latent atlas prior distribution and the lesion posterior distributions jointly from the image data. It delineates lesion areas individually in each channel, allowing for differences in lesion appearance across modalities, an important feature of many brain tumor imaging sequences. We also propose discriminative model extensions to map the output of the generative model to arbitrary labels with semantic and biological meaning, such as “tumor core” or “fluid-filled structure”, but without a one-to-one correspondence to the hypo- or hyper-intense lesion areas identified by the generative model. We test the approach in two image sets: the publicly available BRATS set of glioma patient scans, and multimodal brain images of patients with acute and subacute ischemic stroke. We find the generative model that has been designed for tumor lesions to generalize well to stroke images, and the extended discriminative -discriminative model to be one of the top ranking methods in the BRATS evaluation., National Alliance for Medical Image Computing (U.S.) (NIH NIBIB NAMIC U54- EB005149), Neuroimaging Analysis Center (U.S.) (NIH NIBIB NAC P41EB015902), National Institute for Biomedical Imaging and Bioengineering (U.S.) (R01EB013565), Lundbeck Foundation (R141-2013-13117)

Estimating the damaged area after a forest fire is important for responding to this natural catastrophe. With the support of aerial remote sensing, typically with unmanned aerial vehicles (UAVs), the aerial imagery of forest-fire areas can be easily obtained; however, retrieving the burnt area from the image is still a challenge. We implemented a new approach for segmenting burnt areas from UAV images using deep learning algorithms. First, the data were collected from a forest fire in Andong, the Republic of Korea, in April 2020. Then, the proposed two-patch-level deep-learning models were implemented. A patch-level 1 network was trained using the UNet++ architecture. The output prediction of this network was used as a position input for the second network, which used UNet. It took the reference position from the first network as its input and refined the results. Finally, the final performance of our proposed method was compared with a state-of-the-art image-segmentation algorithm to prove its robustness. Comparative research on the loss functions was also performed. Our proposed approach demonstrated its effectiveness in extracting burnt areas from UAV images and can contribute to estimating maps showing the areas damaged by forest fires.

This paper presents a novel approach to the automated recognition and localization of 3-D objects. The proposed approach uses 3-D object segmentation to segment randomly stacked objects in an unstructured point cloud. Each segmented object is then represented by a regional area-based descriptor, which measures the distribution of surface area in the oriented bounding box (OBB) of the segmented object. By comparing the estimated descriptor with the template descriptors stored in the database, the object can be recognized. With this approach, the detected object can be matched with the model using the iterative closest point (ICP) algorithm to detect its 3-D location and orientation. Experiments were performed to verify the feasibility and effectiveness of the approach. With the measured point clouds having a spatial resolution of 1.05 mm, the proposed method can achieve both a mean deviation and standard deviation below half of the spatial resolution.

Visual attention is often predictive for future actions in humans. In manipulation tasks, the eyes tend to fixate an object of interest even before the reach-to-grasp is initiated. Some recent studies have proposed to exploit this anticipatory gaze behavior to improve the control of dexterous upper limb prostheses. This requires a detailed understanding of visuomotor coordination to determine in which temporal window gaze may provide helpful information. In this paper, we verify and quantify the gaze and motor behavior of 14 transradial amputees who were asked to grasp and manipulate common household objects with their missing limb. For comparison, we also include data from 30 able-bodied subjects who executed the same protocol with their right arm. The dataset contains gaze, first person video, angular velocities of the head, and electromyography and accelerometry of the forearm. To analyze the large amount of video, we developed a procedure based on recent deep learning methods to automatically detect and segment all objects of interest. This allowed us to accurately determine the pixel distances between the gaze point, the target object, and the limb in each individual frame. Our analysis shows a clear coordination between the eyes and the limb in the reach-to-grasp phase, confirming that both intact and amputated subjects precede the grasp with their eyes by more than 500 ms. Furthermore, we note that the gaze behavior of amputees was remarkably similar to that of the able-bodied control group, despite their inability to physically manipulate the objects., Frontiers in Bioengineering and Biotechnology, 7, ISSN:2296-4185

Evaluating object segmentation is a topic of great interest for shape comparison techniques. In this work, ad-hoc metrics for a detailed segmentation analysis and a novel keypoint based method for comparing pairs of shapes are presented. As references, two different segmentation approaches were used: a handmade segmentation and an automatic one based on a Convolutional Neural Network (CNN). The proposed comparison approach consists of a combination between a keypoint extractor and an invariant scale shape identifier. The overall validation process is established according to different steps, which allow to measure the similarity between shapes. First, Reinforced Matched (RM) and Reinforced Ratio (RR) strategies are implemented. Moreover, five different state-of-the-art keypoint extractors are compared, i.e., SIFT, SURF, ORB, A-KAZE, and BRISK. Experimental tests were performed on a popular collection of images, i.e., the Berkeley Segmentation Dataset and Benchmark 300 (BSDS300), which contains shapes segmented both manually and automatically. The experimental results have shown the effectiveness of the proposed method.

This paper presents a light-weight process for 3D reconstruction and measurement of chronic wounds using a commonly available smartphone as an image capturing device. The first stage of our measurement pipeline comprises the creation of a dense 3D point cloud using structure-from-motion (SfM). Furthermore, the wound area is segmented from the surrounding skin using dynamic thresholding in CIELAB color space and a surface is estimated to simulate the missing skin in the wound area. Together with a mesh reconstruction of the wound, the skin surface and the segmented wound is used to calculate the wound dimensions, i.e., its length, surface area and volume. We evaluate the presented pipeline using three wound phantoms, representing different stages in healing, and compare the subsequently scanned and measured wound dimensions with manually measured ones.

Traditional marine mammal photo-identification is based on recognizing the appearances of the same individuals in pictures taken at different places and times. This task is traditionally performed by Biologists or other Scientists, which may demand a heavy cognitive burden and appreciable processing time searching and selecting information from thousands of pictures. Recently crowdsourcing and citizen science arose as a significant information source of potential scientific use. In particular, the use of non-professional photographs taken by the general public is being leveraged by many scientific projects. This represents an opportunity to enlarge the picture database required in trustable capture-recapture models, but at the same time human-assisted matching becomes unfeasible. Automated image analysis may represent an obvious aid, but applying image analytics to match individuals in large unfiltered datasets may be too slow and full of spurious results. Another strategy may be first to filter out useless images or parts thereof, retaining only the regions of interest (RoIs) in which appears the actual visible portion of the animal to be identified. In this work, we explore and develop a multi-criterion RoI detection for marine mammal pictures taken in the open. Particularly we focus on Commerson's dolphins pictures. Popular RoI detection algorithms, like Haar-wavelet-based methods, are show to perform poorly. For this reason, a convolutional neural network and a multifractal classifier based on color and texture features were developed, achieving significantly better outcomes. The resulting RoIs are much more robust, can be automated, and reduce the further burden of the identification process, either assisted or unassisted. Fil: Pollicelli, Débora. Universidad Nacional de la Patagonia "San Juan Bosco"; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Centro Nacional Patagónico. Centro para el Estudio de Sistemas Marinos; Argentina Fil: Coscarella, Mariano Alberto. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Centro Nacional Patagónico. Centro para el Estudio de Sistemas Marinos; Argentina. Universidad Nacional de la Patagonia "San Juan Bosco"; Argentina Fil: Delrieux, Claudio Augusto. Universidad Nacional de la Patagonia "San Juan Bosco"; Argentina. Universidad Nacional del Sur; Argentina. Consejo Nacional de Investigaciones Científicas y Técnicas; Argentina

Automatic road extraction from remote-sensing imagery plays an important role in many applications. However, accurate and efficient extraction from very high-resolution (VHR) images remains difficult because of, for example, increased data size and superfluous details, the spatial and spectral diversity of road targets, disturbances (e.g., vehicles, shadows of trees, and buildings), the necessity of finding weak road edges while avoiding noise, and the fast-acquisition requirement of road information for crisis response. To solve these difficulties, a two-stage method combining edge information and region characteristics is presented. In the first stage, convolutions are executed by applying Gabor wavelets in the best scale to detect Gabor features with location and orientation information. The features are then merged into one response map for connection analysis. In the second stage, highly complete, connected Gabor features are used as edge constraints to facilitate stable object segmentation and limit region growing. Finally, segmented objects are evaluated by some fundamental shape features to eliminate nonroad objects. The results indicate the validity and superiority of the proposed method to efficiently extract accurate road targets from VHR remote-sensing images.

This paper presents a novel object segmentation technique to extract objects that are potentially scattered or distributed over the whole image. The goal of the proposed approach is to achieve accurate segmentation with minimum and easy user assistance. The user provides input in the form of few mouse clicks on the target object which are used to characterize its statistical properties using Gaussian mixture model. This model determines the primary segmentation of the object which is refined by performing morphological operations to reduce the false positives. We observe that the boundary pixels of the target object are potentially misclassified. To obtain an accurate segmentation, we recast our objective as a graph partitioning problem which is solved using the graph cut technique. The proposed technique is tested on several images to segment various types of distributed objects e.g. fences, railings, flowers. We also show some remote sensing application examples, i.e. segmentation of roads, rivers, etc. from aerial images. The obtained results show the effectiveness of the proposed technique.

An efficient moving object Segmentation is useful for real time content based video surveillance and Object Tracking. Commonly a foreground is extracted using a mixture of Gaussian followed by shadow and noise removal to initialise the Object Trackers. This technique uses a kernel mask to make the system more efficient by decreasing the search area and the number of iterations to converge in the new location of the object. In the background model, the post processing step is applied to the obtained object mask to remove noise region and to smoothen the object boundary which incurs additional delay. In this paper a Background Buffering algorithm (BBA) is proposed to construct a reliable background model based on a sequence of input frames. Except moving object data, the other data is used to build reliable background representation. The result shows significant decrease in run-time for the higher level processing steps of surveillance system.

International audience; Though there are many computational models proposed for saliency detection, few of them take object boundary information into account. This paper presents a hierarchical saliency detection model incorporating probabilistic object boundaries, which is based on the observation that salient objects are generally surrounded by explicit boundaries and show contrast with their surroundings. We perform adaptive thresholding operation on ultrametric contour map, which leads to hierarchical image segmentations, and compute the saliency map for each layer based on the proposed robust center bias, border bias, color dissimilarity and spatial coherence measures. After a linear weighted combination of multi-layer saliency maps, and Bayesian enhancement procedure, the final saliency map is obtained. Extensive experimental results on three challenging benchmark datasets demonstrate that the proposed model outperforms eight state-of-the-art saliency detection models.