Search

Multi-object tracking (MOT) is a crucial technology for security surveillance, which is computationally intensive due to the requirement of processing a large number of video streams within low latency in practice. The input video streams of MOT are processed on a cloud computing center with abundant computational capability, posing heavy pressures on delivering video streams to the cloud. Recent advances in the Internet of Things (IoT) technology provide edge-computing-based solutions for video analytics at scale. However, the gap between MOT’s high computational capability demand and IoT devices’ resource-constrained nature remains significant. In this paper, a resource-efficient multi-object tracking method (REMOT) is proposed for real-time surveillance on IoT embedded devices, including an affinity measurement based on an appearance model with angular triplet loss and a motion association that substitutes the time-consuming graph-based data association stage. Considering the trade-off between latency and accuracy, we design an optimization strategy on the parallel processing of deep learning models’ layers to accelerate the inference speed with less accuracy loss. Besides, we employ a model compression strategy for model size reduction. Experiments on MOT16 and MOT17 benchmarks demonstrate that REMOT reduces 2.4x latency compared with the original implementation and achieves a running speed of 81 frames per second (fps) on an embedded device with only a marginal accuracy loss (6%), which meets the requirements of real-time processing and low-latency response for surveillance.

The appearance of a real-world feather is the result of light interactions with complex, patterned structures of varying scale; however, these have not yet been modeled for accurate rendering of feathers in computer graphics. Previously published works related to production and research have presented simplified curve models to represent the appearance of feathers. In this work we present why these approaches are not sufficient using imaging from real feathers along with motivation for a specific appearance model for feathers. We also propose and compare a new technique that takes into account the detailed substructures of feathers and their role in rendering an accurate far-field appearance. Our proposed method lends a high degree of photorealism for rendering feathers in visual-effects and content-creation applications.

Most existing trackers use bounding boxes for object tracking. However, the background contained in the bounding box inevitably decreases the accuracy of the target model, which affects the performance of the tracker and is particularly pronounced for non-rigid objects. To address the above issue, this paper proposes a novel hybrid level set model, which can robustly address the issue of topology changing, occlusions and abrupt motion in non-rigid object tracking by accurately tracking the object contour. In particular, an appearance model is first obtained by repeatedly training and relabeling the initial labeled frame using competing one-class SVMs. Then, by integrating the trained appearance model, an edge detector and image spatial information into the level set model, a new hybrid level set model is presented, which accurately locates the object contour and feeds back to the competing one-class SVMs to update the appearance model of the next frame. In addition, a motion model is defined to predict the accurate location of the object when occlusion and abrupt motion occur in the next frame. Finally, the experimental results on state-of-the-art benchmarks demonstrate the feasibility and effectiveness of the proposed model and the superiority of the proposed method over existing trackers in terms of accuracy and robustness.

Object tracking is an important research direction of space Earth observation in the field of remote sensing. Although the existing correlation filter-based and deep learning (DL)-based object tracking algorithms have achieved great success, they are still unsatisfactory for the problem of object occlusion. The occlusion caused by the complex change in background, and the deviation of the tracking lens, causes object information to go missing, which leads to the omission of detection. Traditionally, most methods for object tracking under occlusion adopt a complex network model, which redetects the occluded object. To address this issue, we propose a novel object tracking approach. First, an action decision-occlusion handling network (AD-OHNet) based on deep reinforcement learning (DRL) is built to achieve low computational complexity for object tracking under occlusion. Second, the temporal and spatial context, the object appearance model, and the motion vector are adopted to provide the occlusion information, which drives actions in reinforcement learning under complete occlusion and contributes to improving the accuracy of tracking while maintaining speed. Finally, the proposed AD-OHNet is evaluated on three remote sensing video datasets of Bogota, Hong Kong, and San Diego taken from Jilin-1 commercial remote sensing satellites. The video datasets all shared problems of low spatial resolution, background clutter, and small objects. Experimental results on the three video datasets validate the effectiveness and efficiency of the proposed tracker.

Most of the existing single object trackers track the target in a unitary local search window, making them particularly vulnerable to challenging factors such as heavy occlusions and out-of-view movements. Despite the attempts to further incorporate global search, prevailing mechanisms that cooperate local and global search are relatively static, thus are still sub-optimal for improving tracking performance. By further studying the local and global search results, we raise a question: can we allow more dynamics for cooperating both results? In this paper, we propose to introduce more dynamics by devising a dynamic attention-guided multi-trajectory tracking strategy. In particular, we construct dynamic appearance model that contains multiple target templates, each of which provides its own attention for locating the target in the new frame. Guided by different attention, we maintain diversified tracking results for the target to build multi-trajectory tracking history, allowing more candidates to represent the true target trajectory. After spanning the whole sequence, we introduce a multi-trajectory selection network to find the best trajectory that delivers improved tracking performance. Extensive experimental results show that our proposed tracking strategy achieves compelling performance on various large-scale tracking benchmarks. The project page of this paper can be found at https://sites.google.com/view/mt-track/., Comment: Accepted by IEEE T-CSVT 2021

Recently, many discriminative correlation filter (DCF)-basedapproaches have been developed to alleviate the undesired boundary effects of traditional DCF-based methods. However, these approaches either focus on the target area by suppressing the background information or exploit a spatial regularisation term to penalise filter coefficients far from the target centre, where real negative samples are exceedingly rare and irrelevant information is integrated into filters. Based on the observation that the tracking model is strongly dependent on the quality of training samples, this paper proposes a novel DCF-based tracker to precisely process the training set through a smooth and steeply decreasing function in two respects. On the one hand, the surrounding information is suitably incorporated into the target. On the other hand, every real background patch is collected as a negative sample. In addition, the target appearance model is adaptively updated by achieving an appropriate trade-off between current and historical models according to their reliability. Experimental results show that our method outperforms other state-of-the-art trackers in terms of accuracy and efficiency. In particular, compared with the baseline tracker, our method achieves gains of 7.4% and 9.9% in terms of the area under the curve (AUC) on the OTB-100 and LaSOT datasets, respectively.

In this paper, we propose a point-cloudbased algorithm for human-following robots to detect and follow the target person in a complex outdoor environment. Specifically, we exploit AdaBoost to train a binary classifier in a designed feature space based on sparse point-cloud to distinguish the target person from other objects. Then a particle filter is applied to continuously track the target's position. Motivated by the interference of obstacles in long-distance human-following scenarios, a motion plan algorithm based on vector field histogram is adopted. Experiments are carried out both on the dataset we collected and in real application scenarios. The results show that our algorithm has the ability of real-time target detection and tracking, and is robust to deal with complex situations in outdoor environments.

The appearance model of flying basketball obtained by the traditional basketball flight trajectory tracking method is not accurate, which leads the anti-interference performance of trajectory tracking not ideal. Based on data fusion and sparse representation model, a new automatic trajectory tracking method is proposed. Firstly, the relevant technologies of basketball flight trajectory automatic tracking are studied and summarized, and then the method is studied. The specific implementation steps of this method are as follows: the features of flying basketball images were extracted by the target feature extraction algorithm, and the appearance model of flying basketball was built based on sparse representation. Data fusion technology and particle filter algorithm are combined to realize automatic tracking of basketball flight path. Through three axial basketball trajectories of automatic tracking test and noise test and verify the design method under the 3D world coordinate system to achieve the X, Y, and Z axis up more accurate tracking, at the same time, after applying measurement signal to noise, automatic trajectory tracking results affected by some, but still managed to realize the trajectory tracking.

One of the biggest outlooks of face recognition is to create a useful application for human identification at immigration offices for security purposes. The human face is a complex visual pattern that consists of general categorical information, identity specification, primary information, and eccentricity. In this domain, the ethnic identification of humans finds its uses in various real-time applications. Yet, automatic identification does not produce prompt results due to complex characteristics and computational complexity. This paper intends to propose automated human ethnicity identification using facial features, comprising three major processes: pre-processing, feature extraction, and classification. At first, the input image is subjected to a pre-processing method, in which the face detection is carried out using the Viola-Jones face detection algorithm. Then, the pre-processed image is subjected to the feature extraction process, where the color feature, texture feature, forehead area extraction, and the improved active appearance model (AAM) based on unique features are extracted. These extracted features are then subjected to the optimized convolutional neural network (CNN) for ethnicity classification. As the major contribution, training of CNN is carried out by the proposed Moth Spiral adopted Grey Wolf Algorithm (MSGWA) model via tuning the optimal weights. Finally, the performance of the proposed work is compared against the adopted and existing approaches on the basis of certain metrics such as NPV, sensitivity, FDR, accuracy, specificity, FPR, precision, MCC, FNR, and F1-score, respectively.

We propose a volleyball action standardization method based on 3D dynamic model. Firstly, optical motion capture technology and 3D scanning technology are used to collect the volleyball motion images. Then, the human skeleton model and the appearance model are constructed. Finally, the distance between each joint of the volleyball player's posture and the standard posture is calculated. The results show that the wrong actions made by the players can be accurately recognized by the proposed method. This study also gives some standardized suggestions and may improve players' action learning and training.

Aerial object tracking approaches based on discriminative correlation filter (DCF) have attracted wide attention in the tracking community due to their impressive progress recently. Many studies introduce temporal regularization into the DCF-based framework to achieve a more robust appearance model and further enhance the tracking performance. However, existing temporal regularization approaches usually utilize the information of two consecutive frames, which are not robust enough due to limited information. Although some methods attempt to incorporate abundant training samples and generally improve the tracking performance, these improvements are at the expense of significantly increased computing consumption. Besides, most existing methods introduce historical information directly without denoising, which means that background noises are also introduced into the filter training and may degrade the tracking accuracy. To tackle the drawbacks mentioned earlier, this work proposes a novel aerial object tracking approach to exploit disruptor-aware interval-based response inconsistency, i.e., IBRI tracker. The proposed method is able to incorporate historical interval information by utilizing responses in the filter training process, thereby obtaining a robust tracking performance while maintaining the real-time speed. Moreover, to reduce the disruptions caused by similar object, partial occlusion, and other challenging scenes, a novel disruptor-aware scheme based on response bucketing is introduced to detect the disruptor and enforce a spatial penalty for the disruptive area around the tracked object. Exhausted experiments on multiple well-known challenging aerial tracking benchmarks demonstrate the accuracy and robustness of the proposed IBRI tracker against other 35 state-of-the-art trackers. With a real-time speed of ~32 frames/s on a single CPU, the proposed approach can be applied for typical aerial platforms to achieve aerial visual object tracking efficiently.

Multi-object tracking (MOT) has received constant attention from researchers with the development of deep learning and person re-identification (ReID). However, the occlusion caused tracking failure is still far from solved. In this paper, we propose a Hard-Soft Attention Network (HSAN) to improve the ReID performance and get robust appearance features of different targets. The pose information and attention mechanism are combined to distinguish between challenging targets. Besides, the unary and binary costs are constructed to ensure consistency and long-term tracking, which consider not only the appearance-motion affinity of single tracks, but also the interactions between neighboring tracks. For that we cluster the tracks into different groups and choose reliable tracks as anchors to establish the two types of costs. Our HSAN appearance model is evaluated on the Market-1501, DUKE and CUHK03 ReID datasets and the MOT tracking method is conducted on MOTChallenge 15, 16 and 17. The experimental results demonstrate that our method can improve tracking accuracy and reduce fragments.

Currently, attention mechanism receives enormous interest and has been extensively employed in the fields of Person Re-Identification (RE-ID), as it gains superior performance in learning discriminative feature representations. However, most off-the-shelf attention methods are still vulnerable to cross-view inconsistency problem. Besides, they merely exploit imprecise channel attention information and coarse-grained spatial attention of homogeneous scales, being insufficient to capture subtle differences among highly-similar individuals. To this end, we propose a novel Attention-Aligned Network (AANet) to address the aforementioned problems, in which a novel Omnibearing Foreground-aware Attention (OFA) module, Attention Alignment Mechanism (AAM) and an improved triplet loss with hard mining are proposed to learn foreground attentive features for RE-ID. Specifically, AANet firstly leverages OFA module to exploit heterogeneous-scale spatial attention and foreground-aware channel attention information. Then AANet further reduces the impact of background clutter and learns camera-invariant and background-invariant representations by virtue of AAM. Last but not least, an improved triplet loss with hard mining is also introduced to enhance the feature learning capability, which can jointly minimize the intra-class distance and maximize the inter-class distance in each triplet unit. Extensive experiments are carried out to demonstrate that the proposed method outperforms most current methods on three main RE-ID benchmarks.

Age estimation has been paid great attention in the field of intelligent surveillance, face recognition, biometrics, etc. In contrast to other facial variations, aging variation presents several unique characteristics, which make age estimation very challenging. The overall process of age estimation is performed using three important steps. In the first step, the pre-processing is performed from the input image based on Viola-Jones algorithm to detect the face region. In the second step, feature extraction is done based on three important features such as local transform directional pattern (LTDP), active appearance model (AAM), and the new feature, deep appearance model (Deep AM). After feature extraction, the classification is carried out based on the extracted features using deep belief network (DBN), where the DBN classifier is trained optimally using the proposed learning algorithm named as crow-sine cosine algorithm (CS).

In this paper, we propose a generic boosting framework for multiple object tracking (MOT). Unlike other works tracking objects from zero, our framework uses their results (tracklets) and makes further optimizations. The motivation of us derives from the observation that most modern MOT trackers have been acceptable performance and can yield relatively reliable tracklets; accordingly, we straight focus on the tracklet-level re-identification, which is the most challenging issue in this case. To achieve that goal, we simultaneously utilize the techniques of single object tracking, tracking fragment (tracklets) and re-identification mechanism through casting them into a multi-label energy optimization and then innovatively solving it using the $$\alpha -$$ expansion with label costs algorithm. All these techniques inspire recent MOT a lot to mitigate the occlusion problem, but to our knowledge, by far few works explore to reasonably combine them all like us. Furthermore, we introduce a spatial attention to improve the appearance model and a hierarchical clustering as post-process to progressively improve the tracking consistency. Finally, testing results on the most used benchmarks demonstrate the significant effectiveness and generality of our framework, and the importance of each contribution is also verified through ablative studies.

More and more researchers have recently paid attention to video object segmentation because it is an important building block for numerous computer vision applications. Although many algorithms promote its development, there are still some open challenges. Efficient and robust pipelines are needed to address appearance changes and the distraction from similar background objects in the video object segmentation. This paper proposes a novel neural network that integrates a temporal attention based appearance model and a boundary-aware loss. The appearance model fuses the appearance information of the first frame, the previous frame, and the current frame in the feature space, which assists the proposed method to learn a discriminative and robust target representation and avoid the drift problem of traditional propagation schemes. Moreover, the boundary-aware loss is employed for network training. Equipped with the boundary-aware loss, the proposed method achieves more accurate segmentation results with clear boundaries. The proposed method is compared with several recent state-of-the-art algorithms on popular benchmark datasets. Comprehensive experiments show that the proposed method achieves favorable performance with a high frame rate.

In online multi-target tracking, modeling of appearance and geometric similarities between pedestrians visual scenes is of great importance. The higher dimension of inherent information in the appearance model compared to the geometric model is problematic in many ways. However, due to the recent success of deep-learning-based methods, handling of high-dimensional appearance information becomes feasible. Among many deep neural networks, Siamese network with triplet loss has been widely adopted as an effective appearance feature extractor. Since the Siamese network can extract the features of each input independently, one can update and maintain target-specific features. However, it is not suitable for multi-target settings that require comparison with other inputs. To address this issue, we propose a novel track appearance model based on the joint-inference network. The proposed method enables a comparison of two inputs to be used for adaptive appearance modeling and contributes to the disambiguation of target-observation matching and to the consolidation of identity consistency. Diverse experimental results support the effectiveness of our method. Our work was recognized as the 3rd-best tracker in BMTT MOTChallenge 2019, held at CVPR2019. The code is available at https://github.com/yyc9268/Deep-TAMA., Accepted in Information Sciences, Elsevier. 3rd Prize on 4th BMTT MOTChallenge Workshop held in CVPR2019

Distance and Shape signature features in human faces offer formidable significance in recognizing facial expressions. Identifying appropriate landmarks is a crucial as well as challenging issue in the field of expression recognition of human faces. Appearance model has been found useful to detect the salient landmarks on human faces. These salient landmarks induce a grid on the human face along with the formation of possible triangles joining the grid. Normalized distance and shape signatures are determined from the grid. Distance signature as well as shape signature find respective stability indices which play important role to recognize the facial expressions. Statistical measures such as range, moment, skewness, kurtosis and entropy are calculated from normalized distance and shape signature pair to supplement the feature set. This enhanced feature set is fed into a Multilayer Perceptron (MLP) to arrive at different expression categories encompassing anger, sadness, fear, disgust, surprise and happy. We investigated our proposed system on Cohn-Kanade (CK+), JAFFE, MMI and MUG databases to conduct and validate our experiment and establish its performance superiority over other existing competitors.

Automated human facial image deidentification is a much-needed technology for privacy-preserving social media and intelligent surveillance applications. We propose a novel utility preserved facial image de-identification to subtly tinker the appearance of facial images to achieve facial anonymity by creating “averaged identity faces”. This approach is able to preserve the utility of the facial images while achieving the goal of privacy protection. We explore a decomposition of an Active appearance model (AAM) face space by using subspace learning where the loss can be modeled as the difference between two trace ratio items, and each respectively models the level of discriminativeness on identity and utility. Finally, the face space is decomposed into subspaces that are respectively sensitive to face identity and face utility. For the subspace most relevant to face identity, a k-anonymity de-identification procedure is applied. To verify the performance of the proposed facial image de-identification approach, we evaluate the created “ averaged faces” using the extended Cohn-Kanade Dataset (CK+). The experimental results show that our proposed approach is satisfied to preserve the utility of the original image while defying face identity recognition.

Currently, polarization visualization strategies are accomplished by mapping polarization information into a perceptually uniform color appearance model CAM02-UCS. However, the deviation of the CAM02-UCS space from the lightness prediction results in an inaccurate match between the polarization information and the perceptual information. In this paper, we propose a novel polarization visualization strategy based on the perceptual uniform space J z a z b z . The polarization visualization be completed by placing the polarization information into the lightness J z , colorfulness C z and hue angle h z channels of the J z a z b z space. The experimental results show that the proposed method can significantly improve the lightness of the low irradiance and high polarization region, hence more polarization information can be sensed by human visual system.

On-line welding penetration state monitoring based on passive visual sensing technology in pulsed gas tungsten arc welding (GTAW) has been a hot research topic in industry and academia. Passive vision sensing technology has some disadvantages. It is easy to be affected by some common disturbance during pulsed GTAW, like spattering, the change of arc intensity, the slightly change of focal length and viewing angle of visual sensor, etc. This paper proposed an intelligent penetration monitoring methodology for GTAW by combining pattern-based visual feature extraction method and supervised machine learning method. The proposed methodology mainly includes two parts: (a) visual features extraction for weld pool images based on active appearance model (AAM); (b) the implementation of penetration monitoring based on supervised machine learning method, random forests (RF). The extracted weld pool visual features include width, length and area size of weld pool. Then the extracted features are utilized to train RF model. Further, the trained RF model can implement the evaluation of penetration states and backside weld seam width. The experiments verify that the proposed method is effective with high accuracy and robustness, which also lays the foundation for online welding penetration control.

A two-level hierarchical scheme for videobased person re-identification (re-id) is presented, with the aim of learning a pedestrian appearance model through more complete walking cycle extraction. Specifically, given a video with consecutive frames, the objective of the first level is to detect the key frame with lightweight Convolutional neural network (CNN) of PCANet to reflect the summary of the video content. At the second level, on the basis of the detected key frame, the pedestrian walking cycle is extracted from the long video sequence. Moreover, local features of Local maximal occurrence (LOMO) of the walking cycle are extracted to represent the pedestrian' s appearance information. In contrast to the existing walking-cycle-based person re-id approaches, the proposed scheme relaxes the limit on step number for a walking cycle, thus making it flexible and less affected by noisy frames. Experiments are conducted on two benchmark datasets: PRID 2011 and iLIDS-VID. The experimental results demonstrate that our proposed scheme outperforms the six state-of-art video-based re-id methods, and is more robust to the severe video noises and variations in pose, lighting, and camera viewpoint.

Discriminative Correlation Filters (DCF) have been shown to achieve impressive performance in visual object tracking. However, existing DCF-based trackers rely heavily on learning regularised appearance models from invariant image feature representations. To further improve the performance of DCF in accuracy and provide a parsimonious model from the attribute perspective, we propose to gauge the relevance of multi-channel features for the purpose of channel selection. This is achieved by assessing the information conveyed by the features of each channel as a group, using an adaptive group elastic net inducing independent sparsity and temporal smoothness on the DCF solution. The robustness and stability of the learned appearance model are significantly enhanced by the proposed method as the process of channel selection performs implicit spatial regularisation. We use the augmented Lagrangian method to optimise the discriminative filters efficiently. The experimental results obtained on a number of well-known benchmarking datasets demonstrate the effectiveness and stability of the proposed method. A superior performance over the state-of-the-art trackers is achieved using less than $$10\%$$ 10 % deep feature channels.

Currently, visual object tracking is a core research area as it can be applied in many applications of computer vision. However, tracking of a visual object is a difficult task as it can go through different varying conditions like occlusion of the target object, appearance variation, illumination variation, etc. during the tracking process. An efficient and robust visual object tracking based on sparse discriminative classier (SDC) and principal component analysis (PCA) subspace representation is presented in this work. The PCA subspace representation modelled the appearance model of the target object and SDC separates the target object and background object very efficiently. The computational complexity is much better than the other existing methods in the literature. Both quantitative and qualitative analyses of different video sequences are done to compare the proposed tracking algorithm with the other existing tracking algorithms. The experimental results show that the proposed method outperforms the other existing tracking algorithms.

Existing discriminative correlation filters suffer from the defects of potential spatial distractors and the degradation of appearance model caused by hard-temporal correlation. Aiming at this issue, a robust tracker which combines the adaptive spatial regularization and the temporal consistent constraint is proposed in this paper. First, we propose to take the extracted saliency map of the background as a reference weight to construct the spatial regularization term, with which the perceived performance of the filter against distractors is enhanced by learning the spatial sparse constraint adaptively. Second, we further implement the temporal consistent regularization formed by capturing dynamic appearance information from multiple historical frames with a high-confidence strategy to mitigate the model degradation. Third, we employ the alternating direction method of multipliers to solve the constrained optimization problem efficiently, thereby the computational complexity can be reduced. The concrete experimental results on OTB-2013, OTB-2015, Temple-Color-128 and VOT2016 benchmarks demonstrate that our tracker outperforms several state-of-the-art algorithms.

In recent years, high spatial resolution remote sensing technology has made significant progress. High-resolution remote sensing satellites provide great convenience for high-quality image acquisition. In order to adapt to changes in the appearance of the target, mainstream tracking algorithms often use pattern recognition methods to build a target appearance model with learning capabilities, and use the image frames acquired during the tracking process to update the appearance model. This paper mainly studies the object-oriented remote sensing image information extraction method based on multi-classifier combination and deep learning algorithm. In this paper, we use the splitting mechanism of the tree structure to retain the appearance model with diversity, and through the integrated learning integration strategy, the target position is collaboratively predicted. Through the comparative analysis on the OTB and VOT platforms, the algorithm works well when the requirements of the tracking standards are low (the accuracy threshold is greater than 20 pixels and the success threshold is less than 0.4 pixels). The experimental results in this paper show that compared with other advanced classification methods, the proposed method shows better generalization performance in accuracy, recall, f-measure, g-mean and AUC.

In this study, we propose a novel RGB-T tracking framework by jointly modeling both appearance and motion cues. First, to obtain a robust appearance model, we develop a novel late fusion method to infer the fusion weight maps of both RGB and thermal (T) modalities. The fusion weights are determined by using offline-trained global and local multimodal fusion networks, and then adopted to linearly combine the response maps of RGB and T modalities. Second, when the appearance cue is unreliable, we comprehensively take motion cues, i.e., target and camera motions, into account to make the tracker robust. We further propose a tracker switcher to switch the appearance and motion trackers flexibly. Numerous results on three recent RGB-T tracking datasets show that the proposed tracker performs significantly better than other state-of-the-art algorithms.

Pose estimation that locates objects in a bin is necessary for a robotic bin picking system. Although many algorithms have shown high performance in pose estimation, most algorithms estimate the poses of objects regardless of their occlusion. This can reduce the success rate in picking up the object. To resolve this issue, we propose a novel pipeline that estimates a pose only for occlusion-free objects based on point pair feature-based pose estimation with multiple edge appearance model (PPF-MEAM). The proposed method detects occlusion-free objects in the 2D image captured by a camera with a convolutional neural network framework. Next, corresponding point clouds of occlusion-free objects need to be extracted by using their locations in the 2D image. we propose a robust extraction method that finds the 3D points corresponding to image pixels in the 2D image to reduce the effect of the calibration errors between the camera and 3D sensor. The point cloud of the occlusion-free objects is finally input into a pipeline of PPF-MEAM to estimate the pose of the object. The experiment results prove that the proposed method is about 50% faster 30% higher in terms of pose estimation success rate compared with the original PPF. Moreover, it increases the success rate of picking tasks compared with the original PPF-MEAM.

There has been increasing interest in face recognition in the thermal infrared spectrum. A critical step in this process is face landmark detection. However, landmark detection in the thermal spectrum presents a unique set of challenges compared to in the visible spectrum: inherently lower spatial resolution due to longer wavelength, differences in phenomenology, and limited availability of labeled thermal face imagery for algorithm development and training. Thermal infrared imaging does have the advantage of being able to passively acquire facial heat signatures without the need for active or ambient illumination in low light and nighttime environments. In such scenarios, thermal imaging must operate by itself without corresponding/paired visible imagery. Mindful of this constraint, we propose visible-to-thermal parameter transfer learning using a coupled convolutional network architecture as a means to leverage visible face data when training a model for thermal-only face landmark detection. This differentiates our approach from models trained either solely on thermal images or models which require a fusion of visible and thermal images at test time. In this work, we implement and analyze four types of parameter transfer learning methods in the context of thermal face landmark detection: Siamese (shared) layers, Linear Layer Regularization (LLR), Linear Kernel Regularization (LKR), and Residual Parameter Transformations (RPT). These transfer learning approaches are compared against a baseline version of the network and an Active Appearance Model (AAM), both of which are trained only on thermal data. We achieve a 6.5% - 9.5% improvement on the DEVCOM ARL Multi-modal Thermal Face Dataset and a 4% improvement on the RWTH Aachen University Thermal Face Dataset over the baseline model. We show that LLR, LKR, and RPT all result in improved thermal face landmark detection performance compared to the baseline and AAM, demonstrating that transfer learning leveraging visible spectrum data improves thermal face landmarking.

With the wide vision and high flexibility, unmanned aerial vehicle (UAV) has been widely used into object tracking in recent years. However, its limited computing capability poses a great challenges to tracking algorithms. On the other hand, Discriminative Correlation Filter (DCF) based trackers have attracted great attention due to their computational efficiency and superior accuracy. Many studies introduce spatial and temporal regularization into the DCF framework to achieve a more robust appearance model and further enhance the tracking performance. However, such algorithms generally set fixed spatial or temporal regularization parameters, which lack flexibility and adaptability under cluttered and challenging scenarios. To tackle such issue, in this letter, we propose a novel DCF tracking model by introducing dynamic spatial regularization weight, which encourage the filter focuses on more reliable region during training stage. Furthermore, our method could optimize the spatial and temporal regularization weight simultaneously using Alternative Direction Method of Multiplies (ADMM) technique method, where each sub-problem has closed-form solution. Through the joint optimization, our tracker could not only suppress the potential distractors but also construct robust target appearance on the basis of reliable historical information. Experiments on two UAV benchmarks have demonstrated that our tracker performs favorably against other state-of-the-art algorithms.

Pain is an alert state of the human body that can be conveyed to the external world through different modalities. A possible communication channel for human pain is represented by facial expressions, whose role in social interactions has been well established. In this work, the link between pain and transfer entropy (TE), passing through facial expressions, is investigated. A new approach to the vision-based measurement (VBM) of pain is presented, which is based on TE among the time-series of facial landmarks positions. The system is composed of three main blocks: A VBM block for the automatic landmarking and the generation of the time-series from the video-sequences; a second block for the evaluation of TE; and finally a classification model based on machine learning algorithms for pain assessment. A public database of video sequences of patients experiencing pain in a controlled scenario was used for the characterization of the system in terms of accuracy and precision. Different uncertainty contributions related to realistic signals interruptions and fluctuations were modeled and propagated to provide a comprehensive evaluation of the proposed measurement system. The obtained results indicate that TE-based approaches can provide great benefits in automatic pain assessment, opening new perspectives for remote management of patients.

This article develops a practical framework of Area Angle Monitoring (AAM) to monitor in real time the stress of bulk power transfer across an area of a power transmission system. Area angle is calculated from synchrophasor measurements in real time to provide alert to system operators if the area angle exceeds pre-defined thresholds. This article proposes a general method to identify the warning threshold of area angle and a simplified method to quickly update area angle thresholds under significant topology change. A mitigation strategy to relieve the area stress is also proposed. In order to handle the limited coverage of synchrophasor measurements, this article proposes a method to estimate phase angles for boundary buses without synchrophasor measurements, which extends the application scenario of AAM. AAM is verified for a power transmission area in the Western Electricity Coordinating Council system with both simulated data and synchrophasor measurements recorded from real events. A utility deployment for real-time application of AAM with livestream and recorded synchrophasor data is described.

To address the problem of weakly supervised instance segmentation for electrical equipment using only a red, green, and blue (RGB) camera, an automatic annotation of masks of samples (AAMS) method based on thermal image guidance is proposed in this article. With only image-level label supervision, we exploit foreground segmentation results of thermal images to guide the instance mask extraction of electrical equipment in RGB images through the heterogeneous pixel registration algorithm between RGB-thermal (RGB-T) image pairs. It is realized to automatically annotate instance masks, which greatly improves efficiency and decreases costs. In addition, we further propose a progressively optimized model (POM) for instance segmentation, which first utilizes the fully connected conditional random field (CRF) and the constrain-to-boundary loss to specify fine-detailed boundaries of each object and to solve the difficulty of segmenting electrical equipment with complicated structures. This model also explores the self-paced learning technology to solve the issue of resolution differences between RGB-T image pairs for improving the generalization ability. By comparison to the other state-of-the-arts, experimental results show that our method can obtain by far the better performance on the electrical equipment data set.

Visual object tracking has become one of the most active research topics in computer vision, and it has been applied in several commercial applications. Several visual trackers have been presented in the last two decades. They target different tracking objectives. Object tracking from a real-time video is a challenging problem. Therefore, a robust tracker is required to consider many aspects of videos such as camera motion, occlusion, illumination effect, clutter, and similar appearance. In this paper, we propose an efficient object tracking algorithm that adaptively represents the object appearance using CNN-based features. A sparse measurement matrix is proposed to extract the compressed features for the appearance model without sacrificing the performance. We compress sample images of the foreground object and the background by the sparse matrix. When re-detection is needed, the tracking algorithm conducts an SVM classifier on the extracted features with online update in the compressed domain. A search strategy is proposed to reduce the computational burden in the detection step. Extensive simulations with a challenging video dataset demonstrate that the proposed tracking algorithm provides real-time tracking, while delivering substantially better tracking performance than those of the state-of-the-art techniques in terms of robustness, accuracy, and efficiency.

Automatic age estimation from the face images is a growing research interest nowadays. Various literature works have contributed towards the age detection scheme, besides only a few have resulted in providing good performance. This is due to the influence of the external factors, such as environment, lifestyle, and various expressions present in the face image. This paper proposes a deep belief network with the crow optimization algorithm for the age detection purpose. The proposed Crow Deep Belief Network (CDBN) finds the age of the person in the image through the initial training with the face features. The features for the training of the proposed CDBN are provided by the scattering transform and the Active Appearance Model (AAM). The training of the CDBN with the features provides the optimal weights used for the age detection. The experimentation of the proposed CDBN is done by four standard databases, namely the IMDB database, the Adience database, the AFAD database, and the FG-NET database based on the metrics, such as Mean Absolute Error (MAE), Accuracy of error of one age category (AEO) and Accuracy of an Exact Match (AEM). Among them, the proposed model has the minimum MAE with a value of 2.186 for FG-NET database, and maximum AEO and AEM with the values of 0.972, and 0.971, respectively for IMDB database.

Sprifermin is under investigation as a potential disease-modifying osteoarthritis drug. Previously, 2-year results from the FORWARD study showed significant dose-dependent modification of cartilage thickness in the total femorotibial joint (TFTJ), medial and lateral femorotibial compartments (MFTC, LFTC), and central medial and lateral TFTJ subregions, by quantitative magnetic resonance imaging (qMRI) using manual segmentation.To determine whether qMRI findings from FORWARD could be reproduced by an independent method of automated segmentation using an identical dataset and similar anatomical regions in a post-hoc analysis.Cartilage thickness was assessed at baseline and 6, 12, 18 and 24 months, using automated cartilage segmentation with active appearance models, a supervised machine learning method. Images were blinded for treatment and timepoint. Treatment effect was assessed by observed and adjusted changes using a linear mixed model for repeated measures.Based on automated segmentation, statistically significant, dose-dependent structural modification of cartilage thickness was observed over 2 years with sprifermin vs placebo for TFTJ (overall treatment effect and dose response, both P 0.001), MFTC (P = 0.004 and P = 0.044), and LFTC (both P 0.001) regions. For highest dose, in the central medial tibial (P = 0.008), central lateral tibial (P 0.001) and central lateral femoral (P 0.001) regions.Cartilage thickness assessed by automated segmentation provided a consistent dose response in structural modification compared with manual segmentation. This is the first time that two independent quantification methods of image analysis have reached the same conclusions in an interventional trial, strengthening the conclusions that sprifermin modifies structural progression in knee osteoarthritis.

Background Recently, the field of face and facial features has been progressively studied. The features of facial expression have gained increasing attention for related applications. The wrinkle is the most representative feature, and its research and applications have been topics of high interest. Wrinkles play an important role in face feature analysis. They have been widely used in applications, such as age estimation, skin texture classification, expression recognition, and simulation. Purpose Existing approaches to the image‐based analysis of wrinkles as texture not as curvilinear discontinuity and wrinkle detection mainly have focused on detecting wrinkles on forehead position, which is usually horizontal linear shapes, while the detection of the nasolabial wrinkle is not well understood due to their variety of shapes and complexity. Method In this paper, we present a nasolabial wrinkle line detecting effective algorithm based on the Active appearance model and Hessian filter to improve localization results by creating unique initial shapes of the wrinkle lines for each input face image. Results Experimental results show that the proposed method is capable of tracking curve wrinkle lines, thus allowing to detect complexly structured wrinkle lines. This work demonstrates results illustrated the competitiveness of the proposed method in detecting nasolabial wrinkle lines. Conclusion In our study, this was introduced the effectiveness of changing the structure of AAM and successfully applied in wrinkle line localizing, although competitive results are achieved by the proposed wrinkle detection method.

This paper presents the augmented particle samples based optimal convolutional filters that preserve the appearance model robustness for object tracking in both temporal and spatial levels. In temporal level, augmented particle samples provided by Laplacian group reverse sparse representation exploit the potential geometrical correlation among the different patches that keep the inherent potential distribution which facilitates the update scheme of appearance model between continuous frames in the particle filtering framework. In spatial level, structural information of multi-scale patches extraction can preserve highly stable attributes that significantly improve the object representation robustness in multi-scenarios. Moreover, the optimal convolutional filters that resulted from laplacian score exploits the coherence of high similarity in both positive and negative sets effectively that can guarantee the template update procedures discriminatively. Experimental results demonstrate that the proposed approach achieves better performance on multiple dynamic scenes.

Significant progress is made in the field of object tracking recently. Especially, trackers based on deep learning and correlation filters both have achieved excellent performance. However, object tracking still faces some challenging problems such as deformation and illumination. In such kinds of situations, the accuracy and precision of tracking algorithms plunge as a result. It is imminent to find a solution to this situation. In this paper, we propose a tracking algorithm based on features extracted by residual network called Resnet features and cascaded correlation filters to improve precision and accuracy. Firstly, features extracted by a deep residual network trained on other image processing datasets, are robust enough and retain higher resolution, therefore, we exploit Resnet-101 pretrained offline to obtain features extracted by middle and high layers for target appearance model representation. Resnet-101 is deeper compared with other deep neural networks which means it contains more semantic information. Then, the method we propose to combine our correlation filters is superior. We propose cascaded correlation filters generated by handcraft, middle-level and high-level features from residual network to gain better competence. Handcraft features localize target precisely because they contain more spatial details while Resnet features are robust to the target appearance change because they retain more semantic information. Finally, we conduct extensive experiments on OTB2013 and OTB2015 benchmark. The experimental results show that our tracker achieves high performance under all kinds of challenges and performs favorably against other state-of-the-art trackers.

It is essential to build the effective appearance model for object tracking in computer vision. Most object trackers can be roughly divided into two categories according to the appearance model: the bounding box model and the patch model. The bounding box model cannot handle shape deformation and occlusion of the non-rigid moving object effectively. The patch model is prone to be disturbed by complex backgrounds. In this paper, we propose a robust multi-structured-patch appearance model to represent the target for object tracking. The proposed appearance model is aimed to exploit and identify reliable patches that can be tracked effectively through the whole tracking process. According to attention mechanism in biological vision system, a coarse-to-fine strategy is usually used to search the target. Therefore, the proposed appearance model is represented by robust patches in different sizes, in which the bigger patches search the rough region of the target and the smaller patches estimate the accurate location. Experimental results on OTB100 dataset show that the proposed method outperforms state-of-the-art trackers.

In human tracking, sparse representation successfully localises the human in a video with minimal reconstruction error using target templates. However, the state‐of‐the‐art approaches use colour and local appearance of a human to discriminate the human from the background regions, and hence fail when the human is occluded and appears in the varying illumination environment. In this study, a robust tracking algorithm is proposed that utilises gradient orientation and fine and coarse sparse representation of the target template. Sparse representation‐based human appearance model utilises weighted gradient orientation that is insensitive to illumination variation. Coarse and fine representation of sparse code facilitates tracking under varying scales. Subspace learning from image gradient orientation is enforced with occlusion detection during the dictionary updation stage to capture the visual characteristics of the local human appearance that supports tracking under partial occlusion with lesser tracking error. The proposed human tracking algorithm is evaluated on various datasets and shows efficient human tracking performance when compared to the other state‐of‐the‐art approaches. Furthermore, the proposed human tracking algorithm is suitable for surveillance applications.

Directly benefiting from the rapid advancement of deep learning methods, person re-identification (Re-ID) applications have been widespread with remarkable successes in recent years. Nevertheless, cross-scene Re-ID is still hindered by large view variation, since it is challenging to effectively exploit and leverage the temporal clues due to heavy computational burden and the difficulty in flexibly incorporating discriminative features. To alleviate, we articulate a long-short temporal–spatial clues excited network (LSTS-NET) for robust person Re-ID across different scenes. In essence, our LSTS-NET comprises a motion appearance model and a motion-refinement aggregating scheme. Of which, the former abstracts temporal clues based on multi-range low-rank analysis both in consecutive frames and in cross-camera videos, which can augment the person-related features with details while suppressing the clutter background across different scenes. In addition, to aggregate the temporal clues with spatial features, the latter is proposed to automatically activate the person-specific features by incorporating personalized motion-refinement layers and several motion-excitation CNN blocks into deep networks, which expedites the extraction and learning of discriminative features from different temporal clues. As a result, our LSTS-NET can robustly distinguish persons across different scenes. To verify the improvement of our LSTS-NET, we conduct extensive experiments and make comprehensive evaluations on 8 widely-recognized public benchmarks. All the experiments confirm that, our LSTS-NET can significantly boost the Re-ID performance of existing deep learning methods, and outperforms the state-of-the-art methods in terms of robustness and accuracy.

In this paper, we propose a novel visual object tracking method using a part based appearance model. First, a local kernel feature is developed to encode edge information of patches. Next, bounding box of the target is divided into multiple parts. Then, each part uses correlation filter based tracking to predict position in the next frame. The matrix cosine similarity is utilized to measure reliabilities of the patches. Finally, optimal target location is predicted via maximizing the likelihood, which is obtained by adaptively fusing the reliable patches locations. Experimental results illustrate that our algorithm outperforms state-of-the-art tracking methods significantly in terms of accuracy and robustness.

In this paper, we present a method for detecting and tracking waterborne obstacles from an unmanned surface vehicle (USV) for the purpose of short-term obstacle avoidance. A stereo camera system provides a point cloud of the scene in front of the vehicle. The water surface is estimated by fitting a plane to the point cloud and outlying points are further processed to find potential obstacles. We propose a new plane fitting algorithm for water surface detection that applies a fast approximate semantic segmentation to filter the point cloud and utilizes an external IMU reading to constrain the plane orientation. A novel histogram-like depth appearance model is proposed to keep track of the identity of the detected obstacles through time and to filter out false detections that negatively impact the vehicle's automatic guidance system. The improved plane fitting algorithm and the temporal verification using depth fingerprints result in notable improvement on the challenging MODD2 data set by significantly reducing the amount of false positive detections. The proposed method is able to run in real time on board of a small-sized USV, which was used to acquire the MODD2 data set as well.

Visual tracking is a complex problem due to unconstrained appearance variations and a dynamic environment. The extraction of complementary information from the object environment via multiple features and adaption to the target's appearance variations are the key problems of this paper. To this end, we propose a robust object tracking framework based on the unified graph fusion (UGF) of multicue to adapt to the object's appearance. The proposed cross-diffusion of sparse and dense features not only suppresses the individual feature deficiencies but also extracts the complementary information from multicue. This iterative process builds robust unified features which are invariant to object deformations, fast motion, and occlusion. Robustness of the unified feature also enables the random forest classifier to precisely distinguish the foreground from the background, adding resilience to background clutter. In addition, we present a novel kernel-based adaptation strategy using outlier detection and a transductive reliability metric. The adaptation strategy updates the appearance model to accommodate variations in scale, illumination, and rotation. Both qualitative and quantitative analyses on benchmark video sequences from OTB-50, OTB-100, VOT2017/18, and UAV123 show that the proposed UGF tracker performs favorably against 18 other state-of-the-art trackers under various object tracking challenges.

Person re-identification plays important roles in many practical applications. Due to various human poses, complex backgrounds and similarity of person clothes, person re-identification is still a challenging task. In this paper, we mainly focus on the robust and discriminative appearance feature representation and proposed a novel multi-appearance method for person re-identification. First, we proposed a deep feature fusion method and get the multi-appearance feature by combining two Convolutional Neural Networks. Then, in order to further enhance the representation of the appearance feature, the multi-part model was constructed by combining the whole body and the six body parts. Additionally, we optimized the feature extraction process by adding a pooling layer. Comprehensive and comparative experiments with the state-of-the-art methods over publicly available datasets demonstrated that the proposed method can get promising results.

Object tracking is a significant problem of computer vision due to challenging environmental variations. Single cue appearance model is not sufficient to handle the variations. To this end, we propose a multi-cue tracking framework in which complementary cues namely, LBP and HOG were exploited to develop a robust appearance model. The proposed feature fusion captures the high-level relationship between the features and diminishes the low-level relationship. Transductive reliability is also integrated at each frame to make tracker adaptive with the changing environment. In addition, K-Means based classifier creates clear and concise boundary between positive and negative fragments which are further used to update the reference dictionary. This adaptation strategy prevents the erroneous updation of the proposed tracker during background clutters, occlusion, and fast motion. Qualitative and quantitative analysis on challenging video sequences from OTB-100 dataset, VOT dataset and UAV123 reveal that the proposed tracker performs favorably against 13 others state-of-the-art trackers.

Visual tracking using particle filter has been extensively investigated due to its myriad of application in the field of computer vision. However, particle filter framework performance is heavily impaired due to its inherent problems namely, particle degeneracy and impoverishment. In addition, most of the tracking methods using single cue are greatly affected by dynamic environmental challenges. To address these issues, we propose an adaptive multi-cue particle filter based real-time visual tracking framework. Three complementary cues namely, color histogram, LBP and pyramid of histogram of gradient have been exploited for object’s appearance model. These cues are integrated using the proposed adaptive fusion model for the automatic boosting of important particles and suppression of unimportant particles. Resampling method using butterfly search optimization relocate low performing particles to high likelihood area. Proposed outlier detection mechanism not only helps in detecting low performing particles but also aids in updating of the reference dictionary. Online estimation of cue reliability along with its multi-cue fusion leads to quick adaptation of the proposed tracker. On average of the outcome, our tracker achieves average center location error of 6.89 (in pixels) and average F-measure of 0.786 when evaluated on OTB-100 and VOT dataset against 13 others state-of-the-art.

Model drifting is one of the knotty problems that seriously restricts the accuracy of discriminative trackers in visual tracking. Most existing works usually focus on improving the robustness of the target appearance model. However, they are prone to suffer from model drifting due to the inappropriate model updates during the tracking-by-detection. In this paper, we propose a novel update-pacing framework to suppress the occurrence of model drifting in visual tracking. Specifically, the proposed framework first initializes an ensemble of trackers, each of which updates the model in a different update interval. Once the forward tracking trajectory of each tracker is determined, the backward trajectory will also be generated by the current model to measure the difference with the forward one, and the tracker with the smallest deviation score will be selected as the most robust tracker for the remaining tracking. By performing such self-examination on trajectory pairs, the framework can effectively preserve the temporal context consistency of sequential frames to avoid learning corrupted information. To further improve the performance of the proposed method, a multi-feature extension framework is also proposed to incorporate multiple features into the ensemble of the trackers. The extensive experimental results obtained on large-scale object tracking benchmarks demonstrate that the proposed framework significantly increases the accuracy and robustness of the underlying base trackers, such as DSST, Struck, KCF, and CT, and achieves superior performance compared with the state-of-the-art methods without using deep models.