Search

With the increasing popularity of Unmanned Aerial Vehicles (UAVs) in computer vision-related applications, intelligent UAV video analysis has recently attracted the attention of an increasing number of researchers. To facilitate research in the UAV field, this paper presents a UAV dataset with 100 videos featuring approximately 2700 vehicles recorded under unconstrained conditions and 840k manually annotated bounding boxes. These UAV videos were recorded in complex real-world scenarios and pose significant new challenges, such as complex scenes, high density, small objects, and large camera motion, to the existing object detection and tracking methods. These challenges have encouraged us to define a benchmark for three fundamental computer vision tasks, namely, object detection, single object tracking (SOT) and multiple object tracking (MOT), on our UAV dataset. Specifically, our UAV benchmark facilitates evaluation and detailed analysis of state-of-the-art detection and tracking methods on the proposed UAV dataset. Furthermore, we propose a novel approach based on the so-called Context-aware Multi-task Siamese Network (CMSN) model that explores new cues in UAV videos by judging the consistency degree between objects and contexts and that can be used for SOT and MOT. The experimental results demonstrate that our model could make tracking results more robust in both SOT and MOT, showing that the current tracking and detection methods have limitations in dealing with the proposed UAV benchmark and that further research is indeed needed. [ABSTRACT FROM AUTHOR]

Fluidization of non-spherical particles is very common in petroleum engineering. Understanding the complex phenomenon of non-spherical particle flow is of great significance. In this paper, coupled with two-fluid model, the drag coefficient correlation based on artificial neural network was applied in the simulations of a bubbling fluidized bed filled with non-spherical particles. The simulation results were compared with the experimental data from the literature. Good agreement between the experimental data and the simulation results reveals that the modified drag model can accurately capture the interaction between the gas phase and solid phase. Then, several cases of different particles, including tetrahedron, cube, and sphere, together with the nylon beads used in the model validation, were employed in the simulations to study the effect of particle shape on the flow behaviors in the bubbling fluidized bed. Particle shape affects the hydrodynamics of non-spherical particles mainly on microscale. This work can be a basis and reference for the utilization of artificial neural network in the investigation of drag coefficient correlation in the dense gas–solid two-phase flow. Moreover, the proposed drag coefficient correlation provides one more option when investigating the hydrodynamics of non-spherical particles in the gas–solid fluidized bed. [ABSTRACT FROM AUTHOR]

As a post-processing procedure, the diffusion process has demonstrated its ability of substantially improving the performance of various visual retrieval systems. Whereas, great efforts are also devoted to similarity (or metric) fusion, seeing that only one individual type of similarity cannot fully reveal the intrinsic relationship between objects. This stimulates a great research interest of considering similarity fusion in the framework of the diffusion process (i.e., fusion with diffusion) for robust retrieval. In this paper, we first revisit representative methods about fusion with diffusion and provide new insights which are ignored by previous researchers. Then, observing that existing algorithms are susceptible to noisy similarities, the proposed regularized ensemble diffusion (RED) is bundled with an automatic weight learning paradigm, so that the negative impacts of noisy similarities are suppressed. Though formulated as a convex optimization problem, one advantage of RED is that it converts back into the iteration-based solver with the same computational complexity as the conventional diffusion process. At last, we integrate several recently-proposed similarities with the proposed framework. The experimental results suggest that we can achieve new state-of-the-art performances on various retrieval tasks, including 3D shape retrieval on the ModelNet data set, and image retrieval on the Holidays and Ukbench data sets. [ABSTRACT FROM AUTHOR]

Emotion recognition is challenging due to the emotional gap between emotions and audio–visual features. Motivated by the powerful feature learning ability of deep neural networks, this paper proposes to bridge the emotional gap by using a hybrid deep model, which first produces audio–visual segment features with Convolutional Neural Networks (CNNs) and 3D-CNN, then fuses audio–visual segment features in a Deep Belief Networks (DBNs). The proposed method is trained in two stages. First, CNN and 3D-CNN models pre-trained on corresponding large-scale image and video classification tasks are fine-tuned on emotion recognition tasks to learn audio and visual segment features, respectively. Second, the outputs of CNN and 3D-CNN models are combined into a fusion network built with a DBN model. The fusion network is trained to jointly learn a discriminative audio–visual segment feature representation. After average-pooling segment features learned by DBN to form a fixed-length global video feature, a linear Support Vector Machine is used for video emotion classification. Experimental results on three public audio–visual emotional databases, including the acted RML database, the acted eNTERFACE05 database, and the spontaneous BAUM-1s database, demonstrate the promising performance of the proposed method. To the best of our knowledge, this is an early work fusing audio and visual cues with CNN, 3D-CNN, and DBN for audio–visual emotion recognition. [ABSTRACT FROM AUTHOR]

An intrinsic challenge of person re-identification (re-ID) is the annotation difficulty. This typically means (1) few training samples per identity and (2) thus the lack of diversity among the training samples. Consequently, we face high risk of over-fitting when training the convolutional neural network (CNN), a state-of-the-art method in person re-ID. To reduce the risk of over-fitting, this paper proposes a Pseudo-Positive Regularization method to enrich the diversity of the training data. Specifically, unlabeled data from an independent pedestrian database are retrieved using the target training data as query. A small proportion of these retrieved samples are randomly selected as the Pseudo-Positive samples and added to the target training set for the supervised CNN training. The addition of Pseudo-Positive samples is therefore a Data Augmentation method to reduce the risk of over-fitting during CNN training. We implement our idea in the identification CNN models (i.e., CaffeNet, VGGNet-16 and ResNet-50). On CUHK03 and Market-1501 datasets, experimental results demonstrate that the proposed method consistently improves the baseline and yields competitive performance to the state-of-the-art person re-ID methods. [ABSTRACT FROM AUTHOR]

Extreme learning machine algorithm proposed in recent years has been widely used in many fields due to its fast training speed and good generalization performance. Unlike the traditional neural network, the ELM algorithm greatly improves the training speed by randomly generating the relevant parameters of the input layer and the hidden layer. However, due to the randomly generated parameters, some generated “bad” parameters may be introduced to bring negative effect on the final generalization ability. To overcome such drawback, this paper combines the artificial immune system (AIS) with ELM, namely, AIS-ELM. With the help of AIS’s global search and good convergence, the randomly generated parameters of ELM are optimized effectively and efficiently to achieve a better generalization performance. To evaluate the performance of AIS-ELM, this paper compares it with relevant algorithms on several benchmark datasets. The experimental results reveal that our proposed algorithm can always achieve superior performance. [ABSTRACT FROM AUTHOR]

In the early days, content-based image retrieval (CBIR) was studied with global features. Since 2003, image retrieval based on local descriptors (de facto SIFT) has been extensively studied for over a decade due to the advantage of SIFT in dealing with image transformations. Recently, image representations based on the convolutional neural network (CNN) have attracted increasing interest in the community and demonstrated impressive performance. Given this time of rapid evolution, this article provides a comprehensive survey of instance retrieval over the last decade. Two broad categories, SIFT-based and CNN-based methods, are presented. For the former, according to the codebook size, we organize the literature into using large/medium-sized/small codebooks. For the latter, we discuss three lines of methods, i.e., using pre-trained or fine-tuned CNN models, and hybrid methods. The first two perform a single-pass of an image to the network, while the last category employs a patch-based feature extraction scheme. This survey presents milestones in modern instance retrieval, reviews a broad selection of previous works in different categories, and provides insights on the connection between SIFT and CNN-based methods. After analyzing and comparing retrieval performance of different categories on several datasets, we discuss promising directions towards generic and specialized instance retrieval. [ABSTRACT FROM PUBLISHER]

In content-based image retrieval, SIFT feature and the feature from deep convolutional neural network (CNN) have demonstrated promising performance. To fully explore both visual features in a unified framework for effective and efficient retrieval, we propose a collaborative index embedding method to implicitly integrate the index matrices of them. We formulate the index embedding as an optimization problem from the perspective of neighborhood sharing and solve it with an alternating index update scheme. After the iterative embedding, only the embedded CNN index is kept for on-line query, which demonstrates significant gain in retrieval accuracy, with very economical memory cost. Extensive experiments have been conducted on the public datasets with million-scale distractor images. The experimental results reveal that, compared with the recent state-of-the-art retrieval algorithms, our approach achieves competitive accuracy performance with less memory overhead and efficient query computation. [ABSTRACT FROM PUBLISHER]

State-of-the-art image classification approaches are mainly based on robust image representation, such as the bag-of-features (BoF) model or the convolutional neural network (CNN) architecture. In real applications, the orientation (left/right) of an image or an object might vary from sample to sample, whereas some handcrafted descriptors (e.g., SIFT) and network operations (e.g., convolution) are not reversal-invariant, leading to the unsatisfied stability of image features extracted from these models. To deal with, a popular solution is to augment the dataset by adding a left-right reversed copy for each image. This strategy improves the recognition accuracy to some extent, but also brings the price of almost doubled time and memory consumptions on both the training and testing stages. In this paper, we present an alternative solution based on designing reversal-invariant representation of local patterns, so that we can obtain the identical representation for an image and its left-right reversed copy. For the BoF model, we design a reversal-invariant version of SIFT descriptor named Max-SIFT, a generalized RIDE algorithm which can be applied to a large family of local descriptors. For the CNN architecture, we present a simple idea of generating reversal-invariant deep features ( RI-Deep), and, inspired by which, design reversal-invariant convolution ( RI-Conv) layers to increase the CNN capacity without increasing the model complexity. Experiments reveal consistent accuracy gain on various image classification tasks, including scene understanding, fine-grained object recognition, and large-scale visual recognition. [ABSTRACT FROM AUTHOR]

This paper considers the task of image search using the Bag-of-Words (BoW) model. In this model, the precision of visual matching plays a critical role. Conventionally, local cues of a keypoint, e.g., SIFT, are employed. However, such strategy does not consider the contextual evidences of a keypoint, a problem which would lead to the prevalence of false matches. To address this problem and enable accurate visual matching, this paper proposes to integrate discriminative cues from multiple contextual levels, i.e., local, regional, and global, via probabilistic analysis. 'True match' is defined as a pair of keypoints corresponding to the same scene location on all three levels (Fig. 1). Specifically, the Convolutional Neural Network (CNN) is employed to extract features from regional and global patches. We show that CNN feature is complementary to SIFT due to its semantic awareness and compares favorably to several other descriptors such as GIST, HSV, etc. To reduce memory usage, we propose to index CNN features outside the inverted file, communicated by memory-efficient pointers. Experiments on three benchmark datasets demonstrate that our method greatly promotes the search accuracy when CNN feature is integrated. We show that our method is efficient in terms of time cost compared with the BoW baseline, and yields competitive accuracy with the state-of-the-arts. [ABSTRACT FROM AUTHOR]

Gait recognition is an important biometric technology that allows for the remote collection of stakeholders' characteristics, without requiring their explicit cooperation. It has gained considerable attention in the fields of criminal investigation and intelligent security. Previous studies have shown that local gait features can enhance gait recognition performance by improving robustness to disturbances. However, global gait features also play a crucial role in gait recognition. Many researchers have utilized convolutional operations to extract global features, but these operations tend to focus on features within the receptive field, neglecting those outside of it. Therefore, the potential of global gait features has not been fully explored. In this paper, we propose a gait recognition framework based on vision transformers, aiming to enhance the extraction of global gait features. We introduce an adaptive multi-frame global feature mapping (AMGM) method to address the challenge of inconsistent feature dimensions caused by variations in the number of gait frames when fusing global and local features. We evaluate our model on the latest datasets, and the experimental results demonstrate a significant breakthrough. Notably, our model achieves state-of-the-art recognition accuracy, particularly in scenarios where subjects are wearing coats. Additionally, our model achieves remarkable improvements in recognition accuracy through training with small sample sets. [ABSTRACT FROM AUTHOR]

Retinal vessel segmentation (RVS) helps the diagnosis of diabetic retinopathy, which can cause visual impairment and even blindness. Some problems are hindering the application of automatic RVS, including accuracy, robustness and segmentation speed. In this paper, we propose a cross-connected convolutional neural network (CcNet) for the automatic segmentation of retinal vessel trees. In the CcNet, convolutional layers extract the features and predict the pixel classes according to those learned features. The CcNet is trained and tested with full green channel images directly. The cross connections between primary path and secondary path fuse the multi-level features. The experimental results on two publicly available datasets (DRIVE: Sn = 0.7625, Acc = 0.9528; STARE: Sn = 0.7709, Acc = 0.9633) are higher than those of most state-of-the-art methods. In the cross-training phase, CcNte's accuracy fluctuations (△ Accs) on DRIVE and STARE are 0.0042 and 0.007, respectively, which are relatively small compared with those of published methods. In addition, our algorithm has faster computing speed (0.063 s) than those listed algorithms using a GPU (graphics processing unit). These results reveal that our algorithm has potential in practical applications due to promising segmentation performances including advanced specificity, accuracy, robustness and fast processing speed. [ABSTRACT FROM AUTHOR]

Highlights • We propose a simple yet effective quantization to embed deep binary codes. • Activations from multiple CNN layers function together through weighted score fusion in the proposed framework. • Handcrafted local descriptor SIFT, as a kind of low level feature, can also be combined in our fusion procedure. • Regularized diffusion process are customized on the ranking list to make the similarity estimation vary smoothly. • Extensive experiments are conducted on four public datasets, and state-of-the-art results are obtained on Holidays and UKBench datasets. Abstract Convolutional Neural Networks (CNNs) have achieved a breakthrough on a large number of image retrieval benchmarks. However, most previous works make use of the CNNs following the image classification strategy, where the last fully connected layer activations of the whole image are occupied as a single holistic feature vector. To improve the representation power of CNNs, this paper proposes a Multi-layer Fusion (MF) approach to aggregate deep activations for image retrieval task. The key insight of our approach is that different layers of a CNN are sensitive to specific patterns, and are complementary with each other for image representation. Specifically, our approach transforms CNN activations to deep binary codes embedded in the inverted index of Bag-of-Words structure for fast retrieval. Those activations are derived from multiple layers of a CNN on local patches, for features from orderless local areas have proved superior to global ones in the low level handcrafted cases. Corresponding weights and diffusion process are thereafter utilized to penalize and re-rank the individual similarity scores of layers. Our method is efficient, which extracts visual features from different layers only once. Furthermore, the proposed MF approach can be easily extended to include SIFT features to enhance the representation power. Extensive experiments on four public retrieval datasets quantitatively evaluate the effectiveness of our contributions, and the proposed algorithm prove to be the new state-of-the-art on the Holidays and UKBench datasets. [ABSTRACT FROM AUTHOR]

Blind image quality assessment (BIQA) aims at precisely estimating human perceived image quality with no access to a reference. Recently, several attempts have been made to develop BIQA methods based on deep neural networks (DNNs). Although these methods obtained promising performance, they have some limitations: (1) their DNN models are actually ”shallow” in term of depth; and (2) these methods typically use the output of the last layer in the DNN model as the feature representation for quality prediction. Since the representation depth has been demonstrated beneficial for various vision tasks, it is significant to explore very deep networks for learning BIQA models. Besides, the information in the last layer may unduly generalize over local artifacts which are highly related to quality degradation. On the contrary, intermediate layers may be sensitive to local degradations but will not capture high-level semantics. Thus, reasoning at multiple levels of representation is necessary in the IQA task. In this paper, we propose to extract multi-level representations from a very deep DNN model for learning an effective BIQA model, and consequently present a simple but extraordinarily effective BIQA framework, codenamed BLINDER ( BLind Image quality predictioN via multi-level DEep Representations ). Thorough experiments have been conducted on five standard databases, which show that a significant improvement can be achieved by adopting multi-level deep representations. Besides, BLINDER considerably outperforms previous state-of-the-art BIQA methods for authentically distorted images. [ABSTRACT FROM AUTHOR]

One of the major challenges in person Re-Identification (ReID) is the inconsistent visual appearance of a person. Current works on visual feature and distance metric learning have achieved significant achievements, but still suffer from the limited robustness to pose variations, viewpoint changes, etc ., and the high computational complexity. This makes person ReID among multiple cameras still challenging. This work is motivated to learn mid-level human attributes which are robust to visual appearance variations and could be used as efficient features for person matching. We propose a weakly supervised multi-type attribute learning framework which considers the contextual cues among attributes and progressively boosts the accuracy of attributes only using a limited number of labeled data. Specifically, this framework involves a three-stage training. A deep Convolutional Neural Network (dCNN) is first trained on an independent dataset labeled with attributes. Then it is fine-tuned on another dataset only labeled with person IDs using our defined triplet loss. Finally, the updated dCNN predicts attribute labels for the target dataset, which is combined with the independent dataset for the final round of fine-tuning. The predicted attributes, namely deep attributes exhibit promising generalization ability across different datasets. By directly using the deep attributes with simple Cosine distance, we have obtained competitive accuracy on four person ReID datasets. Experiments also show that a simple distance metric learning modular further boosts our method, making it outperform many recent works. [ABSTRACT FROM AUTHOR]

The size of databases is the key to success to face recognition systems. However, building such a database is both time-consuming and labor intensive. In this paper, we address the problem by proposing a database generation framework based on text–video correlation. Specifically, visual content of a video can be presented as a character sequence by face detection, tracking and recognition, while text information extracted from subtitles and scripts provides complementary identity sequence. By correlating these two sequences, faces recognized can be refined without manual intervention. Experiments demonstrate that 90% of the human effort in face database construction can be reduced. [ABSTRACT FROM AUTHOR]

The scale-invariant feature transform (SIFT) feature plays a very important role in multimedia content analysis, such as near-duplicate image and video retrieval. However, the storage and query costs of SIFT become unbearable for large-scale databases. In this paper, SIFT features are robustly encoded with temporal information by tracking the SIFT to generate temporal-concentration SIFT (TCSIFT), which highly compresses the quantity of local features to reduce visual redundancy, and keeps the advantages of SIFT as much as possible at the same time. On the basis of TCSIFT, a novel framework for large-scale video copy retrieval is proposed in which the processes of retrieval and validation are implemented at the feature and frame level. Experimental results for two different datasets, i.e., CC_WEB_VIDEO and TRECVID, demonstrate that our method can yield comparable accuracy, compact storage size, and more efficient execution time, as well as adapt to various video transformations. [ABSTRACT FROM AUTHOR]

Abstract: In this paper we propose an online multiple instance boosting algorithm with instance-level semi-supervised learning, termed SemiMILBoost, to achieve robust object tracking. Our work revisits the multiple instance learning (MIL) formulation to alleviate the drifting problem in tracking, which addresses two key issues in the existing MIL based tracking-by-detection methods, i.e., the unselective treatment of instances in the positive bag during weak classifier updating and the lack of object prior knowledge in instance modeling. We tackle both issues in a principled way by using a robust SemiMILBoost algorithm, which treats instances in the positive bag as unlabeled while the ones in the negative bag as negative. To improve the discriminability of weak classifiers online, we iteratively update them with the pseudo-labels and importance of all instances in the positive bag, which are predicted by employing the instance-level semi-supervised learning technique with object prior knowledge during boosting. Experimental results demonstrate that our proposed algorithm outperforms the state-of-the-art tracking methods on several challenging video sequences. [Copyright &y& Elsevier]