Search

In this paper, we propose a weakly-supervised approach called Order-Constrained Representation Learning (OCRL) to predict future actions from instructional videos by observing incomplete steps of actions. Most conventional methods focus on predicting actions based on partially observed video frames, which mainly study low-level semantics such as motion consistency. Unlike performing a single action, completing a task in an instructional video usually requires several steps of action and longer periods. Motivated by the fact that the order of action steps is key to learning task semantics, we develop a new frame of contrastive loss, called StepNCE, to integrate the shared semantic information between step order and task semantics under the framework of the memory bank-based momentum-updating algorithm. Specifically, we learn the video representations from step order-rearranged trimmed video clips based on the proposed task-consistency rule and order-consistency rule. Our StepNCE loss can be used to pre-train a video feature encoder, which is then fine-tuned to carry out the instructional video prediction task. Our approach digs deeper into the sequential logic between different action steps with respect to a certain task, which is able to promote the video understanding methods to a new semantic level. We evaluate our method on five popular instructional video and action prediction datasets: COIN, CrossTask, UT-Interaction, BIT-Interaction, and ActivityNet v1.2, and the results show that our approach gains improvements from conventional prediction methods. [ABSTRACT FROM AUTHOR]

Efficiently identifying the mated gallery fingerprint of a latent fingerprint in a large database requires a highly accurate and efficient fingerprint matching algorithm. The common strategy to achieve this goal is to combine an efficient indexing algorithm with a slow but accurate matching algorithm. Despite of the importance of latent indexing, it has received far less attention than rolled and plain fingerprint indexing. Due to the small fingerprint area, poor image quality and huge variety in information quantity of latent fingerprints, existing rolled and plain fingerprint indexing approaches cannot be simply migrated to the latent fingerprint indexing. In this paper, we propose (1) a multi-scale fixed-length representation approach for latent fingerprint indexing, and (2) a fingerprint information quantity estimation approach for adaptive candidate list reduction. The representation scheme is designed to deal with small finger area and low image quality of latents. The information quantity of a latent is a predictor of the indexing score of its mated gallery fingerprint and thus can be used to determine a proper threshold for its candidate list. Extensive experimental results on NIST SD27, MOLF, N2N, and Hisign latent fingerprint databases show that the proposed method achieved the state-of-the-art indexing accuracy on latent fingerprints, and significantly improved the efficiency of state-of-the-art latent matching algorithm. [ABSTRACT FROM AUTHOR]

Different people age in different ways. Learning a personalized age estimator for each person is a promising direction for age estimation given that it better models the personalization of aging processes. However, most existing personalized methods suffer from the lack of large-scale datasets due to the high-level requirements: identity labels and enough samples for each person to form a long-term aging pattern. In this paper, we aim to learn personalized age estimators without the above requirements and propose a meta-learning method named MetaAge for age estimation. Unlike most existing personalized methods that learn the parameters of a personalized estimator for each person in the training set, our method learns the mapping from identity information to age estimator parameters. Specifically, we introduce a personalized estimator meta-learner, which takes identity features as the input and outputs the parameters of customized estimators. In this way, our method learns the meta knowledge without the above requirements and seamlessly transfers the learned meta knowledge to the test set, which enables us to leverage the existing large-scale age datasets without any additional annotations. Extensive experimental results on three benchmark datasets including MORPH II, ChaLearn LAP 2015 and ChaLearn LAP 2016 databases demonstrate that our MetaAge significantly boosts the performance of existing personalized methods and outperforms the state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Fingerprint pose estimation is a challenging problem since the pose is not defined by salient anatomical features and fingerprint images usually suffer from noise and small area. In this article, we proposed a method for joint estimation of pose and singular points of fingerprints, with the expectation that the pose and singular points can improve each other. By virtue of that singular points can be located accurately, we hope to improve the accuracy of pose estimation. Meanwhile, the robustness of pose estimation can improve the anti-noise performance of singular point detection. To achieve this, we propose a multi-task deep neural network, which contains a feature extraction body and two estimation heads for singular point and pose respectively. The proposed network can deal with various types of fingerprints, including plain, rolled and latent fingerprints. Experiments on four databases (NIST SD4, SD14, SD27 and FVC2004 DB1A) show that (1) the estimated poses and detected singular points are close to manual annotations despite of different image qualities; (2) the estimated poses for mated fingerprint pairs are consistent; and (3) the proposed pose estimation method outperforms state-of-the-art methods while utilized as pose constraint for a fingerprint indexing algorithm. [ABSTRACT FROM AUTHOR]

Dense registration of fingerprints is a challenging task due to elastic skin distortion, low image quality, and self-similarity of ridge pattern. To overcome the limitation of handcraft features, we propose to train an end-to-end network to directly output pixel-wise displacement field between two fingerprints. The proposed network includes a siamese network for feature embedding, and a following encoder-decoder network for regressing displacement field. By applying displacement fields reliably estimated by tracing high quality fingerprint videos to challenging fingerprints, we synthesize a large number of training fingerprint pairs with ground truth displacement fields. In addition, based on the proposed registration algorithm, we propose a fingerprint mosaicking method based on optimal seam selection. Registration and matching experiments on FVC2004 databases, Tsinghua Distorted Fingerprint (TDF) database, and NIST SD27 latent fingerprint database show that our registration method outperforms previous dense registration methods in accuracy. Mosaicking experiments on FVC2004 DB1_A and a small fingerprint database demonstrate that the proposed algorithm produced higher quality fingerprints and led to higher matching accuracy, which also validates the performance of our registration algorithm. [ABSTRACT FROM AUTHOR]

Optical coherence tomography (OCT), as a non-destructive and high-resolution fingerprint acquisition technology, is robust against poor skin conditions and resistant to spoof attacks. It measures fingertip information on and beneath skin as 3D volume data, containing the surface fingerprint, internal fingerprint and sweat glands. Various methods have been proposed to extract internal fingerprints, which ignore the inter-slice dependence and often require manually selected parameters. In this article, a modified U-Net that combines residual learning, bidirectional convolutional long short-term memory and hybrid dilated convolution (denoted as BCL-U Net) for OCT volume data segmentation and two fingerprint reconstruction approaches are proposed. To the best of our knowledge, it is the first time that simultaneous and automatic extraction is performed for surface fingerprint, internal fingerprint and sweat gland. The proposed BCL-U Net utilizes the spatial dependence in OCT volume data and deals with segmentation of objects with diverse sizes to achieve accurate extraction. Comparisons have been performed to demonstrate the advantages of the proposed method. A thorough evaluation of the recognition abilities of internal and surface fingerprints is conducted using a dataset significantly larger than previous studies. Four databases containing internal and surface fingerprints are generated from 1572 OCT volume data by the proposed method. The internal fingerprint matching experiment has achieved a lowest equal error rate (EER) of 0.95%. Mixed internal and surface fingerprint matching experiment is also performed and achieves an EER of 3.67%, verifying the consistency of the internal and surface fingerprints. The matching experiments for fingers under poor skin conditions show a 2.47% EER of internal fingerprints that is much lower than that of surface fingerprints, which proves the advantage of internal fingerprints and indicates the potential of the internal fingerprints to supplement or replace the surface fingerprints for some specific applications. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the problem of facial kinship verification by learning hierarchical reasoning graph networks. Conventional methods usually focus on learning discriminative features for each facial image of a paired sample and neglect how to fuse the obtained two facial image features and reason about the relations between them. To address this, we propose a Star-shaped Reasoning Graph Network (S-RGN). Our S-RGN first constructs a star-shaped graph where each surrounding node encodes the information of comparisons in a feature dimension and the central node is employed as the bridge for the interaction of surrounding nodes. Then we perform relational reasoning on this star graph with iterative message passing. The proposed S-RGN uses only one central node to analyze and process information from all surrounding nodes, which limits its reasoning capacity. We further develop a Hierarchical Reasoning Graph Network (H-RGN) to exploit more powerful and flexible capacity. More specifically, our H-RGN introduces a set of latent reasoning nodes and constructs a hierarchical graph with them. Then bottom-up comparative information abstraction and top-down comprehensive signal propagation are iteratively performed on the hierarchical graph to update the node features. Extensive experimental results on four widely used kinship databases show that the proposed methods achieve very competitive results. [ABSTRACT FROM AUTHOR]

Palmprints are a recognizable biometric feature that contains multimodal information. Contact-based high-resolution palmprints are the most commonly used image in automatic palmprint identification systems with high-level security requirements because of its high matching accuracy and robustness. Currently, the traditional contact-based high-resolution palmprint acquisition system only captures the total internal reflection (TIR) image, which is similar to fingerprint acquisition systems. However, it is worth noting that the palm is much larger than the finger, and different regions of the palm cannot be easily pressed to the same plane. We observed that the images obtained by these existing contact-based high-resolution palmprint image acquisition systems often have poor integrity and sharpness, especially in the center of the palm. Considering this problem, we proposed an improved contact-based high-resolution palmprint image acquisition system. In addition to the TIR image, it can also simultaneously acquire a diffuse reflection (DR) image. A high-resolution DR image has low contrast and is thus not suitable for direct identification. However, the high integrity and sharing of structural information with the TIR image make it possible to complement the TIR image. Our strategy is to transfer the DR image to the TIR domain through a convolution neural network (CNN) and then fuse it with the corresponding TIR image to obtain a higher quality TIR domain image. The partial-to-“full” identification experimental results show that the rank-1 accuracy of the fused TIR (FTIR) image is improved by approximately 23% compared with the traditional TIR image, which proves the effectiveness of our strategy. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the problem of RGB-D egocentric action recognition. Unlike conventional human action videos that are passively recorded by static cameras, egocentric videos are self-generated from wearable sensors that are more flexible and provide the close-ups with the visual attention of the wearers when they act. Moreover, RGB-D videos contain the spatial appearance and temporal information in the RGB modality and reflect the 3D structure of the scenes in the depth modality. To adequately learn the nonlinear structure of heterogeneous representations from different modalities and exploit their complementary characteristics, we develop a multi-stream deep neural networks (MDNN) method, which aims to preserve the distinctive property for each modality and simultaneously explore their sharable information in a unified deep architecture. Specifically, we deploy a Cauchy estimator to maximize the correlations of the sharable components and enforce the orthogonality constraints on the distinctive components to guarantee their high independencies. Since the egocentric action recognition is usually sensitive to hand poses, we extend our MDNN by integrating with the hand cues to enhance the recognition accuracy. Extensive experimental results on a newly collected data set and two additional benchmarks are presented to demonstrate the effectiveness of our proposed method for RGB-D egocentric action recognition. [ABSTRACT FROM AUTHOR]

In this paper, we propose a uniform and variational deep learning (UVDL) method for RGB-D object recognition and person re-identification. Unlike most existing object recognition and person re-identification methods, which usually use only the visual appearance information from RGB images, our method recognizes visual objects and persons with RGB-D images to exploit more reliable information such as geometric and anthropometric information that are robust to different viewpoints. Specifically, we extract the depth feature and the appearance feature from the depth and RGB images with two deep convolutional neural networks, respectively. In order to combine the depth feature and the appearance feature to exploit their relationship, we design a uniform and variational multi-modal auto-encoder at the top layer of our deep network to seek a uniform latent variable by projecting them into a common space, which contains the whole information of RGB-D images and has small intra-class variation and large inter-class variation, simultaneously. Finally, we optimize the auto-encoder layer and two deep convolutional neural networks jointly to minimize the discriminative loss and the reconstruction error. The experimental results on both RGB-D object recognition and RGB-D person re-identification are presented to show the efficiency of our proposed approach. [ABSTRACT FROM AUTHOR]

Palmprints are a recognizable biometric feature that contains multimodal information. Contact-based high-resolution palmprints are the most commonly used image in automatic palmprint identification systems with high-level security requirements because of its high matching accuracy and robustness. Currently, the traditional contact-based high-resolution palmprint acquisition system only captures the total internal reflection (TIR) image, which is similar to fingerprint acquisition systems. However, it is worth noting that the palm is much larger than the finger, and different regions of the palm cannot be easily pressed to the same plane. We observed that the images obtained by these existing contact-based high-resolution palmprint image acquisition systems often have poor integrity and sharpness, especially in the center of the palm. Considering this problem, we proposed an improved contact-based high-resolution palmprint image acquisition system. In addition to the TIR image, it can also simultaneously acquire a diffuse reflection (DR) image. A high-resolution DR image has low contrast and is thus not suitable for direct identification. However, the high integrity and sharing of structural information with the TIR image make it possible to complement the TIR image. Our strategy is to transfer the DR image to the TIR domain through a convolution neural network (CNN) and then fuse it with the corresponding TIR image to obtain a higher quality TIR domain image. The partial-to-“full” identification experimental results show that the rank-1 accuracy of the fused TIR (FTIR) image is improved by approximately 23% compared with the traditional TIR image, which proves the effectiveness of our strategy. [ABSTRACT FROM AUTHOR]

In this paper, we propose an unsupervised feature learning method called deep binary descriptor with multi-quantization (DBD-MQ) for visual analysis. Existing learning-based binary descriptors such as compact binary face descriptor (CBFD) and DeepBit utilize the rigid sign function for binarization despite of data distributions, which usually suffer from severe quantization loss. In order to address the limitation, we propose a deep multi-quantization network to learn a data-dependent binarization in an unsupervised manner. More specifically, we design a K-Autoencoders (KAEs) network to jointly learn the parameters of feature extractor and the binarization functions under a deep learning framework, so that discriminative binary descriptors can be obtained with a fine-grained multi-quantization. As DBD-MQ simply allocates the same number of quantizers to each real-valued feature dimension ignoring the elementwise diversity of informativeness, we further propose a deep competitive binary descriptor with multi-quantization (DCBD-MQ) method to learn optimal allocation of bits with the fixed binary length in a competitive manner, where informative dimensions gain more bits for complete representation. Moreover, we present a similarity-aware binary encoding strategy based on the earth mover's distance of Autoencoders, so that elements that are quantized into similar Autoencoders will have smaller Hamming distances. Extensive experimental results on six widely-used datasets show that our DBD-MQ and DCBD-MQ outperform most state-of-the-art unsupervised binary descriptors. [ABSTRACT FROM AUTHOR]

In this paper, we propose an ordinal deep learning approach for facial age estimation. Unlike conventional hand-crafted feature-based methods that require prior and expert knowledge, we propose an ordinal deep feature learning (ODFL) method to learn feature descriptors for face representation directly from raw pixels. Motivated by the fact that age labels are chronologically correlated and age estimation is an ordinal learning problem, our proposed ODFL enforces two criteria on the descriptors, which are learned at the top of the deep networks: 1) the topology-preserving ordinal relation is employed to exploit the order information in the learned feature space and 2) the age-difference cost information is leveraged to dynamically measure face pairs with different age value gaps. However, both the procedures of feature extraction and age estimation are learned independently in ODFL, which may lead to a sub-optimal problem. To address this, we further propose an end-to-end ordinal deep learning (ODL) framework, where the complementary information of both the procedures is exploited to reinforce our model. Extensive experimental results on five face aging datasets show that both our ODFL and ODL achieve superior performance in comparisons with most state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, we propose a two-stream transformer networks (TSTN) approach for video-based face alignment. Unlike conventional image-based face alignment approaches which cannot explicitly model the temporal dependency in videos and motivated by the fact that consistent movements of facial landmarks usually occur across consecutive frames, our TSTN aims to capture the complementary information of both the spatial appearance on still frames and the temporal consistency information across frames. To achieve this, we develop a two-stream architecture, which decomposes the video-based face alignment into spatial and temporal streams accordingly. Specifically, the spatial stream aims to transform the facial image to the landmark positions by preserving the holistic facial shape structure. Accordingly, the temporal stream encodes the video input as active appearance codes, where the temporal consistency information across frames is captured to help shape refinements. Experimental results on the benchmarking video-based face alignment datasets show very competitive performance of our method in comparisons to the state-of-the-arts. [ABSTRACT FROM AUTHOR]

Thrombosis has become a global disease threatening human health. The left atrial appendage (LAA) is a major source of thrombosis in patients with atrial fibrillation (AF). Positive correlation exists between LAA volume and AF risk. LAA morphology has been suggested to influence thromboembolic risk in AF patients and to help predict thromboembolic events in low-risk patient groups. Automatic segmentation of LAA can greatly help physicians diagnose AF. In consideration of the large anatomical variations of the LAA, we proposed a robust method for automatic LAA segmentation on computed tomographic angiography (CTA) data using fully convolutional neural networks with three-dimensional (3–D) conditional random fields (CRFs). After manual localization of ROI of LAA, we adopted the FCN in natural image segmentation and transferred their learned models by fine-tuning the networks to segment each 2–D LAA slice. Subsequently, we used a modified dense 3–D CRF that accounts for the 3–D spatial information and larger contextual information to refine the segmentations of all slices. Our method was evaluated on 150 sets of CTA data using five-fold cross validation. Compared with manual annotation, we obtained a mean dice overlap of $\text{94.76}\%$ and a mean volume overlap of $\text{91.10}\%$ with a computation time of less than 40 s per volume. Experimental results demonstrated the robustness of our method in dealing with large anatomical variations and computational efficiency for adoption in a daily clinical routine.) [ABSTRACT FROM AUTHOR]

Metric learning has been widely used in many visual analysis applications, which learns new distance metrics to measure the similarities of samples effectively. Conventional metric learning methods learn a single linear Mahalanobis metric, yet such linear projections are not powerful enough to capture the nonlinear relationships. Recently, deep metric learning approaches, such as discriminative deep metric learning and deep transfer metric learning, have been introduced to fully exploit the nonlinearity of samples by learning hierarchical nonlinear transformations. However, these methods only learn holistic metrics over the input space and are limited for the heterogeneous data sets, where data varies locally. In this paper, we propose a deep localized metric learning approach for visual recognition by learning multiple fine-grained deep localized metrics. We first learn $K$ local subspaces and one holistic subspace with the K-auto-encoders-based clustering. Then, given an input pair, we compute its localized distance on each learned subspace and obtain the final distance representation. Finally, we train the entire neural networks to ensure the distances of positive pairs smaller than negative pairs by a large margin. Experimental results on three visual recognition applications, including face recognition, person re-identification, and scene recognition, show that our DLML outperforms most existing metric learning approaches. [ABSTRACT FROM AUTHOR]

In this paper, we propose a nonlinear structural hashing approach to learn compact binary codes for scalable video search. Unlike most existing video hashing methods which consider image frames within a video separately for binary code learning, we develop a multi-layer neural network to learn compact and discriminative binary codes by exploiting both the structural information between different frames within a video and the nonlinear relationship between video samples. To be specific, we learn these binary codes under two different constraints at the output of our network: 1) the distance between the learned binary codes for frames within the same scene is minimized and 2) the distance between the learned binary matrices for a video pair with the same label is less than a threshold and that for a video pair with different labels is larger than a threshold. To better measure the structural information of the scenes from videos, we employ a subspace clustering method to cluster frames into different scenes. Moreover, we design multiple hierarchical nonlinear transformations to preserve the nonlinear relationship between videos. Experimental results on three video data sets show that our method outperforms state-of-the-art hashing approaches on the scalable video search task. [ABSTRACT FROM AUTHOR]

In this paper, we propose a context-aware local binary feature learning (CA-LBFL) method for face recognition. Unlike existing learning-based local face descriptors such as discriminant face descriptor (DFD) and compact binary face descriptor (CBFD) which learn each feature code individually, our CA-LBFL exploits the contextual information of adjacent bits by constraining the number of shifts from different binary bits, so that more robust information can be exploited for face representation. Given a face image, we first extract pixel difference vectors (PDV) in local patches, and learn a discriminative mapping in an unsupervised manner to project each pixel difference vector into a context-aware binary vector. Then, we perform clustering on the learned binary codes to construct a codebook, and extract a histogram feature for each face image with the learned codebook as the final representation. In order to exploit local information from different scales, we propose a context-aware local binary multi-scale feature learning (CA-LBMFL) method to jointly learn multiple projection matrices for face representation. To make the proposed methods applicable for heterogeneous face recognition, we present a coupled CA-LBFL (C-CA-LBFL) method and a coupled CA-LBMFL (C-CA-LBMFL) method to reduce the modality gap of corresponding heterogeneous faces in the feature level, respectively. Extensive experimental results on four widely used face datasets clearly show that our methods outperform most state-of-the-art face descriptors. [ABSTRACT FROM PUBLISHER]

In this paper, we present a label-sensitive deep metric learning (LSDML) approach for facial age estimation. Motivated by the fact that human age labels are chronologically correlated, our proposed LSDML aims to seek a series of hierarchical nonlinear transformations by deep residual network to project face samples to a latent common space, where the similarity of face pairs is equivalently isotonic to the age difference in a ranking-preserving manner. Since traversal access to total negative samples catastrophically costs and leads to suboptimal, our model learns to mine hard meaningful samples in parallel to learning feature similarity, so that the local manifold of face samples is preserved in the transformed subspace. To better improve the performance on the data set that contains few labeled samples, we further extend our LSDML to a multi-source LSDML method, which aims at maximizing the cross-population correlation of different face aging data sets. Extensive experimental results on four benchmarking data sets show the effectiveness of our proposed approach. [ABSTRACT FROM PUBLISHER]

To facilitate fast similarity search, this paper proposes to encode the nonlinear similarity and image structure as compact binary codes. Rather than adopting single matrix as projection in the literature, we employ a nonlinear transformation in the form of multilayer neural network to generate binary codes to capture the local structure between data samples. Specifically, we train the network such that the quantization loss is minimized and the variance over all bits is maximized. In addition, we capture the salient structure of image samples at the abstract level with sparsity constraint and inherit the generalization power to unseen samples. Furthermore, we incorporate the supervisory label information into the learning procedure to take advantage of the manual label. To obtain the desired binary codes and the parameterized nonlinear transformation, we optimize the formulated objective problem over each variable with an iterative alternating method. To validate the efficacy of the proposed hashing approach, we conduct experiments on three widely used datasets, namely CIFAR10, MNIST, and SUN397, by comparing with several recent proposed hashing methods. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a rotation-invariant local binary descriptor (RI-LBD) learning method for visual recognition. Compared with hand-crafted local binary descriptors, such as local binary pattern and its variants, which require strong prior knowledge, local binary feature learning methods are more efficient and data-adaptive. Unlike existing learning-based local binary descriptors, such as compact binary face descriptor and simultaneous local binary feature learning and encoding, which are susceptible to rotations, our RI-LBD first categorizes each local patch into a rotational binary pattern (RBP), and then jointly learns the orientation for each pattern and the projection matrix to obtain RI-LBDs. As all the rotation variants of a patch belong to the same RBP, they are rotated into the same orientation and projected into the same binary descriptor. Then, we construct a codebook by a clustering method on the learned binary codes, and obtain a histogram feature for each image as the final representation. In order to exploit higher order statistical information, we extend our RI-LBD to the triple rotation-invariant co-occurrence local binary descriptor (TRICo-LBD) learning method, which learns a triple co-occurrence binary code for each local patch. Extensive experimental results on four different visual recognition tasks, including image patch matching, texture classification, face recognition, and scene classification, show that our RI-LBD and TRICo-LBD outperform most existing local descriptors. [ABSTRACT FROM AUTHOR]

Face alignment aims at localizing multiple facial landmarks for a given facial image, which usually suffers from large variances of diverse facial expressions, aspect ratios and partial occlusions, especially when face images were captured in wild conditions. Conventional face alignment methods extract local features and then directly concatenate these features for global shape regression. Unlike these methods which cannot explicitly model the correlation of neighbouring landmarks and motivated by the fact that individual landmarks are usually correlated, we propose a deep sharable and structural detectors (DSSD) method for face alignment. To achieve this, we firstly develop a structural feature learning method to explicitly exploit the correlation of neighbouring landmarks, which learns to cover semantic information to disambiguate the neighbouring landmarks. Moreover, our model selectively learns a subset of sharable latent tasks across neighbouring landmarks under the paradigm of the multi-task learning framework, so that the redundancy information of the overlapped patches can be efficiently removed. To better improve the performance, we extend our DSSD to a recurrent DSSD (R-DSSD) architecture by integrating with the complementary information from multi-scale perspectives. Experimental results on the widely used benchmark datasets show that our methods achieve very competitive performance compared to the state-of-the-arts. [ABSTRACT FROM AUTHOR]