Search

Facial expression recognition (FER) is of great interest to the current studies of human-computer interaction. In this paper, we propose a novel geometry-guided facial expression recognition framework, based on graph convolutional networks and transformers, to perform effective emotion recognition from videos. Specifically, we detect and utilize facial landmarks to construct a spatial-temporal graph, based on both the landmark coordinates and local appearance, for representing a facial expression sequence. The graph convolutional blocks and transformer modules are employed to produce high-semantic emotion-related representations from the structured facial graphs, which facilitate the framework to establish both the local and non-local dependency between the vertices. Moreover, spatial and temporal attention mechanisms are introduced into graph-based learning to promote FER reasoning, via the emphasis on the most informative facial components and frames. Extensive experiments demonstrate that the proposed framework achieves promising performance for geometry-based FER and shows great generalization and robustness in real-world applications. [ABSTRACT FROM AUTHOR]

Learning discriminative representations with good robustness from facial observations serves as a fundamental step towards intelligent facial expression recognition (FER). In this article, we propose a novel geometry-aware FER framework to boost the FER performance based on both the geometric and appearance knowledge. Specifically, we propose an encoding strategy for facial landmarks, and adopt a graph convolutional network (GCN) to fully explore the structural information of the facial components behind different expressions. A convolutional neural network (CNN) is further applied to the whole facial observation to learn the global characteristics of different expressions. The features from these two networks are fused into a comprehensive high-semantic representation, which promotes the FER reasoning from both visual and structural perspectives. Moreover, to facilitate the networks to concentrate on the most informative facial regions and components, we introduce multi-level attention mechanisms into the proposed framework, which enhance the reliability of the learned representations for effective FER. Experiments on two challenging FER benchmarks demonstrate that the attentive graph-based learning on the facial geometry boosts the FER accuracy. Furthermore, the insensitivity of the geometric information to the appearance variations also improves the generalization of the proposed framework. [ABSTRACT FROM AUTHOR]

This paper presents a new method, named implicit and explicit feature purification (IEFP), for age-invariant face recognition. Facial features extracted from a face image contain the information about the identity, age, and other attributes. For age-invariant face recognition, it is important to remove the irrelevant information, and retain the identity information only, in the facial features. Through the two proposed feature purification mechanisms, our framework can produce facial-feature embeddings that preserve identity information as much as possible and are insensitive to age variations. Specifically, on the one hand, a special network module is devised to implicitly purify the original facial features obtained from a face encoder. On the other hand, to obtain purer facial feature representations for age-invariant face recognition, irrelevant information within the implicitly purified features, such as the age, is further removed. This is realized by using a regularizer, based on information theory, to explicitly minimize the correlation between identity-related features and age-related features. Comprehensive ablation studies show that these two feature purification schemes can work independently, as well as collaboratively, to achieve better performance. Extensive evaluations on several benchmark data sets show that the IEFP method is on par with those competitors learned on far more favorable training samples, and it achieves the best performance in a fair comparison. Furthermore, we provide mathematical interpretation to explain the effectiveness of our approach, and find that it tends to generate low-rank, yet high-dimensional, representations for age-invariant face recognition. [ABSTRACT FROM AUTHOR]

The emergence of wearable devices has opened up new potentials for egocentric activity recognition. Although some methods integrate attention mechanisms into deep neural networks to capture fine-grained human-object interactions in a weak-supervision manner, they either ignore exploiting the temporal consistency or generate attention based on considering appearance cues only. To address these limitations, in this paper, we propose an enhanced attention-tracking method, combined with multi-branch network (EAT-MBNet), for egocentric activity recognition. Specifically, we propose class-aware attention maps (CAAMs) by employing a self-attention-based module to refine the class activation maps (CAMs). Our proposed method can enhance the semantic dependency between the activity categories and the feature maps. To highlight the discriminative features from the regions of interest across frames, we propose a flow-guided attention-tracking (F-AT) module, by simultaneously leveraging historical attention and motion patterns. Furthermore, we propose a cross-modality modeling branch based on an interactive GRU module, which captures the time-synchronized long-term relationships between the appearance and motion branches. Experimental results on four egocentric activity benchmarks demonstrate that the proposed method achieves state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Corn-plant counting is an important process for predicting corn yield and analyzing corn-plant phenotypes. In this letter, an effective corn-plant counting method is proposed, which is based on utilizing the scale-aware (SA) contextual feature and channel interdependence (CI). Given the Visual Geometry Group (VGG) Network features, the SA features are extracted by spatial pyramid pooling to derive multiscale context information. In order to utilize the channel interdependent information, the VGG features are integrated via a channel attention module. Moreover, an encoder–decoder structure is constructed to fuse the SA features and the CI-based features. Considering the sparsity of a corn plant, a hybrid loss function is adopted to train the network, by considering a density map loss function and an absolute count loss function. Experimental results demonstrate the effectiveness of the proposed method for corn-plant counting. [ABSTRACT FROM AUTHOR]

How to achieve competitive accuracy and less computation time simultaneously for background estimation is still an intractable task. In this paper, an effective background subtraction approach for video sequences is proposed based on a subsuperpixel model. In our algorithm, the superpixels of the first frame are constructed using a simple linear iterative clustering method. After transforming the frame from a color format to gray level, the initial superpixels are divided into K smaller units, i.e., subsuperpixels, via the k-means clustering algorithm. The background model is then initialized by representing each subsuperpixel as a multidimensional feature vector. For the subsequent frames, moving objects are detected by the subsuperpixel representation and a weighting measure. In order to deal with ghost artifacts, a background model updating strategy is devised, based on the number of pixels represented by each cluster center. As each superpixel is refined via the subsuperpixel representation, the proposed method is more efficient and achieves a competitive accuracy for background subtraction. Experimental results demonstrate the effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Highlights • This paper provides a systematic review of current histogram-based local feature descriptors, which have been applied for facial-expression recognition. • The weaknesses and strengths of the existing descriptors are discussed and analysed. • A comprehensive evaluation of the performances of 27 local descriptors is presented. • The best overall performances are obtained by LPQ and LGBPHS. This paper aims to present histogram-based local descriptors applied to Facial Expression Recognition (FER) from static images and provide a systematic review and analysis of them. First, we describe the main steps in encoding binary patterns in a local patch, which are required in every histogram-based local descriptor. Then, we list the existing local descriptors, while analysing their strengths and weaknesses. Finally, we present the experimental results of all these descriptors on commonly used facial expression databases, with varying resolution, noise, occlusion, and number of sub-regions, as well as comparing them with the results obtained by the state-of-the-art deep learning methods. This paper aims to bring together different studies of the visual features for FER by evaluating their performances under the same experimental setup, and critically reviewing various classifiers making use of the local descriptors. [ABSTRACT FROM AUTHOR]

In this paper, a novel and efficient framework by exploiting Quaternionic Distance Based Weber Local Descriptor (QDWLD) and object cues is proposed for image saliency detection. In contrast to the existing saliency detection models, the advantage of the proposed approach is that it can combine quaternion number system and object cues simultaneously, which is independent of image contents and scenes. Firstly, QDWLD, which was initially designed for detecting outliers in color images, is used to represent the directional cues in an image. Meanwhile, two low-level priors, namely the Convex-Hull-Based center and color contrast cue of the image, are utilized and fused as an object-level cue. Finally, by combining QDWLD with object cues, a reliable saliency map of the image can be computed and estimated. Experimental results, based on two widely used and openly available database, show that the proposed method is able to produce reliable and promising salient maps/estimations, compared to other state-of-the-art saliency-detection models. [ABSTRACT FROM AUTHOR]

The human visual system (HVS) can reliably perceive salient objects in an image, but, it remains a challenge to computationally model the process of detecting salient objects without prior knowledge of the image contents. This paper proposes a visual-attention-aware model to mimic the HVS for salient-object detection. The informative and directional patches can be seen as visual stimuli, and used as neuronal cues for humans to interpret and detect salient objects. In order to simulate this process, two typical patches are extracted individually and in parallel from the intensity channel and the discriminant color channel, respectively, as the primitives. In our algorithm, an improved wavelet-based salient-patch detector is used to extract the visually informative patches. In addition, as humans are sensitive to orientation features, and as directional patches are reliable cues, we also propose a method for extracting directional patches. These two different types of patches are then combined to form the most important patches, which are called preferential patches and are considered as the visual stimuli applied to the HVS for salient-object detection. Compared with the state-of-the-art methods for salient-object detection, experimental results using publicly available datasets show that our produced algorithm is reliable and effective. [ABSTRACT FROM PUBLISHER]

Face recognition methods, which usually represent face images using holistic or local facial features, rely heavily on alignment. Their performances also suffer a severe degradation under variations in expressions or poses, especially when there is one gallery per subject only. With the easy access to high-resolution (HR) face images nowadays, some HR face databases have recently been developed. However, few studies have tackled the use of HR information for face recognition or verification. In this paper, we propose a pose-invariant face-verification method, which is robust to alignment errors, using the HR information based on pore-scale facial features. A new keypoint descriptor, namely, pore-Principal Component Analysis (PCA)-Scale Invariant Feature Transform (PPCASIFT)—adapted from PCA-SIFT—is devised for the extraction of a compact set of distinctive pore-scale facial features. Having matched the pore-scale features of two-face regions, an effective robust-fitting scheme is proposed for the face-verification task. Experiments show that, with one frontal-view gallery only per subject, our proposed method outperforms a number of standard verification methods, and can achieve excellent accuracy even the faces are under large variations in expression and pose. [ABSTRACT FROM AUTHOR]

Establishing correct correspondences between two faces with different viewpoints has played an important role in 3D face reconstruction and other computer-vision applications. Usually, face images are considered to lack sufficient distinctive features to establish a large number of correspondences on uncalibrated images. In this paper, we investigate pore-scale facial features, which are formed from pores, fine wrinkles, and hair. These features have many characteristics that make them suitable for matching facial images under different variations. Using both biological observation and computer-vision consideration, a new framework is devised for pore-scale facial-feature extraction and matching. The matching difficulty under various skin appearances of different subjects and imaging distortion is also analyzed. For further improving the matching performance and tackling distortions such as varying illuminations and unfocused blurring, a pore-to-pore correspondences dataset is established for training a more distinctive and compact descriptor. Experiments are conducted on a face database containing 105 subjects, and the results prove that the pore-scale features are highly distinctive; face images with a minimum resolution of 600×700 (0.4 mega) pixels contain sufficient details to perform a reliable matching in different poses. Generally, our algorithm can establish between 500 and 2000 correct correspondences on a pair of uncalibrated face images of the same person. Furthermore, the proposed methods can be applied to face recognition, 3D reconstruction, etc. [ABSTRACT FROM AUTHOR]

Image classification usually requires complicated segmentation to separate foreground objects from the background scene. However, the statistical content of a background scene can actually provide very useful information for classification. In this paper, we propose a new hybrid pyramid kernel which incorporates local features extracted from both dense regular grids and interest points for image classification, without requiring segmentation. Features extracted from dense regular grids can better capture information about the background scene, while interest points detected at corners and edges can better capture information about the salient objects. In our algorithm, these two local features are combined in both the spatial and the feature-space domains, and are organized into pyramid representations. In order to obtain better classification accuracy, we fine-tune the parameters involved in the similarity measure, and we determine discriminative regions by means of relevance feedback. From the experimental results, we observe that our algorithm can achieve a 6.37 % increase in performance as compared to other pyramid-representation-based methods. To evaluate the applicability of the proposed hybrid kernel to large-scale databases, we have performed a cross-dataset experiment and investigated the effect of foreground/background features on each of the kernels. In particular, the proposed hybrid kernel has been proven to satisfy Mercer's condition and is efficient in measuring the similarity between image features. For instance, the computational complexity of the proposed hybrid kernel is proportional to the number of features. [ABSTRACT FROM AUTHOR]

The reliable and accurate identification of cancer categories is crucial to a successful diagnosis and a proper treatment of the disease. In most existing work, samples of gene expression data are treated as one-dimensional signals, and are analyzed by means of some statistical signal processing techniques or intelligent computation algorithms. In this paper, from an image-processing viewpoint, a spectral-feature-based Tikhonov-regularized least-squares (TLS) ensemble algorithm is proposed for cancer classification using gene expression data. In the TLS model, a test sample is represented as a linear combination of the atoms of a dictionary. Two types of dictionaries, namely singular value decomposition (SVD)-based eigenassays and independent component analysis (ICA)-based eigenassays, are proposed for the TLS model, and both are extracted via a two-stage approach. The proposed algorithm is inspired by our finding that, among these eigenassays, the categories of some of the testing samples can be assigned correctly by using the TLS models formed from some of the spectral features, but not for those formed from the original samples only. In order to retain the positive characteristics of these spectral features in making correct category assignments, a strategy of classifier committee learning (CCL) is designed to combine the results obtained from the different spectral features. Experimental results on standard databases demonstrate the feasibility and effectiveness of the proposed method. [ABSTRACT FROM PUBLISHER]

Abstract: In this paper, an efficient method based on singular values and potential-field representation is proposed for face-image retrieval. Firstly, we theoretically prove that the leading singular values of an image can be used as a rotation-shift-scale-invariant global feature. Then, for the feature-extraction stage, we exploit these special properties of the singular values to devise a compact, global feature for face-image representation. We also use the singular values of the potential field derived from edge gradients to enhance the retrieval performance. Experimental results based on the GTAV database show that the use of singular values as rotation-shift-scale-invariant global features is able to produce plausible retrieval results. [Copyright &y& Elsevier]

Abstract: For face recognition, image features are first extracted and then matched to those features in a gallery set. The amount of information and the effectiveness of the features used will determine the recognition performance. In this paper, we propose a novel face recognition approach using information about face images at higher and lower resolutions so as to enhance the information content of the features that are extracted and combined at different resolutions. As the features from different resolutions should closely correlate with each other, we employ the cascaded generalized canonical correlation analysis (GCCA) to fuse the information to form a single feature vector for face recognition. To improve the performance and efficiency, we also employ “Gabor-feature hallucination”, which predicts the high-resolution (HR) Gabor features from the Gabor features of a face image directly by local linear regression. We also extend the algorithm to low-resolution (LR) face recognition, in which the medium-resolution (MR) and HR Gabor features of a LR input image are estimated directly. The LR Gabor features and the predicted MR and HR Gabor features are then fused using GCCA for LR face recognition. Our algorithm can avoid having to perform the interpolation/super-resolution of face images and having to extract HR Gabor features. Experimental results show that the proposed methods have a superior recognition rate and are more efficient than traditional methods. [Copyright &y& Elsevier]

Recently, over-the-top live gaming content video (GCV) services have significantly contributed to the overall internet traffic. Consequently, there is a growing demand of GCV quality assessment (GCVQA) to maintain service quality. Although recent literature has proposed a few GCVQA methods, these mainly focus on extracting spatial features and temporal fusion separately, limiting their performance due to the neglect of spatiotemporal feature learning, which is crucial for GCV as it typically shares spatial and temporal features across frames. To address this, we propose a novel GCVQA model, focusing on GCV spatiotemporal feature learning. First, we employ a multi-task self-supervised learning spatiotemporal pyramid convolutional neural network (STP-CNN) model to extract short-term spatiotemporal quality feature representations (STQFR) of GCVs. Our STP-CNN model specifically extracts multiscale spatiotemporal features from various temporal scales of multi-frames in pyramid mode, enabling dynamic learning of diverse spatiotemporal cues. Subsequently, we propose the differential Transformer model to process all short-term STQFR within a GCV, extracting global spatiotemporal features of GCV to assess the overall quality of GCV. To evaluate the effectiveness of our proposed method, we conducted experiments using four GCVQA datasets. The results demonstrate that our method outperforms existing approaches in predicting the perceived quality of GCV. • Propose a deep learning-based NR-GCVQA method tailored explicitly for GCV, emphasizing spatiotemporal feature learning in short-term and long-term. • Propose the multi-task, self-supervised STP-CNN model that employs a pyramid mode to extract multiscale spatiotemporal features, optimizing the representation of short-term spatiotemporal features. • Introduce the Diff-Transformer model to distill global spatiotemporal features of GCV to achieve a precise GCV quality prediction. • Provide experimental evidence to substantiate the accuracy and effectiveness of our proposed NR-GCVQA methodology. [ABSTRACT FROM AUTHOR]

Most existing person re-identification (ReID) methods rely only on the spatial appearance information from either one or multiple person images, whilst ignore the space-time cues readily available in video or image-sequence data. Moreover, they often assume the availability of exhaustively labelled cross-view pairwise data for every camera pair, making them non-scalable to ReID applications in real-world large scale camera networks. In this work, we introduce a novel video based person ReID method capable of accurately matching people across views from arbitrary unaligned image-sequences without any labelled pairwise data. Specifically, we introduce a new space-time person representation by encoding multiple granularities of spatio-temporal dynamics in form of time series. Moreover, a Time Shift Dynamic Time Warping (TS-DTW) model is derived for performing automatically alignment whilst achieving data selection and matching between inherently inaccurate and incomplete sequences in a unified way. We further extend the TS-DTW model for accommodating multiple feature-sequences of an image-sequence in order to fuse information from different descriptions. Crucially, this model does not require pairwise labelled training data (i.e. unsupervised) therefore readily scalable to large scale camera networks of arbitrary camera pairs without the need for exhaustive data annotation for every camera pair. We show the effectiveness and advantages of the proposed method by extensive comparisons with related state-of-the-art approaches using two benchmarking ReID datasets, PRID2011 and iLIDS-VID. [ABSTRACT FROM AUTHOR]

Abstract: Gabor wavelets (GWs) are commonly used for extracting local features for various applications such as object detection, recognition and tracking. However, extracting Gabor features is computationally intensive, so the features are impractical for real-time applications. In this paper, we propose a simplified version of Gabor wavelets (SGWs) and an efficient algorithm for extracting the features based on an integral image. We evaluate the performance of the SGW features for face recognition. Experimental results show that using SGWs can achieve a performance level similar to using GWs, while the runtime for feature extraction using SGWs is, at most, 4.39 times faster than that of GWs implemented by using the fast Fourier transform (FFT). [Copyright &y& Elsevier]