Search

We present Hand-CNN, a novel convolutional network architecture for detecting hand masks and predicting hand orientations in unconstrained images. Hand-CNN extends MaskRCNN with a novel attention mechanism to incorporate contextual cues in the detection process. This attention mechanism can be implemented as an efficient network module that captures non-local dependencies between features. This network module can be inserted at different stages of an object detection network, and the entire detector can be trained end-to-end. We also introduce a large-scale annotated hand dataset containing hands in unconstrained images for training and evaluation. We show that Hand-CNN outperforms existing methods on several datasets, including our hand detection benchmark and the publicly available PASCAL VOC human layout challenge. We also conduct ablation studies on hand detection to show the effectiveness of the proposed contextual attention module., 9 pages, 9 figures

Trajectory-pooled Deep-learning Descriptors have been the state-of-the-art feature descriptors for human action recognition in video on many datasets. This paper improves their performance by applying the proposed eigen-evolution pooling to each trajectory, encoding the temporal evolution of deep learning features computed along the trajectory. This leads to Eigen-Evolution Trajectory (EET) descriptors, a novel type of video descriptor that significantly outperforms Trajectory-pooled Deep-learning Descriptors. EET descriptors are defined based on dense trajectories, and they provide complimentary benefits to video descriptors that are not based on trajectories. Empirically, we observe that the combination of EET descriptors and VideoDarwin outperforms the state-of-the-art methods on the Hollywood2 dataset, and its performance on the UCF101 dataset is close to the state-of-the-art.

The National Synchrotron Light Source II (NSLS-II) at Brookhaven National Laboratory (BNL) is now providing some of the world's brightest x-ray beams. A suite of imaging and diffraction methods, exploiting megapixel detectors with kilohertz frame-rates at NSLS-II beamlines, generate a variety of image streams in unprecedented velocities and volumes. A complete understanding of a complex material system often requires a cluster of x-ray characterization tools that can reveal its elemental, structural, chemical and physical properties at different length-scales and time-scales. The flourish and continuing refinement of x-ray probes enable that the same sample may be studied with different perspectives and granularities, and at different time and locations; these powerful tools generate a correspondingly daunting big data challenge, with multiple image streams that outpaces any manual efforts and traditional data analysis practice. In this paper, we applied deep learning methods, in particular, deep convolutional neural network (CNN) to automatically recognize image features from image streams from NSLS-II, and integrated our deep-learning methods into the Google Tensorflow to cluster and label both real and synthetic 2-D scattering image patterns. These methods would empower scientists by providing timely insights, allowing them to steer experiments efficiently during their precious x-ray beamtime allocation. Experiment shows that the CNN-based image labeling attains a 10% improvement over traditional K-mean and Support Vector Machine.

Selecting relevant features for support vector machine (SVM) classifiers is important for a variety of reasons such as generalization performance, computational efficiency, and feature interpretability. Traditional SVM approaches to feature selection typically extract features and learn SVM parameters independently. Independently performing these two steps might result in a loss of information related to the classification process. This paper proposes a convex energy-based framework to jointly perform feature selection and SVM parameter learning for linear and non-linear kernels. Experiments on various databases show significant reduction of features used while maintaining classification performance.

Image alignment has been a long standing problem in computer vision. Parameterized Appearance Models (PAMs) such as the Lucas-Kanade method, Eigentracking, and Active Appearance Models are commonly used to align images with respect to a template or to a previously learned model. While PAMs have numerous advantages relative to alternate approaches, they have at least two drawbacks. First, they are especially prone to local minima in the registration process. Second, often few, if any, of the local minima of the cost function correspond to acceptable solutions. To overcome these problems, this paper proposes a method to learn a metric for PAMs that explicitly optimizes that local minima occur at and only at the places corresponding to the correct fitting parameters. To the best of our knowledge, this is the first paper to address the problem of learning a metric to explicitly model local properties of the PAMs' error surface. Synthetic and real examples show improvement in alignment performance in comparison with traditional approaches. In addition, we show how the proposed criteria for a good metric can be used to select good features to track.

This chapter will demonstrate the various robotics applications that can be achieved using evolutionary multi-objective optimization (EMO) techniques. The main objective of this chapter is to demonstrate practical ways of generating simple legged locomotion for simulated robots with two, four and six legs using EMO. The operational performance as well as complexities of the resulting evolved Pareto solutions that act as controllers for these robots will then be analyzed. Additionally, the operational dynamics of these evolved Pareto controllers in noisy and uncertain environments, limb dynamics and effects of using a different underlying EMO algorithm will also be discussed.

Automatic facial action unit (AU) detection from video is a long-standing problem in computer vision. Two main approaches have been pursued: (1) static modeling — typically posed as a discriminative classification problem in which each video frame is evaluated independently; (2) temporal modeling — frames are segmented into sequences and typically modeled with a variant of dynamic Bayesian networks. We propose a segment-based approach, kSeg-SVM, that incorporates benefits of both approaches and avoids their limitations. kSeg-SVM is a temporal extension of the spatial bag-of-words. kSeg-SVM is trained within a structured output SVM framework that formulates AU detection as a problem of detecting temporal events in a time series of visual features. Each segment is modeled by a variant of the BoW representation with soft assignment of the words based on similarity. Our framework has several benefits for AU detection: (1) both dependencies between features and the length of action units are modeled; (2) all possible segments of the video may be used for training; and (3) no assumptions are required about the underlying structure of the action unit events (e.g., i.i.d.). Our algorithm finds the best k-or-fewer segments that maximize the SVM score. Experimental results suggest that the proposed method outperforms state-of-the-art static methods for AU detection.

Current methods of assessing psychopathology depend almost entirely on verbal report (clinical interview or questionnaire) of patients, their family, or caregivers. They lack systematic and efficient ways of incorporating behavioral observations that are strong indicators of psychological disorder, much of which may occur outside the awareness of either individual. We compared clinical diagnosis of major depression with automatically measured facial actions and vocal prosody in patients undergoing treatment for depression. Manual FACS coding, active appearance modeling (AAM) and pitch extraction were used to measure facial and vocal expression. Classifiers using leave-one-out validation were SVM for FACS and for AAM and logistic regression for voice. Both face and voice demonstrated moderate concurrent validity with depression. Accuracy in detecting depression was 88% for manual FACS and 79% for AAM. Accuracy for vocal prosody was 79%. These findings suggest the feasibility of automatic detection of depression, raise new issues in automated facial image analysis and machine learning, and have exciting implications for clinical theory and practice.

Visual categorization problems, such as object classification or action recognition, are increasingly often approached using a detection strategy: a classifier function is first applied to candidate subwindows of the image or the video, and then the maximum classifier score is used for class decision. Traditionally, the subwindow classifiers are trained on a large collection of examples manually annotated with masks or bounding boxes. The reliance on time-consuming human labeling effectively limits the application of these methods to problems involving very few categories. Furthermore, the human selection of the masks introduces arbitrary biases (e.g. in terms of window size and location) which may be suboptimal for classification. In this paper we propose a novel method for learning a discriminative subwindow classifier from examples annotated with binary labels indicating the presence of an object or action of interest, but not its location. During training, our approach simultaneously localizes the instances of the positive class and learns a subwindow SVM to recognize them. We extend our method to classification of time series by presenting an algorithm that localizes the most discriminative set of temporal segments in the signal. We evaluate our approach on several datasets for object and action recognition and show that it achieves results similar and in many cases superior to those obtained with full supervision.

Automatic facial feature localization has been a long-standing challenge in the field of computer vision for several decades. This can be explained by the large variation a face in an image can have due to factors such as position, facial expression, pose, illumination, and background clutter. Support Vector Machines (SVMs) have been a popular statistical tool for facial feature detection. Traditional SVM approaches to facial feature detection typically extract features from images (e.g. multiband filter, SIFT features) and learn the SVM parameters. Independently learning features and SVM parameters might result in a loss of information related to the classification process. This paper proposes an energy-based framework to jointly perform relevant feature weighting and SVM parameter learning. Preliminary experiments on standard face databases have shown significant improvement in speed with our approach.

Active appearance models (AAMs) have been extensively used for face alignment during the last 20 years. While AAMs have numerous advantages relative to alternate approaches, they suffer from two major drawbacks: (i) AAMs are especially prone to local minima in the fitting process; (ii) few if any of the local minima of the cost function correspond to acceptable solutions. To minimize these problems, this paper proposes a method to learn the fitting cost function that explicitly optimizes that the local minima occur at and only at the places corresponding to the correct fitting parameters. The paper explores two methods to parameterize the cost function: pixel weighting and subspace learning. Experiments on synthetic and real data show the effectiveness of our approach for face alignment.

Parameterized appearance models (PAMs) (e.g. eigen-tracking, active appearance models, morphable models) use principal component analysis (PCA) to model the shape and appearance of objects in images. Given a new image with an unknown appearance/shape configuration, PAMs can detect and track the object by optimizing the modelpsilas parameters that best match the image. While PAMs have numerous advantages for image registration relative to alternative approaches, they suffer from two major limitations: First, PCA cannot model non-linear structure in the data. Second, learning PAMs requires precise manually labeled training data. This paper proposes parameterized kernel principal component analysis (PKPCA), an extension of PAMs that uses Kernel PCA (KPCA) for learning a non-linear appearance model invariant to rigid and/or non-rigid deformations. We demonstrate improved performance in supervised and unsupervised image registration, and present a novel application to improve the quality of manual landmarks in faces. In addition, we suggest a clean and effective matrix formulation for PKPCA.

Parameterized appearance models (PAMs) (e.g. eigen-tracking, active appearance models, morphable models) are commonly used to model the appearance and shape variation of objects in images. While PAMs have numerous advantages relative to alternate approaches, they have at least two drawbacks. First, they are especially prone to local minima in the fitting process. Second, often few if any of the local minima of the cost function correspond to acceptable solutions. To solve these problems, this paper proposes a method to learn a cost function by explicitly optimizing that the local minima occur at and only at the places corresponding to the correct fitting parameters. To the best of our knowledge, this is the first paper to address the problem of learning a cost function to explicitly model local properties of the error surface to fit PAMs. Synthetic and real examples show improvement in alignment performance in comparison with traditional approaches.

Many categories of objects, such as human faces, can be naturally viewe d as a composition of several different layers. For example, a bearded face with glasses can be decomposed in three layers: a layer for glasses, a layer for the beard and a layer for other permanent facial features. While m odeling such a face with a linear subspace model could be very difficult, layer separation allows for easymodeling and modification of some certain structures while leaving others unchanged. In this paper, we present a method for automatic layer extraction and its applications to face synthesis and editing. Layers are automatically extracted by utilizing the differences between subspaces and modeled separately. We show that our method can be us ed for tasks such beard removal (virtual shaving), beard synthesis, and beard transfer, among others.

SharedPlans is an agent teamwork model that provides a formalization of the conditions under which a group of agents has a collaborative plan. This paper describes a general framework for implementing SharedPlans theory that addresses the computational issues of team formation, group plan elaboration and plan execution, involving coordination, communication and monitoring. The framework includes a team-oriented programming language for specifying recipes for SharedPlans, and an extension to a BDI architecture with several meta-plans for interpreting the plan language. We indicate how the formal requirements for the establishment of SharedPlans are fulfilled within the framework.