Search

This document summarizes the technology, procedures, and database organization of the CMU Multi-Modal Activity Database (CMU-MMAC). The CMU-MMAC database contains multimodal measures of the human activity of subjects performing the tasks involved in cooking and food preparation. The CMU-MMAC database was collected in Carnegie Mellon University’s Motion Capture Lab. A kitchen was built and to date five subjects have been recorded cooking five different recipes: brownies, pizza, sandwich, salad and scrambled eggs. The following modalities were recorded: • Video: (1) Three high spatial resolution (1024 × 768) color video cameras at low temporal resolution (30 Hertz). (2) Two low spatial resolution (640 × 480) color video cameras at high temporal resolution (60 Hertz). (3) One wearable low spatial resolution (640×480) camera at low temporal resolution (12 Hertz). • Audio: (1) Five balanced microphones. (2) Wearable watch. • Motion capture: A Vicon motion capture system with 12 infrared MX-40 cameras. Each camera records images of 4 megapixel resolution at 120 Hertz. • Five 3-axis accelerometers and gyroscopes. Several computers were used for recording the various modalities. The computers were synchronized using the Network Time Protocol (NTP).

Kernel Principal Component Analysis (KPCA) is a popular generalization of linear PCA that allows non-linear feature extraction. In KPCA, data in the input space is mapped to higher (usually) dimensional feature space where the data can be linearly modeled. The feature space is typically induced implicitly by a kernel function, and linear PCA in the feature space is performed via the kernel trick. However, due to the implicitness of the feature space, some extensions of PCA such as robust PCA cannot be directly generalized to KPCA. This paper presents a technique to overcome this problem, and extends it to a unified framework for treating noise, missing data, and outliers in KPCA. Our method is based on a novel cost function to perform inference in KPCA. Extensive experiments, in both synthetic and real data, show that our algorithm outperforms existing methods.

Appearance Models (AM) are commonly used to model appearance and shape variation of objects in images. In particular, they have proven useful to detection, tracking, and synthesis of people’s faces from video. While AM have numerous advantages relative to alternative approaches, they have at least two important drawbacks. First, they are especially prone to local minima in fitting; this problem becomes increasingly problematic as the number of parameters to estimate grows. Second, often few if any of the local minima correspond to the correct location of the model error. To address these problems, we propose Filtered Component Analysis (FCA), an extension of traditional Principal Component Analysis (PCA). FCA learns an optimal set of filters with which to build a multi-band representation of the object. FCA representations were found to be more robust than either grayscale or Gabor filters to problems of local minima. The effectiveness and robustness of the proposed algorithm is demonstrated in both synthetic and real data.

Meetings are a very important part of every days life for professionals working in universities, companies or governmental institutions. In fact, it is estimated that a midlevel manager or professional spends around 35% of his/her time in meetings. We have designed a physical awareness system called CAMEO (Camera Assisted Meeting Event Observer), a hardware/software component to record and monitor people’s activities in meetings. CAMEO captures the audio and a high resolution omnidirectional view of the meeting by stitching images coming from almost concentric cameras. Besides recording capabilities, CAMEO automatically detects people and automatically learns a personspecific facial appearance model (PSFAM) for each of the participants. The PSFAMs allow more robust, reliable and faster tracking and identification. Several novelties in the video capturing device, multiple person identification and tracking are proposed. The effectiveness and robustness of the proposed system is demonstrated over several real time experiments and a large data set of videos.

Calcineurin B-like interacting protein kinases (CIPKs) decode calcium signals upon interaction with the calcium sensors calcineurin B like proteins into phosphorylation events that result into adaptation to environmental stresses. Few phosphorylation targets of CIPKs are known and therefore the molecular mechanisms underlying their downstream output responses are not fully understood. Tomato (Solanum lycopersicum) Cipk6 regulates immune and susceptible Programmed cell death in immunity transforming Ca signals into reactive oxygen species (ROS) signaling. To investigate SlCipk6-induced molecular mechanisms and identify putative substrates, a yeast two-hybrid approach was carried on and a protein was identified that contained a Universal stress protein (Usp) domain present in bacteria, protozoa and plants, which we named “SlRd2”. SlRd2 was an ATP-binding protein that formed homodimers in planta. SlCipk6 and SlRd2 interacted using coimmunoprecipitation and bimolecular fluorescence complementation (BiFC) assays in Nicotiana benthamiana leaves and the complex localized in the cytosol. SlCipk6 phosphorylated SlRd2 in vitro, thus defining, to our knowledge, a novel target for CIPKs. Heterologous SlRd2 overexpression in yeast conferred resistance to highly toxic LiCl, whereas SlRd2 expression in Escherichia coli UspA mutant restored bacterial viability in response to HO treatment. Finally, transient expression of SlCipk6 in transgenic N. benthamiana SlRd2 overexpressors resulted in reduced ROS accumulation as compared to wild-type plants. Taken together, our results establish that SlRd2, a tomato UspA, is, to our knowledge, a novel interactor and phosphorylation target of a member of the CIPK family, SlCipk6, and functionally regulates SlCipk6-mediated ROS generation.

Endotracheal intubation (ETI) is a very crucial medical procedure performed on critically ill patients. It involves insertion of a breathing tube into the trachea i.e. the windpipe connecting the larynx and the lungs. Often, this procedure is performed by the paramedics (aka providers) under challenging prehospital settings e.g. roadside, ambulances or helicopters. Successful intubations could be lifesaving, whereas, failed intubation could potentially be fatal. Under prehospital environments, ETI success rates among the paramedics are surprisingly low and this necessitates better training and performance evaluation of ETI skills. Currently, few objective metrics exist to quantify the differences in ETI techniques between providers of various skill levels. In this pilot study, we develop a quantitative framework for discriminating the kinematic characteristics of providers with different experience levels. The system utilizes statistical analysis on spatio-temporal multimodal features extracted from optical motion capture, accelerometers and electromyography (EMG) sensors. Our experiments involved three individuals performing intubations on a dummy, each with different levels of expertise. Quantitative performance analysis on multimodal features revealed distinctive differences among different skill levels. In future work, the feedback from these analysis could potentially be harnessed towards enhancing ETI training.

Over the last century, Component Analysis (CA) methods such as Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Canonical Correlation Analysis (CCA), Locality Preserving Projections (LPP), and Spectral Clustering (SC) have been extensively used as a feature extraction step for modeling, classification, visualization, and clustering. CA techniques are appealing because many can be formulated as eigen-problems, offering great potential for learning linear and nonlinear representations of data in closed-form. However, the eigen-formulation often conceals important analytic and computational drawbacks of CA techniques, such as solving generalized eigen-problems with rank deficient matrices (e.g., small sample size problem), lacking intuitive interpretation of normalization factors, and understanding commonalities and differences between CA methods. This paper proposes a unified least-squares framework to formulate many CA methods. We show how PCA, LDA, CCA, LPP, SC, and its kernel and regularized extensions correspond to a particular instance of least-squares weighted kernel reduced rank regression (LS--WKRRR). The LS-WKRRR formulation of CA methods has several benefits: 1) provides a clean connection between many CA techniques and an intuitive framework to understand normalization factors; 2) yields efficient numerical schemes to solve CA techniques; 3) overcomes the small sample size problem; 4) provides a framework to easily extend CA methods. We derive weighted generalizations of PCA, LDA, SC, and CCA, and several new CA techniques.

Active Appearance Models (AAMs) have been extensively used for face alignment during the last 20 years. While AAMs have numerous advantages relative to alternate approaches, they suffer from two major drawbacks: (i) AAMs are especially prone to local minima in the fitting process; (ii) few if any of the local minima of the cost function correspond to acceptable solutions. To minimize these problems, this paper proposes a method to learn the fitting cost function that explicitly optimizes that the local minima occur at and only at the places corresponding to the correct fitting parameters. The paper explores two methods to parameterize the cost function: pixel weighting and subspace learning. Experiments on synthetic and real data show the effectiveness of our approach for face alignment.

Parameterized Appearance Models (PAMs) (e.g. eigentracking, active appearance models, morphable models) use Principal Component Analysis (PCA) to model the shape and appearance of objects in images. Given a new image with an unknown appearance/shape configuration, PAMs can detect and track the object by optimizing the model’s parameters that best match the image. While PAMs have numerous advantages for image registration relative to alternative approaches, they suffer from two major limitations: First, PCA cannot model non-linear structure in the data. Second, learning PAMs requires precise manually labeled training data. This paper proposes Parameterized Kernel Principal Component Analysis (PKPCA), an extension of PAMs that uses Kernel PCA (KPCA) for learning a non-linear appearance model invariant to rigid and/or non-rigid deformations. We demonstrate improved performance in supervised and unsupervised image registration, and present a novel application to improve the quality of manual landmarks in faces. In addition, we suggest a clean and effective matrix formulation for PKPCA.

Parameterized Appearance Models (PAMs) (e.g. Eigen-tracking, Active Appearance Models, Morphable Models) are commonly used to model the appearance and shapevariation of objects in images. While PAMs have numerous advantages relative to alternate approaches, they have at least two drawbacks. First, they are especially prone to local minima in the fitting process. Second, often few if any of the local minima of the cost function correspond to acceptable solutions. To solve these problems, this paper proposes a method to learn a cost function by explicitly optimizing that the local minima occur at and only at the places corresponding to the correct fitting parameters. To the best of our knowledge, this is the first paper to address the problem of learning a cost function to explicitly model local properties of the error surface to fit PAMs. Synthetic and real examples show improvement in alignment performance in comparison with traditional approaches.

Kernel machines (e.g. SVM, KLDA) have shown state-ofthe- art performance in several visual classification tasks. The classification performance of kernel machines greatly depends on the choice of kernels and its parameters. In this paper, we propose a method to search over a space of parameterized kernels using a gradient-descent based method. Our method effectively learns a non-linear representation of the data useful for classification and simultaneously performs dimensionality reduction. In addition, we suggest a new matrix formulation that simplifies and unifies previous approaches. The effectiveness and robustness of the proposed algorithm is demonstrated in both synthetic and real examples of pedestrian and mouth detection in images.

Meetings are an integral part of business life. In previous work, we have developed a physical awareness system called CAMEO (Camera Assisted Meeting Event Observer) to record and process audio/visual information of a meeting. A very important task in meeting understanding is to know who is attending to the meeting and CAMEO's task is to infer people's identity from video. In this paper, we present an approach to identify people from an omnidirectional video sequence. Two main novelties are proposed: First a new dimensionality reduction technique MODA (Multimodal Oriented Discriminant Analysis) is used to perform fast matching and second we show that using multiple spatiotemporal constraints the recognition performance greatly improves. The effectiveness and robustness of the proposed system is demonstrated over several real time experiments and a large data set of videos.

Static cameras are pervasive in a variety of environments. However it remains a challenging problem to extract and reason about high-level features from real-time and continuous observation of an environment. In this paper, we present CAMEO, the Camera Assisted Meeting Event Observer, which is a physical awareness system designed for use by an agent-based electronic assistant. CAMEO is an inexpensive high-resolution omnidirectional vision system designed to be used in meeting environments. The multiple camera design achieves the desired high image resolution and lower cost that can be achieved when compared to traditional omnicameras that make use of a single camera and mirror solution.

Linear discriminant analysis (LDA) has been an active topic of research during the last century. However, the existing algorithms have several limitations when applied to visual data. LDA is only optimal for gaussian distributed classes with equal covariance matrices and just classes-1 features can be extracted. On the other hand, LDA does not scale well to high dimensional data (over-fitting) and it does not necessarily minimize the classification error. In this paper, we introduce Oriented Discriminant Analysis (ODA), a LDA extension which can overcome these drawbacks. Three main novelties are proposed: • An optimal dimensionality reduction which maximizes the Kullback- Liebler divergence between classes is proposed. This allows us to model class covariances and to extract more than classes-1 features. • Several covariance approximations are introduced to improve classification in the small sample case. • A linear time iterative majorization method is introduced in order to find a local optimal solution. Several synthetic and real experiments on face recognition are reported.

We present a meeting recorder infrastructure used to record and annotate events that occur in meetings. Multiple data streams are recorded and analyzed in order to infer a higher-level state of the group’s activities. We describe the hardware and software systems used to capture people’s activities as well as the methods used to characterize them.

A 250/101(11)

Wearable headsets for interacting in virtual worlds are becoming more and more popular as new interaction platforms, such as Metaverse, are rising. An opportunity to overcome communication gaps between the Deaf and Hearing communities is presented. Sign language recognition, translation and production are challenging problems difficult to assess, among other reasons, due to the absence of good datasets. Towards this end, in 2021 How2Sign was published, a multimodal and multiview continuous American Sign Language dataset. In this project we introduce, EgoSign, an egocentric dataset built up as an extension of the existing How2Sign. This new dataset uses the hand tracking functionality of the new Oculus Quest 2 headset to obtain high quality hand pose estimation, a central feature for sign language understanding that cannot be obtained from the How2Sign videos with the publicly available human pose estimators.