Search

This dissertation consists of three chapters on semi-parametric Bayesian Econometric methods. Chapter 1 applies a semi-parametric method to demand systems, and compares the abilities to recover the true elasticities of different approaches to linearly estimating the widely used Almost Ideal demand model, by either iteration or approximation. Chapter 2 co-authored with Dr. Melvyn Weeks introduces a new semi-parametric Bayesian Generalized Least Square estimator, which employs the Dirichlet Process prior to cope with potential heterogeneity in the error distributions. Two methods are discussed as special cases of the GLS estimator, the Seemingly Unrelated Regression for equation systems, and the Random Effects Model for panel data, which can be applied to many fields such as the demand analysis in Chapter 1. Chapter 3 focuses on the subset selection for the efficiencies of firms, which addresses the influence of heterogeneity in the distributions of efficiencies on subset selections by applying the semi-parametric Bayesian Random Effects Model introduced in Chapter 2.

In this dissertation we examine two topics relevant to modern machine learning research: 1) Subgraph counting and 2) High-dimensional subset selection. The former can be used to construct features for performing graph analytics. The latter has applications in sparse modeling such as feature selection, sparse regression, and interpretable machine learning. Since these problems become intractable for large datasets, we design efficient approximation algorithms for both tasks with data-dependent performance guarantees. In the first part of the dissertation, we study the problem of approximating all three and four node induced subgraphs in a large graph. These counts are called the 3 and 4-profile, respectively, and describe a graph's connectivity properties. This problem generalizes graphlet counting and has found applications ranging from bioinformatics to spam detection. Our algorithms use the novel concept of graph profile sparsifiers: sparse graphs that can be used to approximate the full profile counts for a given large graph. We obtain novel concentration results showing that graphs can be substantially sparsified and still retain good approximation quality for the global graph profile. We also study the problem of counting local and ego profiles centered at each vertex of the graph. These quantities embed every vertex into a low-dimensional space that characterizes the local geometry of its neighborhood. We introduce the concept of edge pivots and show that all local 3 and 4-profiles can be computed as vertex programs using compressed two-hop information. Our algorithms are local, distributed message-passing schemes and compute all graph profiles in parallel. We empirically evaluate these algorithms with a distributed GraphLab implementation, and show improvements over previous state-of-the-art in experiments scaling up to 640 cores on Amazon EC2. In the second part we shift to the problem of subset selection: for example, selecting a few features from a large feature set. Motivated by the need for interpretable, nonlinear regression models for high-dimensional data, we draw a novel connection between this and submodular maximization. We extend an earlier concept of weak submodularity from the setting of sparse linear regression to a broad class of objective functions, including generalized linear model likelihoods. We then show that three greedy algorithms (Oblivous, Forward Stepwise, and Orthogonal Matching Pursuit) perform within a constant factor from the best possible subset. Our methods do not require any statistical modeling assumptions and allow direct control over the number of obtained features. This contrasts with other work that uses regularization parameters to control sparsity only implicitly. Our proof technique connecting convex analysis and submodular set function theory may be of independent interest for other statistical learning applications that have combinatorial structure. In the third part, we consider the problem of explaining the predictions of a given black-box classifier. For example, why does a deep neural network assign an image to a particular class? We cast interpretability of black-box classifiers as a subset selection problem and propose to solve it with an efficient streaming algorithm. We provide a constant factor approximation guarantee for this algorithm in the case of a random stream order and a weakly submodular objective function. This is the first such theoretical guarantee for this general class of functions, and we also show that no such algorithm exists for a worst case stream order. Our algorithm obtains similar explanations of Inception V3 predictions 10 times faster than the state-of-the-art LIME framework.

Most of the computer vision methods assume that data will be labeled and available beforehand in order to train a good recognition model. However, it becomes infeasible and unrealistic to know all the labels beforehand with the huge corpus of visual data being generated on a daily basis. In most image and video analysis tasks, selection of the most informative samples from a huge pool of training data in order to learn a good recognition model is an active research problem. Furthermore, it is also useful to reduce the annotation cost, as it is time-consuming to annotate unlabeled samples. In this thesis, we aim to design information-theoretic approaches which exploit inter-relationships between data instances in order to find informative samples in image or videos. Moreover, in recent years, the advent of high-tech journaling tools facilitates an image to be forged in a way that can easily evade state-of-the-art image tampering detection approaches. The recent success of the deep learning approaches in different recognition tasks inspires us to develop a high-confidence detection framework which can localize forged/manipulated regions in an image. Unlike semantic object segmentation where all meaningful regions (objects) are segmented, the localization of image forgery focuses only the possible tampered region which makes the problem even more challenging.We present two distinct information-theoretic approaches for selecting samples to learn recognition models, and a deep learning based method for localizing manipulation from images. In first approach, we show how models for joint scene and object classification can be learned online. A major motivation for this approach is to exploit the hierarchical relationships between scenes and objects, represented as a graphical model, in an active learning framework. To select the samples on graph, which need to be labeled by a human, we formulate an optimization function that reduces the joint entropy of scene and object variables. The second approach we propose is motivated by the theories in data compression, which exploits the concept of typicality from the domain of information theory in order to find informative samples in videos. Typicality is a simple and powerful technique which can be applied to compress the training data to learn a good classification model. Both of the approaches lead to a significant reduction in the amount of manual labeling effort for similar or better performance when compared with a model trained with the full dataset. In the final chapter, we explore a deep learning architecture to localize manipulated regions from an image. Our proposed framework utilizes resampling features, Long-Short Term Memory (LSTM) cells, and encoder-decoder network to segment out manipulated regions from non-manipulated ones. The overall framework is capable of detecting different types of image forgeries.

Orthostatic stress tests, such as postural change from sitting to standing and head-up tilt, are common noninvasive procedures used to study short-term regulation of the cardiovascular system. The regulation involves a complex chain of reactions including regulation of heart rate, the topic of this dissertation. This dissertation focuses on developing and analyzing a mathematical model predicting heart rate regulation in response to changes in blood pressure during postural changes from sitting to standing and during head-up tilt. Two studies will be analyzed: 1) sit-to-stand versus head-up tilt and 2) healthy elderly versus elderly diabetic. Both studies are based on the same mathematical model predicting heart rate using a series of nonlinear delay differential equations with blood pressure as an input. The model is adapted to individuals by estimation of an optimal set of parameters and validated against clinical data. Model complexity is a major obstacle to parameter estimation. Therefore, a focus of this research is on model reduction. In particular, our approach is to reduce the number of estimated parameters, but not at the expense of calibrating the model to data. We use a sensitivity analysis to rank the parameters from the most to least sensitive. The parameters that are insensitive are kept fixed during the parameter estimation procedure. In addition, we use subset selection to identify a subset of model parameters that are most identifiable. These modeling techniques allow us to identify a set of sensitive and identifiable parameters that could be estimated using nonlinear optimization procedures. The resulting model and parameter estimation techniques allow us to predict heart rate during sit-to-stand and head-up tilt. Using data from five healthy elderly subjects we are able to determine significant differences between the two groups of subjects. In the second study, a comparison of healthy elderly and subjects with diabetes during sit-to-stand reveals that the sympathetic response is delayed longer for subjects with diabetes than for healthy elderly subjects, and that vestibular, muscle sympathetic nerve, and central command response is more pronounced in healthy elderly subjects than in subjects with diabetes.

In this dissertation, we extend several relatively new developments in statistical model selection and data mining in order to improve one of the workhorse statistical tools - mixture modeling (Pearson, 1894). The traditional mixture model assumes data comes from several populations of Gaussian distributions. Thus, what remains is to determine how many distributions, their population parameters, and the mixing proportions. However, real data often do not fit the restrictions of normality very well. It is likely that data from a single population exhibiting either asymmetrical or nonnormal tail behavior could be erroneously modeled as two populations, resulting in suboptimal decisions. To avoid these pitfalls, we develop the mixture model under a broader distributional assumption by fitting a group of multivariate elliptically-contoured distributions (Anderson and Fang, 1990; Fang et al., 1990). Special cases include the multivariate Gaussian and power exponential distributions, as well as the multivariate generalization of the Student’s T. This gives us the flexibility to model nonnormal tail and peak behavior, though the symmetry restriction still exists. The literature has many examples of research generalizing the Gaussian mixture model to other distributions (Farrell and Mersereau, 2004; Hasselblad, 1966; John, 1970a), but our effort is more general. Further, we generalize the mixture model to be non-parametric, by developing two types of kernel mixture model. First, we generalize the mixture model to use the truly multivariate kernel density estimators (Wand and Jones, 1995). Additionally, we develop the power exponential product kernel mixture model, which allows the density to adjust to the shape of each dimension independently. Because kernel density estimators enforce no functional form, both of these methods can adapt to nonnormal asymmetric, kurtotic, and tail characteristics. Over the past two decades or so, evolutionary algorithms have grown in popularity, as they have provided encouraging results in a variety of optimization problems. Several authors have applied the genetic algorithm - a subset of evolutionary algorithms - to mixture modeling, including Bhuyan et al. (1991), Krishna and Murty (1999), and Wicker (2006). These procedures have the benefit that they bypass computational issues that plague the traditional methods. We extend these initialization and optimization methods by combining them with our updated mixture models. Additionally, we “borrow” results from robust estimation theory (Ledoit and Wolf, 2003; Shurygin, 1983; Thomaz, 2004) in order to data-adaptively regularize population covariance matrices. Numerical instability of the covariance matrix can be a significant problem for mixture modeling, since estimation is typically done on a relatively small subset of the observations. We likewise extend various information criteria (Akaike, 1973; Bozdogan, 1994b; Schwarz, 1978) to the elliptically-contoured and kernel mixture models. Information criteria guide model selection and estimation based on various approximations to the Kullback-Liebler divergence. Following Bozdogan (1994a), we use these tools to sequentially select the best mixture model, select the best subset of variables, and detect influential observations - all without making any subjective decisions. Over the course of this research, we developed a full-featured Matlab toolbox (M3) which implements all the new developments in mixture modeling presented in this dissertation. We show results on both simulated and real world datasets. Keywords: mixture modeling, nonparametric estimation, subset selection, influence detection, evidence-based medical diagnostics, unsupervised classification, robust estimation.

The parameter identification problem attempts to find parameter values that cause the solution of a predictive model to match data. In this work, parameters in cardiovascular and respiratory models are identified. This work’s main contribution is in its application of gradient based optimization techniques and insight into methods to identify parameters that can be estimated given subject specific data. The models presented in this paper are lumped compartment models of the cardiovascular and respiratory systems. Lumped compartment models treat the cardiovascular and respiratory systems as collections of interconnected compartments transporting blood and exchanging oxygen and carbon dioxide. Using these compartments, a system of ordinary differential equations (ODE) is generated that incorporates several physiological parameters representing vascular resistances, compliances, and tissue metabolic rates. The solution to this ODE system is used to predict cerebral blood flow, systemic arterial blood pressure, and expired carbon dioxide partial pressures, which are then compared to subject data. Minimizing the two-norm difference between between the result of the predictive model and the experimental data is a non-linear least squares problem. Although the least squares problem is overdetermined, the data do not contain enough information to determine all model parameters. A combination of sensitivity analysis, expert knowledge, and subset selection techniques reduce the number of model parameters estimated.

Many variable selection procedures have been developed in the literature for linear regression models. We propose a new and general approach, the False Selection Rate (FSR) method, to control variable selection with the advantage of being applicable to a broader class of regression models; for example, binary regression, Poisson regression, etc. By adding a number of pseudo-variables to the real set of data and monitoring the proportion of pseudo-variables falsely selected in the model, we are able to control the model false selection rate, selecting as many important variables as possible while selecting a relatively low proportion of false important ones. We focus on forward selection because it is applicable in the case where there are more variables than observations. Due to the difficulty of obtaining analytical results, we study our approach by Monte Carlo and compare it with a variety of commonly used procedures. We first focus on linear regression models, and then extend the approach to logistic regression models. The new method is illustrated on four real data sets.