Search

Recent years have witnessed an increasing need for the automated classification of sequential data, such as activities of daily living, social media interactions, financial series, and others. With the continuous flow of new data, it is critical to classify the observations on-the-fly and without being limited by a predetermined number of classes. In addition, a model should be able to update its parameters in response to a possible evolution in the distributions of the classes. This compelling problem, however, does not seem to have been adequately addressed in the literature, since most studies focus on offline classification over predefined class sets. In this paper, we present a principled solution for this problem based on an adaptive online system leveraging Markov switching models and hierarchical Dirichlet process priors. This adaptive online approach is capable of classifying the sequential data over an unlimited number of classes while meeting the memory and delay constraints typical of streaming contexts. In this paper, we introduce an adaptive “learning rate” that is responsible for balancing the extent to which the model retains its previous parameters or adapts to new observations. Experimental results on stationary and evolving synthetic data and two video data sets, TUM Assistive Kitchen and collated Weizmann, show a remarkable performance in terms of segmentation and classification, particularly for sequences from evolutionary distributions and/or those containing previously unseen classes. [ABSTRACT FROM AUTHOR]

Sparse nonnegative matrix factorization (SNMF) aims to factorize a data matrix into two optimized nonnegative sparse factor matrices, which could benefit many tasks, such as document-word co-clustering. However, the traditional SNMF typically assumes the number of latent factors (i.e., dimensionality of the factor matrices) to be fixed. This assumption makes it inflexible in practice. In this paper, we propose a doubly sparse nonparametric NMF framework to mitigate this issue by using dependent Indian buffet processes (dIBP). We apply a correlation function for the generation of two stick weights associated with each column pair of factor matrices while still maintaining their respective marginal distribution specified by IBP. As a consequence, the generation of two factor matrices will be columnwise correlated. Under this framework, two classes of correlation function are proposed: 1) using bivariate Beta distribution and 2) using Copula function. Compared with the single IBP-based NMF, this paper jointly makes two factor matrices nonparametric and sparse, which could be applied to broader scenarios, such as co-clustering. This paper is seen to be much more flexible than Gaussian process-based and hierarchial Beta process-based dIBPs in terms of allowing the two corresponding binary matrix columns to have greater variations in their nonzero entries. Our experiments on synthetic data show the merits of this paper compared with the state-of-the-art models in respect of factorization efficiency, sparsity, and flexibility. Experiments on real-world data sets demonstrate the efficiency of this paper in document-word co-clustering tasks. [ABSTRACT FROM AUTHOR]

This paper proposes a Bayesian nonparametric framework for clustering axially symmetric data. Our approach is based on a Dirichlet processes mixture model with Watson distributions, which can also be considered as the infinite Watson mixture model. In this paper, first, we extend the finite Watson mixture model into its infinite counterpart based on the framework of truncated Dirichlet process mixture model with a stick-breaking representation. Second, we propose a coordinate ascent mean-field variational inference algorithm that can effectively learn the parameters of our model with closed-form solutions; Third, to cope with a massive data set, we develop a stochastic variational inference algorithm to learn the proposed model through the method of stochastic gradient ascent; Finally, the proposed nonparametric Bayesian model is evaluated through simulated axially symmetric data sets and a real-world application, namely, gene expression data clustering. [ABSTRACT FROM AUTHOR]

Markov blankets (MBs) in Bayesian networks (BNs) play an important role in both local causal discovery and large-scale BN structure learning. Almost all existing MB discovery algorithms are designed under the assumption of causal sufficiency, which states that there are no latent common causes for two or more of the observed variables in data. However, latent common causes are ubiquitous in many applications, and hence, this assumption is often violated in practice. Thus, developing algorithms for discovering MBs without assuming causal sufficiency is of practical significance, and it is crucial for causal structure learning in real-world data. In this paper, we focus on addressing this problem. Specifically, we adopt a maximal ancestral graph (MAG) model to represent latent common causes and the concept of MBs without assuming causal sufficiency. Then, we propose an effective and efficient algorithm to discover the MB of a target variable in an MAG. Using benchmark and real-world data sets, the experiments validate the algorithm proposed in this paper. [ABSTRACT FROM AUTHOR]

This paper determines the weight decay parameter value of a deep convolutional neural network (CNN) that yields a good generalization. To obtain such a CNN in practice, numerical trials with different weight decay values are needed. However, the larger the CNN architecture is, the higher is the computational cost of the trials. To address this problem, this paper formulates an analytical solution for the decay parameter through a proposed objective function in conjunction with Bayesian probability distributions. For computational efficiency, a novel method to approximate this solution is suggested. This method uses a small amount of information in the Hessian matrix. Theoretically, the approximate solution is guaranteed by a provable bound and is obtained by a proposed algorithm, where its time complexity is linear in terms of both the depth and width of the CNN. The bound provides a consistent result for the proposed learning scheme. By reducing the computational cost of determining the decay value, the approximation allows for the fast investigation of a deep CNN (DCNN) which yields a small generalization error. Experimental results show that our assumption verified with different DCNNs is suitable for real-world image data sets. In addition, the proposed method significantly reduces the time cost of learning with setting the weight decay parameter while achieving good classification performances. [ABSTRACT FROM AUTHOR]

In this paper, we introduce the concept of principal paths in data space; we show that this is a well-characterized problem from the point of view of cognition, and that it can lead to salient insights in the analyzed data enabling topological/holistic descriptions. These paths, interestingly, can be interpreted as local principal curves, and in this paper, we suggest that they are analogous to what, in the statistical mechanics realm, are called minimum free-energy paths. Here, we move that concept from physics to data space and compute them in both the original and the kernel space. The algorithm is a regularized version of the well-known $k$ -means clustering algorithm. The regularization parameter is derived via an in-sample model selection process based on the Bayesian evidence maximization. Interestingly, we show that this choice for the regularization parameter consistently leads to the same manifold even when changing the number of clusters. We apply the method to common data sets, dynamical systems, and, in particular, to molecular dynamics trajectories showing the generality, the usefulness of the approach and its superiority with respect to other related approaches. [ABSTRACT FROM AUTHOR]

This paper addresses the duality between the deterministic feed-forward neural networks (FF-NNs) and linear Bayesian networks (LBNs), which are the generative stochastic models representing probability distributions over the visible data based on a linear function of a set of latent (hidden) variables. The maximum entropy principle is used to define a unique generative model corresponding to each FF-NN, called projected belief network (PBN). The FF-NN exactly recovers the hidden variables of the dual PBN. The large- $N$ asymptotic approximation to the PBN has the familiar structure of an LBN, with the addition of an invertible nonlinear transformation operating on the latent variables. It is shown that the exact nature of the PBN depends on the range of the input (visible) data details for the three cases of input data range are provided. The likelihood function of the PBN is straightforward to calculate, allowing it to be used as a generative classifier. An example is provided in which a generative classifier based on the PBN has comparable performance to a deep belief network in classifying handwritten characters. In addition, several examples are provided that demonstrate the duality relationship, for example, by training networks from either side of the duality. [ABSTRACT FROM AUTHOR]

Variational Bayesian (VB) learning has been successfully applied to instantaneous blind source separation. However, the traditional VB learning is restricted to the separation of independent source signals. Moreover, it has the difficulty to recover source signals with a sizable number of samples because of its rapidly increasing computational requirement. To overcome such shortcomings, frame-based VB (FVB) learning is proposed to address both independent and dependent source separation with a large number of samples in this paper. Specifically, a Gaussian process (GP) is employed to model independent or dependent source signals. To our knowledge, GP has been only used to model each of independent source signals. For dependent source signals, this paper proposes a novel modeling process: initial source signals are zigzag concatenated into a long serial and GP is then used to model it. In order to obtain a reliable covariance function for GP, first, we apply singular value decomposition to give initial estimated source signals and then we select an appropriate covariance function with which GP can perfectly fit them. In order to alleviate the computational burden of VB learning, we split observed signals into frames, and then model and infer source signals for each frame. Compared with the state-of-the-art algorithms, the experimental results show that the FVB learning has potential to provide improvement in separation performance not only for independent source signals but also for dependent ones, especially for long data records. [ABSTRACT FROM AUTHOR]

This paper proposes a Bayesian nonparametric regression model of panel data for sequential pattern classification. The proposed method provides a flexible and parsimonious model that allows both time-independent spatial variables and time-dependent exogenous variables to be predictors. Not only this method improves the accuracy of parameter estimation for limited data, but also it facilitates model interpretation by identifying statistically significant predictors with hypothesis testing. Moreover, as the data length approaches infinity, posterior consistency of the model is guaranteed for general data-generating processes under regular conditions. The resulting model of panel data can also be used for sequential classification. The proposed method has been tested by numerical simulation, then validated on an econometric public data set, and subsequently validated for detection of combustion instabilities with experimental data that have been generated in a laboratory environment. [ABSTRACT FROM AUTHOR]

Multitask learning (MTL) aims to learn multiple tasks simultaneously through the interdependence between different tasks. The way to measure the relatedness between tasks is always a popular issue. There are mainly two ways to measure relatedness between tasks: common parameters sharing and common features sharing across different tasks. However, these two types of relatedness are mainly learned independently, leading to a loss of information. In this paper, we propose a new strategy to measure the relatedness that jointly learns shared parameters and shared feature representations. The objective of our proposed method is to transform the features of different tasks into a common feature space in which the tasks are closely related and the shared parameters can be better optimized. We give a detailed introduction to our proposed MTL method. Additionally, an alternating algorithm is introduced to optimize the nonconvex objection. A theoretical bound is given to demonstrate that the relatedness between tasks can be better measured by our proposed MTL algorithm. We conduct various experiments to verify the superiority of the proposed joint model and feature MTL method. [ABSTRACT FROM AUTHOR]

This paper presents a nearest neighbor partitioning method designed to improve the performance of a neural-network classifier. For neural-network classifiers, usually the number, positions, and labels of centroids are fixed in partition space before training. However, that approach limits the search for potential neural networks during optimization; the quality of a neural network classifier is based on how clear the decision boundaries are between classes. Although attempts have been made to generate floating centroids automatically, these methods still tend to generate sphere-like partitions and cannot produce flexible decision boundaries. We propose the use of nearest neighbor classification in conjunction with a neural-network classifier. Instead of being bound by sphere-like boundaries (such as the case with centroid-based methods), the flexibility of nearest neighbors increases the chance of finding potential neural networks that have arbitrarily shaped boundaries in partition space. Experimental results demonstrate that the proposed method exhibits superior performance on accuracy and average f-measure. [ABSTRACT FROM AUTHOR]

Most of the existing Bayesian clustering algorithms perform well on the balanced data. When the data are highly imbalanced, these Bayesian clustering algorithms tend to strongly favor the larger clusters, but provide a notably low detection of the smaller clusters. In this paper, we present an incremental local distribution-based clustering algorithm with the Bayesian adaptive resonance theory (ILBART). This algorithm is developed to adapt itself to a changing environment without using any predefined parameters. The algorithm not only accurately finds the clusters, even in data sets with a severely imbalanced distribution, but also efficiently processes the dynamic data according to the evolving relationships among the clusters. We test our proposed algorithm with experiments conducted on several imbalanced data sets. The experimental results show that our proposed algorithm performs well for clustering imbalanced data and can also obtain a better performance than many other relevant clustering algorithms in several performance indices. [ABSTRACT FROM AUTHOR]

This paper considers the binary classification with the probit model under the expectation–maximization (EM) algorithm. Usually, in the Bayesian approach of the probit model, the latent variables are introduced to handle with the intractable problem. For each training sample, there is a corresponding latent variable. However, the EM algorithm requires matrix inversions which demand for the expansive computational cost when the number of training samples is large. To overcome this problem, we employ the group latent-variable approach where for each of the training samples there are corresponding multiple latent variables instead of just one. The major advantage of this approach, which is originated from Bayesian backfitting, is that there are no requirements for matrix inversions in the EM algorithm for the probit model. In this paper, to obtain sparse classifiers the Laplacian prior is employed and the method to control the degree of sparseness is presented. Although the sparsity of the classifier is not determined in a full automatic way, it can be controlled by specifying just one parameter. In other words, we are free to choose the degree of sparseness. The proposed method is compared with support vector machine, relevance vector machine, and generalized LASSO. [ABSTRACT FROM AUTHOR]

This paper presents the branching approach, which can improve deterministic implementation methods (such as variational Bayesian inference and expectation propagation method) for Bayesian mixture distributions. This proposed approach utilizes a set of artificial conditions defined by using the latent variables of the mixture distribution. This condition set is updated iteratively by branching based on a condition selected from the previous condition set. The target approximate Bayesian inference is obtained by merging the approximate conditional inferences under each condition in the condition set. The proposed approach is compared with several standard implementation methods by using a numerical example and a real-world example. [ABSTRACT FROM AUTHOR]

Neural decoding, which aims to predict external visual stimuli information from evoked brain activities, plays an important role in understanding human visual system. Many existing methods are based on linear models, and most of them only focus on either the brain activity pattern classification or visual stimuli identification. Accurate reconstruction of the perceived images from the measured human brain activities still remains challenging. In this paper, we propose a novel deep generative multiview model for the accurate visual image reconstruction from the human brain activities measured by functional magnetic resonance imaging (fMRI). Specifically, we model the statistical relationships between the two views (i.e., the visual stimuli and the evoked fMRI) by using two view-specific generators with a shared latent space. On the one hand, we adopt a deep neural network architecture for visual image generation, which mimics the stages of human visual processing. On the other hand, we design a sparse Bayesian linear model for fMRI activity generation, which can effectively capture voxel correlations, suppress data noise, and avoid overfitting. Furthermore, we devise an efficient mean-field variational inference method to train the proposed model. The proposed method can accurately reconstruct visual images via Bayesian inference. In particular, we exploit a posterior regularization technique in the Bayesian inference to regularize the model posterior. The quantitative and qualitative evaluations conducted on multiple fMRI data sets demonstrate the proposed method can reconstruct visual images more accurately than the state of the art. [ABSTRACT FROM AUTHOR]

In this big data era, interpretable machine learning models are strongly demanded for the comprehensive analytics of large-scale multiclass data. Characterizing all features from such data is a key but challenging step to understand the complexity. However, existing feature selection methods do not meet this need. In this paper, to address this problem, we propose a Bayesian multiclass nonnegative matrix factorization (MC-NMF) model with structured sparsity that is able to discover ubiquitous and class-specific features. Variational update rules were derived for efficient decomposition. In order to relieve the need of model selection and stably describe feature patterns, we further propose MC-NMF with stability selection, an ensemble method that collectively detects feature patterns from many runs of MC-NMF using different hyperparameter values and training subsets. We assessed our models on both simulated count data and multitumor ribonucleic acid-seq data. The experiments revealed that our models were able to recover predefined feature patterns from the simulated data and identify biologically meaningful patterns from the pan-cancer data. [ABSTRACT FROM AUTHOR]

In this paper, we develop a novel variational Bayesian learning method for the Dirichlet process (DP) mixture of the inverted Dirichlet distributions, which has been shown to be very flexible for modeling vectors with positive elements. The recently proposed extended variational inference (EVI) framework is adopted to derive an analytically tractable solution. The convergency of the proposed algorithm is theoretically guaranteed by introducing single lower bound approximation to the original objective function in the EVI framework. In principle, the proposed model can be viewed as an infinite inverted Dirichlet mixture model that allows the automatic determination of the number of mixture components from data. Therefore, the problem of predetermining the optimal number of mixing components has been overcome. Moreover, the problems of overfitting and underfitting are avoided by the Bayesian estimation approach. Compared with several recently proposed DP-related methods and conventional applied methods, the good performance and effectiveness of the proposed method have been demonstrated with both synthesized data and real data evaluations. [ABSTRACT FROM AUTHOR]

Sparse Bayesian learning has emerged as a powerful tool to tackle various image classification tasks. The existing sparse Bayesian models usually use independent Gaussian distribution as the prior knowledge for the noise. However, this assumption often contradicts to the practical observations in which the noise is long tail and pixels containing noise are spatially correlated. To handle the practical noise, this paper proposes to partition the noise image into several 2-D groups and adopt the long-tail distribution, i.e., the scale mixture of the matrix Gaussian distribution, to model each group to capture the intragroup correlation of the noise. Under the nonparametric Bayesian estimation, the low-rank-induced prior and the matrix Gamma distribution prior are imposed on the covariance matrix of each group, respectively, to induce two Bayesian correlated group regression (BCGR) methods. Moreover, the proposed methods are extended to the case with unknown group structure. Our BCGR method provides an effective way to automatically fit the noise distribution and integrates the long-tail attribute and structure information of the practical noise into model. Therefore, the estimated coefficients are better for reconstructing the desired data. We apply BCGR to address image classification task and utilize the learned covariance matrices to construct a grouped Mahalanobis distance to measure the reconstruction residual of each class in the design of a classifier. Experimental results demonstrate the effectiveness of our new BCGR model. [ABSTRACT FROM AUTHOR]

Directional and pairwise measurements are often used to model interactions in a social network setting. The mixed-membership stochastic blockmodel (MMSB) was a seminal work in this area, and its ability has been extended. However, models such as MMSB face particular challenges in modeling dynamic networks, for example, with the unknown number of communities. Accordingly, this paper proposes a dynamic infinite mixed-membership stochastic blockmodel, a generalized framework that extends the existing work to potentially infinite communities inside a network in dynamic settings (i.e., networks are observed over time). Additional model parameters are introduced to reflect the degree of persistence among one’s memberships at consecutive time stamps. Under this framework, two specific models, namely mixture time variant and mixture time invariant models, are proposed to depict two different time correlation structures. Two effective posterior sampling strategies and their results are presented, respectively, using synthetic and real-world data. [ABSTRACT FROM AUTHOR]

Successful biological systems adapt to change. In this paper, we are principally concerned with adaptive systems that operate in environments where data arrives sequentially and is multivariate in nature, for example, sensory streams in robotic systems. We contribute two reservoir inspired methods: 1) the online echostate Gaussian process (OESGP) and 2) its infinite variant, the online infinite echostate Gaussian process (OIESGP) Both algorithms are iterative fixed-budget methods that learn from noisy time series. In particular, the OESGP combines the echo-state network with Bayesian online learning for Gaussian processes. Extending this to infinite reservoirs yields the OIESGP, which uses a novel recursive kernel with automatic relevance determination that enables spatial and temporal feature weighting. When fused with stochastic natural gradient descent, the kernel hyperparameters are iteratively adapted to better model the target system. Furthermore, insights into the underlying system can be gleamed from inspection of the resulting hyperparameters. Experiments on noisy benchmark problems (one-step prediction and system identification) demonstrate that our methods yield high accuracies relative to state-of-the-art methods, and standard kernels with sliding windows, particularly on problems with irrelevant dimensions. In addition, we describe two case studies in robotic learning-by-demonstration involving the Nao humanoid robot and the Assistive Robot Transport for Youngsters (ARTY) smart wheelchair. [ABSTRACT FROM AUTHOR]

In this paper, we proposed a generative model for feature selection under the unsupervised learning context. The model assumes that data are independently and identically sampled from a finite mixture of Student’s t distributions, which can reduce the sensitiveness to outliers. Latent random variables that represent the features’ salience are included in the model for the indication of the relevance of features. As a result, the model is expected to simultaneously realize clustering, feature selection, and outlier detection. Inference is carried out by a tree-structured variational Bayes algorithm. Full Bayesian treatment is adopted in the model to realize automatic model selection. Controlled experimental studies showed that the developed model is capable of modeling the data set with outliers accurately. Furthermore, experiment results showed that the developed algorithm compares favorably against existing unsupervised probability model-based Bayesian feature selection algorithms on artificial and real data sets. Moreover, the application of the developed algorithm on real leukemia gene expression data indicated that it is able to identify the discriminating genes successfully. [ABSTRACT FROM PUBLISHER]

In this paper, we consider the problem of learning multiple related tasks simultaneously with the goal of improving the generalization performance of individual tasks. The key challenge is to effectively exploit the shared information across multiple tasks as well as preserve the discriminative information for each individual task. To address this, we propose a novel probabilistic model for multitask learning (MTL) that can automatically balance between low-rank and sparsity constraints. The former assumes a low-rank structure of the underlying predictive hypothesis space to explicitly capture the relationship of different tasks and the latter learns the incoherent sparse patterns private to each task. We derive and perform inference via variational Bayesian methods. Experimental results on both regression and classification tasks on real-world applications demonstrate the effectiveness of the proposed method in dealing with the MTL problems. [ABSTRACT FROM PUBLISHER]

This paper analyzes data-based online nonlinear extremum-seeker (DONE), an online optimization algorithm that iteratively minimizes an unknown function based on costly and noisy measurements. The algorithm maintains a surrogate of the unknown function in the form of a random Fourier expansion. The surrogate is updated whenever a new measurement is available, and then used to determine the next measurement point. The algorithm is comparable with Bayesian optimization algorithms, but its computational complexity per iteration does not depend on the number of measurements. We derive several theoretical results that provide insight on how the hyperparameters of the algorithm should be chosen. The algorithm is compared with a Bayesian optimization algorithm for an analytic benchmark problem and three applications, namely, optical coherence tomography, optical beam-forming network tuning, and robot arm control. It is found that the DONE algorithm is significantly faster than Bayesian optimization in the discussed problems while achieving a similar or better performance. [ABSTRACT FROM AUTHOR]

It has received much attention in recent years to use Markov blankets in a Bayesian network for feature selection. The Markov blanket of a class attribute in a Bayesian network is a unique yet minimal feature subset for optimal feature selection if the probability distribution of a data set can be faithfully represented by this Bayesian network. However, if a data set violates the faithful condition, Markov blankets of a class attribute may not be unique. To tackle this issue, in this paper, we propose a new concept of representative sets and then design the selection via group alpha-investing (SGAI) algorithm to perform Markov blanket feature selection with representative sets for classification. Using a comprehensive set of real data, our empirical studies have demonstrated that SGAI outperforms the state-of-the-art Markov blanket feature selectors and other well-established feature selection methods. [ABSTRACT FROM PUBLISHER]

In this paper, a novel statistical generative model based on hierarchical Pitman-Yor process and generalized Dirichlet distributions (GDs) is presented. The proposed model allows us to perform joint clustering and feature selection thanks to the interesting properties of the GD distribution. We develop an online variational inference algorithm, formulated in terms of the minimization of a Kullback-Leibler divergence, of our resulting model that tackles the problem of learning from high-dimensional examples. This variational Bayes formulation allows simultaneously estimating the parameters, determining the model’s complexity, and selecting the appropriate relevant features for the clustering structure. Moreover, the proposed online learning algorithm allows data instances to be processed in a sequential manner, which is critical for large-scale and real-time applications. Experiments conducted using challenging applications, namely, scene recognition and video segmentation, where our approach is viewed as an unsupervised technique for visual learning in high-dimensional spaces, showed that the proposed approach is suitable and promising. [ABSTRACT FROM PUBLISHER]

Regularization has been one of the most popular approaches to prevent overfitting in electroencephalogram (EEG) classification of brain–computer interfaces (BCIs). The effectiveness of regularization is often highly dependent on the selection of regularization parameters that are typically determined by cross-validation (CV). However, the CV imposes two main limitations on BCIs: 1) a large amount of training data is required from the user and 2) it takes a relatively long time to calibrate the classifier. These limitations substantially deteriorate the system’s practicability and may cause a user to be reluctant to use BCIs. In this paper, we introduce a sparse Bayesian method by exploiting Laplace priors, namely, SBLaplace, for EEG classification. A sparse discriminant vector is learned with a Laplace prior in a hierarchical fashion under a Bayesian evidence framework. All required model parameters are automatically estimated from training data without the need of CV. Extensive comparisons are carried out between the SBLaplace algorithm and several other competing methods based on two EEG data sets. The experimental results demonstrate that the SBLaplace algorithm achieves better overall performance than the competing algorithms for EEG classification. [ABSTRACT FROM PUBLISHER]

For most of the non-Gaussian statistical models, the data being modeled represent strongly structured properties, such as scalar data with bounded support (e.g., beta distribution), vector data with unit length (e.g., Dirichlet distribution), and vector data with positive elements (e.g., generalized inverted Dirichlet distribution). In practical implementations of non-Gaussian statistical models, it is infeasible to find an analytically tractable solution to estimating the posterior distributions of the parameters. Variational inference (VI) is a widely used framework in Bayesian estimation. Recently, an improved framework, namely, the extended VI (EVI), has been introduced and applied successfully to a number of non-Gaussian statistical models. EVI derives analytically tractable solutions by introducing lower bound approximations to the variational objective function. In this paper, we compare two approximation strategies, namely, the multiple lower bounds (MLBs) approximation and the single lower bound (SLB) approximation, which can be applied to carry out the EVI. For implementation, two different conditions, the weak and the strong conditions, are discussed. Convergence of the EVI depends on the selection of the lower bound, regardless of the choice of weak or strong condition. We also discuss the convergence properties to clarify the differences between MLB and SLB. Extensive comparisons are made based on some EVI-based non-Gaussian statistical models. Theoretical analysis is conducted to demonstrate the differences between the weak and strong conditions. Experimental results based on real data show advantages of the SLB approximation over the MLB approximation. [ABSTRACT FROM AUTHOR]

Hyperspectral (HS) image can describe subtle differences in the spectral signatures of materials, but it has low spatial resolution limited by the existing technical and budget constraints. In this paper, we propose a promising HS pansharpening method with deep priors (HPDP) to fuse a low-resolution (LR) HS image with a high-resolution (HR) panchromatic (PAN) image. Different from the existing methods, we redefine the spectral response function (SRF) based on the larger eigenvalue of structure tensor (ST) matrix for the first time that is more in line with the characteristics of HS imaging. Then, we introduce HFNet to capture deep residual mapping of high frequency across the upsampled HS image and the PAN image in a band-by-band manner. Specifically, the learned residual mapping of high frequency is injected into the structural transformed HS images, which are the extracted deep priors served as additional constraint in a Sylvester equation to estimate the final HR HS image. Comparative analyses validate that the proposed HPDP method presents the superior pansharpening performance by ensuring higher quality both in spatial and spectral domains for all types of data sets. In addition, the HFNet is trained in the high-frequency domain based on multispectral (MS) images, which overcomes the sensitivity of deep neural network (DNN) to data sets acquired by different sensors and the difficulty of insufficient training samples for HS pansharpening. [ABSTRACT FROM AUTHOR]

This paper proposes a novel unified framework to solve the 3-D localization and tracking problem that occurs multiple camera settings with overlapping views. The main challenge is to overcome the uncertainty of the back projection arising from the challenges of ground point detection in an environment that includes severe occlusions and the unknown heights of people. To tackle this challenge, we establish a Bayesian learning framework that maximizes a posterior over the trajectory assignments and 3-D positions for given detections from multiple cameras. To solve the Bayesian learning problem in a tractable form, we develop an expectation–maximization scheme based on the variation inference approximation, where the probability distributions are designed to follow Boltzmann distributions of seven terms that are induced from multicamera tracking settings. The experimental results show that the proposed method outperforms the state-of-the-art methods on the challenging multicamera data sets. [ABSTRACT FROM AUTHOR]

Linear discriminant analysis (LDA) has been a widely used supervised feature extraction and dimension reduction method in pattern recognition and data analysis. However, facing high-order tensor data, the traditional LDA-based methods take two strategies. One is vectorizing original data as the first step. The process of vectorization will destroy the structure of high-order data and result in high dimensionality issue. Another is tensor LDA-based algorithms that extract features from each mode of high-order data and the obtained representations are also high-order tensor. This paper proposes a new probabilistic LDA (PLDA) model for tensorial data, namely, tensor PLDA. In this model, each tensorial data are decomposed into three parts: the shared subspace component, the individual subspace component, and the noise part. Furthermore, the first two parts are modeled by a linear combination of latent tensor bases, and the noise component is assumed to follow a multivariate Gaussian distribution. Model learning is conducted through a Bayesian inference process. To further reduce the total number of model parameters, the tensor bases are assumed to have tensor CandeComp/PARAFAC (CP) decomposition. Two types of experiments, data reconstruction and classification, are conducted to evaluate the performance of the proposed model with the convincing result, which is superior or comparable against the existing LDA-based methods. [ABSTRACT FROM AUTHOR]

Saliency detection is used to identify the most important and informative area in a scene, and it is widely used in various vision tasks, including image quality assessment, image matching, and object recognition. Manifold ranking (MR) has been used to great effect for the saliency detection, since it not only incorporates the local spatial information but also utilizes the labeling information from background queries. However, MR completely ignores the feature information extracted from each superpixel. In this paper, we propose an MR-based matrix factorization (MRMF) method to overcome this limitation. MRMF models the ranking problem in the matrix factorization framework and embeds query sample labels in the coefficients. By incorporating spatial information and embedding labels, MRMF enforces similar saliency values on neighboring superpixels and ranks superpixels according to the learned coefficients. We prove that the MRMF has good generalizability, and develops an efficient optimization algorithm based on the Nesterov method. Experiments using popular benchmark data sets illustrate the promise of MRMF compared with the other state-of-the-art saliency detection methods. [ABSTRACT FROM PUBLISHER]

A language model (LM) is calculated as the probability of a word sequence that provides the solution to word prediction for a variety of information systems. A recurrent neural network (RNN) is powerful to learn the large-span dynamics of a word sequence in the continuous space. However, the training of the RNN-LM is an ill-posed problem because of too many parameters from a large dictionary size and a high-dimensional hidden layer. This paper presents a Bayesian approach to regularize the RNN-LM and apply it for continuous speech recognition. We aim to penalize the too complicated RNN-LM by compensating for the uncertainty of the estimated model parameters, which is represented by a Gaussian prior. The objective function in a Bayesian classification network is formed as the regularized cross-entropy error function. The regularized model is constructed not only by calculating the regularized parameters according to the maximum a posteriori criterion but also by estimating the Gaussian hyperparameter by maximizing the marginal likelihood. A rapid approximation to a Hessian matrix is developed to implement the Bayesian RNN-LM (BRNN-LM) by selecting a small set of salient outer-products. The proposed BRNN-LM achieves a sparser model than the RNN-LM. Experiments on different corpora show the robustness of system performance by applying the rapid BRNN-LM under different conditions. [ABSTRACT FROM AUTHOR]

Reliably predicting software defects is one of the most significant tasks in software engineering. Two of the major components of modern software reliability modeling approaches are: 1) extraction of salient features for software system representation, based on appropriately designed software metrics and 2) development of intricate regression models for count data, to allow effective software reliability data modeling and prediction. Surprisingly, research in the latter frontier of count data regression modeling has been rather limited. More specifically, a lack of simple and efficient algorithms for posterior computation has made the Bayesian approaches appear unattractive, and thus underdeveloped in the context of software reliability modeling. In this paper, we try to address these issues by introducing a novel Bayesian regression model for count data, based on the concept of max-margin data modeling, effected in the context of a fully Bayesian model treatment with simple and efficient posterior distribution updates. Our novel approach yields a more discriminative learning technique, making more effective use of our training data during model inference. In addition, it allows of better handling uncertainty in the modeled data, which can be a significant problem when the training data are limited. We derive elegant inference algorithms for our model under the mean-field paradigm and exhibit its effectiveness using the publicly available benchmark data sets. [ABSTRACT FROM PUBLISHER]

In the operational optimization and scheduling problems of actual industrial processes, such as iron and steel, and microelectronics, the operational indices and process parameters usually need to be predicted. However, for some input and output variables of these prediction models, there may exist a lot of uncertainties coming from themselves, the measurement error, the rough representation, and so on. In such cases, constructing a prediction interval (PI) for the output of the corresponding prediction model is very necessary. In this paper, two twin extreme learning machine (TELM) models for constructing PIs are proposed. First, we propose a regularized asymmetric least squares extreme learning machine (RALS-ELM) method, in which different weights of its squared error loss function are set according to whether the error of the model output is positive or negative in order that the above error can be differentiated in the parameter learning process, and Tikhonov regularization is introduced to reduce overfitting. Then, we propose an asymmetric Bayesian extreme learning machine (AB-ELM) method based on the Bayesian framework with the asymmetric Gaussian distribution (AB-ELM), in which the weights of its likelihood function are determined as the same method in RALS-ELM, and the type II maximum likelihood algorithm is derived to learn the parameters of AB-ELM. Based on RALS-ELM and AB-ELM, we use a pair of weights following the reciprocal relationship to obtain two nonparallel regressors, including a lower-bound regressor and an upper-bound regressor, respectively, which can be used for calculating the PIs. Finally, some discussions are given, about how to adjust the weights adaptively to meet the desired PI, how to use the proposed TELMs for nonlinear quantile regression, and so on. Results of numerical comparison on data from one synthetic regression problem, three University of California Irvine benchmark regression problems, and two actual industrial regression problems show the effectiveness of the proposed models. [ABSTRACT FROM AUTHOR]

Multiple kernel learning (MKL) and classifier ensemble are two mainstream methods for solving learning problems in which some sets of features/views are more informative than others, or the features/views within a given set are inconsistent. In this paper, we first present a novel probabilistic interpretation of MKL such that maximum entropy discrimination with a noninformative prior over multiple views is equivalent to the formulation of MKL. Instead of using the noninformative prior, we introduce a novel data-dependent prior based on an ensemble of kernel predictors, which enhances the prediction performance of MKL by leveraging the merits of the classifier ensemble. With the proposed probabilistic framework of MKL, we propose a hierarchical Bayesian model to learn the proposed data-dependent prior and classification model simultaneously. The resultant problem is convex and other information (e.g., instances with either missing views or missing labels) can be seamlessly incorporated into the data-dependent priors. Furthermore, a variety of existing MKL models can be recovered under the proposed MKL framework and can be readily extended to incorporate these priors. Extensive experiments demonstrate the benefits of our proposed framework in supervised and semisupervised settings, as well as in tasks with partial correspondence among multiple views. [ABSTRACT FROM AUTHOR]

The use of domain knowledge in learning systems is expected to improve learning efficiency and reduce model complexity. However, due to the incompatibility with knowledge structure of the learning systems and real-time exploratory nature of reinforcement learning (RL), domain knowledge cannot be inserted directly. In this paper, we show how self-organizing neural networks designed for online and incremental adaptation can integrate domain knowledge and RL. Specifically, symbol-based domain knowledge is translated into numeric patterns before inserting into the self-organizing neural networks. To ensure effective use of domain knowledge, we present an analysis of how the inserted knowledge is used by the self-organizing neural networks during RL. To this end, we propose a vigilance adaptation and greedy exploitation strategy to maximize exploitation of the inserted domain knowledge while retaining the plasticity of learning and using new knowledge. Our experimental results based on the pursuit-evasion and minefield navigation problem domains show that such self-organizing neural network can make effective use of domain knowledge to improve learning efficiency and reduce model complexity. [ABSTRACT FROM PUBLISHER]

The L1 -norm low-rank matrix factorization (LRMF) has been attracting much attention due to its wide applications to computer vision and pattern recognition. In this paper, we construct a new hierarchical Bayesian generative model for the L1 -norm LRMF problem and design a mean-field variational method to automatically infer all the parameters involved in the model by closed-form equations. The variational Bayesian inference in the proposed method can be understood as solving a weighted LRMF problem with different weights on matrix elements based on their significance and with L2 -regularization penalties on parameters. Throughout the inference process of our method, the weights imposed on the matrix elements can be adaptively fitted so that the adverse influence of noises and outliers embedded in data can be largely suppressed, and the parameters can be appropriately regularized so that the generalization capability of the problem can be statistically guaranteed. The robustness and the efficiency of the proposed method are substantiated by a series of synthetic and real data experiments, as compared with the state-of-the-art L1 -norm LRMF methods. Especially, attributed to the intrinsic generalization capability of the Bayesian methodology, our method can always predict better on the unobserved ground truth data than existing methods. [ABSTRACT FROM PUBLISHER]

Most current model reference adaptive control (MRAC) methods rely on parametric adaptive elements, in which the number of parameters of the adaptive element are fixed a priori, often through expert judgment. An example of such an adaptive element is radial basis function networks (RBFNs), with RBF centers preallocated based on the expected operating domain. If the system operates outside of the expected operating domain, this adaptive element can become noneffective in capturing and canceling the uncertainty, thus rendering the adaptive controller only semiglobal in nature. This paper investigates a Gaussian process-based Bayesian MRAC architecture (GP-MRAC), which leverages the power and flexibility of GP Bayesian nonparametric models of uncertainty. The GP-MRAC does not require the centers to be preallocated, can inherently handle measurement noise, and enables MRAC to handle a broader set of uncertainties, including those that are defined as distributions over functions. We use stochastic stability arguments to show that GP-MRAC guarantees good closed-loop performance with no prior domain knowledge of the uncertainty. Online implementable GP inference methods are compared in numerical simulations against RBFN-MRAC with preallocated centers and are shown to provide better tracking and improved long-term learning. [ABSTRACT FROM AUTHOR]

With the development of digital multimedia technologies, image matting has gained increasing interests from both academic and industrial communities. The purpose of image matting is to precisely extract the foreground objects with arbitrary shapes from an image or a video frame for further editing. It is generally known that image matting is inherently an ill-posed problem because we need to output three images out of only one input image. In this paper, we provide a comprehensive survey of the existing image matting algorithms and evaluate their performance. In addition to the blue screen matting, we systematically divide all existing natural image matting methods into four categories: 1) color sampling-based; 2) propagation-based; 3) combination of sampling-based and propagation-based; and 4) learning-based approaches. Sampling-based methods assume that the foreground and background colors of an unknown pixel can be explicitly estimated by examining nearby pixels. Propagation-based methods are instead based on the assumption that foreground and background colors are locally smooth. Learning-based methods treat the matting process as a supervised or semisupervised learning problem. Via the learning process, users can construct a linear or nonlinear model between the alpha mattes and the image colors using a training set to estimate the alpha matte of an unknown pixel without any assumption about the characteristics of the testing image. With three benchmark data sets, the various matting algorithms are evaluated and compared using several metrics to demonstrate the strengths and weaknesses of each method both quantitatively and qualitatively. Finally, we conclude this paper by outlining the research trends and suggesting a number of promising directions for future development. [ABSTRACT FROM PUBLISHER]

In this paper, we propose a Gaussian process (GP) model for analysis of nonlinear time series. Formulation of our model is based on the consideration that the observed data are functions of latent variables, with the associated mapping between observations and latent representations modeled through GP priors. In addition, to capture the temporal dynamics in the modeled data, we assume that subsequent latent representations depend on each other on the basis of a hidden Markov prior imposed over them. Derivation of our model is performed by marginalizing out the model parameters in closed form using GP priors for observation mappings, and appropriate stick-breaking priors for the latent variable (Markovian) dynamics. This way, we eventually obtain a nonparametric Bayesian model for dynamical systems that accounts for uncertainty in the modeled data. We provide efficient inference algorithms for our model on the basis of a truncated variational Bayesian approximation. We demonstrate the efficacy of our approach considering a number of applications dealing with real-world data, and compare it with the related state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Dimension reduction for high-order tensors is a challenging problem. In conventional approaches, dimension reduction for higher order tensors is implemented via Tucker decomposition to obtain lower dimensional tensors. This paper introduces a probabilistic vectorial dimension reduction model for tensorial data. The model represents a tensor by using a linear combination of the same order basis tensors, thus it offers a learning approach to directly reduce a tensor to a vector. Under this expression, the projection base of the model is based on the tensor CandeComp/PARAFAC (CP) decomposition and the number of free parameters in the model only grows linearly with the number of modes rather than exponentially. A Bayesian inference has been established via the variational Expectation Maximization (EM) approach. A criterion to set the parameters (a factor number of CP decomposition and the number of extracted features) is empirically given. The model outperforms several existing principal component analysis-based methods and CP decomposition on several publicly available databases in terms of classification and clustering accuracy. [ABSTRACT FROM AUTHOR]

In this paper, we propose a Bayesian model for the data association problem, in which trajectory smoothness is enforced through the use of Gaussian process priors. This model allows to score candidate associations using the evidence framework, thus casting the data association problem into an optimization problem. Under some additional mild assumptions, this optimization problem is shown to be equivalent to a constrained Max $K$ -section problem. Furthermore, for $K=2$ , a MaxCut formulation is obtained, to which an approximate solution can be efficiently found using an SDP relaxation. Solving this MaxCut problem is equivalent to finding the optimal association out of the combinatorially many possibilities. The obtained clustering depends only on two hyperparameters, which can also be selected by maximum evidence. [ABSTRACT FROM PUBLISHER]

This paper presents a Gaussian classifier-based evolutionary strategy (GCES) to solve multimodal optimization problems. An evolutionary technique for them must answer two crucial questions to guarantee its success: how to distinguish among the different basins of attraction and how to safeguard the already discovered good-quality solutions including both global and local optima. In GCES, multimodal optimization problems are regarded as classification ones, and Gaussian mixture models are employed to save the locations and basins of already and presently identified local or global optima. A sequential estimation technique for the covariance of a Gaussian model is introduced into GCES. To best adjust the global step size, a strategy named top-ranked sample selection is introduced, and a classification method instead of a common but problematic radius-triggered manner is proposed. Experiments are performed on a series of benchmark test functions to compare GCES with the state-of-the-art multimodal optimization approaches. The results show that GCES is not only simple to program and understand, but also provides better and consistent performance. [ABSTRACT FROM AUTHOR]

Probabilistic classification vector machine (PCVM) is a sparse learning approach aiming to address the stability problems of relevance vector machine for classification problems. Because PCVM is based on the expectation maximization algorithm, it suffers from sensitivity to initialization, convergence to local minima, and the limitation of Bayesian estimation making only point estimates. Another disadvantage is that PCVM was not efficient for large data sets. To address these problems, this paper proposes an efficient PCVM (EPCVM) by sequentially adding or deleting basis functions according to the marginal likelihood maximization for efficient training. Because of the truncated prior used in EPCVM, two approximation techniques, i.e., Laplace approximation and expectation propagation (EP), have been used to implement EPCVM to obtain full Bayesian solutions. We have verified Laplace approximation and EP with a hybrid Monte Carlo approach. The generalization performance and computational effectiveness of EPCVM are extensively evaluated. Theoretical discussions using Rademacher complexity reveal the relationship between the sparsity and the generalization bound of EPCVM. [ABSTRACT FROM PUBLISHER]

In this paper, both Bayesian and mutual-information classifiers are examined for binary classifications with or without a reject option. The general decision rules are derived for Bayesian classifiers with distinctions on error types and reject types. A formal analysis is conducted to reveal the parameter redundancy of cost terms when abstaining classifications are enforced. The redundancy implies an intrinsic problem of nonconsistency for interpreting cost terms. If no data are given to the cost terms, we demonstrate the weakness of Bayesian classifiers in class-imbalanced classifications. On the contrary, mutual-information classifiers are able to provide an objective solution from the given data, which shows a reasonable balance among error types and reject types. Numerical examples of using two types of classifiers are given for confirming the differences, including the extremely class-imbalanced cases. Finally, we briefly summarize the Bayesian and mutual-information classifiers in terms of their application advantages and disadvantages, respectively. [ABSTRACT FROM PUBLISHER]

Learning a distance metric in feature space potentially improves the performance of the $K$ nearest neighbor classifier and is useful in many real-world applications. Many metric learning (ML) algorithms are, however, based on the point estimation of a quadratic optimization problem, which is time-consuming, susceptible to overfitting, and lacks a natural mechanism to reason with parameter uncertainty—a property useful especially when the training set is small and/or noisy. To deal with these issues, we present a novel Bayesian ML (BML) method, called Bayesian neighborhood component analysis (NCA), based on the well-known NCA method, in which the metric posterior is characterized by the local label consistency constraints of observations, encoded with a similarity graph instead of independent pairwise constraints. For efficient Bayesian inference, we explore the variational lower bound over the log-likelihood of the original NCA objective. Experiments on several publicly available data sets demonstrate that the proposed method is able to learn robust metric measures from small size data set and/or from challenging training set with labels contaminated by errors. The proposed method is also shown to outperform a previous pairwise constrained BML method. [ABSTRACT FROM AUTHOR]

The learning automaton (LA) that simulates the interaction between an intelligent agent and a stochastic environment to learn the optimal action is an important tool in reinforcement learning. Being confronted with an unknown environment, most learning automata have more than one parameters to be tuned during a pretraining process in which the LA interacts with the environment. Only after the parameters are tuned properly, an LA can act most properly during the training procedure to obtain the optimal behavior. The cost of parameter tuning can be enormous, e.g., possibly millions of interactions are required to seek the best parameter configuration. Therefore, the parameter-free LA that uses identical parameters for every environment and saves further tuning has become the hot spot of this research. This article proposes an efficient parameter-free learning automaton (EPFLA) that depends on a separating function (SF). Taking advantage of both frequentist inference and Bayesian inference, the SF plays a dual role in the proposed scheme: 1) evaluating the difference in performance between actions in the environment and 2) exploring actions by coining an action selection strategy. A proof is provided to ensure the $\epsilon $ -optimality of EPFLA. Comprehensive comparisons verify the privileges of EPFLA over both parameter-based schemes and existing parameter-free schemes. [ABSTRACT FROM AUTHOR]

Bayesian-directed acyclic discrete-variable graphs are reduced to a simplified normal form made up of only replicator units (or equal constraint units), source, and single-input/single-output blocks. In this framework, the same adaptation algorithm can be applied to all the parametric blocks. We obtain and compare adaptation rules derived from a constrained maximum likelihood formulation and a minimum Kullback–Leibler divergence criterion using Karush–Kuhn–Tucker conditions. The learning algorithms are compared with two other updating equations based on localized decisions and on a variational approximation, respectively. The performance of the various algorithms is verified on synthetic data sets for various architectures. Factor graphs in reduced normal form provide an appealing framework for rapid deployment of Bayesian-directed graphs in the applications. [ABSTRACT FROM PUBLISHER]

An accelerated failure time (AFT) model has been widely used for the analysis of censored survival or failure time data. However, the AFT imposes the restrictive log-linear relation between the survival time and the explanatory variables. In this paper, we introduce a relevance vector machine survival (RVMS) model based on Weibull AFT model that enables the use of kernel framework to automatically learn the possible nonlinear effects of the input explanatory variables on target survival times. We take advantage of the Bayesian inference technique in order to estimate the model parameters. We also introduce two approaches to accelerate the RVMS training. In the first approach, an efficient smooth prior is employed that improves the degree of sparsity. In the second approach, a fast marginal likelihood maximization procedure is used for obtaining a sparse solution of survival analysis task by sequential addition and deletion of candidate basis functions. These two approaches, denoted by smooth RVMS and fast RVMS, typically use fewer basis functions than RVMS and improve the RVMS training time; however, they cause a slight degradation in the RVMS performance. We compare the RVMS and the two accelerated approaches with the previous sparse kernel survival analysis method on a synthetic data set as well as six real-world data sets. The proposed kernel survival analysis models have been discovered to be more accurate in prediction, although they benefit from extra sparsity. The main advantages of our proposed models are: 1) extra sparsity that leads to a better generalization and avoids overfitting; 2) automatic relevance sample determination based on data that provide more accuracy, in particular for highly censored survival data; and 3) flexibility to utilize arbitrary number and types of kernel functions (e.g., non-Mercer kernels and multikernel learning). [ABSTRACT FROM PUBLISHER]

Recent advances in deep learning have brought to the fore models that can make multiple computational steps in the service of completing a task; these are capable of describing long-term dependencies in sequential data. Novel recurrent attention models over possibly large external memory modules constitute the core mechanisms that enable these capabilities. Our work addresses learning subtler and more complex underlying temporal dynamics in language modeling tasks that deal with sparse sequential data. To this end, we improve upon these recent advances by adopting concepts from the field of Bayesian statistics, namely, variational inference. Our proposed approach consists in treating the network parameters as latent variables with a prior distribution imposed over them. Our statistical assumptions go beyond the standard practice of postulating Gaussian priors. Indeed, to allow for handling outliers, which are prevalent in long observed sequences of multivariate data, multivariate $t$ -exponential distributions are imposed. On this basis, we proceed to infer corresponding posteriors; these can be used for inference and prediction at test time, in a way that accounts for the uncertainty in the available sparse training data. Specifically, to allow for our approach to best exploit the merits of the $t$ -exponential family, our method considers a new $t$ -divergence measure, which generalizes the concept of the Kullback–Leibler divergence. We perform an extensive experimental evaluation of our approach, using challenging language modeling benchmarks, and illustrate its superiority over existing state-of-the-art techniques. [ABSTRACT FROM AUTHOR]