Search

Multi-view learning deals with data that is described through multiple representations, or views. While various real-world data can be represented by three or more views, several existing multi-view classification methods can only handle two views. Previously proposed methods usually solve this issue by optimizing pairwise combinations of views. Although this can numerically deal with the issue of multiple views, it ignores the higher order correlations which can only be examined by exploring all views simultaneously. In this work new multi-view classification approaches are introduced which aim to include higher order statistics when three or more views are available. The proposed model is an extension to the recently proposed Restricted Kernel Machine classifier model and assumes shared hidden features for all views, as well as a newly introduced model tensor. Experimental results show an improvement with respect to state-of-the art pairwise multi-view learning methods, both in terms of classification accuracy and runtime. • A novel multi-view classification model, called ϱ TMV-RKM, is proposed. • Tensor learning is incorporated to account for higher order correlations. • Experimental results show the merit of using a weight tensor. • Comparisons with other state-of-the-art methods show improvement. • Multiple approaches to handle a large-scale dataset are proposed. [ABSTRACT FROM AUTHOR]

Long Short-Term Memory (LSTM) has shown significant performance on many real-world applications due to its ability to capture long-term dependencies. In this paper, we utilize LSTM to obtain a data-driven forecasting model for an application of weather forecasting. Moreover, we propose Transductive LSTM (T-LSTM) which exploits the local information in time-series prediction. In transductive learning, the samples in the test point vicinity are considered to have higher impact on fitting the model. In this study, a quadratic cost function is considered for the regression problem. Localizing the objective function is done by considering a weighted quadratic cost function at which point the samples in the neighborhood of the test point have larger weights. We investigate two weighting schemes based on the cosine similarity between the training samples and the test point. In order to assess the performance of the proposed method in different weather conditions, the experiments are conducted on two different time periods of a year. The results show that T-LSTM results in better performance in the prediction task. [ABSTRACT FROM AUTHOR]

Abstract This work proposes a new algorithm for training a re-weighted ℓ 2 Support Vector Machine (SVM), inspired on the re-weighted Lasso algorithm of Candès et al. and on the equivalence between Lasso and SVM shown recently by Jaggi. In particular, the margin required for each training vector is set independently, defining a new weighted SVM model. These weights are selected to be binary, and they are automatically adapted during the training of the model, resulting in a variation of the Frank–Wolfe optimization algorithm with essentially the same computational complexity as the original algorithm. As shown experimentally, this algorithm is computationally cheaper to apply since it requires less iterations to converge, and it produces models with a sparser representation in terms of support vectors and which are more stable with respect to the selection of the regularization hyper-parameter. [ABSTRACT FROM AUTHOR]

This paper introduces a novel hybrid deep neural kernel framework. The proposed deep learning model makes a combination of a neural networks based architecture and a kernel based model. In particular, here an explicit feature map, based on random Fourier features, is used to make the transition between the two architectures more straightforward as well as making the model scalable to large datasets by solving the optimization problem in the primal. Furthermore, the introduced framework is considered as the first building block for the development of even deeper models and more advanced architectures. Experimental results show an improvement over shallow models and the standard non-hybrid neural networks architecture on several medium to large scale real-life datasets. [ABSTRACT FROM AUTHOR]

Spectral clustering suffers from a scalability problem in both memory usage and computational time when the number of data instances N is large. To solve this issue, we present a fast spectral clustering algorithm able to effectively handle millions of datapoints at a desktop PC scale. The proposed technique relies on a kernel-based formulation of the spectral clustering problem, also known as kernel spectral clustering. In this framework, the Nyström approximation of the feature map of size m , with m ≪ N , is used to solve the primal optimization problem. This leads to a reduction of time complexity from O ( N 3 ) to O ( mN ) and space complexity from O ( N 2 ) to O ( mN ). The effectiveness of the proposed algorithm in terms of computational efficiency and clustering quality is illustrated on several datasets. [ABSTRACT FROM AUTHOR]

Recent advances in stochastic learning, such as dual averaging schemes for proximal subgradient-based methods and simple but theoretically well-grounded solvers for linear Support Vector Machines (SVMs), revealed an ongoing interest in making these approaches consistent, robust and tailored towards sparsity inducing norms. In this paper we study reweighted schemes for stochastic learning (specifically in the context of classification problems) based on linear SVMs and dual averaging methods with primal–dual iterate updates. All these methods favor properties of a convex and composite optimization objective. The latter consists of a convex regularization term and loss function with Lipschitz continuous subgradients, e.g. l 1 -norm ball together with hinge loss. Some approaches approximate in a limit the l 0 -type of a penalty. In our analysis we focus on a regret and convergence criteria of such an approximation. We derive our results in terms of a sequence of convex and strongly convex optimization objectives. These objectives are obtained via the smoothing of a generic sub-differential and possibly non-smooth composite function by the global proximal operator. We report an extended evaluation and comparison of the reweighted schemes against different state-of-the-art techniques and solvers for linear SVMs. Our experimental study indicates the usefulness of the proposed methods for obtaining sparser and better solutions. We show that reweighted schemes can outperform state-of-the-art traditional approaches in terms of generalization error as well. [ABSTRACT FROM AUTHOR]

In this letter a new variational principle to the matrix singular value decomposition (SVD) is proposed. It is formulated as a constrained optimization problem where two sets of constraints are expressed in terms of compatible feature maps, which are evaluated on data vectors that relate to the rows and columns of the given matrix. Provided that a compatibility condition holds the solution can be related to Lanczos' decomposition theorem. The method is further extended to nonlinear SVD, which is illustrated also on image examples. [ABSTRACT FROM AUTHOR]

To be physically interpretable, sub-pixel land cover fractions or abundances should fulfill two constraints, the Abundance Non-negativity Constraint (ANC) and the Abundance Sum-to-one Constraint (ASC). This paper focuses on the effect of imposing these constraints onto the MultiLayer Perceptron (MLP) for a multi-class sub-pixel land cover classification of a time series of low resolution MODIS-images covering the northern part of Belgium. Two constraining modes were compared, (i) an in-training approach that uses ‘softmax’ as the transfer function in the MLP’s output layer and (ii) a post-training approach that linearly rescales the outputs of the unconstrained MLP. Our results demonstrate that the pixel-level prediction accuracy is markedly increased by the explicit enforcement, both in-training and post-training, of the ANC and the ASC. For aggregations of pixels (municipalities), the constrained perceptrons perform at least as well as their unconstrained counterparts. Although the difference in performance between the in-training and post-training approach is small, we recommend the former for integrating the fractional abundance constraints into MLPs meant for sub-pixel land cover estimation, regardless of the targeted level of spatial aggregation. [ABSTRACT FROM AUTHOR]

This paper proposes an approach based on Least Squares Support Vector Machines (LS-SVMs) for solving second order partial differential equations (PDEs) with variable coefficients. Contrary to most existing techniques, the proposed method provides a closed form approximate solution. The optimal representation of the solution is obtained in the primal–dual setting. The model is built by incorporating the initial/boundary conditions as constraints of an optimization problem. The developed method is well suited for problems involving singular, variable and constant coefficients as well as problems with irregular geometrical domains. Numerical results for linear and nonlinear PDEs demonstrate the efficiency of the proposed method over existing methods. [ABSTRACT FROM AUTHOR]

In this paper we study the problem of finding a support of unknown high-dimensional distributions in the presence of labeling information, called Supervised Novelty Detection (SND). The One-Class Support Vector Machine (SVM) is a widely used kernel-based technique to address this problem. However with the latter approach it is difficult to model a mixture of distributions from which the support might be constituted. We address this issue by presenting a new class of SVM-like algorithms which help to approach multi-class classification and novelty detection from a new perspective. We introduce a new coupling term between classes which leverages the problem of finding a good decision boundary while preserving the compactness of a support with the $l_2$ -norm penalty. First we present our optimization objective in the primal and then derive a dual QP formulation of the problem. Next we propose a Least-Squares formulation which results in a linear system which drastically reduces computational costs. Finally we derive a Pegasos-based formulation which can effectively cope with large data sets that cannot be handled by many existing QP solvers. We complete our paper with experiments that validate the usefulness and practical importance of the proposed methods both in classification and novelty detection settings. [ABSTRACT FROM AUTHOR]

Abstract: Although over a thousand scientific papers address the topic of load forecasting every year, only a few are dedicated to finding a general framework for load forecasting that improves the performance, without depending on the unique characteristics of a certain task such as geographical location. Meta-learning, a powerful approach for algorithm selection has so far been demonstrated only on univariate time-series forecasting. Multivariate time-series forecasting is known to have better performance in load forecasting. In this paper we propose a meta-learning system for multivariate time-series forecasting as a general framework for load forecasting model selection. We show that a meta-learning system built on 65 load forecasting tasks returns lower forecasting error than 10 well-known forecasting algorithms on 4 load forecasting tasks for a recurrent real-life simulation. We introduce new metafeatures of fickleness, traversity, granularity and highest ACF. The meta-learning framework is parallelized, component-based and easily extendable. [Copyright &y& Elsevier]

Abstract: Kernel spectral clustering fits in a constrained optimization framework where the primal problem is expressed in terms of high-dimensional feature maps and the dual problem is expressed in terms of kernel evaluations. An eigenvalue problem is solved at the training stage and projections onto the eigenvectors constitute the clustering model. The formulation allows out-of-sample extensions which are useful for model selection in a learning setting. In this work, we propose a methodology to reveal the hierarchical structure present on the data. During the model selection stage, several clustering model parameters leading to good clusterings can be found. These results are then combined to display the underlying cluster hierarchies where the optimal depth of the tree is automatically determined. Simulations with toy data and real-life problems show the benefits of the proposed approach. [Copyright &y& Elsevier]

Abstract: This paper discusses a numerical method based on Least Squares Support Vector Machines (LS-SVMs) for solving linear time varying initial and boundary value problems in Differential Algebraic Equations (DAEs). The method generates a closed form (model-based) approximate solution. The results of numerical experiments on different systems with index from to , are presented and compared with analytic solutions to confirm the validity and applicability of the proposed method. [Copyright &y& Elsevier]

Abstract: Kernel spectral clustering has been formulated within a primal–dual optimization setting allowing natural extensions to out-of-sample data together with model selection in a learning framework. This becomes important for predictive purposes and for good generalization capabilities. The clustering model is formulated in the primal in terms of mappings to high-dimensional feature spaces typical of support vector machines and kernel-based methodologies. The dual problem corresponds to an eigenvalue decomposition of a centered Laplacian matrix derived from pairwise similarities within the data. The out-of-sample extension can also be used to introduce sparsity and to reduce the computational complexity of the resulting eigenvalue problem. In this paper, we propose several methods to obtain sparse and highly sparse kernel spectral clustering models. The proposed approaches are based on structural properties of the solutions when the clusters are well formed. Experimental results with difficult toy examples and images show the applicability of the proposed sparse models with predictive capabilities. [Copyright &y& Elsevier]

Abstract: Least squares support vector machine (LS-SVM) classifiers are a class of kernel methods whose solution follows from a set of linear equations. In this work we present low rank modifications to the LS-SVM classifiers that are useful for fast and efficient variable selection. The inclusion or removal of a candidate variable can be represented as a low rank modification to the kernel matrix (linear kernel) of the LS-SVM classifier. In this way, the LS-SVM solution can be updated rather than being recomputed, which improves the efficiency of the overall variable selection process. Relevant variables are selected according to a closed form of the leave-one-out (LOO) error estimator, which is obtained as a by-product of the low rank modifications. The proposed approach is applied to several benchmark data sets as well as two microarray data sets. When compared to other related algorithms used for variable selection, simulations applying our approach clearly show a lower computational complexity together with good stability on the generalization error. [Copyright &y& Elsevier]

Abstract: A new kernel based contrast function for independent component analysis (ICA) is proposed. This criterion corresponds to a regularized correlation measure in high dimensional feature spaces induced by kernels. The formulation is a multivariate extension of the least squares support vector machine (LS-SVM) formulation to kernel canonical correlation analysis (CCA). The regularization is incorporated naturally in the primal problem leading to a dual generalized eigenvalue problem. The smallest generalized eigenvalue is a measure of correlation in the feature space and a measure of independence in the input space. Due to the primal-dual nature of the proposed approach, the measure of independence can also be extended to out-of-sample points which is important for model selection ensuring statistical reliability of the proposed measure. Computational issues due to the large size of the matrices involved in the eigendecomposition are tackled via the incomplete Cholesky factorization. Simulations with toy data, images and speech signals show improved performance on the estimation of independent components compared with existing kernel-based contrast functions. [Copyright &y& Elsevier]

Introduces a master-slave synchronization scheme for Lur'e systems. Utilization of vector field modulation and dynamic output feedback to cover message signals; Representation of the synchronization scheme using modern control theory plant form; Analysis of the input and output properties of the scheme; Benefit of the scheme.

Proposes a method of robust nonlinear...master-slave synchronization for chaotic Lur'e systems with applications to secure communication. Use of vector field modulation in this scheme; Assumptions made about the master-slave systems; How binary valued continuous time message signals are recovered; Illustration of the method on Chua's circuit.

Principal Component Analysis (PCA) and its nonlinear extension Kernel PCA (KPCA) are widely used across science and industry for data analysis and dimensionality reduction. Modern deep learning tools have achieved great empirical success, but a framework for deep principal component analysis is still lacking. Here we develop a deep kernel PCA methodology (DKPCA) to extract multiple levels of the most informative components of the data. Our scheme can effectively identify new hierarchical variables, called deep principal components, capturing the main characteristics of high-dimensional data through a simple and interpretable numerical optimization. We couple the principal components of multiple KPCA levels, theoretically showing that DKPCA creates both forward and backward dependency across levels, which has not been explored in kernel methods and yet is crucial to extract more informative features. Various experimental evaluations on multiple data types show that DKPCA finds more efficient and disentangled representations with higher explained variance in fewer principal components, compared to the shallow KPCA. We demonstrate that our method allows for effective hierarchical data exploration, with the ability to separate the key generative factors of the input data both for large datasets and when few training samples are available. Overall, DKPCA can facilitate the extraction of useful patterns from high-dimensional data by learning more informative features organized in different levels, giving diversified aspects to explore the variation factors in the data, while maintaining a simple mathematical formulation. • A deep kernel principal component analysis (DKPCA) framework is proposed. • Forward and backward couplings between the levels are identified. • Theoretical analysis presents error bounds and higher explained variance than KPCA. • Generative DKPCA is introduced for the pre-image problem in deep kernel methods. • DKPCA is competitive in learning informative multi-level disentangled features. [ABSTRACT FROM AUTHOR]

• A general framework of neural grey system model is developed. • Levenberg-Marquardt algorithm and Bayesian regularization are used for the semiparametric estimation. • Bayesian regularization can tune the regularization parameters automatically. • The proposed model outperforms the other existing models in 6 real-world cases. Most machine learning models are essentially "Black-box" models, of which the performance heavily relies on large scale data sets. In this work the idea of "Grey-box" modelling is adopted in order to take most advantage of known information represented by deterministic structure, and then the neural grey system model is developed. Levenberg-Marquardt algorithm is used to train the proposed model, and the Bayesian regularization is used to tune the regularized parameter automatically. Six real world case studies are presented to show the performance of the proposed model, comparing to 6 existing machine learning models and 17 grey system models. The results show that the proposed model can significantly overperform the other models and has very good generality, illustrating its high potential in a wide variety of real world applications and the efficiency of the proposed modelling method. [ABSTRACT FROM AUTHOR]

We introduce Constr-DRKM, a deep kernel method for the unsupervised learning of disentangled data representations. We propose augmenting the original deep restricted kernel machine formulation for kernel PCA by orthogonality constraints on the latent variables to promote disentanglement and to make it possible to carry out optimization without first defining a stabilized objective. After discussing a number of algorithms for end-to-end training, we quantitatively evaluate the proposed method's effectiveness in disentangled feature learning. We demonstrate on four benchmark datasets that this approach performs similarly overall to β -VAE on several disentanglement metrics when few training points are available while being less sensitive to randomness and hyperparameter selection than β -VAE. We also present a deterministic initialization of Constr-DRKM's training algorithm that significantly improves the reproducibility of the results. Finally, we empirically evaluate and discuss the role of the number of layers in the proposed methodology, examining the influence of each principal component in every layer and showing that components in lower layers act as local feature detectors capturing the broad trends of the data distribution, while components in deeper layers use the representation learned by previous layers and more accurately reproduce higher-level features. • A deep kernel method for unsupervised representation learning is proposed. • A constrained optimization problem on the Stiefel manifold is formulated. • Experiments are conducted in denoising and in disentangled feature learning. • The influence of each principal component in every layer is studied by denoising complex 2D data. • In experiments, our approach was less sensitive than β -VAE to randomness and hyperparameter selection. [ABSTRACT FROM AUTHOR]

This paper introduces a novel framework for generative models based on Restricted Kernel Machines (RKMs) with joint multi-view generation and uncorrelated feature learning, called Gen-RKM. To enable joint multi-view generation, this mechanism uses a shared representation of data from various views. Furthermore, the model has a primal and dual formulation to incorporate both kernel-based and (deep convolutional) neural network based models within the same setting. When using neural networks as explicit feature-maps, a novel training procedure is proposed, which jointly learns the features and shared subspace representation. The latent variables are given by the eigen-decomposition of the kernel matrix, where the mutual orthogonality of eigenvectors represents the learned uncorrelated features. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of generated samples on various standard datasets. [ABSTRACT FROM AUTHOR]

In multi-view clustering, datasets are comprised of different representations of the data, or views. Although each view could individually be used, exploiting information from all views together could improve the cluster quality. In this paper a new model Multi-View Kernel Spectral Clustering (MVKSC) is proposed that performs clustering when two or more views are available. This model is formulated as a weighted kernel canonical correlation analysis in a primal-dual optimization setting typical of Least Squares Support Vector Machines (LS-SVM). The primal model includes, in particular, a coupling term, which enforces the clustering scores corresponding to the different views to align. Because of the out-of-sample extension, this model is easily applied to large-scale datasets. The performance of the proposed model is shown on synthetic and real-world datasets, as well as on some large-scale datasets. Experimental comparisons with a number of other methods show that using multiple views improves the clustering results and that the proposed method is competitive with other state-of-the-art algorithms in terms of clustering accuracy and runtime. Especially on the large-scale datasets the advantage of the proposed method is clearly shown, as it is able to handle larger datasets than the other state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

We propose a new methodology for identifying Wiener systems using the data acquired from two separate experiments. In the first experiment, we feed the system with a sinusoid at a prescribed frequency and use the steady state response of the system to estimate the static nonlinearity. In the second experiment, the estimated nonlinearity is used to identify a model of the linear block, feeding the system with a persistently exciting input. We discuss both parametric and nonparametric approaches to estimate the static nonlinearity. In the parametric case, we show that modeling the static nonlinearity as a polynomial results into a fast least-squares based estimation procedure. In the nonparametric case, least squares support vector machines (LS-SVM) are employed to obtain a flexible model. The effectiveness of the method is demonstrated through numerical experiments. [ABSTRACT FROM AUTHOR]

The use of indefinite kernels has attracted many research interests in recent years due to their flexibility. They do not possess the usual restrictions of being positive definite as in the traditional study of kernel methods. This paper introduces the indefinite unsupervised and semi-supervised learning in the framework of least squares support vector machines (LS-SVM). The analysis is provided for both unsupervised and semi-supervised models, i.e., Kernel Spectral Clustering (KSC) and Multi-Class Semi-Supervised Kernel Spectral Clustering (MSS-KSC). In indefinite KSC models one solves an eigenvalue problem whereas indefinite MSS-KSC finds the solution by solving a linear system of equations. For the proposed indefinite models, we give the feature space interpretation, which is theoretically important, especially for the scalability using Nyström approximation. Experimental results on several real-life datasets are given to illustrate the efficiency of the proposed indefinite kernel spectral learning. [ABSTRACT FROM AUTHOR]

In multi-view learning, data is described using different representations, or views. Multi-view classification methods try to exploit information from all views to improve the classification performance. Here a new model is proposed that performs classification when two or more views are available. The model is called Multi-View Least Squares Support Vector Machines (MV-LSSVM) Classification and is based on solving a constrained optimization problem. The primal objective includes a coupling term, which minimizes a combination of the errors from all views. The model combines the benefits from both early and late fusion, it is able to incorporate information from all views in the training phase while still allowing for some degree of freedom to model the views differently. Experimental comparisons with similar methods show that using multiple views improves the results with regard to the single view classifiers and that it outperforms other state-of-the-art algorithms in terms of classification accuracy. [ABSTRACT FROM AUTHOR]

• Introduce jigsaw puzzle solving auxiliary loss into vision transformer-based models. • Removing positional embeddings, randomly masking patches as techniques. • Improve vision transformers' generalization on large-scale image classification. • Improve vision transformers' robustness against label noise. • Improve vision transformers' robustness against adversarial examples. The success of Vision Transformer (ViT) in various computer vision tasks has promoted the ever-increasing prevalence of this convolution-free network. The fact that ViT works on image patches makes it potentially relevant to the problem of jigsaw puzzle solving, which is a classical self-supervised task aiming at reordering shuffled sequential image patches back to their original form. Solving jigsaw puzzle has been demonstrated to be helpful for diverse tasks using Convolutional Neural Networks (CNNs), such as feature representation learning, domain generalization and fine-grained classification. In this paper, we explore solving jigsaw puzzle as a self-supervised auxiliary loss in ViT for image classification, named Jigsaw-ViT. We show two modifications that can make Jigsaw-ViT superior to standard ViT: discarding positional embeddings and masking patches randomly. Yet simple, we find that the proposed Jigsaw-ViT is able to improve on both generalization and robustness over the standard ViT, which is usually rather a trade-off. Numerical experiments verify that adding the jigsaw puzzle branch provides better generalization to ViT on large-scale image classification on ImageNet. Moreover, such auxiliary loss also improves robustness against noisy labels on Animal-10N, Food-101N, and Clothing1M, as well as adversarial examples. Our implementation is available at https://yingyichen-cyy.github.io/Jigsaw-ViT. [ABSTRACT FROM AUTHOR]

Evolutionary spectral clustering (ESC) represents a state-of-the-art algorithm for grouping objects evolving over time. It typically outperforms traditional static clustering by producing clustering results that can adapt to data drifts while being robust to short-term noise. A major drawback of ESC is given by its cubic complexity, e.g. O( N 3 ), and high memory demand, namely O( N 2 ), that make it unfeasible to handle datasets characterized by a large number N of patterns. In this paper, we propose a solution to this issue by presenting the efficient evolutionary spectral clustering algorithm (E 2 SC). First we introduce the notion of a smoothed graph Laplacian , then we exploit the incomplete Cholesky decomposition (ICD) to construct an approximation of the this smoothed Laplacian and reduce the size of the related eigenvalue problem from N to m , with m ≪ N . Furthermore, in contrast to the standard ICD algorithm, a stopping criterion based on the convergence of the cluster assignments after the selection of each pivot is used, which is effective also when there is not a fast decay of the Laplacian spectrum. Overall, the proposed approach scales linearly with respect to the number of input datapoints N and has low memory requirements because only matrices of size N × m and m × m are constructed. [ABSTRACT FROM AUTHOR]

Although time delay is an important element in both system identification and control performance assessment, its computation remains elusive. This paper proposes the application of a least squares support vector machines driven approach to the problem of determining constant time delay for a chemical process. The approach consists of two steps, where in the first step the state of the system and its derivative are approximated based on the LS-SVM model. The second step consists of modeling the delay term and estimating the unknown model parameters as well as the time delay of the system. Therefore the proposed approach avoids integrating the given differential equation that can be computationally expensive. This time delay estimation method is applied to both simulation and experimental data obtained from a continuous, stirred, heated tank. The results show that the proposed method can provide accurate estimates even if significant noise or unmeasured additive disturbances are present. [ABSTRACT FROM AUTHOR]

This paper introduces an on-line semi-supervised learning algorithm formulated as a regularized kernel spectral clustering (KSC) approach. We consider the case where new data arrive sequentially but only a small fraction of it is labeled. The available labeled data act as prototypes and help to improve the performance of the algorithm to estimate the labels of the unlabeled data points. We adopt a recently proposed multi-class semi-supervised KSC based algorithm (MSS-KSC) and make it applicable for on-line data clustering. Given a few user-labeled data points the initial model is learned and then the class membership of the remaining data points in the current and subsequent time instants are estimated and propagated in an on-line fashion. The update of the memberships is carried out mainly using the out-of-sample extension property of the model. Initially the algorithm is tested on computer-generated data sets, then we show that video segmentation can be cast as a semi-supervised learning problem. Furthermore we show how the tracking capabilities of the Kalman filter can be used to provide the labels of objects in motion and thus regularizing the solution obtained by the MSS-KSC algorithm. In the experiments, we demonstrate the performance of the proposed method on synthetic data sets and real-life videos where the clusters evolve in a smooth fashion over time. [ABSTRACT FROM AUTHOR]

We present a novel approach to learn binary classifiers when only positive and unlabeled instances are available (PU learning). This problem is routinely cast as a supervised task with label noise in the negative set. We use an ensemble of SVM models trained on bootstrap resamples of the training data for increased robustness against label noise. The approach can be considered in a bagging framework which provides an intuitive explanation for its mechanics in a semi-supervised setting. We compared our method to state-of-the-art approaches in simulations using multiple public benchmark data sets. The included benchmark comprises three settings with increasing label noise: (i) fully supervised, (ii) PU learning and (iii) PU learning with false positives. Our approach shows a marginal improvement over existing methods in the second setting and a significant improvement in the third. [ABSTRACT FROM AUTHOR]

Without an inclusion of process dynamics, traditional data-driven soft sensors are termed as static because only single snapshots of process samples are used. It leads to a series of limitations, such as sensitivity to temporal noises and inaccurate description in process dynamics. To this end, static models have been extended to dynamic versions thereof like dynamic partial least squares (DPLS) with lagged inputs for the sake of process dynamics. The dimension of soft sensor models’ inputs, however, could be considerably larger than static ones, which leads to the over-fitting problem. In this paper, we introduce the concept of temporal smoothness as a novel approach to DPLS-based dynamic soft sensor modeling. The starting point is to not only include historical process data but also impose smoothness regularization on proximal dynamic parameters. The smoothness regularization assumes that historical inputs have smoothly varying impacts on the latent variables as a valid prior knowledge, which is in consensus with the physical truth of industrial processes. Therefore abrupt changes in model dynamics are desirably penalized and the DPLS-based soft sensors enjoy better generalizations and interpretations. A numerical example is given to demonstrate the advantages of temporal smoothness. A simulated Tennessee Eastman process study and a real quality prediction task in a crude distillation unit process are provided to show the feasibility as well as effectiveness of our method. [ABSTRACT FROM AUTHOR]

In this paper we present a novel method for unraveling the hierarchical clusters in a given dataset using the Gershgorin circle theorem. The Gershgorin circle theorem provides upper bounds on the eigenvalues of the normalized Laplacian matrix. This can be utilized to determine the ideal range for the number of clusters ( k ) at different levels of hierarchy in a given dataset. The obtained intervals help to reduce the search space for identifying the ideal value of k at each level. Another advantage is that we don’t need to perform the computationally expensive eigen-decomposition step to obtain the eigenvalues and eigenvectors. The intervals provided for k can be considered as input for any spectral clustering method which uses a normalized Laplacian matrix. We show the effectiveness of the method in combination with a spectral clustering method to generate hierarchical clusters for several synthetic and real world datasets. [ABSTRACT FROM AUTHOR]

To pursue the insensitivity to feature noise and the stability to re-sampling, a new type of support vector machine (SVM) has been established via replacing the hinge loss in the classical SVM by the pinball loss and was hence called a pin-SVM . Though a different loss function is used, pin-SVM has a similar structure as the classical SVM. Specifically, the dual problem of pin-SVM is a quadratic programming problem with box constraints, for which the sequential minimal optimization (SMO) technique is applicable. In this paper, we establish SMO algorithms for pin-SVM and its sparse version. The numerical experiments on real-life data sets illustrate both the good performance of pin-SVMs and the effectiveness of the established SMO methods. [ABSTRACT FROM AUTHOR]

Multi-view Spectral Clustering (MvSC) partitions data into clusters according to multiple views for higher performance. However, most existing works overlook model interpretability and involve iterative and alternating updates on parameters with expensive computations, where out-of-sample predictions are also commonly prohibitive. In this paper, we construct a novel weighted conjugate feature duality for formulating the MvSC problem cast as a weighted principal components analysis. In this new method, a common latent space is derived with shared dual variables, which achieves the couplings by learning projections from different views into the latent space. To boost higher-order correlations, the tensor-based modeling is introduced. Our method leads to a closed-form solution in the dual through a single eigendecomposition thanks to the primal–dual representation, leading to easy algorithmic setups in practice. Our novel conjugate duality and the conceived shared latent space greatly boost the efficiency in model representation and computation: element-wise operations of kernels are involved in the dual instead of the explicit tensor computation in the primal and the size of our eigendecomposition is independent of the number of views. Besides, we can effectively incorporate the visualization of the clusters in a single latent space, which is particularly useful with many data views. For predicting unseen data under generic applications, our method can be flexibly applied with out-of-sample extensions, which can also be utilized for further efficiency in large-scale cases with fixed-size kernel schemes. Numerical experiments verify our superiority in accuracy, efficiency, and diverse data explorations. • A new method for MvSC is proposed via a novel conjugate feature duality. • With our duality, a common latent space can be imposed for different data sources. • A tensor-based modeling is derived for higher-order couplings and diversified fusion. • Our method gives an efficient representation and a closed-form solution in the dual. • Experiments verify our superiority in accuracy, efficiency, and interpretation. [ABSTRACT FROM AUTHOR]

Abstract: As the simplest extension to linear classifiers, piecewise linear (PWL) classifiers have attracted a lot of attention, because of their simplicity and classification capability. In this paper, a PWL feature mapping is introduced by investigating the property of the PWL classification boundary. Then support vector machines (SVM) with PWL feature mappings are proposed, called PWL-SVMs. In this paper, it is shown that some widely used PWL classifiers, such as k-nearest-neighbor, adaptive boosting of linear classifier and intersection kernel support vector machine, can be represented by the proposed feature mapping. That means the proposed PWL-SVMs at least can achieve the performance of the above PWL classifiers. Moreover, PWL-SVMs enjoy good properties of SVM and the performance on numerical experiments illustrates the effectiveness. Then some extensions are discussed and the application of PWL-SVMs can be expected. [Copyright &y& Elsevier]

is paper shows the feasibility of utilizing the Kernel Spectral Clustering (KSC) method for the purpose of community detection in big data networks. KSC employs a primal-dual framework to construct a model. It results in a powerful property of effectively inferring the community affiliation for out-of-sample extensions. The original large kernel matrix cannot fitinto memory. Therefore, we select a smaller subgraph that preserves the overall community structure to construct the model. It makes use of the out-of-sample extension property for community membership of the unseen nodes. We provide a novel memory- and computationally efficient model selection procedure based on angular similarity in the eigenspace. We demonstrate the effectiveness of KSC on large scale synthetic networks and real world networks like the YouTube network, a road network of California and the Livejournal network. These networks contain millions of nodes and several million edges. [ABSTRACT FROM AUTHOR]

Abstract: Evolving graphs describe many natural phenomena changing over time, such as social relationships, trade markets, metabolic networks etc. In this framework, performing community detection and analyzing the cluster evolution represents a critical task. Here we propose a new model for this purpose, where the smoothness of the clustering results over time can be considered as a valid prior knowledge. It is based on a constrained optimization formulation typical of Least Squares Support Vector Machines (LS-SVM), where the objective function is designed to explicitly incorporate temporal smoothness. The latter allows the model to cluster the current data well and to be consistent with the recent history. We also propose new model selection criteria in order to carefully choose the hyper-parameters of our model, which is a crucial issue to achieve good performances. We successfully test the model on four toy problems and on a real world network. We also compare our model with Evolutionary Spectral Clustering, which is a state-of-the-art algorithm for community detection of evolving networks, illustrating that the kernel spectral clustering with memory effect can achieve better or equal performances. [Copyright &y& Elsevier]

This paper shows the feasibility of utilizing the Kernel Spectral Clustering (KSC) method for the purpose of community detection in big data networks. KSC employs a primal-dual framework to construct a model. It results in a powerful property of effectively inferring the community affiliation for out-of-sample extensions. The original large kernel matrix cannot fitinto memory. Therefore, we select a smaller subgraph that preserves the overall community structure to construct the model. It makes use of the out-of-sample extension property for community membership of the unseen nodes. We provide a novel memory- and computationally efficient model selection procedure based on angular similarity in the eigenspace. We demonstrate the effectiveness of KSC on large scale synthetic networks and real world networks like the You Tube network, a road network of California and the Livejournal network. These networks contain millions of nodes and several million edges. [ABSTRACT FROM AUTHOR]

Abstract: Tensor-based techniques for learning allow one to exploit the structure of carefully chosen representations of data. This is a desirable feature in particular when the number of training patterns is small which is often the case in areas such as biosignal processing and chemometrics. However, the class of tensor-based models is somewhat restricted and might suffer from limited discriminative power. On a different track, kernel methods lead to flexible nonlinear models that have been proven successful in many different contexts. Nonetheless, a naïve application of kernel methods does not exploit structural properties possessed by the given tensorial representations. The goal of this work is to go beyond this limitation by introducing non-parametric tensor-based models. The proposed framework aims at improving the discriminative power of supervised tensor-based models while still exploiting the structural information embodied in the data. We begin by introducing a feature space formed by multilinear functionals. The latter can be considered as the infinite dimensional analogue of tensors. Successively we show how to implicitly map input patterns in such a feature space by means of kernels that exploit the algebraic structure of data tensors. The proposed tensorial kernel links to the MLSVD and features an interesting invariance property; the approach leads to convex optimization and fits into the same primal–dual framework underlying SVM-like algorithms. [Copyright &y& Elsevier]

It is a well-known problem that obtaining a correct bandwidth and/or smoothing parameter in nonparametric regression is difficult in the presence of correlated errors. There exist a wide variety of methods coping with this problem, but they all critically depend on a tuning procedure which requires accurate information about the correlation structure. We propose a bandwidth selection procedure based on bimodal kernels which successfully removes the correlation without requiring any prior knowledge about its structure and its parameters. Further, we show that the form of the kernel is very important when errors are correlated which is in contrast to the independent and identically distributed (i.i.d.) case. Finally, some extensions are proposed to use the proposed criterion in support vector machines and least squares support vector machines for regression. [ABSTRACT FROM AUTHOR]

Summary: Objective: This study investigates the use of automated pattern recognition methods on magnetic resonance data with the ultimate goal to assist clinicians in the diagnosis of brain tumours. Recently, the combined use of magnetic resonance imaging (MRI) and magnetic resonance spectroscopic imaging (MRSI) has demonstrated to improve the accuracy of classifiers. In this paper we extend previous work that only uses binary classifiers to assess the type and grade of a tumour to a multiclass classification system obtaining class probabilities. The important problem of input feature selection is also addressed. Methods and material: Least squares support vector machines (LS-SVMs) with radial basis function kernel are applied and compared with linear discriminant analysis (LDA). Both a Bayesian framework and cross-validation are used to infer the parameters of the LS-SVM classifiers. Four different techniques to obtain multiclass probabilities as a measure of accuracy are compared. Four variable selection methods are explored. MRI and MRSI data are selected from the INTERPRET project database. Results: The results illustrate the significantly better performance of automatic relevance determination (ARD), in combination with LS-SVMs in a Bayesian framework and coupling of class probabilities, compared to classical LDA. Conclusion: It is demonstrated that binary LS-SVMs can be extended to a multiclass classifier system obtaining class probabilities by Bayesian techniques and pairwise coupling. Feature selection based on ARD further improves the results. This classifier system can be of great help in the diagnosis of brain tumours. [Copyright &y& Elsevier]

Abstract: Model-free estimates of the noise variance are important in model selection and setting tuning parameters. In this paper a data representation is discussed which leads to such an estimator suitable for multivariate data. Its visual representation—called the differogram cloud here—is based on the 2-norm of the differences of input and output data. The crucial concept of locality in this representation is translated as the increasing variance of the difference, which does not rely explicitly on an extra hyper-parameter. Connections with -statistics, Taylor series expansions and other related methods are given. Numerical simulations indicate a convergence of the estimator. This paper extends results towards a time-dependent setting and to the case of non-Gaussian noise models or outliers. As an application, this paper focuses on model selection for Least Squares Support Vector Machines. For this purpose, a variant of the LS-SVM regressor is derived based on Morozov''s discrepancy principle relating the regularization constant directly with the (observed) noise level. [Copyright &y& Elsevier]

Abstract: This paper studies a method for the identification of Hammerstein models based on least squares support vector machines (LS-SVMs). The technique allows for the determination of the memoryless static nonlinearity as well as the estimation of the model parameters of the dynamic ARX part. This is done by applying the equivalent of Bai''s overparameterization method for identification of Hammerstein systems in an LS-SVM context. The SISO as well as the MIMO identification cases are elaborated. The technique can lead to significant improvements with respect to classical overparameterization methods as illustrated in a number of examples. Another important advantage is that no stringent assumptions on the nature of the nonlinearity need to be imposed except for a certain degree of smoothness. [Copyright &y& Elsevier]

Coupled local minimizers (CLM) is a new method applicable to global optimization of functions with multiple local minima. In CLM a cooperative search mechanism is set up using a population of local optimizers, which are coupled during the search process by synchronization constraints. CLM is characterised by a relative fast convergence since the local optimizers are gradient-based. The combination of both, the coupled parallel strategy and the fast convergence, offers an efficient global optimization algorithm. In the paper the CLM method is described and is illustrated with a test function. Due to the simultaneous and coupled search of a whole population of optimizers, CLM is able to find the global minimum of the test function. Next, CLM is successfully applied to FE model updating using experimental modal data. In an example the damage pattern of a reinforced concrete beam is identified. [Copyright &y& Elsevier]

In this paper, we propose a kernel principal component analysis model for multi-variate time series forecasting, where the training and prediction schemes are derived from the multi-view formulation of Restricted Kernel Machines. The training problem is simply an eigenvalue decomposition of the summation of two kernel matrices corresponding to the views of the input and output data. When a linear kernel is used for the output view, it is shown that the forecasting equation takes the form of kernel ridge regression. When that kernel is non-linear, a pre-image problem has to be solved to forecast a point in the input space. We evaluate the model on several standard time series datasets, perform ablation studies, benchmark with closely related models and discuss its results. • A novel time series forecasting model based on restricted kernel machines (RKM). • Training is an eigenvalue decomposition of a summation of two kernel matrices. • Future time series values can be recursively forecasted using the trained RKM model. • The dynamics of the data are captured by the model's orthogonal latent components. [ABSTRACT FROM AUTHOR]

Abstract The one-bit quantization is implemented by one single comparator that operates at low power and a high rate. Hence one-bit compressive sensing (1bit-CS) becomes attractive in signal processing. When measurements are corrupted by noise during signal acquisition and transmission, 1bit-CS is usually modeled as minimizing a loss function with a sparsity constraint. The one-sided ℓ 1 loss and the linear loss are two popular loss functions for 1bit-CS. To improve the decoding performance on noisy data, we consider the pinball loss , which provides a bridge between the one-sided ℓ 1 loss and the linear loss. Using the pinball loss, two convex models, an elastic-net pinball model and its modification with the ℓ 1 -norm constraint, are proposed. To efficiently solve them, the corresponding dual coordinate ascent algorithms are designed and their convergence is proved. The numerical experiments confirm the effectiveness of the proposed algorithms and the performance of the pinball loss minimization for 1bit-CS. [ABSTRACT FROM AUTHOR]