Search

Ensemble methods have shown to be more effective than monolithic classifiers, in particular when diversity holds among their components. How to enforce diversity in classifier ensembles has received much attention from machine learning researchers, yielding a variety of different techniques and algorithms. In this paper, a novel algorithm for ensemble classifiers is proposed, in which ensemble components are trained with focus on different regions of the sample space. In so doing, diversity is mainly a consequence of the intention to limit the scope of base classifiers. The algorithm proposed in this paper shares roots with several ensemble paradigms, in particular with random forests, as it generates forests of decision trees as well. As decision trees are trained with focus on specific subsets of the sample space, the resulting ensemble is in fact a forest of “local” trees. Comparative experimental results highlight that, on average, these ensembles perform better than other relevant kinds of ensemble classifiers, including random forests. [ABSTRACT FROM AUTHOR]

• The first attempt to design a GEP-based codematrix generation algorithm. The individual structure of GEP allows the codematrix to be optimized in a much faster and more effective way. • The feature selection scheme is combined with the design of codematrix, to accomplish the feature selection and codematrix optimization simultaneously. • A greedy strategy-based adaptive optimization method to further improve the individual's quality to deal with hard classes by appending proper columns to the codematrix. The multi-class classification task is one of the most common tasks in machine learning. As a typical solution based on a partitioning strategy, Error-Correcting Output Codes (ECOC) can transform a multi-class classification problem into multiple binary classification problems. The key of ECOC is to construct an effective codematrix to represent a set of class decomposition schemes, which transforms a multiclass problem into a group of binary class problems. Consequently, the design of a fast and effective ECOC codematrix generation method is of great research significance and value for solving multi-class classification problems. In ECOC algorithms, the design of codematrix is treated as a combination problem between different code columns, in which the evolutionary algorithm shows a great advantage. Based on this consideration, the Gene Expression Programming (GEP) is applied to search for the codematrix with high performance because its expressive tree structure makes it well represent codematrcies for subsequent optimization operations. This paper proposes an adaptive ECOC algorithm based on Gene Expression Programming (GEP) and similarity measurement matrix, named GEP-ECOC. In our GEP, each individual represents a set of columns to form a random ECOC codematrix, which is optimized in the evolutionary process. Meanwhile, the crossover and mutation operations are modified to include a legality checking process to ensure that the generated codematrix satisfies the ECOC constraints. The GEP-based ECOC codematrix generation algorithm can quickly produce a codematrix with better performance, which ensures the efficiency of the algorithm to a certain extent. In addition, an adaptive algorithm based on a similarity measurement matrix is proposed to add new columns to the current codematrix, aiming to better handle hard classes. Our algorithm is compared with other algorithms on various data sets, and the experimental results confirm that our GEP-ECOC can balance the efficiency and performance of the algorithm and achieve higher performance. [ABSTRACT FROM AUTHOR]

• Convolutional Neural Networks can be improved in terms of the classification performance and robustness by using variable weight structures. • Analysis of different data processing methods, models' robustness and statistical properties. • Comparative analysis of variable weight convolutional neural networks and other widely used machine learning techniques. • Medical applications to the classification of seizure phases and types. Deep learning techniques have recently achieved impressive results and raised expectations in the domains of medical diagnosis and physiological signal processing. The widely adopted methods include convolutional neural networks (CNNs) and recurrent neural networks (RNNs). However, the existing models possess static connection weights between layers, which might limit the generalization capability and the classification performance of the models as the weights of different layers are fixed after training. Furthermore, to deal with a large amount of data, a neural network with a sufficiently large size is required. This paper proposes the variable weight convolutional neural networks (VWCNNs), which are a type of network structure employing dynamic weights instead of static weights in their convolutional layers and fully-connected layers. VWCNNs are able to adapt to different characteristics of input data and can be viewed as an infinite number of traditional, fixed-weight CNNs. We will show that the proposed VWCNN structure outperforms the conventional CNN in terms of the classification accuracy, generalization capability, and robustness when the inputs are contaminated by noise. In this paper, VWCNNs are applied to the classification of three seizure phases (seizure-free, pre-seizure and seizure) based on measured electroencephalography (EEG) data. VWCNNs achieve 100% test accuracy and show strong robustness in the classification of the three seizure phases, and thus show the potential to be a useful classification tool for medical diagnosis. Furthermore, the classification of seven types of seizures is investigated in this paper using the world's largest open source database of seizure recordings, TUH EEG seizure corpus. Comparisons with conventional CNNs, RNN, MobileNet, ResNet, DenseNet and traditional machine learning methods including random forest, decision tree, support vector machine, K-nearest neighbours, standard neural networks, and Naïve Bayes are being conducted using realistic test data sets. The results demonstrate that VWCNNs have advantages over other classifiers in terms of classification accuracy and robustness. [ABSTRACT FROM AUTHOR]

The paper is devoted to two problems: (1) reinforcement of SVM algorithms, and (2) justification of memorization mechanisms for generalization. (1) Current SVM algorithm was designed for the case when the risk for the set of nonnegative slack variables is defined by l 1 norm. In this paper, along with that classical l 1 norm, we consider risks defined by l 2 norm and l ∞ norm. Using these norms, we formulate several modifications of the existing SVM algorithm and show that the resulting modified SVM algorithms can improve (sometimes significantly) the classification performance. (2) Generalization ability of existing learning algorithms is usually explained by arguments involving uniform convergence of empirical losses to the corresponding expected losses over a given set of functions. However, along with bounds for uniform convergence of empirical losses to the expected losses, the VC theory also provides bounds for relative uniform convergence. These bounds lead to a more accurate estimate of the expected loss. Advanced methods of estimating of expected risk of error have to leverage these bounds, which also support mechanisms of training data memorization, which, as the paper demonstrates, can improve classification performance. [ABSTRACT FROM AUTHOR]

Highlights • Optimization is an important issue in machine learning because many machine learning models are reformulated as optimization problems. Different kinds of machine learning algorithms mainly focus on minimizing their emiprical loss like deep learning, logisitic regression, and support vector machine. Because data is explosively growing, it is challenging to deal with a large-scale optimization problem. We propose a fast and most condition number free algorithm for train machine learning algorithms. This algorithm has several advantages. • Contrast to previous work which approximates Hessian matrix by sketched or subsampled Hessian directly, we use the sketched or subsampled Hessian matrix as preconditioner and get the direction vector by preconditioned conjugate gradient method. Accordingly, we propose a novel stochastic second-order algorithm called Preconditioned Newton Conjugate Gradient with Sketched Hessian (PNCG). This strategy makes our algorithm almost independent on the condition number. • We numerically demonstrate the effectiveness and robustness of our algorithms in widely used machine learning algorithm Ridge Logistic Regression on several real datasets with different condition number. Our algorithms show good performance compared with existing second order methods in all cases, especially when condition number is large. Abstract Optimization is an important issue in machine learning because many machine learning models are reformulated as optimization problems. Different kinds of machine learning algorithms mainly focus on minimizing their empirical loss like deep learning, logistic regression, and support vector machine. Because data is explosively growing, it is challenging to deal with a large-scale optimization problem. Recently, stochastic second-order methods have emerged to attract much attention due to their efficiency in each iteration. These methods show good performance on training machine learning algorithms like logistic regression and support vector machine. However, the computational complexity of existing stochastic second-order methods heavily depends on the condition number of the Hessian. In this paper, we propose a new Newton-like method called Preconditioned Newton Conjugate Gradient with Sketched Hessian (PNCG). The runtime complexity of PNCG is at most logarithmic in the condition number of the Hessian. PNCG exhibits advantages over existing subsampled Newton methods especially when the Hessian matrix in question is ill-conditioned. We also show that our method has good performance on training machine learning algorithm empirically. The results show consistent improvements in computational efficiency. [ABSTRACT FROM AUTHOR]

Abstract In streaming data classification, most of the existing methods assume that all arrived evolving data are completely labeled. One challenge is that some applications where only small amount of labeled examples are available for training. Incremental semi-supervised learning algorithms have been proposed for regularizing neural networks by incorporating various side information, such as pairwise constraints or user-provided labels. However, it is hard to put them into practice, especially for non-stationary environments due to the effectiveness and parameter sensitivity of such algorithms. In this paper, we propose a novel incremental semi-supervised learning framework on streaming data. Each layer of model is comprised of a generative network, a discriminant structure and the bridge. The generative network uses dynamic feature learning based on autoencoders to learn generative features from streaming data which has been demonstrated its potential in learning latent feature representations. In addition, the discriminant structure regularizes the network construction via building pairwise similarity and dissimilarity constraints. It is also used for facilitating the parameter learning of the generative network. The network and structure are integrated into a joint learning framework and bridged by enforcing the correlation of their parameters, which balances the flexible incorporation of supervision information and numerical tractability for non-stationary environments as well as explores the intrinsic data structure. Moreover, an efficient algorithm is designed to solve the proposed optimization problem and we also give an ensemble method. Particularly, when multiple layers of model are stacked, the performance is significantly boosted. Finally, to validate the effectiveness of the proposed method, extensive experiments are conducted on synthetic and real-life datasets. The experimental results demonstrate that the performance of the proposed algorithms is superior to some state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

Highlights • Networks and graphs are naturally embedded in Euclidean hyperspheres. • Communicability embedding of networks/graphs reveals clusters in networks. • Nonmetric multidimensional scaling allows visualization of networks in 3D communicability space. • Communicability clusters of papers in a citation network reveal levels of mathematization. • Communicability clusters in a gene-gene network reveal genes that co-participate in cancer and other diseases. Abstract A complex network is a condensed representation of the relational topological framework of a complex system. A main reason for the existence of such networks is the transmission of items through the entities of these complex systems. Here, we consider a communicability function that accounts for the routes through which items flow on networks. Such a function induces a natural embedding of a network in a Euclidean high-dimensional sphere. We use one of the geometric parameters of this embedding, namely the angle between the position vectors of the nodes in the hyperspheres, to extract structural information from networks. First we propose a simple method for visualizing networks by reducing the dimensionality of the communicability space to 3-dimensional spheres. Secondly, we use clustering analysis to cluster the nodes of the networks based on their similarities in terms of their capacity to successfully deliver information through the network. After testing these approaches in benchmark networks and compare them with the most used clustering methods in networks we analyze two real-world examples. In the first, consisting of a citation network, we discover citation groups that reflect the level of mathematics used in their publications. In the second, we discover groups of genes that coparticipate in human diseases, reporting a few genes that coparticipate in cancer and other diseases. Both examples emphasize the potential of the current methodology for the discovery of new patterns in relational data. [ABSTRACT FROM AUTHOR]

Abstract In this paper, we propose the use of data symmetries, in the sense of equivalences under signal transformations, as priors for learning symmetry-adapted data representations, i.e., representations that are equivariant to these transformations. We rely on a group-theoretic definition of equivariance and provide conditions for enforcing a learned representation, for example the weights in a neural network layer or the atoms in a dictionary, to have the structure of a group and specifically the group structure in the distribution of the input. By reducing the analysis of generic group symmetries to permutation symmetries, we devise a regularization scheme for representation learning algorithm, using an unlabeled training set. The proposed regularization is aimed to be a conceptual, theoretical and computational proof of concept for symmetry-adapted representation learning, where the learned data representations are equivariant or invariant to transformations, without explicit knowledge of the underlying symmetries in the data. [ABSTRACT FROM AUTHOR]

Due to the ability of capturing the geometry structure of data manifold, context-sensitive similarity has demonstrated impressive performances in the retrieval task. The key idea of context-sensitive similarity is that the similarity between two data points can be more reliably estimated with the local context of other points in the affinity graph. Therefore, neighborhood selection is a crucial factor for those algorithms, which affects the performance dramatically. In this paper, we propose a new algorithm called Smooth Neighborhood (SN) that mines the neighborhood structure to satisfy the manifold assumption. By doing so, nearby points on the underlying manifold are guaranteed to yield similar neighbors as much as possible. Moreover, SN is adjusted to tackle multiple affinity graphs by imposing a weight learning paradigm, and this is the primary difference compared with related works which are only applicable with one affinity graph. Finally, we integrate SN with Sparse Contextual Activation (SCA), a representative context-sensitive similarity proposed recently. Extensive experimental results and comparisons manifest that with the neighborhood structure generated by SN, the proposed framework can yield state-of-the-art performances on shape retrieval, image retrieval and 3D model retrieval. [ABSTRACT FROM AUTHOR]

The convex clustering formulation of Chi and Lange (2015) is revisited. While this formulation can be precisely and efficiently solved, it uses the standard Euclidean metric to measure the distance between the data points and their corresponding cluster centers and hence its performance deteriorates significantly in the presence of outlier features. To address this issue, this paper considers a formulation that combines convex clustering with metric learning. It is shown that: (1) for any given positive definite Mahalanobis distance metric, the problem of convex clustering can be precisely and efficiently solved using the Alternating Direction Method of Multipliers; (2) the problem of learning a positive definite Mahalanobis distance metric admits a closed-form solution; (3) an algorithm that alternates between convex clustering and metric learning can provide a significant performance boost over not only the original convex clustering formulation but also the recently proposed robust convex clustering formulation of Wang et al. (2017). [ABSTRACT FROM AUTHOR]

Many real-world applications propose the request for sharing knowledge among different tasks or datasets. Transfer learning has been proposed to solve this kind of problems and it has been successfully applied in supervised learning and semi-supervised learning settings. However, its adoption in clustering, one of the most classical research problems in machine learning and data mining, is still scarce. Spectral clustering, as a major clustering algorithm with wide applications and better performance than k -means typically, has not been well incorporated with knowledge transfer. In this paper, we first consider the problem of learning from only one auxiliary unlabeled dataset for spectral clustering and propose a novel algorithm called transfer spectral clustering (TSC). Then, it is extended to the settings with multiple auxiliary tasks. TSC assumes the feature embeddings being shared with the auxiliary tasks and utilizes co-clustering to extract useful information from the auxiliary datasets to improve the clustering performance. TSC involves not only the data manifold information of individual task but also the feature manifold information shared between related tasks. An in-depth explanation of our algorithm together with a convergence analysis are provided. As demonstrated by the extensive experiments, TSC can effectively improve the clustering performance by using auxiliary unlabeled data when compared with other state-of-the-art clustering algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel approach for learning multi-label classifiers with the help of privileged information. Specifically, we use similarity constraints to capture the relationship between available information and privileged information, and use ranking constraints to capture the dependencies among multiple labels. By integrating similarity constraints and ranking constraints into the learning process of classifiers, the privileged information and the dependencies among multiple labels are exploited to construct better classifiers during training. A maximum margin classifier is adopted, and an efficient learning algorithm of the proposed method is also developed. We evaluate the proposed method on two applications: multiple object recognition from images with the help of implicit information about object importance conveyed by the list of manually annotated image tags; and multiple facial action unit detection from low-resolution images augmented by high-resolution images. Experimental results demonstrate that the proposed method can effectively take full advantage of privileged information and dependencies among multiple labels for better object recognition and better facial action unit detection. [ABSTRACT FROM AUTHOR]

• Providing the condition under which an AHC does not effectively extract clusters. • Introducing the entanglement to measure the dendrogram quality. • Identifying the root cause of a density bias of traditional AHC algorithms. • Improving AHC performance with a data-dependent kernel. Agglomerative hierarchical clustering (AHC) is one of the popular clustering approaches. AHC generates a dendrogram that provides richer information and insights from a dataset than partitioning clustering. However, a major problem with existing distance-based AHC methods is: it fails to effectively identify adjacent clusters with varied densities, regardless of the cluster extraction methods applied to the resultant dendrogram. This paper aims to reveal the root cause of this issue and provides a solution by using a data-dependent kernel. We analyse the condition under which existing AHC methods fail to effectively extract clusters, and give the reason why the data-dependent kernel is an effective remedy. This leads to a new approach to kernerlise existing hierarchical clustering algorithms including the traditional AHC algorithms, HDBSCAN, GDL, PHA and HC-OT. Our extensive empirical evaluation shows that the recently introduced Isolation Kernel produces a higher quality or purer dendrogram than distance, Gaussian Kernel and adaptive Gaussian Kernel in all the above mentioned AHC algorithms. [ABSTRACT FROM AUTHOR]

Conventional Gaussian kernel c-means clustering algorithms are widely used in applications. However, Gaussian kernel functions have an important parameter, the width hyper-parameter, which needs to be tuned. Usually this parameter is tuned once and for all and it is the same for all variables. Thus, implicitly, all the variables are equally rescaled and therefore, they have equal importance on the clustering task. This paper presents Gaussian kernel c-means hard clustering algorithms with automated computation of the width hyper-parameters. In these kernel-based clustering algorithms, the hyper-parameters change at each iteration of the algorithm, they differ from variable to variable and can differ from cluster to cluster. Because each variable is rescaled differently according to its own hyper-parameter, these algorithms can select the important variables in the clustering process. Experiments using synthetic data sets and using UCI machine learning repository data sets corroborate the usefulness of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Manifold learning aims to discover the low dimensional space where the input high dimensional data are embedded by preserving the geometric structure. Unfortunately, almost all the existing manifold learning methods were proposed under single view scenario, and they cannot be straightforwardly applied to multiple feature sets. Although concatenating multiple views into a single feature provides a plausible solution, it remains a question on how to better explore the independence and interdependence of different views while conducting manifold learning. In this paper, we propose a multi-view manifold learning with locality alignment (MVML-LA) framework to learn a common yet discriminative low-dimensional latent space that contain sufficient information of original inputs. Both supervised algorithm (S-MVML-LA) and unsupervised algorithm (U-MVML-LA) are developed. Experiments on benchmark real-world datasets demonstrate the superiority of our proposed S-MVML-LA and U-MVML-LA over existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Adaptive graph learning methods for clustering, which adjust a data similarity matrix while taking into account its clustering capability, have drawn increasing attention in recent years due to their promising clustering performance. Existing adaptive graph learning methods are based on either original data or linearly projected data and thus rely on the assumption that either representation is a good indicator of the underlying data structure. However, this assumption is sometimes not met in high dimensional data. Studies have shown that high-dimensional data in many problems tend to lie on an embedded nonlinear manifold structure. Motivated by this observation, in this paper, we develop dual data representations, i.e., original data and a nonlinear embedding of the data obtained via an Extreme Learning Machine (ELM)-based neural network, and propose to use them as the more reliable basis for graph learning. The resulting algorithm based on ELM and Constrained Laplacian Rank (ELM-CLR) further improves the clustering capability and robustness, while retaining the advantages of adaptive graph learning, such as not requiring any post-processing to extract cluster indicators. The empirical study shows that the proposed algorithm outperforms the state-of-the-art graph-based clustering methods on a broad range of benchmark datasets. [ABSTRACT FROM AUTHOR]

• First, we propose a fast implementation on the ESDE method. The key is to equivalently transform the large matrix problem of size d into a much smaller one of size n, where d is the data dimension and n is the number of training samples, with d ≫ n in practice. Numerical results demonstrate that the proposed algorithms are at least two orders of magnitude faster than their original counterparts, with no recognition accuracy lost. • Second, to the best of our knowledge, there are no incremental algorithms for matrix exponential discriminant methods till now. To fill in this gap, the second contribution of this paper is to propose incremental ESDE algorithms for incremental learning problems. • Third, the proposed fast implementation strategy and the incremental techniques also apply to other exponential discriminant analysis methods. In various pattern classification problems, semi-supervised learning methods have shown its effectiveness in utilizing unlabeled data to yield better performance than some supervised and unsupervised learning methods. Semi-supervised discriminant embedding (SDE) is a semi-supervised extension of local discriminant embedding (LDE). However, when dealing with high dimensional data, SDE often suffers from the small-sample-size (SSS) problem. In order to settle this problem, an exponential semi-supervised discriminant embedding (ESDE) method was proposed in F. Dornaika, Y. EI Traboulsi. Matrix exponential based semi-supervised discriminant embedding for image classification , Pattern Recognition, 61 (2017): 92–103], which makes use of the tool of matrix exponential. Despite its high discriminative ability, the computational overhead of ESDE is very large for high dimensional data. In order to cure this drawback, the first contribution of this paper is to propose a fast implementation on the ESDE method. The key is to equivalently transform the large matrix problem of size d into a much smaller one of size n , where d is the data dimension and n is the number of training samples, with d ≫ n in practice. On the other hand, in many real world applications, it is likely that whole labeled training set is unavailable beforehand, and the training data is obtained incrementally. Many incremental semi-supervised learning methods have been proposed to deal with this problem, to the best of our knowledge, however, there are no incremental algorithms for matrix exponential discriminant methods till now. To fill in this gap, the second contribution of this paper is to propose incremental ESDE algorithms for incremental learning problems. Numerical experiments on some real-world data sets show the numerical behavior of the proposed algorithms. [ABSTRACT FROM AUTHOR]

Kernel Fisher discriminant analysis (KFDA) is a very popular learning method for the purpose of classification. In this paper, we propose a novel learning algorithm to improve KFDA and make it very suitable for dealing with the large-scale and high-dimensional data sets. The proposed algorithm is termed individualized KFDA (IKFDA). IKFDA is based on individualized learning , i.e., a strategy to learn and classify the individual test samples one by one. Our approach seeks to find the appropriate training subset, referred to as learning area , for each individual test sample, and then employ the learning area to construct the KFDA model for the test sample. For each individual test sample, IKFDA exploits some types of similarity measures to determine a learning area that consists of the training samples that are most similar to the test sample. Compared with the traditional learning algorithms that often exploit the whole training set to construct the learning models without considering the distribution property of the test samples, IKFDA can adaptively learn the individual test samples. It is a powerful tool to deal with the real-world complicated data sets that are often very large-scale and high-dimensional, and are usually drawn from the different distributions. Extensive experiments show that the proposed algorithm can obtain good classification results. [ABSTRACT FROM AUTHOR]

• A hybrid Machine Learning (ML) approach for an efficient wrapper feature selection is proposed for enhancing the performance and reducing the computation time of IDS while increasing the accuracy to detect the intruder in IoT networks. • The hybrid ML method consists of two techniques: Differential Evolution (DE) and Extreme Learning Machine (ELM). The DE is responsible for choosing the beneficial features while the ELM is employed to assess the selected features. • Extensive experiments are implemented to evaluate the proposed approach using different performance metrics. In addition, to verify the efficiency of proposed model, the comparison is achieved with some related existing methods such as CFS + SVM [20] , CFS + ANN [19] , GA + Bagging [18] , CFS + RF [30] , and Chi-squared + RF [30]. The intrusion detection system (IDS) has gained a rapid increase of interest due to its widely recognized potential in various security fields, however, it suffers from several challenges. Different network datasets have several redundant and irrelevant features that affect the decision of the IDS classifier. Therefore, it is essential to decrease these features to improve the system performance. In this paper, an efficient wrapper feature selection method is proposed for improving the performance and decreasing the processing time of the IDS. The proposed approach employs a differential evaluation algorithm to select the useful features whilst the extreme learning machine classifier is applied after feature selection to evaluate the selected features. Many experiments are performed using the full NSL-KDD dataset to evaluate the performance of the proposed method. The results prove that the proposed approach can efficiently reduce the features, increase the accuracy, reduce the false alarm rates, and improve the processing time of the IDS in comparison to other recent related works. [ABSTRACT FROM AUTHOR]

This paper proposes a framework for binary classification of the time series with cyclic characteristics. The framework presents an iterative algorithm for learning the cyclic characteristics by introducing the discriminative frequency bands (DFBs) using the discriminant analysis along with k-means clustering method. The DFBs are employed by a hybrid model for learning dynamic characteristics of the time series within the cycles, using statistical and structural machine learning techniques. The framework offers a systematic procedure for finding the optimal design parameters associated with the hybrid model. The proposed model is optimized to detect the changes of the heart sound recordings (HSRs) related to aortic stenosis. Experimental results show that the proposed framework provides efficient tools for the classification of the HSRs based on the heart murmurs. It is also evidenced that the hybrid model, proposed by the framework, substantially improves the classification performance when it comes to detection of the heart disease. [ABSTRACT FROM AUTHOR]

• A novel explainable neural network is proposed for modeling main effects and structured interactions. • The GAMI-Net is a disentangled feedforward network with multiple additive subnetworks. • GAMI-Net takes into account three interpretability constraints: sparsity, heredity, marginal clarity. • An adaptive training algorithm is developed for training GAMI-Net efficiently. • GAMI-Net enjoys superior interpretability and outperforms benchmark methods. The lack of interpretability is an inevitable problem when using neural network models in real applications. In this paper, an explainable neural network based on generalized additive models with structured interactions (GAMI-Net) is proposed to pursue a good balance between prediction accuracy and model interpretability. GAMI-Net is a disentangled feedforward network with multiple additive subnetworks; each subnetwork consists of multiple hidden layers and is designed for capturing one main effect or one pairwise interaction. Three interpretability aspects are further considered, including a) sparsity, to select the most significant effects for parsimonious representations; b) heredity, a pairwise interaction could only be included when at least one of its parent main effects exists; and c) marginal clarity, to make main effects and pairwise interactions mutually distinguishable. An adaptive training algorithm is developed, where main effects are first trained and then pairwise interactions are fitted to the residuals. Numerical experiments on both synthetic functions and real-world datasets show that the proposed model enjoys superior interpretability and it maintains competitive prediction accuracy in comparison to the explainable boosting machine and other classic machine learning models. [ABSTRACT FROM AUTHOR]

• we first show that the solution paths produced by various algorithms have the property of piecewise linearity. • We point out model function builds the bridge between solution path and error path, and show that the piecewise linearity of solution path leads to the piecewise linearity of model function. • Based on the piecewise linearity of model function, we prove that a large class of error (or loss) functions are piecewise constant, linear, or quadratic w.r.t. the regularization parameter. Finally, we propose our GEP for the generalized error (or loss) functions and solution path algorithms, which guarantees to find the models with the minimum CV error. • The experimental results on a variety of datasets not only confirm our theoretical findings, but also show that the best model with our GEP has better generalization error on the test data, compared to the grid search, manual search, and random search. Model selection with cross validation (CV) is very popular in machine learning. However, CV with grid and other common search strategies cannot guarantee to find the model with minimum CV error, which is often the ultimate goal of model selection. Recently, various solution path algorithms have been proposed for several important learning algorithms including support vector classification, Lasso, and so on. However, they still do not guarantee to find the model with minimum CV error. In this paper, we first show that the solution paths produced by various algorithms have the property of piecewise linearity. Then, we prove that a large class of error (or loss) functions are piecewise constant, linear, or quadratic w.r.t. the regularization parameter, based on the solution path. Finally, we propose a new generalized error path algorithm (GEP), and prove that it will find the model with minimum CV error in a finite number of steps for the entire range of the regularization parameter. The experimental results on a variety of datasets not only confirm our theoretical findings, but also show that the best model with our GEP has better generalization error on the test data, compared to the grid search, manual search, and random search. [ABSTRACT FROM AUTHOR]

• We propose a guideline to distill the architecture and knowledge of pre-trained standard CNNs simultaneously for fast compression and acceleration. • The effectiveness is first verified on offline HCTR. Compared with the baseline CNN, the corresponding compact network can reduce the computational cost by > 10 × and model size by > 8 × with negligible accuracy loss. • Furthermore, the proposed method is successfully used to reduce resource consumption of the mainstream backbone networks on CTW and MNIST. The distillation technique helps transform cumbersome neural networks into compact networks so that models can be deployed on alternative hardware devices. The main advantage of distillation-based approaches include a simple training process, supported by most off-the-shelf deep learning software and no special hardware requirements. In this paper, we propose a guideline for distilling the architecture and knowledge of pretrained standard CNNs. The proposed algorithm is first verified on a large-scale task: offline handwritten Chinese text recognition (HCTR). Compared with the CNN in the state-of-the-art system, the reconstructed compact CNN can reduce the computational cost by > 10 × and the model size by > 8 × with negligible accuracy loss. Then, by conducting experiments on two additional classification task datasets: Chinese Text in the Wild (CTW) and MNIST, we demonstrate that the proposed approach can also be successfully applied on mainstream backbone networks. [ABSTRACT FROM AUTHOR]

• We propose a new unsupervised feature selection based on information theory. • A single optimization problem is designed considering dependence among features. • We analyze the convergence of the proposed iterative algorithm. Many research topics present very high dimensional data. Because of the heavy execution times and large memory requirements, many machine learning methods have difficulty in processing these data. In this paper, we propose a new unsupervised feature selection method considering the pairwise dependence of features (feature dependency-based unsupervised feature selection, or DUFS). To avoid selecting redundant features, the proposed method calculates the dependence among features and applies this information to a regression-based unsupervised feature selection process. We can select small feature set with the dependence among features by eliminating redundant features. To consider the dependence among features, we used mutual information widely used in supervised feature selection area. To our best knowledge, it is the first study to consider the pairwise dependence of features in the unsupervised feature selection method. Experimental results for six data sets demonstrate that the proposed method outperforms existing state-of-the-art unsupervised feature selection methods in most cases. [ABSTRACT FROM AUTHOR]

• A novel hybrid approach which integrates a Faster R-CNN for crack patch detection, a DCNN for crack orientation recognition, and a Bayesian algorithm for integration. It provides a novel framework to combine deep learning models and Bayesian analysis to address challenging vision problems where the deep learning approaches with simple end-to-end learning strategy might not be effective. • A distinctive approach to apply Faster R-CNN for the challenging task of crack detection by training it to detect crack patches of suitable SNR, and a semi-automatic method to annotate crack patches of suitable scales to train a Faster R-CNN. • A new Bayesian integration algorithm based on local spatial proximity, orientation consistency and alignment consistency to connect associated neighboring crack patches and suppress false detections, as well as an efficient algorithm to learn the optimal parameters. Vision-based crack detection is of crucial importance in various industries, and it is very challenging due to weak signals in noisy backgrounds. In this paper, we propose a novel hybrid approach for crack detection in raw images, which combines deep learning models and Bayesian probabilistic analysis for robust crack detection. First, we re-train a state-of-the-art object detector (e.g. a Faster R-CNN) to detect crack patches of suitable SNR (signal-noise-ratio). We design a semi-automatic method to generate ground truths of crack patches along crack lines for training. To further improve the accuracy of crack detections over the whole image, we propose a Bayesian integration algorithm to suppress false detections. Specifically, we use a deep CNN to recognize the orientation of the crack segment in each detected patch. Then, a Bayesian probability is computed on the accumulated evidence from detected adjacent patches within a neighborhood based on spatial proximity, orientation consistency and alignment consistency. The patch which lacks local supports is suppressed as false detection. An algorithm to learn the parameters of Bayesian integration is also derived. Extensive experiments and evaluations are performed on a new comprehensive dataset of crack images. The results show that our approach outperforms the state-of-the-art baseline approach on deep CNN classifier. Ablation experiments are also conducted to show the effectiveness of proposed techniques. [ABSTRACT FROM AUTHOR]