Search

Recently, the minimum error entropy (MEE) criterion has been used as an information theoretic alternative to traditional mean-square error criterion in supervised learning systems. MEE yields nonquadratic, nonconvex performance surface even for adaptive linear neuron (ADALINE) training, which complicates the theoretical analysis of the method. In this paper, we develop a unified approach for mean-square convergence analysis for ADALINE training under MEE criterion. The weight update equation is formulated in the form of block-data. Based on a block version of energy conservation relation, and under several assumptions, we carry out the mean-square convergence analysis of this class of adaptation algorithm, including mean-square stability, mean-square evolution (transient behavior) and the mean-square steady-state performance. Simulation experimental results agree with the theoretical predictions very well.

The multivariate autoregressive (MVAR) model is widely used in describing the dynamics of nonlinear systems, in which the estimates of model parameters and underlying states can be achieved by dual extended Kalman filter (DEKF). However, when the measurements are corrupted by complicated non-Gaussian noises, the DEKF based on the minimum mean-square error (MMSE) criterion may provide biased estimates. In the present article, we develop a novel dual Kalman-type filter, referred to as DEKF under minimum error entropy (MEE) with fiducial points (MEEFs-DEKF) to deal with the non-Gaussian noises. First, the equivalent state-space model and parameter-space model are presented based on the MVAR model. Then, an optimality criterion based on MEE with fiducial points (MEEFs) is applied in the batch-mode regression equation to improve the robustness. Finally, a fixed-point iteration algorithm gives the posterior estimates of state and parameter. Simulation results confirm that the proposed MEEF-DEKF can achieve excellent performance in various noises with different distributions, especially in heavy-tailed and multimodal noises. [ABSTRACT FROM AUTHOR]

Monitoring the consciousness states of patients and ensuring the appropriate depth of anesthesia (DOA) is critical for the safe implementation of surgery. In this study, a high-density electroencephalogram (EEG) combined with blood drug concentration and behavioral response indicators was used to monitor propofol-induced sedation and evaluate the alterations in consciousness states. Microstate analysis, which can reflect the semi-stable state of the sub-second activation of the brain functional network, can be used to assess the brain’s consciousness states. In this research, the EEG microstate sequences were constructed to compare the characteristics of corresponding sequences. Compared with the baseline (BS) state, the microstate sequences in the moderate sedation (MD) state exhibited higher complexity indexes of the multiscale sample entropy. With respect to the transition probability (TP) of microstates, most microstates tended to be converted into microstate C in the BS state. In contrast, they tended to be converted into microstate F in the MD state. The significant difference between the expected TP and observed TP could lead to the conclusion that hidden layers were present when there were changes in the consciousness states. According to the hidden Markov model, the accuracy of distinguishing the BS and MD states was 80.16%. The characteristics of microstate sequence revealed the variations in the brain states caused by alterations in consciousness states during anesthesia from a new perspective and presented a new idea for monitoring the DOA. [ABSTRACT FROM AUTHOR]

The spiking neural network (SNN) is regarded as a promising candidate to deal with the great challenges presented by current machine learning techniques, including the high energy consumption induced by deep neural networks. However, there is still a great gap between SNNs and the online meta-learning performance of artificial neural networks. Importantly, existing spike-based online meta-learning models do not target the robust learning based on spatio-temporal dynamics and superior machine learning theory. In this invited article, we propose a novel spike-based framework with minimum error entropy, called MeMEE, using the entropy theory to establish the gradient-based online meta-learning scheme in a recurrent SNN architecture. We examine the performance based on various types of tasks, including autonomous navigation and the working memory test. The experimental results show that the proposed MeMEE model can effectively improve the accuracy and the robustness of the spike-based meta-learning performance. More importantly, the proposed MeMEE model emphasizes the application of the modern information theoretic learning approach on the state-of-the-art spike-based learning algorithms. Therefore, in this invited paper, we provide new perspectives for further integration of advanced information theory in machine learning to improve the learning performance of SNNs, which could be of great merit to applied developments with spike-based neuromorphic systems. [ABSTRACT FROM AUTHOR]

To date, most linear and nonlinear Kalman filters (KFs) have been developed under the Gaussian assumption and the well-known minimum mean square error (MMSE) criterion. In order to improve the robustness with respect to impulsive (or heavy-tailed) non-Gaussian noises, the maximum correntropy criterion (MCC) has recently been used to replace the MMSE criterion in developing several robust Kalman-type filters. To deal with more complicated non-Gaussian noises such as noises from multimodal distributions, in this article, we develop a new Kalman-type filter, called minimum error entropy KF (MEE-KF), by using the minimum error entropy (MEE) criterion instead of the MMSE or MCC. Similar to the MCC-based KFs, the proposed filter is also an online algorithm with the recursive process, in which the propagation equations are used to give prior estimates of the state and covariance matrix, and a fixed-point algorithm is used to update the posterior estimates. In addition, the MEE extended KF (MEE-EKF) is also developed for performance improvement in the nonlinear situations. The high accuracy and strong robustness of MEE-KF and MEE-EKF are confirmed by experimental results. [ABSTRACT FROM AUTHOR]

In recent years, the sparse system identification (SSI) has received increasing attention, and various sparsity-aware adaptive algorithms based on the minimum mean square error (MMSE) criterion have been developed, which are optimal under the assumption of Gaussian distributions. However, the Gaussian assumption does not always hold in real-world environments. The maximum correntropy criterion (MCC) is used to replace the MMSE criterion to suppress the heavy-tailed non-Gaussian noises. For some more complex non-Gaussian noises such as those from multimodal distributions, the minimum error entropy (MEE) criterion can outperform MCC although it is computationally somewhat more expensive. To improve the performance of SSI in non-Gaussian noises, in this brief we develop a class of sparsity-aware MEE algorithms with the fixed point iteration (MEE-FP) by incorporating the zero-attracting ($\ell _{1}$ -norm), reweighted zero-attracting (reweighted $\ell _{1}$ -norm) and correntropy induced metric (CIM) penalty terms into the cost function. The corresponding algorithms are termed as ZA-MEE-FP, RZA-MEE-FP, and CIM-MEE-FP, which can achieve better performance than the original MEE-FP algorithm and the MCC based sparsity-aware algorithms. Simulation results confirm the excellent performance of the new algorithms. [ABSTRACT FROM AUTHOR]

The unscented Kalman filter (UKF) provides a powerful tool for power system forecasting-aided state estimation (FASE). However, when the power systems are affected by the abnormal operating situations, i.e., the non-Gaussian communication noises, sudden loads or state changes, and instrument failures, the original UKF based on the minimum mean square error (MMSE) criterion may suffer from performance degradation. In contrast to the MMSE criterion, the minimum error entropy (MEE) exhibits the robustness with respect to complex non-Gaussian disturbances. In this article, we develop a new unscented Kalman-type filter based on the MEE criterion, termed MEE-UKF. To derive the MEE-UKF, a statistical linearization approach is adopted in the augmented model such that the state and measurement errors are combined in the MEE cost function simultaneously. Then, a fixed-point iteration algorithm is used to recursively update the posterior estimates and covariance matrix. Apart from the impulsive noises, the MEE-UKF can deal with complex multimodal distribution noises in both process and measurement. The high accuracy and strong robustness of MEE-UKF are confirmed by the simulation results on IEEE 14, 30, and 57 bus test systems under different non-Gaussian disturbances. [ABSTRACT FROM AUTHOR]

Common spatial pattern (CSP) is a classic method commonly used in multichannel electroencephalogram (EEG) signal processing, which aims to extract effective features for binary classification by solving spatial filters that maximize the ratio of filtered dispersion between two classes. The aim of this paper is to improve the performance of the conventional CSP method, which will be badly influenced by noises. The recently proposed quantized minimum error entropy (QMEE) criterion is applied to structure a new objective function instead of the ${L} _{2}$ -norm in the conventional CSP. Quantization is utilized to reduce the computational complexity. The new objective function is optimized by a gradient-based iterative algorithm. The desirable performance of the QMEE-based CSP method, namely CSP-QMEE, is demonstrated with a toy example and two real EEG datasets, including Dataset IIb of the brain–computer interfaces (BCIs) Competition IV (three channels) and Dataset IIIa of the BCI Competition III (60 channels). The new method can achieve satisfactory performance compared to existing methods on all datasets. The promising results in this paper suggest that the CSP-QMEE may become a powerful tool for BCIs. [ABSTRACT FROM AUTHOR]

Linear regression model (LRM) based on mean square error (MSE) criterion is widely used in Granger causality analysis (GCA), which is the most commonly used method to detect the causality between a pair of time series. However, when signals are seriously contaminated by non-Gaussian noises, the LRM coefficients will be inaccurately identified. This may cause the GCA to detect a wrong causal relationship. Minimum error entropy (MEE) criterion can be used to replace the MSE criterion to deal with the non-Gaussian noises. But its calculation requires a double summation operation, which brings computational bottlenecks to GCA especially when sizes of the signals are large. To address the aforementioned problems, in this letter, we propose a new method called GCA based on the quantized MEE (QMEE) criterion (GCA-QMEE), in which the QMEE criterion is applied to identify the LRM coefficients and the quantized error entropy is used to calculate the causality indexes. Compared with the traditional GCA, the proposed GCA-QMEE not only makes the results more discriminative, but also more robust. Its computational complexity is also not high because of the quantization operation. Illustrative examples on synthetic and EEG datasets are provided to verify the desirable performance and the availability of the GCA-QMEE. [ABSTRACT FROM AUTHOR]

Minimum error entropy (MEE), as a robust adaption criterion, has received considerable attention due to its broad applicability, especially in the presence of non-Gaussian noises. In this brief, we propose a constrained adaptive filtering algorithm under MEE criterion, called CMEE, which is derived by incorporating a set of linear equality constrains into MEE to handle a constrained MEE optimization problem. In addition, convergence analysis of the proposed CMEE including the stability and steady-state mean square deviation is studied. Simulation results validate the theoretical conclusions, and confirm the effectiveness of the new algorithm in non-Gaussian noises. [ABSTRACT FROM AUTHOR]

Minimum error entropy with fiducial points (MEEF) has received a lot of attention, due to its outstanding performance to curb the negative influence caused by non-Gaussian noises in the fields of machine learning and signal processing. However, the estimate of the information potential of MEEF involves a double summation operator based on all available error samples, which can result in large computational burden in many practical scenarios. In this paper, an efficient quantization method is therefore adopted to represent the primary set of error samples with a smaller subset, generating a quantized MEEF (QMEEF). Some basic properties of QMEEF are presented and proved from theoretical perspectives. In addition, we have applied this new criterion to train a class of linear-in-parameters models, including the commonly used linear regression model, random vector functional link network, and broad learning system as special cases. Experimental results on various datasets are reported to demonstrate the desirable performance of the proposed methods to perform regression tasks with contaminated data. [ABSTRACT FROM AUTHOR]

Permutation entropy (PE) has been recently suggested as a novel measure to characterize the complexity of nonlinear time series. In this paper, we propose a simple method to address some of PE's limitations, mainly its inability to differentiate between distinct patterns of a certain motif and the sensitivity of patterns close to the noise floor. The method relies on the fact that patterns may be too disparate in amplitudes and variances and proceeds by assigning weights for each extracted vector when computing the relative frequencies associated with every motif. Simulations were conducted over synthetic and real data for a weighting scheme inspired by the variance of each pattern. Results show better robustness and stability in the presence of higher levels of noise, in addition to a distinctive ability to extract complexity information from data with spiky features or having abrupt changes in magnitude. [ABSTRACT FROM AUTHOR]

Recent studies suggest that the minimum error entropy (MEE) criterion can outperform the traditional mean square error criterion in supervised machine learning, especially in nonlinear and non-Gaussian situations. In practice, however, one has to estimate the error entropy from the samples since in general the analytical evaluation of error entropy is not possible. By the Parzen windowing approach, the estimated error entropy converges asymptotically to the entropy of the error plus an independent random variable whose probability density function (PDF) corresponds to the kernel function in the Parzen method. This quantity of entropy is called the smoothed error entropy, and the corresponding optimality criterion is named the smoothed MEE (SMEE) criterion. In this paper, we study theoretically the SMEE criterion in supervised machine learning where the learning machine is assumed to be nonparametric and universal. Some basic properties are presented. In particular, we show that when the smoothing factor is very small, the smoothed error entropy equals approximately the true error entropy plus a scaled version of the Fisher information of error. We also investigate how the smoothing factor affects the optimal solution. In some special situations, the optimal solution under the SMEE criterion does not change with increasing smoothing factor. In general cases, when the smoothing factor tends to infinity, minimizing the smoothed error entropy will be approximately equivalent to minimizing error variance, regardless of the conditional PDF and the kernel. [ABSTRACT FROM AUTHOR]

The minimum error entropy (MEE) criterion has been receiving increasing attention due to its promising perspectives for applications in signal processing and machine learning. In the context of Bayesian estimation, the MEE criterion is concerned with the estimation of a certain random variable based on another random variable, so that the error's entropy is minimized. Several theoretical results on this topic have been reported. In this work, we present some further results on the MEE estimation. The contributions are twofold: (1) we extend a recent result on the minimum entropy of a mixture of unimodal and symmetric distributions to a more general case, and prove that if the conditional distributions are generalized uniformly dominated (GUD), the dominant alignment will be the MEE estimator; (2) we show by examples that the MEE estimator (not limited to singular cases) may be non-unique even if the error distribution is restricted to zero-mean (unbiased). [ABSTRACT FROM AUTHOR]

Recently, the information potential (IP) of order \alpha, defined as the argument of the log in the \alpha-order Renyi entropy, has been successfully used as an information theoretic criterion for supervised adaptive system training. In this paper, we use the survival function (or equivalently the distribution function) of an absolute value transformed random variable to define a new information potential, named the survival information potential (SIP). Compared with the IP, the SIP has some advantages, such as validity in a wide range of distributions, robustness, and the simplicity in computation. The properties of SIP and a simple formula for computing the empirical SIP are given in the paper. Finally, the SIP criterion is applied in adaptive system training, and simulation examples on FIR adaptive filtering, kernel adaptive filtering, and time delay neural networks (TDNNs) training are presented to demonstrate the performance. [ABSTRACT FROM PUBLISHER]