Preserving Similarity and Staring Decisis for Feature Selection

Feature selection plays an important role in many areas that are relevant to machine learning. Information theory is widely applied to feature selection methods because it can measure linear and nonlinear correlations among variables. However, previous information theory-based feature selection methods ignore two critical problems. First, features are collected or annotated manually, it is inevitable to make some incorrect features or outliers. Second, previous information theory-based methods pay much attention to the correlations among features while ignoring the correlations between samples. To address these problems, this paper proposes an information theory-based feature selection method: Preserving Similarity and Staring Decisis. The proposed method firstly employs the similarity between samples to obtain a new transformed data that avoids outliers or incorrect features in original data. Meanwhile, the new transformed data enhances the separability of features, which is beneficial to feature selection. Additionally, different from previous methods that consider all the already-selected features in the feature selection process, the proposed method stares the last already-selected feature only, which reduces the computational cost. Extensive experiments demonstrate that the proposed method outperforms nine state-of-the-art methods in terms of multiple criteria significantly.