Search

Dimension reduction is a critical technology for high-dimensional data processing, where Linear Discriminant Analysis (LDA) and its variants are effective supervised methods. However, LDA prefers to feature with smaller variance, which causes feature with weak discriminative ability retained. In this paper, we propose a novel Ratio Sum for Linear Discriminant Analysis (RSLDA), which aims at maximizing discriminative ability of each feature in subspace. To be specific, it maximizes the sum of ratio of the between-class distance to the within-class distance in each dimension of subspace. Since the original RSLDA problem is difficult to obtain the closed solution, an equivalent problem is developed which can be solved by an alternative optimization algorithm. For solving the equivalent problem, it is transformed into two sub-problems, one of which can be solved directly, the other is changed into a convex optimization problem, where singular value decomposition is employed instead of matrix inversion. Consequently, performance of algorithm cannot be affected by the non-singularity of covariance matrix. Furthermore, Kernel RSLDA (KRSLDA) is presented to improve the robustness of RSLDA. Additionally, time complexity of RSLDA and KRSLDA are analyzed. Extensive experiments show that RSLDA and KRSLDA outperforms other comparison methods on toy datasets and multiple public datasets. [ABSTRACT FROM AUTHOR]

Linear discriminant analysis (LDA) has been proven to be effective in dimensionality reduction. However, the performance of LDA depends on the consistency assumption of the global structure and the local structure. Some work extended LDA along this line of research and proposed local formulations of LDA. Unfortunately, the learning scheme of these algorithms is suboptimal in that the intrinsic relationship between data points is pre-learned in the original space, which is usually affected by the noise and redundant features. Besides, the time cost is relatively high. To alleviate these drawbacks, we propose a Fast Locality Discriminant Analysis framework (FLDA), which has three advantages: (1) It can divide a non-Gaussian distribution class into many sub-blocks that obey Gaussian distributions by using the anchor-based strategy. (2) It captures the manifold structure of data by learning the fuzzy membership relationship between data points and the corresponding anchor points, which can reduce computation time. (3) The weights between data points and anchor points are adaptively updated in the subspace where the irrelevant information and the noise in high-dimensional space have been effectively suppressed. Extensive experiments on toy data sets, UCI benchmark data sets and imbalanced data sets demonstrate the efficiency and effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Semi-supervised learning as one of most attractive problems in machine learning research field has aroused broad attentions in recent years. In this paper, we propose a novel locality preserved dimensionality reduction framework, named Semi-supervised Adaptive Local Embedding learning (SALE), which learns a local discriminative embedding by constructing a $k_1$ k 1 Nearest Neighbors ($k_1$ k 1 NN) graph on labeled data, so as to explore the intrinsic structure, i.e., sub-manifolds from non-Gaussian labeled data. Then, mapping all samples into learned embedding and constructing another $k_2$ k 2 NN graph on all embedded data to explore the global structure of all samples. Therefore, the unlabeled data and their corresponding labeled neighbors can be clustered into same sub-manifold, so as to improve the discriminative power of embedded data. Furthermore, we propose two semi-supervised dimensionality reduction methods with orthogonal and whitening constraints based on proposed SALE framework. An efficient alternatively iterative optimization algorithm is developed to solve the NP-hard problem in our models. Extensive experiments conducted on several synthetic and real-world data sets demonstrate the superiorities of our methods on local structure exploration and classification task. [ABSTRACT FROM AUTHOR]

Linear Discriminant Analysis (LDA) has been widely used in supervised dimensionality reduction fields. However, LDA is usually weak in tackling data with Non-Gaussian distribution due to its incapability of extracting the intrinsic structure of data. In order to learn the intrinsic information more effectively, some dimensionality reduction methods incorporate the adaptive full-connected graph into the algorithm frame, but the defect is that the calculation of each pairwise distance is very time-consuming. In this letter, we propose a novel fast adaptive local subspace learning with regressive regularization model to solve the supervised dimensional reduction problem. Firstly, the adaptive anchor point graph is used to capture local structure information, which can greatly reduce computation complexity. Secondly, by using regressive regularization, the samples from different classes can be better separated in the projected space and the workload of selecting the optimal reduced dimension is easier. Moreover, entropy regularization is used to derive more appropriate weights. Finally, extensive experiments are conducted on real world data sets to verify the superiority of our model. [ABSTRACT FROM AUTHOR]

Linear discriminant analysis (LDA) is a well-known technique for supervised dimensionality reduction and has been extensively applied in many real-world applications. LDA assumes that the samples are Gaussian distributed, and the local data distribution is consistent with the global distribution. However, real-world data seldom satisfy this assumption. To handle the data with complex distributions, some methods emphasize the local geometrical structure and perform discriminant analysis between neighbors. But the neighboring relationship tends to be affected by the noise in the input space. In this research, we propose a new supervised dimensionality reduction method, namely, locality adaptive discriminant analysis (LADA). In order to directly process the data with matrix representation, such as images, the 2-D LADA (2DLADA) is also developed. The proposed methods have the following salient properties: 1) they find the principle projection directions without imposing any assumption on the data distribution; 2) they explore the data relationship in the desired subspace, which contains less noise; and 3) they find the local data relationship automatically without the efforts for tuning parameters. The performance of dimensionality reduction shows the superiorities of the proposed methods over the state of the art. [ABSTRACT FROM AUTHOR]

Principal component analysis (PCA) is warmly welcomed in dimensionality reduction and its applications. Due to the high sensitivity of PCA to outliers, a series of PCA methods are proposed to enhance the robustness of PCA. Besides, the representation ability of the existing PCA methods has limitations as well. To enhance the robustness and representation ability of robust PCA, we elaborate a novel Graph Convolution Robust PCA method (GRPCA) to incorporate the manifold structure into PCA. It constructs a sparse graph based on the local connectivity structure of samples. Graph auto-encoder is utilized to solve the robust PCA problem under the low-rank and sparse constraints. With the dual-decoder, GRPCA learns the low-dimensional embeddings that reconstruct the manifold structure and low-rank approximation simultaneously. Furthermore, since the graph suffers from misconnection triggered by occlusions, the local connectivity structure of low-dimensional embeddings is utilized to modify the graph. Our proposed method excels in both the clustering of low-dimensional embeddings and the low-rank recovery. Lastly, extensive experiments conducted on six real-world datasets demonstrated the efficiency and superiority of the proposed GRPCA. [ABSTRACT FROM AUTHOR]

There are many successful spectral based unsupervised dimensionality reduction methods, including Laplacian Eigenmap (LE), Locality Preserving Projection (LPP), Spectral Regression (SR), etc. We find that LPP and SR are equivalent if the symmetric similarity matrix is doubly stochastic, Positive Semi-Definite (PSD) and with rank $p$ p , where $p$ p is the reduced dimension. Since solving SR is believed faster than solving LPP based on some related literature, the discovery promotes us to seek to construct such specific similarity matrix to speed up LPP solving procedures. We then propose an unsupervised linear method called Unsupervised Large Graph Embedding (ULGE). ULGE starts with a similar idea as LPP but adopts an efficient approach to construct anchor-based similarity matrix and then performs spectral analysis on it. Moreover, since conventional anchor generation strategies suffer kinds of problems, we propose an efficient and effective anchor generation strategy, called Balanced $K$ K -means based Hierarchical $K$ K -means (BHKH). The computational complexity of ULGE can reduce to $O(ndm)$ O (n d m) , which is a significant improvement compared to conventional methods need $O(n^2d)$ O (n 2 d) at least, where $n$ n , $d$ d and $m$ m are the number of samples, dimensions, and anchors, respectively. Extensive experiments on several publicly available datasets demonstrate the efficiency and effectiveness of the proposed method. [ABSTRACT FROM AUTHOR]

Fuzzy K-Means (FKM) clustering is of great importance for analyzing unlabeled data. FKM algorithms assign each data point to multiple clusters with some degree of certainty measured by the membership function. In these methods, the fuzzy membership degree matrix is obtained based on the calculation of the distance between data points in the original space. However, this operation may lead to suboptimal results because of the influence of noises and redundant features. Besides, some FKM clustering methods ignore the importance of the weighting exponent. In this paper, we propose a novel FKM method called Fuzzy K-Means Clustering With Discriminative Embedding. Within this method, we simultaneously conduct dimensionality reduction along with fuzzy membership degree learning. To retain most information in the embedding subspace and improve the robustness of this method, principal component analysis is incorporated into our framework. An iterative optimization algorithm is proposed to solve the model. To validate the efficacy of the proposed method, we perform comprehensive analyses, including convergence behavior, parameter determination and computational complexity. Moreover, we also match a appropriate weighting exponent for each data set. Experimental results on benchmark data sets show that the proposed method is more discriminative and effective for clustering tasks. [ABSTRACT FROM AUTHOR]

Linear Discriminant Analysis (LDA) is one of the most successful supervised dimensionality reduction methods and has been widely used in many real-world applications. However, ℓ2-norm is employed as the distance metric in the objective of LDA, which is sensitive to outliers. Many previous works improve the robustness of LDA by using ℓ1-norm distance. However, the robustness against outliers is limited and the solver of ℓ1-norm is mostly based on the greedy search strategy, which is time-consuming and easy to get stuck in a local optimum. In this paper, we propose a novel robust LDA measured by ℓ2,1-norm to learn robust discriminative projections. The proposed model is challenging to solve since it needs to minimize and maximize (minmax) ℓ2,1-norm terms simultaneously. As a result, we first systematically derive an efficient iterative optimization algorithm to solve a general ratio minimization problem, and then rigorously prove its convergence. More importantly, an alternately non-greedy iterative re-weighted optimization algorithm is developed based on the preceding approach for solving proposed ℓ2,1-norm minmax problem. Besides, an optimal weighted mean mechanism is driven according to the designed objective and solver, which can be applied to other approaches for robustness improvement. Experimental results on several real-world datasets show the effectiveness of proposed method. [ABSTRACT FROM AUTHOR]

Unsupervised feature selection has attracted more and more attention due to the rapid growth of the large amount of unlabelled and high-dimensional data. The performance of traditional spectral-based unsupervised methods always depends on the quality of constructed similarity matrix. However, real world data always contain a large number of noise samples and features that make the similarity matrix created by original data cannot be fully relied. We propose an unsupervised feature selection method which conducts feature selection and local structure learning simultaneously. Moreover, we add an important constraint on the similarity matrix to allow it to capture more accurate information of the data structure. To perform feature selection, orthogonal constraint and ℓ2,p-norm are adopted on the projection matrix. An efficient and simple algorithm is derived to tackle the problem. We conduct comprehensive experiments on various benchmark data sets, including handwritten digit, face image, and biomedical data, to validate the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Dimensionality reduction has attracted much research interest in the past few decades. Existing dimensionality reduction methods like linear discriminant analysis and principal component analysis have achieved promising performance, but the single and linear projection properties limit further improvements of performance. A novel convolutional two-dimensional nonlinear discriminant analysis method is proposed for dimensionality reduction in this article. In order to handle nonlinear data properly, we present a newly designed structure with convolutional neural networks (CNNs) to realize an equivalent objective function with classical two-dimensional linear discriminant analysis (2DLDA) and thus embed the original 2DLDA into an end-to-end network. In this way, the proposed dimensionality reduction network can utilize the nonlinearity of the CNN and benefit from the learning ability. The results of experiment on different image-related applications demonstrate that our method outperforms other comparable approaches, and its effectiveness is proved. [ABSTRACT FROM AUTHOR]

In many real-world applications, we are often confronted with high dimensional data which are represented by various heterogeneous views. How to cluster this kind of data is still a challenging problem due to the curse of dimensionality and effectively integration of different views. To address this problem, we propose two parameter-free weighted multi-view projected clustering methods which perform structured graph learning and dimensionality reduction simultaneously. We can use the obtained structured graph directly to extract the clustering indicators, without performing other discretization procedures as previous graph-based clustering methods have to do. Moreover, two parameter-free strategies are adopted to learn an optimal weight for each view automatically, without introducing a regularization parameter as previous methods do. Extensive experiments on several public datasets demonstrate that the proposed methods outperform other state-of-the-art approaches and can be used more practically. [ABSTRACT FROM AUTHOR]

2-D linear discriminant analysis (2DLDA) has been widely used in pattern recognition and image classification. 2DLDA selects discriminative features from the up and left corner of images. However, 2DLDA uses the Frobenius norm (F-norm), which is sensitive to noise or outliers in data, as a metric. In this paper, we propose a novel framework, called horizontal and vertical nuclear norm-based 2DLDA (HVNN-2DLDA) for image representation. In the proposed framework, HVNN-2DLDA methods (i.e., HNN-2DLDA and VNN-2DLDA) are proposed, and both use the nuclear norm as a criterion. The nuclear norm can provide more structure and global information for the reconstruction of noisy images. HNN-2DLDA and VNN-2DLDA represent images in the row and column directions, respectively. In addition, by combining the row and column directions, we propose a bilateral nuclear norm-based 2DLDA method called BNN-2DLDA. The advantage of BNN-2DLDA over HNN-2DLDA and VNN-2DLDA is that an image sample can be represented by both the row and the column directions instead of only the row or column direction. HVNN-2DLDA learns a set of local optimal projection vectors by maximizing the ratio of the nuclear norm of the between-class scatter matrix and the nuclear norm of the within-class scatter matrix. To verify the robustness and recognition performance in image classification of HVNN-2DLDA, six public image databases are used for experiments. The experimental results demonstrate the effectiveness and the feasibility of the proposed framework. [ABSTRACT FROM AUTHOR]

Dimensionality reduction is one of the most fundamental topic in machine learning. A range of methods focus on dimensionality reduction have been proposed in various areas. Among the unsupervised dimensionality reduction methods, graph-based dimensionality reduction has begun to draw more and more attention due to its effectiveness. However, most existing graph-based methods have high computation complexity, which is not applicable to large-scale problems. To solve this problem, an unsupervised graph-based dimensionality reduction method called fast and flexible large graph embedding (FFLGE) based on anchors is proposed. FFLGE uses an anchor-based strategy to construct an anchor-based graph and design similarity matrix and then perform the dimensionality reduction efficiently. The computational complexity of the proposed FFLGE reduces to $O(ndm)$ , where $n$ is the number of samples, $d$ is the number of dimensions and $m$ is the number of anchors. Furthermore, it is interesting to note that locality preserving projection and principal component analysis are two special cases of FFLGE. In the end, the experiments based on several publicly large-scale datasets proves the effectiveness and efficiency of the method proposed. [ABSTRACT FROM AUTHOR]

We consider the problem of image representation from the perspective of statistical design. Recent studies have shown that images are possibly sampled from a low dimensional manifold despite of the fact that the ambient space is usually very high dimensional. Learning low dimensional image representations is crucial for many image processing tasks such as recognition and retrieval. Most of the existing approaches for learning low dimensional representations, such as principal component analysis (PCA) and locality preserving projections (LPP), aim at discovering the geometrical or discriminant structures in the data. In this paper, we take a different perspective from statistical experimental design, and propose a novel dimensionality reduction algorithm called A-Optimal Projection (AOP). AOP is based on a linear regression model. Specifically, AOP finds the optimal basis functions so that the expected prediction error of the regression model can be minimized if the new representations are used for training the model. Experimental results suggest that the proposed approach provides a better representation and achieves higher accuracy in image retrieval. [ABSTRACT FROM PUBLISHER]

Semisupervised dimensionality reduction has been attracting much attention as it not only utilizes both labeled and unlabeled data simultaneously, but also works well in the situation of out-of-sample. This paper proposes an effective approach of semisupervised dimensionality reduction through label propagation and label regression. Different from previous efforts, the new approach propagates the label information from labeled to unlabeled data with a well-designed mechanism of random walks, in which outliers are effectively detected and the obtained virtual labels of unlabeled data can be well encoded in a weighted regression model. These virtual labels are thereafter regressed with a linear model to calculate the projection matrix for dimensionality reduction. By this means, when the manifold or the clustering assumption of data is satisfied, the labels of labeled data can be correctly propagated to the unlabeled data; and thus, the proposed approach utilizes the labeled and the unlabeled data more effectively than previous work. Experimental results are carried out upon several databases, and the advantage of the new approach is well demonstrated. [ABSTRACT FROM AUTHOR]

The Gaussian process latent variable model (GP-LVM) has been identified to be an effective probabilistic approach for dimensionality reduction because it can obtain a low-dimensional manifold of a data set in an unsupervised fashion. Consequently, the GP-LVM is insufficient for supervised learning tasks (e.g., classification and regression) because it ignores the class label information for dimensionality reduction. In this paper, a supervised GP-LVM is developed for supervised learning tasks, and the maximum a posteriori algorithm is introduced to estimate positions of all samples in the latent variable space. We present experimental evidences suggesting that the supervised GP-LVM is able to use the class label information effectively, and thus, it outperforms the GP-LVM and the discriminative extension of the GP-LVM consistently. The comparison with some supervised classification methods, such as Gaussian process classification and support vector machines, is also given to illustrate the advantage of the proposed method. [ABSTRACT FROM AUTHOR]

Manifold learning has been demonstrated as an effective way to represent intrinsic geometrical structure of samples. In this paper, a new manifold learning approach, named Local Coordinates Alignment (LCA), is developed based on the alignment technique. LCA first obtains local coordinates as representations of local neighborhood by preserving proximity relations on a patch, which is Euclidean. Then, these extracted local coordinates are aligned to yield the global embeddings. To solve the out of sample problem, linearization of LCA (LLCA) is proposed. In addition, in order to solve the non-Euclidean problem in real world data when building the locality, kernel techniques are utilized to represent similarity of the pairwise points on a local patch. Empirical studies on both synthetic data and face image sets show effectiveness of the developed approaches. [ABSTRACT FROM AUTHOR]

A suitable feature representation can faithfully preserve the intrinsic structure of data. However, traditional dimensionality reduction (DR) methods commonly use the original input features to define the intrinsic structure, which makes the estimated intrinsic structure unreliable since redundant or noisy features may exist in the original input features. Thus a dilemma is that (1) one needs the most suitable feature representation to define the intrinsic structure of data and (2) one should use the proper intrinsic structure of data to perform feature extraction. To address the problem, in this paper we propose a unified learning framework to simultaneously obtain the optimal feature representation and intrinsic structure of data. The structure is learned from the results of feature learning, and the features are learned to preserve the refined structure of data. By leveraging the interactions between the process of determining the most suitable feature representation and intrinsic structure of data, we can capture accurate structure and obtain the optimal feature representation of data. Experimental results demonstrate that our method outperforms state-of-the-art methods in DR and subspace clustering. The code of the proposed method is available at “ http://www.yongxu.org/lunwen.html ”. [ABSTRACT FROM AUTHOR]

Abstract: Latent variable models are powerful dimensionality reduction approaches in machine learning and pattern recognition. However, this kind of methods only works well under a necessary and strict assumption that the training samples and testing samples are independent and identically distributed. When the samples come from different domains, the distribution of the testing dataset will not be identical with the training dataset. Therefore, the performance of latent variable models will be degraded for the reason that the parameters of the training model do not suit for the testing dataset. This case limits the generalization and application of the traditional latent variable models. To handle this issue, a transfer learning framework for latent variable model is proposed which can utilize the distance (or divergence) of the two datasets to modify the parameters of the obtained latent variable model. So we do not need to rebuild the model and only adjust the parameters according to the divergence, which will adopt different datasets. Experimental results on several real datasets demonstrate the advantages of the proposed framework. [Copyright &y& Elsevier]

Dimensionality reduction is ubiquitous in biomedical applications. A newly proposed spectral dimensionality reduction method, named kernel entropy component analysis (KECA), can reveal the structure related to Renyi entropy of an input space data set. However, each principal component in the Hilbert space depends on all training samples in KECA, causing degraded performance. To overcome this drawback, a sparse KECA (SKECA) algorithm based on a recursive divide-and-conquer (DC) method is proposed in this work. The original large and complex problem of KECA is decomposed into a series of small and simple sub-problems, and then they are solved recursively. The performance of SKECA is evaluated on four biomedical datasets, and compared with KECA, principal component analysis (PCA), kernel PCA (KPCA), sparse PCA and sparse KPCA. Experimental results indicate that the SKECA outperforms conventional dimensionality reduction algorithms, even for high order dimensional features. It suggests that SKECA is potentially applicable to biomedical data processing. [ABSTRACT FROM AUTHOR]