Search

Graph Convolution Networks (GCNs) have been shown to be very effective in utilizing pair-wise relationships across samples. They have been successfully applied to solve various machine learning problems in practice. In many applications, the construction of GCNs involves more than one layer. However, their generalization and stability analysis are limited. The main aim of this paper is to analyze GCNs with two layers. The formulation is based on transductive semi-supervised learning and the filtering is done in the eigen-domain. We show the uniform stability of the neural network and the convergence of the generalization gap to zero. The analysis of two-layer GCN is more involved than the single-layer case and requires some new estimates of the neural network's quantities. The analysis confirms the usefulness of GCNs. It also sheds light on the design of the neural network, for instance, how the data should be scaled to achieve the uniform stability of the learning process. Some experimental results on benchmark datasets are presented to illustrate the theory. [ABSTRACT FROM AUTHOR]

The class of multi-relational graph convolutional networks (MRGCNs) is a recent extension of standard graph convolutional networks (GCNs) to handle heterogenous graphs with multiple types of relationships. MRGCNs have been shown to yield results superior than traditional GCNs in various machine learning tasks. The key idea is to introduce a new kind of convolution operated on tensors that can effectively exploit correlations exhibited in multiple relationships. The main objective of this paper is to analyze the algorithmic stability and generalization guarantees of MRGCNs to confirm the usefulness of MRGCNs. Our contributions are of three folds. First, we develop a matrix representation of various tensor operations underneath MRGCNs to simplify the analysis significantly. Next, we prove the uniform stability of MRGCNs and deduce the convergence of the generalization gap to support the usefulness of MRGCNs. The analysis sheds lights on the design of MRGCNs, for instance, how the data should be scaled to achieve the uniform stability of the learning process. Finally, we provide experimental results to demonstrate the stability results. [ABSTRACT FROM AUTHOR]

Recently, deep-learning-based denoising methods for hyperspectral images (HSIs) have been comprehensively studied and achieved impressive performance because they can effectively extract complex and nonlinear image features. Compared with deep-learning-based methods, the nonlocal similarity-based denoising methods are more suitable for images containing edges or regular textures. We propose a powerful HSI denoising method, termed non-local 3-D convolutional neural network (NL-3DCNN), combining traditional machine learning and deep learning techniques. NL-3DCNN exploits the high spectral correlation of an HSI using subspace representation, and the corresponding representation coefficients are termed eigenimages. The high spatial correlation in eigenimages is exploited by grouping nonlocal similar patches, which are denoised by a 3-D convolutional neural network. The numerical and graphical denoising results of the simulated and real data show that the proposed method is superior to the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

We study a class of spectral learning methods with dependent observations, including popular ridge regression, Landweber iteration, spectral cut-off and so on. We derive an explicit risk bound in terms of the correlation of the observations, regularity of the regression function, and effective dimension of the reproducing kernel Hilbert space. By appropriately choosing regularization parameter according to the sample size, the risk bound yields a nearly optimal learning rate with a logarithmic term for strongly mixing sequences. We thus extend the applicable range of spectral algorithm to non-i.i.d. sampling process. Particularly, it is shown that the learning rates for i.i.d. samples in the literature refer to our special case, i.e., the mixing condition parameter tends to zero. [ABSTRACT FROM AUTHOR]

Although image denoising as a basic task of image restoration has been widely studied in the past decades, there are not many studies on mixed noise denoising. In this paper, we propose two structured dictionary learning models to recover images corrupted by mixed Gaussian and impulse noise. These two models can be merged as $\ell _{p}$ -norm fidelity plus $\ell _{q}$ -norm regularization. The fidelity term is used to fit image patches and the regularization term is employed for sparse coding. Particularly, we utilize proximal (and proximal linearized) alternating minimization methods as the main solvers to deal with these two models. We remove the Gaussian noise under the assumption that the uncorrupted image can be approximated with a linear representation under an appropriate orthogonal basis. We use different ways to remove impulse noise for these two models. The experimental results are reported to compare the existing methods and demonstrate the performance of the proposed denoising model is better than the other existing methods in terms of some quality assessment metrics. [ABSTRACT FROM AUTHOR]

Multi-instance multi-label (MIML) learning is one of challenging research problems in machine learning. In the literature, there are several methods for solving MIML problems. However, they may take a long computational time and have a huge storage cost for large MIML data sets. The main aim of this paper is to propose and develop an efficient Markov Chain learning algorithm for MIML problems, especially for data represented by non-negative features. Our idea is to perform labels classification iteratively through two Markov chains constructed by using objects and features respectively. The classification of objects can be obtained by using labels propagation via training data in the iterative method. Moreover, we demonstrate that the proposed method can be formulated by considering normalized linear kernel. Because linear kernel function is explicit and separable, it is not necessary to compute and store a huge affinity matrix among objects/instances compared with the use of other kernel functions. Therefore, both the storage and computational time of the proposed algorithm are very efficient. Experimental results are presented to show that the classification performance of the proposed method using normalized linear kernel function is about the same as those using the other kernel functions, while the required computational time is much less, which together suggest that the linear kernel can be good enough for MIML problem. Also experimental results on some benchmark data sets are reported to illustrate the effectiveness of the proposed method in one-error, ranking loss, coverage and average precision, and show that it is competitive with the other MIML methods. [ABSTRACT FROM AUTHOR]

In this paper, we study a physics-informed algorithm for Wasserstein Generative Adversarial Networks (WGANs) for uncertainty quantification in solutions of partial differential equations. By using groupsort activation functions in adversarial network discriminators, network generators are utilized to learn the uncertainty in solutions of partial differential equations observed from the initial/boundary data. Under mild assumptions, we show that the generalization error of the computed generator converges to the approximation error of the network with high probability, when the number of samples are sufficiently taken. According to our established error bound, we also find that our physics-informed WGANs have higher requirement for the capacity of discriminators than that of generators. Numerical results on synthetic examples of partial differential equations are reported to validate our theoretical results and demonstrate how uncertainty quantification can be obtained for solutions of partial differential equations and the distributions of initial/boundary data. However, the quality or the accuracy of the uncertainty quantification theory in all the points in the interior is still the theoretical vacancy, and required for further research. • Wasserstein generative adversarial networks need little priors for uncertainty quantifications. • Physics-informed regularization term enforces the satisfaction of governing PDEs. • The proposed probabilistic model (WGAN-PINNs) is able to deal with non-Gaussian and spatially dependent noise. • WGAN-PINNs have good convergence and generalization properties. • Theoretically we have higher requirements on discriminators than generators. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved great success in solving many machine learning and computer vision problems. In this paper, we propose a deep neural network called the Tucker network derived from the Tucker format and analyze its expressive power. The results demonstrate that the Tucker network has exponentially higher expressive power than the shallow network. In other words, a shallow network with an exponential width is required to realize the same score function as that computed by the Tucker network. Moreover, we discuss the expressive power between the hierarchical Tucker tensor network (HT network) and the proposed Tucker network. To generalize the Tucker network into a deep version, we combine the hierarchical Tucker format and Tucker format to propose a deep Tucker tensor decomposition. Its corresponding deep Tucker network is presented. Experiments are conducted on three datasets: MNIST, CIFAR-10 and CIFAR-100. The results experimentally validate the theoretical results and show that the Tucker network and deep Tucker network have better performance than the shallow network and HT network. [ABSTRACT FROM AUTHOR]

Abstract: In this paper, we study the problem of multi-label collective classification (MLCC) where instances are related and associated with multiple class labels. Such correlation of class labels among interrelated instances exists in a wide variety of data, e.g., a web page can belong to multiple categories since its semantics can be recognized in different ways, and the linked web pages are more likely to have the same classes than the unlinked pages. We propose an effective and novel Markov chain based learning method for MLCC problems. Our idea is to model the problem as a Markov chain with restart on transition probability graphs, and to propagate the ranking score of labeled instances to unlabeled instances based on the affinity among instances. The affinity among instances is set up by explicitly using the attribute features derived from the content of instances as well as the correlation features constructed from the links of instances. Intuitively, an instance which contains linked neighbors that are highly similar to the other instances with a high rank of a particular class label, has a high chance of this class label. Extensive experiments have been conducted on two DBLP datasets to demonstrate the effectiveness of the proposed algorithm. The performance of the proposed algorithm is shown to be better than those of the binary relevance multi-label algorithm, collective classification algorithms (wvRN, ICA and Gibbs), and the ICML algorithm for the tested MLCC problems. [Copyright &y& Elsevier]

The problem of multilabel classification has attracted great interest in the last decade, where each instance can be assigned with a set of multiple class labels simultaneously. It has a wide variety of real-world applications, e.g., automatic image annotations and gene function analysis. Current research on multilabel classification focuses on supervised settings which assume existence of large amounts of labeled training data. However, in many applications, the labeling of multilabeled data is extremely expensive and time consuming, while there are often abundant unlabeled data available. In this paper, we study the problem of transductive multilabel learning and propose a novel solution, called Trasductive Multilabel Classification (TraM), to effectively assign a set of multiple labels to each instance. Different from supervised multilabel learning methods, we estimate the label sets of the unlabeled instances effectively by utilizing the information from both labeled and unlabeled data. We first formulate the transductive multilabel learning as an optimization problem of estimating label concept compositions. Then, we derive a closed-form solution to this optimization problem and propose an effective algorithm to assign label sets to the unlabeled instances. Empirical studies on several real-world multilabel learning tasks demonstrate that our TraM method can effectively boost the performance of multilabel classification by using both labeled and unlabeled data. [ABSTRACT FROM AUTHOR]

The main aim of this paper is to propose a statistical indicator for wind shear prediction from Light Detection and Ranging (LIDAR) observational data. Accurate warning signal of wind shear is particularly important for aviation safety. The main challenges are that wind shear may result from a sustained change of the headwind and the possible velocity of wind shear may have a wide range. Traditionally, aviation models based on terrain-induced setting are used to detect wind shear phenomena. Different from traditional methods, we study a statistical indicator which is used to measure the variation of headwinds from multiple headwind profiles. Because the indicator value is nonnegative, a decision rule based on one-side normal distribution is employed to distinguish wind shear cases and non-wind shear cases. Experimental results based on real data sets obtained at Hong Kong International Airport runway are presented to demonstrate that the proposed indicator is quite effective. The prediction performance of the proposed method is better than that by the supervised learning methods (LDA, KNN, SVM, and logistic regression). This model would also provide more accurate warnings of wind shear for pilots and improve the performance of Wind shear and Turbulence Warning System. [ABSTRACT FROM AUTHOR]

The image nonlocal self-similarity (NSS) prior refers to the fact that a local patch often has many nonlocal similar patches to it across the image and has been widely applied in many recently proposed machining learning algorithms for image processing. However, there is no theoretical analysis on its working principle in the literature. In this paper, we discover a potential causality between NSS and low-rank property of color images, which is also available to grey images. A new patch group based NSS prior scheme is proposed to learn explicit NSS models of natural color images. The numerical low-rank property of patched matrices is also rigorously proved. The NSS-based QMC algorithm computes an optimal low-rank approximation to the high-rank color image, resulting in high PSNR and SSIM measures and particularly the better visual quality. A new tensor NSS-based QMC method is also presented to solve the color video inpainting problem based on quaternion tensor representation. The numerical experiments on color images and videos indicate the advantages of NSS-based QMC over the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Detection of flowering and quantification of flowering phenology are key to monitoring the reproduction of tropical trees and their response to global change. However, effective monitoring of flowering over various scales from individuals to forest ecosystem levels is lacking due to the relatively small sizes of flowers, diverse flowering strategies across species, and the short duration of flowering, making accurate flower detection difficult. Drone-based surveys require less time and human resources than traditional ground-based flower surveys and thus may be able to help address this in a cost-effective manner but remain underexplored in species-rich tropical forest ecosystems. Here, we developed a method that integrated the Residual Networks 50 (ResNet50) deep learning algorithm with high resolution imagery (c. 0.05 m) from monthly drone surveys done in a 50-ha tropical forest plot on Barro Colorado Island (BCI), Panamá, over 2018–2020 to detect a diversity of flowering species in this tree community and to track the timing of flowering throughout the year. We built a comprehensive training library of canopy components (flower, leaf, branch, and shade) from this forest plot throughout the study period, trained a single deep learning model across all drone imagery, and validated it using five-fold cross validation at the pixel scale. We further generated image- and tree-crown-specific supervised classifications to evaluate the deep learning model at the tree-crown scale. Our deep learning method accurately classified flowers (User's accuracy = 95.3 %, Producer's accuracy = 85.8 %) while maintaining high predictive power for the other three classes (Overall accuracy = 98.4 %). Our results also demonstrated high consistency against tree-crown-specific supervised classifications for flower (r2 = 0.85), leaf (r2 = 0.84), and branch (r2 = 0.92) components, with lower agreement observed for the shade component (r2 = 0.59). These results demonstrate the effectiveness of our method in advancing fine-scale flower monitoring in the tropics, with potential to be extended to other regions or other remote sensing platforms with frequent high-resolution monitoring. The method will allow us to better monitor flowering in tropical forests and improve our understanding of how phenology and reproductive success may be affected by climate change. [ABSTRACT FROM AUTHOR]

We study a multi-instance (MI) learning dimensionality-reduction algorithm through sparsity and orthogonality, which is especially useful for high-dimensional MI data sets. We develop a novel algorithm to handle both sparsity and orthogonality constraints that existing methods do not handle well simultaneously. Our main idea is to formulate an optimization problem where the sparse term appears in the objective function and the orthogonality term is formed as a constraint. The resulting optimization problem can be solved by using approximate augmented Lagrangian iterations as the outer loop and inertial proximal alternating linearized minimization (iPALM) iterations as the inner loop. The main advantage of this method is that both sparsity and orthogonality can be satisfied in the proposed algorithm. We show the global convergence of the proposed iterative algorithm. We also demonstrate that the proposed algorithm can achieve high sparsity and orthogonality requirements, which are very important for dimensionality reduction. Experimental results on both synthetic and real data sets show that the proposed algorithm can obtain learning performance comparable to that of other tested MI learning algorithms. [ABSTRACT FROM AUTHOR]

Heterogeneous transfer learning has recently gained much attention as a new machine learning paradigm in which the knowledge can be transferred from source domains to target domains in different feature spaces. Existing works usually assume that source domains can provide accurate and useful knowledge to be transferred to target domains for learning. In practice, there may be noise appearing in given source (text) and target (image) domains data, and thus, the performance of transfer learning can be seriously degraded. In this paper, we propose a robust and non-negative collective matrix factorization model to handle noise in text-to-image transfer learning, and make a reliable bridge to transfer accurate and useful knowledge from the text domain to the image domain. The proposed matrix factorization model can be solved by an efficient iterative method, and the convergence of the iterative method can be shown. Extensive experiments on real data sets suggest that the proposed model is able to effectively perform transfer learning in noisy text and image domains, and it is superior to the popular existing methods for text-to-image transfer learning. [ABSTRACT FROM AUTHOR]

This paper proposes a novel inverse method based on the deep convolutional neural network (ConvNet) to extract snow's layer thickness and temperature via passive microwave remote sensing (PMRS). The proposed ConvNet is trained using simulated data obtained through conventional computational electromagnetic methods. Compared with the traditional inverse method, the trained ConvNet can predict the result with higher accuracy. Besides, the proposed method has a strong tolerance for noise. The proposed ConvNet composes three pairs of convolutional and activation layers with one additional fully connected layer to realize regression, i.e., the inversion of snow parameters. The feasibility of the proposed method in learning the inversion of snow parameters is validated by numerical examples. The inversion results indicate that the correlation coefficient ( R 2 ) ratio between the proposed ConvNet and conventional methods reaches 4.8, while the ratio for the root mean square error (RMSE) is only 0.18. Hence, the proposed method experiments with a novel path to improve the inversion of passive microwave remote sensing through deep learning approaches. [ABSTRACT FROM AUTHOR]

In this article, we study the problem of online heterogeneous transfer learning, where the objective is to make predictions for a target data sequence arriving in an online fashion, and some offline labeled instances from a heterogeneous source domain are provided as auxiliary data. The feature spaces of the source and target domains are completely different, thus the source data cannot be used directly to assist the learning task in the target domain. To address this issue, we take advantage of unlabeled co-occurrence instances as intermediate supplementary data to connect the source and target domains, and perform knowledge transition from the source domain into the target domain. We propose a novel online heterogeneous transfer learning algorithm called Online Heterogeneous Knowledge Transition (OHKT) for this purpose. In OHKT, we first seek to generate pseudo labels for the co-occurrence data based on the labeled source data, and then develop an online learning algorithm to classify the target sequence by leveraging the co-occurrence data with pseudo labels. Experimental results on real-world data sets demonstrate the effectiveness and efficiency of the proposed algorithm. [ABSTRACT FROM AUTHOR]

Abstract Multi-Instance learning (MIL) aims to predict labels of unlabeled bags by training a model with labeled bags. The usual assumption of existing MIL methods is that the underlying distribution of training data is the same as that of the testing data. However, this assumption may not be valid in practice, especially when training data from a source domain and testing data from a target domain are drawn from different distributions. In this paper, we put forward a novel algorithm Multi-Instance Transfer Metric Learning (MITML). Specially, MITML first attempts to bridge the distributions of different domains by using the bag weighting method. Then a consistent maximum likelihood estimation method is learned to construct an optimal distance metric and exploited to classify testing bags. Comprehensive experimental results on benchmark datasets have demonstrated that the learning performance of the proposed MITML algorithm is better than those of other state-of-the-art MIL algorithms. [ABSTRACT FROM AUTHOR]

In tropical forests, leaf phenology signals leaf-on/off status and exhibits considerable variability across scales from a single tree-crown to the entire forest ecosystem. Such phenology signals importantly regulate large-scale biogeochemical cycles and regional climate. PlanetScope CubeSats data with a 3-m resolution and near-daily global coverage provide an unprecedented opportunity to monitor both fine- and ecosystem-scale phenology variability along large environmental gradients. However, a scalable method that accurately characterizes leaf phenology from PlanetScope with biophysically meaningful metrics remains lacking. We developed an index-guided, ecologically constrained autoencoder (IG-ECAE) method to automatically derive a deciduousness metric (percentage of upper tree canopies with leaf-off status within an image pixel) from PlanetScope. The IG-ECAE first estimated the reflectance spectra of leafy/leafless canopies based on their spectral indices characteristics, then used the derived reflectance spectra to guide an autoencoder deep learning method with additional ecological constraints to refine the reflectance spectra, and finally used linear spectral unmixing to estimate the relative abundance of leafless canopies (or deciduousness) per PlanetScope image pixel. We tested the IG-ECAE method at 16 tropical forest sites spanning multiple continents and a large precipitation gradient (1470–2819 mm year−1). Among these sites, we evaluated the PlanetScope-derived deciduousness against corresponding measures derived from WorldView-2 (n = 9 sites) and local phenocams (n = 9 sites). Our results show that PlanetScope-derived deciduousness agrees: 1) with that derived from WorldView-2 at the patch level (90 m × 90 m) with r2 = 0.89 across all sites; and 2) with that derived from phenocams to quantify ecosystem-scale seasonality with r2 ranging from 0.62 to 0.96. These results demonstrate the effectiveness and scalability of IG-ECAE in characterizing the wide variability in deciduousness across scales from pixels to forest ecosystems, and from a single date to the full annual cycle, indicating the potential for using high-resolution satellites to track the large-scale phenological patterns and response of tropical forests to climate change. • An ecologically constrained deep learning method is developed for tropical phenology monitoring with PlanetScope. • The method is evaluated across 16 tropical sites spanning a large rainfall gradient. • The method assesses cross-scale phenology variability with a deciduousness metric. • The method demonstrates effectiveness and scalability with local measurements. [ABSTRACT FROM AUTHOR]

In this paper, we propose a new random forest (RF) based ensemble method, ForesTexter, to solve the imbalanced text categorization problems. RF has shown great success in many real-world applications. However, the problem of learning from text data with class imbalance is a relatively new challenge that needs to be addressed. A RF algorithm tends to use a simple random sampling of features in building their decision trees. As a result, it selects many subspaces that contain few, if any, informative features for the minority class. Furthermore, the Gini measure for data splitting is considered to be skew sensitive and bias towards the majority class. Due to the inherent complex characteristics of imbalanced text datasets, learning RF from such data requires new approaches to overcome challenges related to feature subspace selection and cut-point choice while performing node splitting. To this end, we propose a new tree induction method that selects splits, both feature subspace selection and splitting criterion, for RF on imbalanced text data. The key idea is to stratify features into two groups and to generate effective term weighting for the features. One group contains positive features for the minority class and the other one contains the negative features for the majority class. Then, for feature subspace selection, we effectively select features from each group based on the term weights. The advantage of our approach is that each subspace contains adequate informative features for both minority and majority classes. One difference between our proposed tree induction method and the classical RF method is that our method uses Support Vector Machines (SVM) classifier to split the training data into smaller and more balance subsets at each tree node, and then successively retrains the SVM classifiers on the data partitions to refine the model while moving down the tree. In this way, we force the classifiers to learn from refined feature subspaces and data subsets to fit the imbalanced data better. Hence, the tree model becomes more robust for text categorization task with imbalanced dataset. Experimental results on various benchmark imbalanced text datasets (Reuters-21578, Ohsumed, and imbalanced 20 newsgroup) consistently demonstrate the effectiveness of our proposed ForesTexter method. The performance of our proposed approach is competitive against the standard random forest and different variants of SVM algorithms. [ABSTRACT FROM AUTHOR]

Domain adaptation aims to transfer auxiliary knowledge from a source domain to enhance the learning performance on a target domain. Recent studies have suggested that deep networks are able to achieve promising results for domain adaptation problems. However, deep neural networks cannot reveal the underlying geometric information from input data. Indeed, such geometric information is very useful for describing the relationship between the samples from source and target domains. In this paper, we propose a novel learning algorithm named GKE, which stands for Geometric Knowledge Embedding. In GKE, we use a graph-based model to explore the underlying geometric structure of the input source and target data based on their similarities. Concretely, we develop a graph convolutional network to learn discriminative representations based on the constructed graph. To obtain effective transferable representations, we match source and target domains by reducing the Maximum Mean Discrepancy (MMD) between their learned representations. Extensive experiments on real-world data sets demonstrate that the proposed method outperforms existing domain adaption methods. • We exploit the geometric information of the source and target data to learn discriminative representations. In this sense, the features of each sample and its neighbors are both considered during the learning procedure. • We introduce MMD into the graph convolutional network to explore geometric knowledge for learning transferable embeddings. • GCN has not been applied to domain adaptation problems before, and this makes our proposed method a decent supplement to existing domain adaptation approaches. • We conduct comprehensive experiments on four real-world applications, including object recognition, image classification and text categorization, to demonstrate the effectiveness of our proposed method. [ABSTRACT FROM AUTHOR]