Search

Reweighting adversarial data during training has been recently shown to improve adversarial robustness, where data closer to the current decision boundaries are regarded as more critical and given larger weights. However, existing methods measuring the closeness are not very reliable: they are discrete and can take only a few values, and they are path-dependent, i.e., they may change given the same start and end points with different attack paths. In this paper, we propose three types of probabilistic margin (PM), which are continuous and path-independent, for measuring the aforementioned closeness and reweighting adversarial data. Specifically, a PM is defined as the difference between two estimated class-posterior probabilities, e.g., such the probability of the true label minus the probability of the most confusing label given some natural data. Though different PMs capture different geometric properties, all three PMs share a negative correlation with the vulnerability of data: data with larger/smaller PMs are safer/riskier and should have smaller/larger weights. Experiments demonstrate that PMs are reliable measurements and PM-based reweighting methods outperform state-of-the-art methods., Comment: 17 pages, 4 figures

It is important to learn various types of classifiers given training data with noisy labels. Noisy labels, in the most popular noise model hitherto, are corrupted from ground-truth labels by an unknown noise transition matrix. Thus, by estimating this matrix, classifiers can escape from overfitting those noisy labels. However, such estimation is practically difficult, due to either the indirect nature of two-step approaches, or not big enough data to afford end-to-end approaches. In this paper, we propose a human-assisted approach called Masking that conveys human cognition of invalid class transitions and naturally speculates the structure of the noise transition matrix. To this end, we derive a structure-aware probabilistic model incorporating a structure prior, and solve the challenges from structure extraction and structure alignment. Thanks to Masking, we only estimate unmasked noise transition probabilities and the burden of estimation is tremendously reduced. We conduct extensive experiments on CIFAR-10 and CIFAR-100 with three noise structures as well as the industrial-level Clothing1M with agnostic noise structure, and the results show that Masking can improve the robustness of classifiers significantly., Comment: NIPS 2018 camera-ready version

Supervised learning needs a huge amount of labeled data, which can be a big bottleneck under the situation where there is a privacy concern or labeling cost is high. To overcome this problem, we propose a new weakly-supervised learning setting where only similar (S) data pairs (two examples belong to the same class) and unlabeled (U) data points are needed instead of fully labeled data, which is called SU classification. We show that an unbiased estimator of the classification risk can be obtained only from SU data, and the estimation error of its empirical risk minimizer achieves the optimal parametric convergence rate. Finally, we demonstrate the effectiveness of the proposed method through experiments.

Collecting labeled data is costly and thus a critical bottleneck in real-world classification tasks. To mitigate this problem, we propose a novel setting, namely learning from complementary labels for multi-class classification. A complementary label specifies a class that a pattern does not belong to. Collecting complementary labels would be less laborious than collecting ordinary labels, since users do not have to carefully choose the correct class from a long list of candidate classes. However, complementary labels are less informative than ordinary labels and thus a suitable approach is needed to better learn from them. In this paper, we show that an unbiased estimator to the classification risk can be obtained only from complementarily labeled data, if a loss function satisfies a particular symmetric condition. We derive estimation error bounds for the proposed method and prove that the optimal parametric convergence rate is achieved. We further show that learning from complementary labels can be easily combined with learning from ordinary labels (i.e., ordinary supervised learning), providing a highly practical implementation of the proposed method. Finally, we experimentally demonstrate the usefulness of the proposed methods., Comment: NIPS 2017 camera-ready version

Distributionally Robust Supervised Learning (DRSL) is necessary for building reliable machine learning systems. When machine learning is deployed in the real world, its performance can be significantly degraded because test data may follow a different distribution from training data. DRSL with f-divergences explicitly considers the worst-case distribution shift by minimizing the adversarially reweighted training loss. In this paper, we analyze this DRSL, focusing on the classification scenario. Since the DRSL is explicitly formulated for a distribution shift scenario, we naturally expect it to give a robust classifier that can aggressively handle shifted distributions. However, surprisingly, we prove that the DRSL just ends up giving a classifier that exactly fits the given training distribution, which is too pessimistic. This pessimism comes from two sources: the particular losses used in classification and the fact that the variety of distributions to which the DRSL tries to be robust is too wide. Motivated by our analysis, we propose simple DRSL that overcomes this pessimism and empirically demonstrate its effectiveness., Comment: ICML 2018 camera-ready (final submission version)

Non-Gaussian component analysis (NGCA) is an unsupervised linear dimension reduction method that extracts low-dimensional non-Gaussian "signals" from high-dimensional data contaminated with Gaussian noise. NGCA can be regarded as a generalization of projection pursuit (PP) and independent component analysis (ICA) to multi-dimensional and dependent non-Gaussian components. Indeed, seminal approaches to NGCA are based on PP and ICA. Recently, a novel NGCA approach called least-squares NGCA (LSNGCA) has been developed, which gives a solution analytically through least-squares estimation of log-density gradients and eigendecomposition. However, since pre-whitening of data is involved in LSNGCA, it performs unreliably when the data covariance matrix is ill-conditioned, which is often the case in high-dimensional data analysis. In this paper, we propose a whitening-free LSNGCA method and experimentally demonstrate its superiority.

Non-Gaussian component analysis (NGCA) is aimed at identifying a linear subspace such that the projected data follows a non-Gaussian distribution. In this paper, we propose a novel NGCA algorithm based on log-density gradient estimation. Unlike existing methods, the proposed NGCA algorithm identifies the linear subspace by using the eigenvalue decomposition without any iterative procedures, and thus is computationally reasonable. Furthermore, through theoretical analysis, we prove that the identified subspace converges to the true subspace at the optimal parametric rate. Finally, the practical performance of the proposed algorithm is demonstrated on both artificial and benchmark datasets.

This thesis consists of four chapters on two topics. The first topic, covered in chapter 2, 3, and 4, is about subjective expectations. Economists have long understood that expectations are important determinants of economic decisions. However, expectations are rarely observed. One way to overcome the problem is to elicit beliefs of individuals, or so-called subjective expectations, directly from survey questions. The three chapters study directly measured expectations on two important assets: housing and stock. Home ownership is very high in many countries and housing is typically the largest asset in most households' portfolios. Stock is often the major component of households' financial wealth. Chapter 2 investigates how house price expectations are related to macro and micro characteristics. Chapter 3 focuses on stock price expectations. Both chapters are based on panel data analysis of individual expectations at the micro level. Chapter 4 is also about house price expectations, but is from a macroeconomic perspective and relies on time series analysis of aggregate data. The second topic, discussed in Chapter 5, is about mortality trends. This chapter introduces a mortality forecasting model, which links mortality trends to trends in economic growth, and studies mortality dynamics for six developed countries.

Can we learn a binary classifier from only positive data, without any negative data or unlabeled data? We show that if one can equip positive data with confidence (positive-confidence), one can successfully learn a binary classifier, which we name positive-confidence (Pconf) classification. Our work is related to one-class classification which is aimed at "describing" the positive class by clustering-related methods, but one-class classification does not have the ability to tune hyper-parameters and their aim is not on "discriminating" positive and negative classes. For the Pconf classification problem, we provide a simple empirical risk minimization framework that is model-independent and optimization-independent. We theoretically establish the consistency and an estimation error bound, and demonstrate the usefulness of the proposed method for training deep neural networks through experiments., Comment: NeurIPS 2018 camera-ready version (this paper was selected for spotlight presentation)

Deep learning with noisy labels is practically challenging, as the capacity of deep models is so high that they can totally memorize these noisy labels sooner or later during training. Nonetheless, recent studies on the memorization effects of deep neural networks show that they would first memorize training data of clean labels and then those of noisy labels. Therefore in this paper, we propose a new deep learning paradigm called Co-teaching for combating with noisy labels. Namely, we train two deep neural networks simultaneously, and let them teach each other given every mini-batch: firstly, each network feeds forward all data and selects some data of possibly clean labels; secondly, two networks communicate with each other what data in this mini-batch should be used for training; finally, each network back propagates the data selected by its peer network and updates itself. Empirical results on noisy versions of MNIST, CIFAR-10 and CIFAR-100 demonstrate that Co-teaching is much superior to the state-of-the-art methods in the robustness of trained deep models., Comment: NIPS 2018 camera-ready version

Most of the semi-supervised classification methods developed so far use unlabeled data for regularization purposes under particular distributional assumptions such as the cluster assumption. In contrast, recently developed methods of classification from positive and unlabeled data (PU classification) use unlabeled data for risk evaluation, i.e., label information is directly extracted from unlabeled data. In this paper, we extend PU classification to also incorporate negative data and propose a novel semi-supervised classification approach. We establish generalization error bounds for our novel methods and show that the bounds decrease with respect to the number of unlabeled data without the distributional assumptions that are required in existing semi-supervised classification methods. Through experiments, we demonstrate the usefulness of the proposed methods., Accepted to the 34th International Conference on Machine Learning (ICML 2017)

Empirical risk minimization (ERM), with proper loss function and regularization, is the common practice of supervised classification. In this paper, we study training arbitrary (from linear to deep) binary classifier from only unlabeled (U) data by ERM. We prove that it is impossible to estimate the risk of an arbitrary binary classifier in an unbiased manner given a single set of U data, but it becomes possible given two sets of U data with different class priors. These two facts answer a fundamental question---what the minimal supervision is for training any binary classifier from only U data. Following these findings, we propose an ERM-based learning method from two sets of U data, and then prove it is consistent. Experiments demonstrate the proposed method could train deep models and outperform state-of-the-art methods for learning from two sets of U data.

In this letter, we explore in detail the potential of nanoheteroepitaxy to controllably fabricate high quality GeSn nano-structures and to further improve the crystallinity of GeSn alloys directly grown on Si(001). The GeSn was grown by molecular beam epitaxy at relatively high temperatures up to 750 degrees C on pre-patterned Si nano-pillars embedded in a SiO2 matrix. The best compromise between selective GeSn growth and homogenous Sn incorporation of 1.4% was achieved at a growth temperature of 600 degrees C. X-ray diffraction measurements confirmed that our growth approach results in both fully relaxed GeSn nano-islands and negligible Si interdiffusion into the core of the nanostructures. Detailed transmission electron microscopy characterizations show that only the small GeSn/Si interface area reveals defects, such as stacking faults. Importantly, the main part of the GeSn islands is defect-free and of high crystalline quality. The latter was further demonstrated by photoluminescence measurements where a clear redshift of the direct CC-CV transition was observed with increasing Sn content.

We present the Local Self-Balancing sampler (LSB), a local Markov Chain Monte Carlo (MCMC) method for sampling in purely discrete domains, which is able to autonomously adapt to the target distribution and to reduce the number of target evaluations required to converge. LSB is based on (i) a parametrization of locally balanced proposals, (ii) a newly proposed objective function based on mutual information and (iii) a self-balancing learning procedure, which minimises the proposed objective to update the proposal parameters. Experiments on energy-based models and Markov networks show that LSB converges using a smaller number of queries to the oracle distribution compared to recent local MCMC samplers., ICML 2022

Dislocation networks are one of the most principle sources deteriorating the performances of devices based on lattice-mismatched heteroepitaxial systems. We demonstrate here a technique enabling fully coherent germanium (Ge) islands selectively grown on nanotip-patterned Si(001) substrates. The silicon (Si)-tip-patterned substrate, fabricated by complementary metal oxide semiconductor compatible nanotechnology, features ∼50-nm-wide Si areas emerging from a SiO2 matrix and arranged in an ordered lattice. Molecular beam epitaxy growths result in Ge nanoislands with high selectivity and having homogeneous shape and size. The ∼850 °C growth temperature required for ensuring selective growth has been shown to lead to the formation of Ge islands of high crystalline quality without extensive Si intermixing (with 91 atom % Ge). Nanotip-patterned wafers result in geometric, kinetic-diffusion-barrier intermixing hindrance, confining the major intermixing to the pedestal region of Ge islands, where kinetic diffusion barriers are, however, high. Theoretical calculations suggest that the thin Si/Ge layer at the interface plays, nevertheless, a significant role in realizing our fully coherent Ge nanoislands free from extended defects especially dislocations. Single-layer graphene/Ge/Si-tip Schottky junctions were fabricated, and thanks to the absence of extended defects in Ge islands, they demonstrate high-performance photodetection characteristics with responsivity of ∼45 mA W(-1) and an Ion/Ioff ratio of ∼10(3).

Integration of functional oxides on Si substrates could open a pathway to integrate diverse devices on Si-based technology. Oxygen vacancies (Vo(··)) can strongly affect solid state properties of oxides, including the room temperature ferromagnetism (RTFM) in diluted magnetic oxides. Here, we report a systematical study on the RTFM of oxygen vacancy engineered (by Pr(3+) doping) CeO2 epitaxial thin films on Si substrates. High quality, mixed single crystalline Ce1-xPrxO2-δ (x = 0-1) solid solution films were obtained. The Ce ions in CeO2 with a fluorite structure show a Ce(4+)-dominant valence state in all films. The local crystal structures of the films were analyzed in detail. Pr doping creates both Vo(··) and PrO8-complex defects in CeO2 and their relative concentrations vary with the Pr-doping level. The RTFM properties of the films reveal a strong dependence on the relative Vo(··) concentration. The RTFM in the films initially increases with higher Pr-doping levels due to the increase of the F(+) center (Vo(··) with one occupied electron) concentration and completely disappears when x > 0.2, where the magnetic polaron concentration is considered to decline below the percolation threshold, thus long-range FM order can no longer be established. We thus demonstrate the possibility to directly grow RTFM Pr-doped CeO2 films on Si substrates, which can be an interesting candidate for potential magneto-optic or spintronic device applications.

In the last years decision-focused learning framework, also known as predict-and-optimize, have received increasing attention. In this setting, the predictions of a machine learning model are used as estimated cost coefficients in the objective function of a discrete combinatorial optimization problem for decision making. Decision-focused learning proposes to train the ML models, often neural network models, by directly optimizing the quality of decisions made by the optimization solvers. Based on a recent work that proposed a noise contrastive estimation loss over a subset of the solution space, we observe that decision-focused learning can more generally be seen as a learning-to-rank problem, where the goal is to learn an objective function that ranks the feasible points correctly. This observation is independent of the optimization method used and of the form of the objective function. We develop pointwise, pairwise and listwise ranking loss functions, which can be differentiated in closed form given a subset of solutions. We empirically investigate the quality of our generic methods compared to existing decision-focused learning approaches with competitive results. Furthermore, controlling the subset of solutions allows controlling the runtime considerably, with limited effect on regret., Comment: Accepted for presentation at ICML, 2022