Search

Reward models (RMs) are essential for aligning large language models (LLMs) with human preferences to improve interaction quality. However, the real world is pluralistic, which leads to diversified human preferences with respect to different religions, politics, cultures, etc. Moreover, each individual can have their unique preferences on various topics. Neglecting the diversity of human preferences, current human feedback aligning methods only consider a general reward model, which is below satisfaction for customized or personalized application scenarios. To explore customized preference learning, we collect a domain-specific preference (DSP) dataset, which includes preferred responses for each given query from four practical domains. Besides, from the perspective of data efficiency, we propose a three-stage customized RM learning scheme, then empirically verify its effectiveness on both general preference datasets and our DSP set. Furthermore, we test multiple training and data strategies on the three learning stages. We find several ways to better preserve the general preferring ability while training the customized RMs, especially general preference enrichment, and customized preference imitation learning. The DSP dataset and code are available at https://github.com/Linear95/DSP.

Open world classification is a task in natural language processing with key practical relevance and impact. Since the open or {\em unknown} category data only manifests in the inference phase, finding a model with a suitable decision boundary accommodating for the identification of known classes and discrimination of the open category is challenging. The performance of existing models is limited by the lack of effective open category data during the training stage or the lack of a good mechanism to learn appropriate decision boundaries. We propose an approach based on \underline{a}daptive \underline{n}egative \underline{s}amples (ANS) designed to generate effective synthetic open category samples in the training stage and without requiring any prior knowledge or external datasets. Empirically, we find a significant advantage in using auxiliary one-versus-rest binary classifiers, which effectively utilize the generated negative samples and avoid the complex threshold-seeking stage in previous works. Extensive experiments on three benchmark datasets show that ANS achieves significant improvements over state-of-the-art methods., Comment: Accepted by EMNLP 2021 (Main Track, Long Paper)

In recommendation systems, items are likely to be exposed to various users and we would like to learn about the familiarity of a new user with an existing item. This can be formulated as an anomaly detection (AD) problem distinguishing between "common users" (nominal) and "fresh users" (anomalous). Considering the sheer volume of items and the sparsity of user-item paired data, independently applying conventional single-task detection methods on each item quickly becomes difficult, while correlations between items are ignored. To address this multi-task anomaly detection problem, we propose collaborative anomaly detection (CAD) to jointly learn all tasks with an embedding encoding correlations among tasks. We explore CAD with conditional density estimation and conditional likelihood ratio estimation. We found that: $i$) estimating a likelihood ratio enjoys more efficient learning and yields better results than density estimation. $ii$) It is beneficial to select a small number of tasks in advance to learn a task embedding model, and then use it to warm-start all task embeddings. Consequently, these embeddings can capture correlations between tasks and generalize to new correlated tasks.

Nickel ditelluride (NiTe2), a new discovered type-II Dirac semimetal whose Dirac node lies close to its Fermi level, is expected to exhibit exotic phenomena including Lifshitz transition and superconductivity. As we know, defects are inevitable for transition metal dichalcogenides and have significant impacts on the optical and electronic properties. However, the systematic study of defects in NiTe2 is still lack. Here, by using high-resolution scanning tunneling microscopy combined with first-principles calculations, the point defects including the vacancy, intercalation and antisite defects in NiTe2 are systematically investigated. We identified five main types native defects and revealed that the growth condition could affect the type of native defects. By controlling the ratio of ingredient during synthesis, the types of point defects are expected to be manipulated, especially antisite defects. Additionally, we find native defects could slightly dope the topological surface states. Our results provide a facile way to manipulate defects for future optimizing the electronic properties of NiTe2 and other related materials., Comment: 17 pages, 4 figures, 1 table

The recent introduction of thermodynamic integration techniques has provided a new framework for understanding and improving variational inference (VI). In this work, we present a careful analysis of the thermodynamic variational objective (TVO), bridging the gap between existing variational objectives and shedding new insights to advance the field. In particular, we elucidate how the TVO naturally connects the three key variational schemes, namely the importance-weighted VI, Renyi-VI, and MCMC-VI, which subsumes most VI objectives employed in practice. To explain the performance gap between theory and practice, we reveal how the pathological geometry of thermodynamic curves negatively affects TVO. By generalizing the integration path from the geometric mean to the weighted Holder mean, we extend the theory of TVO and identify new opportunities for improving VI. This motivates our new VI objectives, named the Holder bounds, which flatten the thermodynamic curves and promise to achieve a one-step approximation of the exact marginal log-likelihood. A comprehensive discussion on the choices of numerical estimators is provided. We present strong empirical evidence on both synthetic and real-world datasets to support our claims.

Total correlation (TC) is a fundamental concept in information theory that measures statistical dependency among multiple random variables. Recently, TC has shown noticeable effectiveness as a regularizer in many learning tasks, where the correlation among multiple latent embeddings requires to be jointly minimized or maximized. However, calculating precise TC values is challenging, especially when the closed-form distributions of embedding variables are unknown. In this paper, we introduce a unified framework to estimate total correlation values with sample-based mutual information (MI) estimators. More specifically, we discover a relation between TC and MI and propose two types of calculation paths (tree-like and line-like) to decompose TC into MI terms. With each MI term being bounded, the TC values can be successfully estimated. Further, we provide theoretical analyses concerning the statistical consistency of the proposed TC estimators. Experiments are presented on both synthetic and real-world scenarios, where our estimators demonstrate effectiveness in all TC estimation, minimization, and maximization tasks. The code is available at https://github.com/Linear95/TC-estimation., Comment: Accepted by AISTATS 2023

Cross-domain alignment between image objects and text sequences is key to many visual-language tasks, and it poses a fundamental challenge to both computer vision and natural language processing. This paper investigates a novel approach for the identification and optimization of fine-grained semantic similarities between image and text entities, under a weakly-supervised setup, improving performance over state-of-the-art solutions. Our method builds upon recent advances in optimal transport (OT) to resolve the cross-domain matching problem in a principled manner. Formulated as a drop-in regularizer, the proposed OT solution can be efficiently computed and used in combination with other existing approaches. We present empirical evidence to demonstrate the effectiveness of our approach, showing how it enables simpler model architectures to outperform or be comparable with more sophisticated designs on a range of vision-language tasks., Comment: Accepted to BMVC 2020 (Oral)

Generative feature matching network (GFMN) is an approach for training implicit generative models for images by performing moment matching on features from pre-trained neural networks. In this paper, we present new GFMN formulations that are effective for sequential data. Our experimental results show the effectiveness of the proposed method, SeqGFMN, for three distinct generation tasks in English: unconditional text generation, class-conditional text generation, and unsupervised text style transfer. SeqGFMN is stable to train and outperforms various adversarial approaches for text generation and text style transfer., Comment: ACL 2020

In this paper, we build an outfit evaluation system which provides feedbacks consisting of a judgment with a convincing explanation. The system is trained in a supervised manner which faithfully follows the domain knowledge in fashion. We create the EVALUATION3 dataset which is annotated with judgment, the decisive reason for the judgment, and all corresponding attributes (e.g. print, silhouette, and material \etc.). In the training process, features of all attributes in an outfit are first extracted and then concatenated as the input for the intra-factor compatibility net. Then, the inter-factor compatibility net is used to compute the loss for judgment. We penalize the gradient of judgment loss of so that our Grad-CAM-like reason is regularized to be consistent with the labeled reason. In inference, according to the obtained information of judgment, reason, and attributes, a user-friendly explanation sentence is generated by the pre-defined templates. The experimental results show that the obtained network combines the advantages of high precision and good interpretation., Comment: 10 pages

Coulomb interaction is of central importance in localized energy levels (bound states) or electronic flat bands and could result in many exotic quantum phases, such as magnetic, superconducting, and topological phases in graphene systems1-14. In graphene monolayer, the relativistic massless Dirac fermion nature of the charge carriers enables us to realize unprecedented quasibound states, which are trapped temporarily via whispering-gallery modes (WGMs) with the lifetime (trapping time) of ~ 10 fs, in circular graphene quantum dots (GQDs)15-20. Here we show that Coulomb interaction still plays a dominating role in determining the electronic properties of the temporarily-confined quasibound states. Our scanning tunneling microscopy (STM) and spectroscopy (STS) measurements demonstrate that the discrete quasibound state in a GQD will split into two peaks when it is partially filled. The energy separation of the two split peaks increases linear with inverse effective radius of the GQDs, indicating that the splitting arises from the Coulomb interaction. Moreover, we show that the real space distribution of the two split states separates in different regions of the GQD to reduce the Coulomb interaction, leading to the breaking of the WGM of the quasibound states.

Within many machine learning algorithms, a fundamental problem concerns efficient calculation of an unbiased gradient wrt parameters $\gammav$ for expectation-based objectives $\Ebb_{q_{\gammav} (\yv)} [f(\yv)]$. Most existing methods either (i) suffer from high variance, seeking help from (often) complicated variance-reduction techniques; or (ii) they only apply to reparameterizable continuous random variables and employ a reparameterization trick. To address these limitations, we propose a General and One-sample (GO) gradient that (i) applies to many distributions associated with non-reparameterizable continuous or discrete random variables, and (ii) has the same low-variance as the reparameterization trick. We find that the GO gradient often works well in practice based on only one Monte Carlo sample (although one can of course use more samples if desired). Alongside the GO gradient, we develop a means of propagating the chain rule through distributions, yielding statistical back-propagation, coupling neural networks to common random variables.

We investigate adversarial learning in the case when only an unnormalized form of the density can be accessed, rather than samples. With insights so garnered, adversarial learning is extended to the case for which one has access to an unnormalized form u(x) of the target density function, but no samples. Further, new concepts in GAN regularization are developed, based on learning from samples or from u(x). The proposed method is compared to alternative approaches, with encouraging results demonstrated across a range of applications, including deep soft Q-learning., Comment: Published in AISTATS 2019; Code: https://github.com/ChunyuanLI/RAS

Ordered atomic-scale superlattices on surface hold great interest both for basic science and for potential applications in advanced technology. However, controlled fabrication of superlattices down to atomic scale has proven exceptionally challenging. Here we demonstrate the segregation-growth and self-organization of ordered S atomic superlattices confined at the interface between graphene and S-rich Cu substrates. Scanning tunneling microscope (STM) studies show that, by finely controlling the growth temperature, we obtain well-ordered S (sub)nanometer-cluster superlattice and monoatomic superlattices with various periods at the interface. These atomic superlattices are stable in atmospheric environment and robust even after high-temperature annealing (~ 350 oC). Our experiments demonstrate that the S monoatomic superlattice can drive graphene into the electronic Kekul\'e distortion phase when the period of the ordered S adatoms is commensurate with graphene lattice. Our results not only open a road to realize atomic-scale superlattices at interfaces, but also provide a new route to realize exotic electronic states in graphene., Comment: 4 figures

With considering the great success of scanning tunnelling microscopy (STM) studies of graphene in the past few years, it is quite surprising to notice that there is still a fundamental contradiction about the reported tunnelling spectra of quasi-free-standing graphene monolayer. Many groups observed V-shape spectra with linearly vanishing density-of-state (DOS) at the Dirac point, whereas, the others reported spectra with a gap of 60 meV pinned to the Fermi level in the quasi-free-standing graphene monolayer. Here we systematically studied the two contradicted tunnelling spectra of the quasi-free-standing graphene monolayer on several different substrates and provided a consistent interpretation about the result. The gap in the spectra arises from the out-of-plane phonons in graphene, which mix the Dirac electrons at the Brillouin zone corners with the nearly free-electron states at the zone center. Our experiment indicated that interactions with substrates could effectively suppress effects of the out-of-plane phonons in graphene and enable us to detect only the DOS of the Dirac electrons in the spectra. We also show that it is possible to switch on and off the out-of-plane phonons of graphene at the nanoscale, i.e., the tunnelling spectra show switching between the two distinct features, through voltage pulses applied to the STM tip., Comment: 4 Figures

Creation of high quality p-n junctions in graphene monolayer is vital in studying many exotic phenomena of massless Dirac fermions. However, even with the fast progress of graphene technology for more than ten years, it remains conspicuously difficult to generate nanoscale and atomically-sharp p-n junctions in graphene. Here, we employ monolayer-vacancy-island engineering of Cu surface to realize nanoscale p-n junctions with atomically-sharp boundaries in graphene monolayer. The variation of graphene-Cu separations around the edges of the Cu monolayer-vacancy-island affects the positions of the Dirac point in graphene, which consequently lead to atomically-sharp p-n junctions with the height as high as 660 meV in graphene. The generated sharp p-n junctions isolate the graphene above the Cu monolayer-vacancy-island as nanoscale graphene quantum dots (GQDs) in a continuous graphene sheet. Massless Dirac fermions are confined by the p-n junctions for a finite time to form quasi-bound states in the GQDs. By using scanning tunneling microscopy, we observe resonances of quasi-bound states in the GQDs with various sizes and directly visualize effects of geometries of the GQDs on the quantum interference patterns of the quasi-bound states, which allow us to test the quantum electron optics based on graphene in atomic scale., Comment: 4 Figures in main text

Room-temperature ferromagnetic semiconductor is vital in nonvolatile digital circuits and it can provide an idea system where we can make use of both charge and spin of electrons. However, seeking room-temperature ferromagnetic semiconductors is still just an appealing idea that has never been realized in practice up to now. Here we demonstrate that graphene monolayer, hybridized with underlying Ni(111) substrate, is the room-temperature ferromagnetic semiconductor that has been continuously searched for decades. Our spin-polarized scanning tunnelling microscopy (STM) experiments, complemented by first-principles calculations, demonstrate explicitly that the interaction between graphene and the Ni substrate generates a large gap in graphene and simultaneously leads to a relatively shift between majority- and minority-spin bands. Consequently, the graphene sheet on the Ni substrate exhibits a spin-polarized gap with energy of several tens meV even at room-temperature. This result makes the science and applications of room-temperature ferromagnetic semiconductors achievable and raises hopes of graphene-based novel information technologies., Comment: 4 Figures in main text

When electrons are confined in a two dimensional (2D) system, typical quantum mechanical phenomena such as Landau quantization can be detected. Graphene systems, including the single atomic layer and few-layer stacked crystals, are ideal 2D materials for studying a variety of quantum mechanical problems. In this article, we review the experimental progress in the unusual Landau quantized behaviors of Dirac fermions in monolayer and multilayer graphene by using scanning tunneling microscopy(STM) and scanning tunneling spectroscopy(STS). Through STS measurement of the strong magnetic fields, distinct Landau-level spectra and rich level splitting phenomena are observed in different graphene layers. These unique properties provide an effective method for identifying the number of layers, as well as the stacking orders, and investigating the fundamentally physical phenomena of graphene. Moreover, in the presence of a strain and charged defects, the Landau quantization of graphene can be significantly modified, leading to unusual spectroscopic and electronic properties.

Massless Dirac fermions in graphene provide unprecedented opportunities to realize the Klein paradox, which is one of the most exotic and striking properties of relativistic particles. In the seminal theoretical work [Katsnelson et al., Nat. Phys. 2 620 (2006)], it was predicted that the massless Dirac fermions can pass through one-dimensional (1D) potential barriers unimpededly at normal incidence. Such a result seems to preclude confinement of the massless Dirac fermions in graphene by using 1D potential barriers. Here, we demonstrate, both experimentally and theoretically, that massless Dirac fermions can be trapped in quasi-1D npn junction of a continuous graphene monolayer. Because of highly anisotropic transmission of the massless Dirac fermions at n-p junction boundaries (the so-called Klein tunneling in graphene), charge carries incident at large oblique angles will be reflected from one edge of the junction with high probability and continue to bounce from the opposite edge. Consequently, these electrons are trapped for a finite time to form quasi-bound states in the quasi-1D npn junction. The quasi-bound states seen as pronounced resonances are probed and the quantum interference patterns arising from these states are directly visualized in our scanning tunneling microscope measurements., Comment: 4 Figures in main text

The electron-hole symmetry in graphene monolayer, which is analogous to the inherent symmetric structure between electrons and positrons of the Universe, plays a crucial role in the chirality and chiral tunnelling of massless Dirac fermions. Here we demonstrate that both strain and charged-defect scattering could break this symmetry dramatically in graphene monolayer. In our experiment, the Fermi velocities of electrons and holes are measured directly through Landau level spectroscopy. In strained graphene with lattice deformation and curvature, the and are measured as 1.2 x 106 m/s and 1.02 x106 m/s, respectively. This giant asymmetry originates from enhanced next-nearest-neighbor hopping in the strained region. Around positively charged-defect, we observe opposite electron-hole asymmetry, and the and are measured to be 0.86x 106 m/s and 1.14 x106 m/s, respectively. Such a large asymmetry is attributed to the fact that the massless Dirac fermions in graphene monolayer are scattered more strongly when they are attracted to the charged-defect than when they are repelled from it., Comment: 4 Figures in main text