Search

Decoder-only large language models (LLMs) excel in high-resource languages across various tasks through few-shot or even zero-shot in-context learning (ICL). However, their performance often does not transfer well to low-resource languages, especially those written in non-Latin scripts. Inspired by recent work that leverages transliteration in encoder-only models, we investigate whether transliteration is also effective in improving LLMs' performance for low-resource languages written in non-Latin scripts. To this end, we propose three prompt templates, where the target-language text is represented in (1) its original script, (2) Latin script, or (3) both. We apply these methods to several representative LLMs of different sizes on various tasks including text classification and sequential labeling. Our findings show that the effectiveness of transliteration varies by task type and model size. For instance, all models benefit from transliterations for sequential labeling (with increases of up to 25%).

Transliterating related languages that use different scripts into a common script shows effectiveness in improving crosslingual transfer in downstream tasks. However, this methodology often makes pretraining a model from scratch unavoidable, as transliteration brings about new subwords not covered in existing multilingual pretrained language models (mPLMs). This is not desired because it takes a lot of computation budget for pretraining. A more promising way is to make full use of available mPLMs. To this end, this paper proposes a simple but effective framework: Transliterate-Merge-Initialize (TransMI), which can create a strong baseline well-suited for data that is transliterated into a common script by exploiting an mPLM and its accompanied tokenizer. TransMI has three stages: (a) transliterate the vocabulary of an mPLM into a common script; (b) merge the new vocabulary with the original vocabulary; and (c) initialize the embeddings of the new subwords. We applied TransMI to three recent strong mPLMs, and our experiments demonstrate that TransMI not only preserves their ability to handle non-transliterated data, but also enables the models to effectively process transliterated data: the results show a consistent improvement of 3% to 34%, varying across different models and tasks. We make our code and models publicly available at \url{https://github.com/cisnlp/TransMI}., Comment: preprint

The world's more than 7000 languages are written in at least 293 scripts. Due to various reasons, many closely related languages use different scripts, which poses a difficulty for multilingual pretrained language models (mPLMs) in learning crosslingual knowledge through lexical overlap. As a consequence, mPLMs are faced with a script barrier: representations from different scripts are located in different subspaces, which can result in crosslingual transfer involving languages of different scripts performing suboptimally. To address this problem, we propose TransliCo, a framework that optimizes the Transliteration Contrastive Modeling (TCM) objective to fine-tune an mPLM by contrasting sentences in its training data and their transliterations in a unified script (in our case Latin), which enhances uniformity in the representation space for different scripts. Using Glot500-m, an mPLM pretrained on over 500 languages, as our source model, we fine-tune it on a small portion (5%) of its training data, and refer to the resulting model as Furina. We show that Furina not only better aligns representations from distinct scripts but also outperforms the original Glot500-m on various zero-shot crosslingual transfer tasks. Additionally, we achieve consistent improvement in a case study on the Indic group where the languages exhibit areal features but use different scripts. We make our code and models publicly available., Comment: ACL 2024

Transformer-based pre-trained language models (PLMs) have achieved remarkable performance in various natural language processing (NLP) tasks. However, pre-training such models can take considerable resources that are almost only available to high-resource languages. On the contrary, static word embeddings are easier to train in terms of computing resources and the amount of data required. In this paper, we introduce MoSECroT Model Stitching with Static Word Embeddings for Crosslingual Zero-shot Transfer), a novel and challenging task that is especially relevant to low-resource languages for which static word embeddings are available. To tackle the task, we present the first framework that leverages relative representations to construct a common space for the embeddings of a source language PLM and the static word embeddings of a target language. In this way, we can train the PLM on source-language training data and perform zero-shot transfer to the target language by simply swapping the embedding layer. However, through extensive experiments on two classification datasets, we show that although our proposed framework is competitive with weak baselines when addressing MoSECroT, it fails to achieve competitive results compared with some strong baselines. In this paper, we attempt to explain this negative result and provide several thoughts on possible improvement.

The two-dimensional 1T-CrTe$_2$ has been an attractive room-temperature van der Waals magnet which has a potential application in spintronic devices. Although it was recognized as a ferromagnetism in the past, the monolayer 1T-CrTe$_2$ was recently found to exhibit zigzag antiferromagnetism with the easy axis oriented at $70^\circ$ to the perpendicular direction of the plane. Therefore, the origin of the intricate anisotropic magnetic behavior therein is well worthy of thorough exploration. Here, by applying density functional theory with spin spiral method, we demonstrate that the Kitaev interaction, together with the single-ion anisotropy and other off-diagonal exchanges, is amenable to explain the magnetic orientation in the metallic 1T-CrTe$_2$. Moreover, the Ruderman-Kittle-Kasuya-Yosida interaction can also be extracted from the dispersion calculations, which explains the metallic behavior of 1T-CrTe$_2$. Our results demonstrate that 1T-CrTe$_2$ is potentially a rare metallic Kitaev material.

The NLP community has mainly focused on scaling Large Language Models (LLMs) vertically, i.e., making them better for about 100 languages. We instead scale LLMs horizontally: we create, through continued pretraining, Glot500-m, an LLM that covers 511 predominantly low-resource languages. An important part of this effort is to collect and clean Glot500-c, a corpus that covers these 511 languages and allows us to train Glot500-m. We evaluate Glot500-m on five diverse tasks across these languages. We observe large improvements for both high-resource and low-resource languages compared to an XLM-R baseline. Our analysis shows that no single factor explains the quality of multilingual LLM representations. Rather, a combination of factors determines quality including corpus size, script, "help" from related languages and the total capacity of the model. Our work addresses an important goal of NLP research: we should not limit NLP to a small fraction of the world's languages and instead strive to support as many languages as possible to bring the benefits of NLP technology to all languages and cultures. Code, data and models are available at https://github.com/cisnlp/Glot500., Comment: ACL 2023

While natural language processing tools have been developed extensively for some of the world's languages, a significant portion of the world's over 7000 languages are still neglected. One reason for this is that evaluation datasets do not yet cover a wide range of languages, including low-resource and endangered ones. We aim to address this issue by creating a text classification dataset encompassing a large number of languages, many of which currently have little to no annotated data available. We leverage parallel translations of the Bible to construct such a dataset by first developing applicable topics and employing a crowdsourcing tool to collect annotated data. By annotating the English side of the data and projecting the labels onto other languages through aligned verses, we generate text classification datasets for more than 1500 languages. We extensively benchmark several existing multilingual language models using our dataset. To facilitate the advancement of research in this area, we will release our dataset and code.

Intrinsic topological superconducting materials are exotic and vital to develop the next-generation topological superconducting devices, topological quantum calculations, and quantum information technologies. Here, we predict the topological and nodal superconductivity of MS (M = Nb and Ta) transition-metal sulfides by using the density functional theory for superconductors combining with the symmetry indicators. We reveal their higher-order topology nature with an index of Z4 = 2. These materials have a higher Tc than the Nb or Ta metal superconductors due to their flat-band and strong electron-phonon coupling nature. Electron doping and lighter isotopes can effectively enhance the Tc. Our findings show that the MS (M = Nb and Ta) systems can be new platforms to study exotic physics in the higher-order topological superconductors, and provide a theoretical support to utilize them as the topological superconducting devices in the field of advanced topological quantum calculations and information technologies., Comment: 7 pages, 4 figures

Electronic phase separation (EPS) originates from an incomplete transformation between electronic phases, causing the inhomogeneous spatial distribution of electronic properties. In the system of two-dimensional electron gas (2DEG), the EPS is usually identified based on a percolative metal-to-superconductor transition. Here, we report a metal-insulator transition (MIT) in KTaO3-based 2DEG with the width of conductive channel decreasing into micrometer scale. Hysteretic resistance-temperature relations are observed due to the competition between metallic and insulating phases, which is tunable by magnetic field. Such a size-dependent MIT effect is attributed to the coexistence and separation of metallic and insulating phases. Combining density functional theory calculation, we propose a theoretical model to simulate the dynamic process of the EPS using the percolation theory, demonstrating the mechanism of size-dependent MIT. Our work suggests a clear and simple 2DEG platform to achieve the spatial coexistence of metallic and insulating phases.

Ferroelectrics are essential in low-dimensional memory devices for multi-bit storage and high-density integration. A polar structure is a necessary premise for ferroelectricity, mainly existing in compounds. However, it is usually rare in elemental materials, causing a lack of spontaneous electric polarization. Here, we report an unexpected room-temperature ferroelectricity in few-chain Te nanowires. Out-of-plane ferroelectric loops and domain reversal are observed by piezoresponse force microscopy. Through density functional theory, we attribute the ferroelectricity to the ion-displacement created by the interlayer interaction between lone pair electrons. Ferroelectric polarization can induce a strong field effect on the transport along the Te chain, supporting a self-gated field-effect transistor. It enables a nonvolatile memory with high in-plane mobility, zero supply voltage, multilevel resistive states, and a high on/off ratio. Our work provides new opportunities for elemental ferroelectrics with polar structures and paves a way towards applications such as low-power dissipation electronics and computing-in-memory devices.

Recently the kagome compounds have inspired enormous interest and made some great progress such as in the field of superconductivity and topology. Here we predict a different kagome magnesium triboride (MgB3) superconductor with a calculated Tc ~12.2 K and Tc ~15.4 K by external stress, the potentially highest among the reported diverse kagome-type superconductors. We reveal its various exotic physical properties including the van Hove singularity, flat-band, multiple Dirac points, and nontrivial topology. The system can be described by a two-band model with highly anisotropic superconducting gaps on Fermi surfaces. Its topological and nodal superconducting nature is unveiled by a recently developed symmetry indicators method. Our results suggest that MgB3 can be a new platform to study exotic physics in the kagome structure, and pave a way to seek for more superconductors and topological materials with XY3-type kagome lattice., Comment: 6 pages, 4 figures

Two-dimensional (2D) magnetic materials are essential for the development of the next-generation spintronic technologies. Recently, layered van der Waals (vdW) compound MnBi2Te4 (MBT) has attracted great interest, and its 2D structure has been reported to host coexisting magnetism and topology. Here, we design several conceptual nanodevices based on MBT monolayer (MBT-ML) and reveal their spin-dependent transport properties by means of the first-principles calculations. The pn-junction diodes and sub-3-nm pin-junction field-effect transistors (FETs) show a strong rectifying effect and a spin filtering effect, with an ideality factor n close to 1 even at a reasonably high temperature. In addition, the pip- and nin-junction FETs give an interesting negative differential resistive (NDR) effect. The gate voltages can tune currents through these FETs in a large range. Furthermore, the MBT-ML has a strong response to light. Our results uncover the multifunctional nature of MBT-ML, pave the road for its applications in diverse next-generation semiconductor spin electric devices., Comment: 6 pages, 6 figures

The intrinsic spin-dependent transport properties of two types of lateral VS2|MoS2 heterojunctions are systematically investigated using first-principles calculations, and their various nanodevices with novel properties are designed. The lateral VS2|MoS2 heterojunction diodes show a perfect rectifying effect and are promising for the applications of Schottky diodes. A large spin-polarization ratio is observed for the A-type device and pure spin-mediated current is then realized. The gate voltage significantly tunes the current and rectification ratio of their field-effect transistors (FETs). In addition, they all have sensitive photoresponse to blue light, and could be used as photodetector and photovoltaic device. Moreover, they generate the effective thermally-driven current when a temperature gratitude appears between the two terminals, suggesting them as potential thermoelectric materials. Hence, the lateral VS2|MoS2 heterojunctions show a multifunctional nature and have various potential applications in spintronics, optoelectronics, and spin caloritronics.

Graphene-based heterostructures are a promising material system for designing the topologically nontrivial Chern insulating devices. Recently, a two-dimensional (2D) monolayer ferromagnetic insulator CrI$_{3}$ was successfully synthesized in experiments [Huang et al., Nature 546, 270 (2017)]. Here, these two interesting materials are proposed to build a heterostructure (Gr/CrI$_{3}$). Our first-principles calculations show that the system forms a van der Waals (vdW) heterostructure, relatively facilely fabricated in experiments. A Chern insulating state is acquired in the Gr/CrI$_{3}$ heterostructure if the vdW gap is compressed to certain extents by applying an external pressure. Amazingly, very strong magnetization (about 150 meV) is found in graphene, induced by the substrate CrI$_{3}$, despite the vdW interactions between them. A low-energy effective model is employed to understand the mechanism. The work functions, contact types, and band alignments of the Gr/CrI$_{3}$ heterostructure system are also studied. Our work demonstrates that the Gr/CrI$_{3}$ heterostructure is a promising system to observe the quantum anomalous Hall effect at high temperatures (up to 45 K) in experiments., Comment: 9 pages, 5 figures