Search

Creating high-quality, large-scale datasets for large language models (LLMs) often relies on resource-intensive, GPU-accelerated models for quality filtering, making the process time-consuming and costly. This dependence on GPUs limits accessibility for organizations lacking significant computational infrastructure. To address this issue, we introduce the Lightweight, Purpose-driven (LP) Data Pipeline, a framework that operates entirely on CPUs to streamline the processes of dataset extraction, filtering, and curation. Based on our four core principles, the LP Data Pipeline significantly reduces preparation time and cost while maintaining high data quality. Importantly, our pipeline enables the creation of purpose-driven datasets tailored to specific domains and languages, enhancing the applicability of LLMs in specialized contexts. We anticipate that our pipeline will lower the barriers to LLM development, enabling a wide range of organizations to access LLMs more easily.

In this paper, we propose the 1 Trillion Token Platform (1TT Platform), a novel framework designed to facilitate efficient data sharing with a transparent and equitable profit-sharing mechanism. The platform fosters collaboration between data contributors, who provide otherwise non-disclosed datasets, and a data consumer, who utilizes these datasets to enhance their own services. Data contributors are compensated in monetary terms, receiving a share of the revenue generated by the services of the data consumer. The data consumer is committed to sharing a portion of the revenue with contributors, according to predefined profit-sharing arrangements. By incorporating a transparent profit-sharing paradigm to incentivize large-scale data sharing, the 1TT Platform creates a collaborative environment to drive the advancement of NLP and LLM technologies.

With the increasing demand for substantial amounts of high-quality data to train large language models (LLMs), efficiently filtering large web corpora has become a critical challenge. For this purpose, KenLM, a lightweight n-gram-based language model that operates on CPUs, is widely used. However, the traditional method of training KenLM utilizes only high-quality data and, consequently, does not explicitly learn the linguistic patterns of low-quality data. To address this issue, we propose an ensemble approach that leverages two contrasting KenLMs: (i) Good KenLM, trained on high-quality data; and (ii) Bad KenLM, trained on low-quality data. Experimental results demonstrate that our approach significantly reduces noisy content while preserving high-quality content compared to the traditional KenLM training method. This indicates that our method can be a practical solution with minimal computational overhead for resource-constrained environments.

We use kinetic theory and the thermodynamic Bethe ansatz to derive the hydrodynamics of spin transport in the Haldane-Shastry chain. We obtain analytical expressions for relaxation from weak-domain-wall initial conditions and corresponding predictions for the spin Drude weight. We corroborate the latter against an exact microscopic calculation at infinite temperature and Luttinger-liquid expectations at zero temperature. We further obtain an analytical expression for the long-wavelength dynamics of the longitudinal spin structure factor at all fillings and temperatures. We show that the latter interpolates between an empirically exact expression that we have found at infinite temperature and the long-wavelength limit of the Haldane-Zirnbauer formula at zero temperature., Comment: v2: 15 pages, 2 figures, minor revisions and close to published version

The competition between scrambling and projective measurements can lead to measurement-induced entanglement phase transitions (MIPT). In this work, we show that the universality class of the MIPT is drastically altered when the system is coupled to a diffusing conserved density. Specifically, we consider a 1+1d random Clifford circuit locally monitored by classically diffusing particles (``measurers''). The resulting diffusive correlations in the measurement density are a relevant perturbation to the usual space-time random MIPT critical point, producing a new universality class for this phase transition. We find ``Griffiths-like'' effects due to rare space-time regions where, e.g., the diffusive measurers have a low or high density, but these are considerably weaker than the Griffiths effects that occur with quenched randomness that produce rare spatial regions with infinite lifetime., Comment: 6 pages, 4 figures + Supplementary Material (11 pages,10 figures)

Many-body localized (MBL) systems fail to reach thermal equilibrium under their own dynamics, even though they are interacting, nonintegrable, and in an extensively excited state. One instability towards thermalization of MBL systems is the so-called ``avalanche'', where a locally thermalizing rare region is able to spread thermalization through the full system. The spreading of the avalanche may be modeled and numerically studied in finite one-dimensional MBL systems by weakly coupling an infinite-temperature bath to one end of the system. We find that the avalanche spreads primarily via strong many-body resonances between rare near-resonant eigenstates of the closed system. Thus we find and explore a detailed connection between many-body resonances and avalanches in MBL systems., Comment: with some revisions, including a new discussion section at the end of the Supplement

When two identical two-dimensional (2D) periodic lattices are stacked in parallel after rotating one layer by a certain angle relative to the other layer, the resulting bilayer system can lose lattice periodicity completely and become a 2D quasicrystal. Twisted bilayer graphene with 30-degree rotation is a representative example. We show that such quasicrystalline bilayer systems generally develop macroscopically degenerate localized zero-energy states (ZESs) in strong coupling limit where the interlayer couplings are overwhelmingly larger than the intralayer couplings. The emergent chiral symmetry in strong coupling limit and aperiodicity of bilayer quasicrystals guarantee the existence of the ZESs. The macroscopically degenerate ZESs are analogous to the flat bands of periodic systems, in that both are composed of localized eigenstates, which give divergent density of states. For monolayers, we consider the triangular, square, and honeycomb lattices, comprised of homogenous tiling of three possible planar regular polygons: the equilateral triangle, square, and regular hexagon. We construct a compact theoretical framework, which we call the quasiband model, that describes the low energy properties of bilayer quasicrystals and counts the number of ZESs using a subset of Bloch states of monolayers. We also propose a simple geometric scheme in real space which can show the spatial localization of ZESs and count their number. Our work clearly demonstrates that bilayer quasicrystals in strong coupling limit are an ideal playground to study the intriguing interplay of flat band physics and the aperiodicity of quasicrystals., Comment: 5 pages, 4 figures + Supplementary Material (12 pages,5 figures)