Search

We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.

Designing new compounds with the coexistence of diverse physical properties is of great significance for broad applications in multifunctional electronic devices. In this work, based on density functional theory, we predict the coexistence of mechanical superhardness and the controllable magnetism in the clathrate material \b{eta}-C3N2 through the implant of the external atom into the intrinsic cage structure. Taking hydrogen-doping (H@\b{eta}-C3N2) and fluorine-doping (F@\b{eta}-C3N2) as examples, our calculations indicate these two doped configurations are stable and discovered that they belong to antiferromagnetic semiconductor and ferromagnetic semi-metal, respectively. These intriguing magnetic phase transitions originate from their distinctive band structure around the Fermi level and can be well understood by the 3D Hubbard model with half-filling occupation and the Stoner model. Moreover, the high Vickers hardness of 49.0 GPa for H@\b{eta}-C3N2 and 48.2 GPa for F@\b{eta}-C3N2 are obtained, suggesting they are clathrate superhard materials as its host. Therefore, the incorporation of H and F in \b{eta}-C3N2 gives rise to a new type of superhard antiferromagnetic semiconductor and superhard ferromagnetic semimetal, respectively, which could have potential applications in harsh conditions. Our work provides an effective strategy to design a new class of highly desirable multifunctional materials with excellent mechanical properties and magnetic properties, which may arouse spintronic applications in superhard materials in the future., Comment: 14 pages, 5 figures

Double holography offers a profound understanding of the island formula by describing a gravitational system on AdS$_d$ coupled to a conformal field theory on $\mathbb{R}^{1,d-1}$, dual to an AdS$_{d+1}$ spacetime with an end-of-the-world (EOW) brane. In this work, we extend the proposal in [A. Almheiri et al. JHEP 03 (2020) 149] by considering that the dual bulk spacetime has two EOW branes: one with a gravitational system and the other with a thermal bath. We demonstrate an equivalence between this proposal and the wedge holographic theory. We examine it in both Anti-de Sitter gravity and de Sitter gravity by calculating the entanglement entropy of the Hawking radiation. Finally, we employ the doubly holographic model to verify the formula for the entanglement entropy in a subregion within conformally flat spacetime.