Search

Pretraining transformers are generally time-consuming. Fully quantized training (FQT) is a promising approach to speed up pretraining. However, most FQT methods adopt a quantize-compute-dequantize procedure, which often leads to suboptimal speedup and significant performance degradation when used in transformers due to the high memory access overheads and low-precision computations. In this work, we propose Jetfire, an efficient and accurate INT8 training method specific to transformers. Our method features an INT8 data flow to optimize memory access and a per-block quantization method to maintain the accuracy of pretrained transformers. Extensive experiments demonstrate that our INT8 FQT method achieves comparable accuracy to the FP16 training baseline and outperforms the existing INT8 training works for transformers. Moreover, for a standard transformer block, our method offers an end-to-end training speedup of 1.42x and a 1.49x memory reduction compared to the FP16 baseline. Our code is open sourced at https://github.com/thu-ml/Jetfire-INT8Training., Comment: 15 pages, 8 figures, 11 tables

Bi2O2Se has attracted intensive attention due to its potential in electronics, optoelectronics, as well as ferroelectric applications. Despite that, there have only been a handful of experimental studies based on ultrafast spectroscopy to elucidate the carrier dynamics in Bi2O2Se thin films, Different groups have reported various ultrafast timescales and associated mechanisms across films of different thicknesses. A comprehensive understanding in relation to thickness and fluence is still lacking. In this work, we have systematically explored the thickness-dependent Raman spectroscopy and ultrafast carrier dynamics in chemical vapor deposition (CVD)-grown Bi2O2Se thin films on mica substrate with thicknesses varying from 22.44 nm down to 4.62 nm at both low and high pump fluence regions. Combining the thickness dependence and fluence dependence of the slow decay time, we demonstrate a ferroelectric transition in the thinner (< 8 nm) Bi2O2Se films, influenced by substrate-induced compressive strain and non-equilibrium states. Moreover, this transition can be manifested under highly non-equilibrium states. Our results deepen the understanding of the interplay between the ferroelectric phase and semiconducting characteristics of Bi2O2Se thin films, providing a new route to manipulate the ferroelectric transition.

The robustness to distribution changes ensures that NLP models can be successfully applied in the realistic world, especially for information extraction tasks. However, most prior evaluation benchmarks have been devoted to validating pairwise matching correctness, ignoring the crucial measurement of robustness. In this paper, we present the first benchmark that simulates the evaluation of open information extraction models in the real world, where the syntactic and expressive distributions under the same knowledge meaning may drift variously. We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique that consists of sentences with structured knowledge of the same meaning but with different syntactic and expressive forms. By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques. We perform experiments on typical models published in the last decade as well as a popular large language model, the results show that the existing successful models exhibit a frustrating degradation, with a maximum drop of 23.43 F1 score. Our resources and code are available at https://github.com/qijimrc/ROBUST., Comment: Accepted by EMNLP 2023 Main Conference

Open Information Extraction models have shown promising results with sufficient supervision. However, these models face a fundamental challenge that the syntactic distribution of training data is partially observable in comparison to the real world. In this paper, we propose a syntactically robust training framework that enables models to be trained on a syntactic-abundant distribution based on diverse paraphrase generation. To tackle the intrinsic problem of knowledge deformation of paraphrasing, two algorithms based on semantic similarity matching and syntactic tree walking are used to restore the expressionally transformed knowledge. The training framework can be generally applied to other syntactic partial observable domains. Based on the proposed framework, we build a new evaluation set called CaRB-AutoPara, a syntactically diverse dataset consistent with the real-world setting for validating the robustness of the models. Experiments including a thorough analysis show that the performance of the model degrades with the increase of the difference in syntactic distribution, while our framework gives a robust boundary. The source code is publicly available at https://github.com/qijimrc/RobustOIE.

Bowers and Stephenson introduced the notion of inversive distance circle packings as a natural generalization of Thurston's circle packings. They conjectured that discrete conformal maps induced by inversive distance circle packings converge to the Riemann mapping. Motivated by the recent work of Luo-Sun-Wu, we prove Bowers-Stephenson's conjecture for Jordan domains by establishing a maximal principle, an infinite rigidity theorem, and a solvability theorem of certain prescribing combinatorial curvature problems for inversive distance circle packings., Comment: 33 pages, 6 figures

The emerging Au-assisted exfoliation technique provides a wealth of large-area and high-quality ultrathin two-dimensional (2D) materials compared with traditional tape-based exfoliation. Fast, damage-free, and reliable determination of the layer number of such 2D films is essential to study layer-dependent physics and promote device applications. Here, an optical method has been developed for simple, high throughput, and accurate determination of the layer number for Au-assisted exfoliated MoS$_2$ and WS$_2$ films in a broad thickness range. The method is based on quantitative analysis of layer-dependent white light reflection spectra, revealing that the reflection peak intensity can be used as a clear indicator for determining the layer number. The simple yet robust method will facilitate the fundamental study on layer-dependent optical, electrical, and thermal properties and device applications of 2D materials. The technique can also be readily combined with photoluminescence and Raman spectroscopies to study other layer-dependent physical properties of 2D materials.

Heterogeneous graph neural networks (HGNNs) have been blossoming in recent years, but the unique data processing and evaluation setups used by each work obstruct a full understanding of their advancements. In this work, we present a systematical reproduction of 12 recent HGNNs by using their official codes, datasets, settings, and hyperparameters, revealing surprising findings about the progress of HGNNs. We find that the simple homogeneous GNNs, e.g., GCN and GAT, are largely underestimated due to improper settings. GAT with proper inputs can generally match or outperform all existing HGNNs across various scenarios. To facilitate robust and reproducible HGNN research, we construct the Heterogeneous Graph Benchmark (HGB), consisting of 11 diverse datasets with three tasks. HGB standardizes the process of heterogeneous graph data splits, feature processing, and performance evaluation. Finally, we introduce a simple but very strong baseline Simple-HGN--which significantly outperforms all previous models on HGB--to accelerate the advancement of HGNNs in the future., Comment: KDD 2021 research track

Video super-resolution has recently become one of the most important mobile-related problems due to the rise of video communication and streaming services. While many solutions have been proposed for this task, the majority of them are too computationally expensive to run on portable devices with limited hardware resources. To address this problem, we introduce the first Mobile AI challenge, where the target is to develop an end-to-end deep learning-based video super-resolution solutions that can achieve a real-time performance on mobile GPUs. The participants were provided with the REDS dataset and trained their models to do an efficient 4X video upscaling. The runtime of all models was evaluated on the OPPO Find X2 smartphone with the Snapdragon 865 SoC capable of accelerating floating-point networks on its Adreno GPU. The proposed solutions are fully compatible with any mobile GPU and can upscale videos to HD resolution at up to 80 FPS while demonstrating high fidelity results. A detailed description of all models developed in the challenge is provided in this paper., Comment: Mobile AI 2021 Workshop and Challenges: https://ai-benchmark.com/workshops/mai/2021/. arXiv admin note: substantial text overlap with arXiv:2105.07825. substantial text overlap with arXiv:2105.08629, arXiv:2105.07809, arXiv:2105.08630