Search

Recent years have seen an increase in the development of large deep learning (DL) models, which makes training efficiency crucial. Common practice is struggling with the trade-off between usability and performance. On one hand, DL frameworks such as PyTorch use dynamic graphs to facilitate model developers at a price of sub-optimal model training performance. On the other hand, practitioners propose various approaches to improving the training efficiency by sacrificing some of the flexibility, ranging from making the graph static for more thorough optimization (e.g., XLA) to customizing optimization towards large-scale distributed training (e.g., DeepSpeed and Megatron-LM). In this paper, we aim to address the tension between usability and training efficiency through separation of concerns. Inspired by DL compilers that decouple the platform-specific optimizations of a tensor-level operator from its arithmetic definition, this paper proposes a schedule language to decouple model execution from definition. Specifically, the schedule works on a PyTorch model and uses a set of schedule primitives to convert the model for common model training optimizations such as high-performance kernels, effective 3D parallelism, and efficient activation checkpointing. Compared to existing optimization solutions, we optimize the model as-needed through high-level primitives, and thus preserving programmability and debuggability for users to a large extent. Our evaluation results show that by scheduling the existing hand-crafted optimizations in a systematic way, we are able to improve training throughput by up to 3.35x on a single machine with 8 NVIDIA V100 GPUs, and by up to 1.32x on multiple machines with up to 64 GPUs, when compared to the out-of-the-box performance of DeepSpeed and Megatron-LM.

Pruning is a popular technique for reducing the model size and computational cost of convolutional neural networks (CNNs). However, a slow retraining or fine-tuning procedure is often required to recover the accuracy loss caused by pruning. Recently, a new research direction on weight pruning, pruning-at-initialization (PAI), is proposed to directly prune CNNs before training so that fine-tuning or retraining can be avoided. While PAI has shown promising results in reducing the model size, existing approaches rely on fine-grained weight pruning which requires unstructured sparse matrix computation, making it difficult to achieve real speedup in practice unless the sparsity is very high. This work is the first to show that fine-grained weight pruning is in fact not necessary for PAI. Instead, the layerwise compression ratio is the main critical factor to determine the accuracy of a CNN model pruned at initialization. Based on this key observation, we propose PreCropping, a structured hardware-efficient model compression scheme. PreCropping directly compresses the model at the channel level following the layerwise compression ratio. Compared to weight pruning, the proposed scheme is regular and dense in both storage and computation without sacrificing accuracy. In addition, since PreCropping compresses CNNs at initialization, the computational and memory costs of CNNs are reduced for both training and inference on commodity hardware. We empirically demonstrate our approaches on several modern CNN architectures, including ResNet, ShuffleNet, and MobileNet for both CIFAR-10 and ImageNet.

Graph neural networks (GNNs) have extended the success of deep neural networks (DNNs) to non-Euclidean graph data, achieving ground-breaking performance on various tasks such as node classification and graph property prediction. Nonetheless, existing systems are inefficient to train large graphs with billions of nodes and edges with GPUs. The main bottlenecks are the process of preparing data for GPUs - subgraph sampling and feature retrieving. This paper proposes BGL, a distributed GNN training system designed to address the bottlenecks with a few key ideas. First, we propose a dynamic cache engine to minimize feature retrieving traffic. By a co-design of caching policy and the order of sampling, we find a sweet spot of low overhead and high cache hit ratio. Second, we improve the graph partition algorithm to reduce cross-partition communication during subgraph sampling. Finally, careful resource isolation reduces contention between different data preprocessing stages. Extensive experiments on various GNN models and large graph datasets show that BGL significantly outperforms existing GNN training systems by 20.68x on average., Comment: Under Review

One-dimensional crystals of fluorinated perylene diimides were achieved by the self-assembly of them via solvent-nonsolvent exchanging. The π-conjugated fluorinated perylene diimides were assembled into highly-ordered nanostructures of well-defined morphologies in organic solvents due to the π-π interaction between the aromatic cores. It was found that with more introduced F atoms, perylene diimides showed remarkably improved solubility and thus were much easier to grow into crystals, due to the increased polarity induced by the strong electron-withdrawing F group. More importantly, single crystal of N,N′-diperfluorophenyl-3,4,9,10-perylenetetracarboxylic diimide(DPFPP) was obtained, and the unit cell-dimensions of triclinic structure were determined by the selected area electron diffraction( SAED) patterns to be a=0.712 nm, b=1.072 nm, c=2.914 nm, α=97.0°, β=89.6°, γ=93.4°. Owing to most of the longest c-axis orienting nearly vertically to the long axis of the needle crystal, the molecular planes are expected to be vertical to the needle axis.

A novel organic electron acceptor, N,N′-dipyrimidinyl-3,4,9,10-perylene-tetracarboxylic diimide (DMP), was designed and synthesized. The molecular structure was characterized by FTIR spectrum and elemental analysis. By cyclic voltammetry measurements, DMP was found to possess a lower LUMO energy level than N,N′-diphenyl-3,4,9,10-perylene-tetracarboxylic diimide due to the stronger electron-withdrawing pyrimidinyl group than the phenyl group. Fluorescence quenching is observed in a dual-layer film consisting of a DMP layer and a C60 layer and was attributed to the charge transfer at the interface due to the energy level offset between DMP and C60.

Synchrotron radiation photoemission was used to investigate the interface between the organic semiconductor (BZP) and polycrystalline silver film. Before complete coverage by one monolayer, the BZP molecules interacted weekly with silver and a gap state appeared at 0.9eV below Fermi level. After the coverage is complete, the growth of BZP is inferred to be in the 3D-island mode and the interaction with the substrate gradually diminishes. Because of the final state effect, the highest occupied molecular orbital (HOMO) level shifts to higher binding energy and reaches ultimately 2.3eV of the HOMO level in the bulk. The deposition of BZP on silver surface resulted in the decrease of sample work function, which demonstrats interfacial dipoles forming on the BZP/Ag contact (Δ=0.3eV) and indicats the electrons being deviated from the organic molecules to the metal substrate. In addition, the effect of annealing and exposure to oxygen on the BZP/Ag interface has been taken into account. It is found that the binding energies of the characteristic peaks of BZP and the gap state hardly change after interface annealing at 250 ℃, but after being exposed to oxygen, the gap state disappears and the valence bands shift to lower binding energies.