Search

The rapid growth of artificial intelligence and deep learning in recent years has led to significant progress in various application domains, including computer vision (CV), natural language processing (NLP), and recommendation models (RM). As the complexity and size of Deep Neural Networks (DNNs) increase, the demand for more effective techniques to train and deploy these models also rises. This dissertation introduces three techniques aimed at enhancing the performance and efficiency of DNNs in three disjoint settings. First, we describe a technique for improving DNN model quality via a training time technique applied to RM-type models. Then, we describe two techniques for model-hardware co-design: (1) Quantization for resource-constrained deployments; (2) Training/tuning for deployments on analog Processing-in-Memory (PIM) accelerators. The dissertation first focuses on the development of better embedding training algorithms for recommendation models. Recommendation systems are widely used in various applications, such as e-commerce, online advertising, and content recommendation. These models often rely on large embedding layers, which can result in substantial memory and computational requirements. We first show that a strikingly simple factorization technique at training can noticeably improve the model size-accuracy trade-off. We show that its quality continues to improve when the inner dimension of factorization increases. The proposed factorization technique enables more efficient and accurate recommendation models without increasing their embedding size. Because factorization does not enlarge the search space of embedding training, its superiority and dependency on the inner dimension is puzzling. To understand the working mechanism behind the proposed factorization, we present a theory based on the dynamics of embedding updates in gradient descent training. Our theory suggest that the factorization leads to dense embedding updates with momentum-like effects, which are not present in conventional training. Then, the dissertation advances the state of the art in mixed-precision quantization algorithms for general DNNs. Quantization is a widely used technique that compresses DNNs by mapping their weights and activations to lower precision values. Conventional quantization methods assign the same precision to all layers in DNNs. Mixed-precision quantization (MPQ) algorithms allow different quantization bitwidths in layers to reduce the numerical precision of neural network weights and activations, leading to lower memory requirements and computational complexity while maintaining minimal loss in accuracy. By improving MPQ techniques, we can enhance the efficiency of DNNs and facilitate their deployment on resource-constrained devices and specialized hardware. We propose a technique based on exploring interactions effects between different layers during quantization. We first show, through empirical studies on several commonly used DNNs, that cross-layer quantization error interactions are non-negligible and that the assumption of layer-independent quantization error in prior work leads to sub-optimal MPQ solutions. Then, we present the first post-training and backpropogation-free MPQ algorithm that is aware of the cross-layer quantization effects. The algorithm is based on the second-order Taylor expansion of the quantization loss and formulates the MPQ problem as an integer quadratic programming (IQP) problem. The formulated IQP is fast to solve and delivers state-of-the-art MPQ performance. Finally, the dissertation addresses the challenges associated with analog PIM DNN accelerators, which have emerged as a promising solution to reduce data movement and improve energy efficiency in memory-intensive workloads. Analog PIM architectures face issues such as device variability and limited precision. To overcome these challenges, we first propose a novel training algorithm that makes use of straight-through estimator and variability sampling to reduce the degradation due to quantization and variability While prior work generally assumes variations are independent, we study a more realistic setting where variation present spatially correlated variability patterns that occur on a chip-to-chip basis. Under correlated variations, training alone is not able to mitigate the destructive degradation. We propose a innovative inference-time tuning architecture to first estimate the degree of variation and then correct the error in outputs based on the estimation. Our proposed training algorithm and inference-time tuning architecture significantly enhance the robustness and accuracy of analog PIM accelerators, while maintaining the benefits of in-memory processing.

The continuously growing computing power and the advent of advanced computational algorithms have enabled the atomistic simulations of materials at an unprecedented accuracy and scalability. In particular, first-principles calculations based on density functional theory (DFT) provide exceptional predictive capability in studying solid-state materials. In this dissertation, I applied DFT calculations to study a wide range of semiconductors that show great promise in electronic and optoelectronic applications, with a focus on the thermodynamics of point defects in semiconductors and computational discovery of novel semiconductor alloys. Point defects are pervasive in semiconductors and it is crucial to understand their impact on the properties. Efficient p-type doping of ultrawide bandgap nitride semiconductors has been a long-standing challenge. By calculating the formation energy and charge-transition levels for intrinsic defects and extrinsic Mg dopant, I discover two avenues towards an enhanced p-doping efficiency for AlN and Al-rich AlGaN alloy. The first approach is to engineer the N chemical potentials to be N-rich, which dramatically reduces the formation energy for Mg dopant by about 2 eV and increases the formation energy of compensating N vacancy by almost 3 eV. Another method is to engineer the position of the Fermi level away from the valence band of AlGaN alloy by the formation of Ga/AlGaN Schottky junction during non-equilibrium growth. This leads to an extremely small formation energy of 0.4 eV for Mg dopant. These two approaches can be generalized to a broader range of ultrawide bandgap semiconductors. Using the same methodology, the electronic properties and defect physics of LaN and Cu2O are investigated. Contrast to previous claim that LaN is a semimetal, I find that LaN has a direct bandgap of 0.62 eV and is an intrinsic n-type semiconductor. The origin of its electrical conductivity is likely due to the formation of N vacancy and substitutional O impurities. The dominant intrinsic defects of Cu2O are the Cu simple vacancy and the Cu split vacancy. However, their large ionization energies lead to low hole concentration. I survey a variety of candidate elements for p-type doping and find that N, S, and Mg are effective p-dopants. Another part of this dissertation studies the thermodynamic and electronic properties of emerging semiconductors and semiconductor alloys. II-IV-N2 materials exhibit unique properties due to disordering on the cation sublattice. The short-range disorder around the N atom has large impact on the thermodynamic stability. Using ZnSnN2 as a model system, I find that the formation energy decreases with respect to an increased fraction of octet-rule conserving Zn2Sn2 motifs. Interestingly, the bandgap of ZnSnN2 is linearly proportional to the square of the long-range order parameter, which enables alloy-free bandgap engineering. In some cases, configurational disorder can improve the thermodynamic stability by maximizing the entropy of the system. Using high-throughput DFT calculations, I investigate the stability over the entire composition space for GeSnPbSSeTe high entropy chalcogenide alloys (HEC). At the growth temperature of 900 K, 99% of the HEC compositions are stabilized by the large entropy. The bandgap calculations and the transport measurement by my collaborators show that equimolar HEC is an ambipolarly dopable semiconductor. The computational studies presented here demonstrate the importance of point defects in controlling the properties of semiconductors and pave the way for the adoption of heterovalent ternary nitrides and high entropy chalcogenide alloys in a wide range of functional applications.

DNNs deployed on analog processing in memory (PIM) architectures are subject to fabrication-time variability. We developed a new joint variability- and quantization-aware DNN training algorithm for highly quantized analog PIM-based models that is significantly more effective than prior work. It outperforms variability-oblivious and post-training quantized models on multiple computer vision datasets/models. For low- bitwidth models and high variation, the gain in accuracy is up to 35.7% for ResNet-18 over the best alternative. We demonstrate that, under a realistic pattern of within- and between-chip components of variability, training alone is unable to prevent large DNN accuracy loss (of up to 54% on CIFAR- 100/ResNet-18). We introduce a self-tuning DNN architecture that dynamically adjusts layer-wise activations during inference and is effective in reducing accuracy loss to below 10%.