Search

Generative artificial intelligence (AI) technology is revolutionizing the computing industry. Not only its applications have broadened to various sectors but also poses new system design and optimization opportunities. The technology is capable of understanding and responding in multiple modalities. However, the advanced capability currently comes with significant system resource demands. To sustainably scale generative AI capabilities to billions of users in the world, inference must be fast and efficient. This paper pinpoints key system design and optimization opportunities by characterizing a family of emerging multi-modal generation models on real systems. Auto-regressive token generation is a critical latency performance bottleneck, typically dominated by GPU idle time. In addition to memory-intensive attention across the generative AI models, linear operations constitute significant inference latency due to the feed forward networks in Transformer-based models. We demonstrate that state-of-the-art optimization levers, spanning from applications to system software and hardware, set a 3.88x better baseline., Comment: 13 pages including references. 8 Figures. Under review to HPCA 2025 Industry Track

Training large-scale machine learning models poses distinct system challenges, given both the size and complexity of today's workloads. Recently, many organizations training state-of-the-art Generative AI models have reported cases of instability during training, often taking the form of loss spikes. Numeric deviation has emerged as a potential cause of this training instability, although quantifying this is especially challenging given the costly nature of training runs. In this work, we develop a principled approach to understanding the effects of numeric deviation, and construct proxies to put observations into context when downstream effects are difficult to quantify. As a case study, we apply this framework to analyze the widely-adopted Flash Attention optimization. We find that Flash Attention sees roughly an order of magnitude more numeric deviation as compared to Baseline Attention at BF16 when measured during an isolated forward pass. We then use a data-driven analysis based on the Wasserstein Distance to provide upper bounds on how this numeric deviation impacts model weights during training, finding that the numerical deviation present in Flash Attention is 2-5 times less significant than low-precision training.

We present LayerSkip, an end-to-end solution to speed-up inference of large language models (LLMs). First, during training we apply layer dropout, with low dropout rates for earlier layers and higher dropout rates for later layers, and an early exit loss where all transformer layers share the same exit. Second, during inference, we show that this training recipe increases the accuracy of early exit at earlier layers, without adding any auxiliary layers or modules to the model. Third, we present a novel self-speculative decoding solution where we exit at early layers and verify and correct with remaining layers of the model. Our proposed self-speculative decoding approach has less memory footprint than other speculative decoding approaches and benefits from shared compute and activations of the draft and verification stages. We run experiments on different Llama model sizes on different types of training: pretraining from scratch, continual pretraining, finetuning on specific data domain, and finetuning on specific task. We implement our inference solution and show speedups of up to 2.16x on summarization for CNN/DM documents, 1.82x on coding, and 2.0x on TOPv2 semantic parsing task. We open source our code and checkpoints at https://github.com/facebookresearch/LayerSkip., Comment: ACL 2024

Large Language Models (LLMs) with hundreds of billions of parameters have transformed the field of machine learning. However, serving these models at inference time is both compute and memory intensive, where a single request can require multiple GPUs and tens of Gigabytes of memory. Multi-Head Attention is one of the key components of LLMs, which can account for over 50% of LLMs memory and compute requirement. We observe that there is a high amount of redundancy across heads on which tokens they pay attention to. Based on this insight, we propose Clustered Head Attention (CHAI). CHAI combines heads with a high amount of correlation for self-attention at runtime, thus reducing both memory and compute. In our experiments, we show that CHAI is able to reduce the memory requirements for storing K,V cache by up to 21.4% and inference time latency by up to 1.73x without any fine-tuning required. CHAI achieves this with a maximum 3.2% deviation in accuracy across 3 different models (i.e. OPT-66B, LLAMA-7B, LLAMA-33B) and 5 different evaluation datasets.

As the development of large-scale Generative AI models evolve beyond text (1D) generation to include image (2D) and video (3D) generation, processing spatial and temporal information presents unique challenges to quality, performance, and efficiency. We present the first work towards understanding this new system design space for multi-modal text-to-image (TTI) and text-to-video (TTV) generation models. Current model architecture designs are bifurcated into 2 categories: Diffusion- and Transformer-based models. Our systematic performance characterization on a suite of eight representative TTI/TTV models shows that after state-of-the-art optimization techniques such as Flash Attention are applied, Convolution accounts for up to 44% of execution time for Diffusion-based TTI models, while Linear layers consume up to 49% of execution time for Transformer-based models. We additionally observe that Diffusion-based TTI models resemble the Prefill stage of LLM inference, and benefit from 1.1-2.5x greater speedup from Flash Attention than Transformer-based TTI models that resemble the Decode phase. Since optimizations designed for LLMs do not map directly onto TTI/TTV models, we must conduct a thorough characterization of these workloads to gain insights for new optimization opportunities. In doing so, we define sequence length in the context of TTI/TTV models and observe sequence length can vary up to 4x in Diffusion model inference. We additionally observe temporal aspects of TTV workloads pose unique system bottlenecks, with Temporal Attention accounting for over 60% of total Attention time. Overall, our in-depth system performance characterization is a critical first step towards designing efficient and deployable systems for emerging TTI/TTV workloads., Comment: Published at 2024 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS)

In 1980 Michelsohn defined a differential operator on sections of the complex Clifford bundle over a compact K\"ahler manifold M . This operator is a differential and its Laplacian agrees with the Laplacian of the Dolbeault operator on forms through a natural identification of differential forms with sections of the Clifford bundle. Relaxing the condition that M be K\"ahler, we introduce two differential operators on sections of the complex Clifford bundle over a compact almost Hermitian manifold which naturally generalize the one introduced by Michelsohn. We show surprising K\"ahler-like symmetries of the kernel of the Laplacians of these operators in the almost Hermitian and almost K\"ahler settings, along with a correspondence of these operators to operators on forms which are of present interest in almost complex geometry.

We obtain a generalization, for a general compact almost complex manifold, of the well-known K\"{a}hler (or Hodge) identities for K\"{a}hler manifolds involving the commutators of the exterior differential and the Lefschetz operator and its adjoint. The main idea is to study the problem on the Clifford bundle via the Dirac operator, and then translate the results to the exterior bundle., Comment: 21 pages, 2 figures

With this work, we investigate the use of Reinforcement Learning (RL) for the generation of spatial assemblies, by combining ideas from Procedural Generation algorithms (Wave Function Collapse algorithm (WFC)) and RL for Game Solving. WFC is a Generative Design algorithm, inspired by Constraint Solving. In WFC, one defines a set of tiles/blocks and constraints and the algorithm generates an assembly that satisfies these constraints. Casting the problem of generation of spatial assemblies as a Markov Decision Process whose states transitions are defined by WFC, we propose an algorithm that uses Reinforcement Learning and Self-Play to learn a policy that generates assemblies that maximize objectives set by the designer. Finally, we demonstrate the use of our Spatial Assembly algorithm in Architecture Design., Comment: Workshop on Machine Learning for Creativity and Design at the 34rd Conference on Neural Information Processing Systems (NeurIPS 2020)

The low-lying energy levels of proton-rich $^{56}$Cu have been extracted using in-beam $\gamma$-ray spectroscopy with the state-of-the-art $\gamma$-ray tracking array GRETINA in conjunction with the S800 spectrograph at the National Superconducting Cyclotron Laboratory at Michigan State University. Excited states in $^{56}$Cu serve as resonances in the $^{55}$Ni(p,$\gamma$)$^{56}$Cu reaction, which is a part of the rp-process in type I x-ray bursts. To resolve existing ambiguities in the reaction Q-value, a more localized IMME mass fit is used resulting in $Q=639\pm82$~keV. We derive the first experimentally-constrained thermonuclear reaction rate for $^{55}$Ni(p,$\gamma$)$^{56}$Cu. We find that, with this new rate, the rp-process may bypass the $^{56}$Ni waiting point via the $^{55}$Ni(p,$\gamma$) reaction for typical x-ray burst conditions with a branching of up to $\sim$40$\%$. We also identify additional nuclear physics uncertainties that need to be addressed before drawing final conclusions about the rp-process reaction flow in the $^{56}$Ni region., Comment: 8 pages, accepted for Phys. Rev. C

The recent advent of wave-shaping methods has demonstrated the focusing of light through and inside even the most strongly scattering materials. Typically in wavefront shaping, light is focused in an area with the size of one speckle spot. It has been shown that the intensity is not only increased in the target speckle spot, but also in an area outside the optimized speckle spot. Consequently, the total transmission is enhanced, even though only the intensity in a single speckle spot is controlled. Here, we experimentally study how the intensity enhancement on both interfaces of a scattering medium depends on the optimization area on the transmission side. We observe that as the optimization radius increases, the enhancement of the total transmitted intensity increases. We find a concomitant decrease of the total reflected intensity, which implies an energy redistribution between transmission and reflection channels. In addition, we find a qualitative evidence of a long-range reflection-transmission correlation. Our result is useful for efficient light harvesting in solar cells, multi-channel quantum secure communications, imaging, and complex beam delivery through a scattering medium., Comment: 9 pages, 8 figures