Search

We have witnessed the unprecedented success of diffusion-based video generation over the past year. Recently proposed models from the community have wielded the power to generate cinematic and high-resolution videos with smooth motions from arbitrary input prompts. However, as a supertask of image generation, video generation models require more computation and are thus hosted mostly on cloud servers, limiting broader adoption among content creators. In this work, we propose a comprehensive acceleration framework to bring the power of the large-scale video diffusion model to the hands of edge users. From the network architecture scope, we initialize from a compact image backbone and search out the design and arrangement of temporal layers to maximize hardware efficiency. In addition, we propose a dedicated adversarial fine-tuning algorithm for our efficient model and reduce the denoising steps to 4. Our model, with only 0.6B parameters, can generate a 5-second video on an iPhone 16 PM within 5 seconds. Compared to server-side models that take minutes on powerful GPUs to generate a single video, we accelerate the generation by magnitudes while delivering on-par quality., Comment: https://snap-research.github.io/snapgen-v/

Existing text-to-image (T2I) diffusion models face several limitations, including large model sizes, slow runtime, and low-quality generation on mobile devices. This paper aims to address all of these challenges by developing an extremely small and fast T2I model that generates high-resolution and high-quality images on mobile platforms. We propose several techniques to achieve this goal. First, we systematically examine the design choices of the network architecture to reduce model parameters and latency, while ensuring high-quality generation. Second, to further improve generation quality, we employ cross-architecture knowledge distillation from a much larger model, using a multi-level approach to guide the training of our model from scratch. Third, we enable a few-step generation by integrating adversarial guidance with knowledge distillation. For the first time, our model SnapGen, demonstrates the generation of 1024x1024 px images on a mobile device around 1.4 seconds. On ImageNet-1K, our model, with only 372M parameters, achieves an FID of 2.06 for 256x256 px generation. On T2I benchmarks (i.e., GenEval and DPG-Bench), our model with merely 379M parameters, surpasses large-scale models with billions of parameters at a significantly smaller size (e.g., 7x smaller than SDXL, 14x smaller than IF-XL).

In this paper, we study the existence of a dense orbit for the diagonal $\PGL(n)$ action on self-products of partial flag varieties. We determine when there exists a dense orbit for flag varieties of the form $F(k_1, \dots, k_r; n)^m$ when $k_i = i$ or when $m=3$ and $k_i = \ell + i$ for some $\ell \geq 0$. We also show that certain infinite families of products do not have a dense orbit., Comment: 15

Vision Transformers (ViT) is known for its scalability. In this work, we target to scale down a ViT to fit in an environment with dynamic-changing resource constraints. We observe that smaller ViTs are intrinsically the sub-networks of a larger ViT with different widths. Thus, we propose a general framework, named Scala, to enable a single network to represent multiple smaller ViTs with flexible inference capability, which aligns with the inherent design of ViT to vary from widths. Concretely, Scala activates several subnets during training, introduces Isolated Activation to disentangle the smallest sub-network from other subnets, and leverages Scale Coordination to ensure each sub-network receives simplified, steady, and accurate learning objectives. Comprehensive empirical validations on different tasks demonstrate that with only one-shot training, Scala learns slimmable representation without modifying the original ViT structure and matches the performance of Separate Training. Compared with the prior art, Scala achieves an average improvement of 1.6% on ImageNet-1K with fewer parameters., Comment: Accepted by NeurIPS 2024

Markov chains are simple yet powerful mathematical structures to model temporally dependent processes. They generally assume stationary data, i.e., fixed transition probabilities between observations/states. However, live, real-world processes, like in the context of activity tracking, biological time series, or industrial monitoring, often switch behavior over time. Such behavior switches can be modeled as transitions between higher-level \emph{modes} (e.g., running, walking, etc.). Yet all modes are usually not previously known, often exhibit vastly differing transition probabilities, and can switch unpredictably. Thus, to track behavior changes of live, real-world processes, this study proposes an online and efficient method to construct Evolving Markov chains (EMCs). EMCs adaptively track transition probabilities, automatically discover modes, and detect mode switches in an online manner. In contrast to previous work, EMCs are of arbitrary order, the proposed update scheme does not rely on tracking windows, only updates the relevant region of the probability tensor, and enjoys geometric convergence of the expected estimates. Our evaluation of synthetic data and real-world applications on human activity recognition, electric motor condition monitoring, and eye-state recognition from electroencephalography (EEG) measurements illustrates the versatility of the approach and points to the potential of EMCs to efficiently track, model, and understand live, real-world processes., Comment: 20 pages, 8 figures

Neural network architecture design requires making many crucial decisions. The common desiderata is that similar decisions, with little modifications, can be reused in a variety of tasks and applications. To satisfy that, architectures must provide promising latency and performance trade-offs, support a variety of tasks, scale efficiently with respect to the amounts of data and compute, leverage available data from other tasks, and efficiently support various hardware. To this end, we introduce AsCAN -- a hybrid architecture, combining both convolutional and transformer blocks. We revisit the key design principles of hybrid architectures and propose a simple and effective \emph{asymmetric} architecture, where the distribution of convolutional and transformer blocks is \emph{asymmetric}, containing more convolutional blocks in the earlier stages, followed by more transformer blocks in later stages. AsCAN supports a variety of tasks: recognition, segmentation, class-conditional image generation, and features a superior trade-off between performance and latency. We then scale the same architecture to solve a large-scale text-to-image task and show state-of-the-art performance compared to the most recent public and commercial models. Notably, even without any computation optimization for transformer blocks, our models still yield faster inference speed than existing works featuring efficient attention mechanisms, highlighting the advantages and the value of our approach., Comment: NeurIPS 2024. Project Page: https://snap-research.github.io/snap_image/

Active metamaterials are engineered structures that possess novel properties that can be changed after the point of manufacture. Their novel properties arise predominantly from their physical structure, as opposed to their chemical composition and can be changed through means such as direct energy addition into wave paths, or physically changing/morphing the structure in response to both a user or environmental input. Active metamaterials are currently of wide interest to the physics community and encompass a range of sub-domains in applied physics (e.g. photonic, microwave, acoustic, mechanical, etc.). They possess the potential to provide solutions that are more suitable to specific applications, or which allow novel properties to be produced which cannot be achieved with passive metamaterials, such as time-varying or gain enhancement effects. They have the potential to help solve some of the important current and future problems faced by the advancement of modern society, such as achieving net-zero, sustainability, healthcare and equality goals. Despite their huge potential, the added complexity of their design and operation, compared to passive metamaterials creates challenges to the advancement of the field, particularly beyond theoretical and lab-based experiments. This roadmap brings together experts in all types of active metamaterials and across a wide range of areas of applied physics. The objective is to provide an overview of the current state of the art and the associated current/future challenges, with the hope that the required advances identified create a roadmap for the future advancement and application of this field.

Direct Preference Optimization (DPO) has emerged as a powerful approach to align text-to-image (T2I) models with human feedback. Unfortunately, successful application of DPO to T2I models requires a huge amount of resources to collect and label large-scale datasets, e.g., millions of generated paired images annotated with human preferences. In addition, these human preference datasets can get outdated quickly as the rapid improvements of T2I models lead to higher quality images. In this work, we investigate a scalable approach for collecting large-scale and fully synthetic datasets for DPO training. Specifically, the preferences for paired images are generated using a pre-trained reward function, eliminating the need for involving humans in the annotation process, greatly improving the dataset collection efficiency. Moreover, we demonstrate that such datasets allow averaging predictions across multiple models and collecting ranked preferences as opposed to pairwise preferences. Furthermore, we introduce RankDPO to enhance DPO-based methods using the ranking feedback. Applying RankDPO on SDXL and SD3-Medium models with our synthetically generated preference dataset "Syn-Pic" improves both prompt-following (on benchmarks like T2I-Compbench, GenEval, and DPG-Bench) and visual quality (through user studies). This pipeline presents a practical and scalable solution to develop better preference datasets to enhance the performance of text-to-image models., Comment: Project Page: https://snap-research.github.io/RankDPO/

The Brill-Noether theory of curves plays a fundamental role in the theory of curves and their moduli and has been intensively studied since the 19th century. In contrast, Brill-Noether theory for higher dimensional varieties is less understood. It is hard to determine when Brill-Noether loci are nonempty and these loci can be reducible and of larger than the expected dimension. Let $E$ be a semistable sheaf on the projective plane. In this paper, we give an upper bound for $h^0(E)$ in terms of the rank $r$ and the slope $\mu$ of $E$. We show that the bound is achieved precisely when $E$ is a twist of a Steiner bundle. We classify the sheaves $E$ such that $h^0(E)$ is sufficiently close to the upper bound. We determine the nonemptiness, irreducibility and dimension of the Brill-Noether loci in the moduli spaces of sheaves with $h^0(E)$ in this range. When they are nonempty, these Brill-Noether loci are irreducible though almost always of larger than the expected dimension., Comment: 22 pages, comments welcome!

Spectral Graph Convolutional Networks (GCNs) have gained popularity in graph machine learning applications due, in part, to their flexibility in specification of network propagation rules. These propagation rules are often constructed as polynomial filters whose coefficients are learned using label information during training. In contrast to learned polynomial filters, explicit filter functions are useful in capturing relationships between network topology and distribution of labels across the network. A number of algorithms incorporating either approach have been proposed; however the relationship between filter functions and polynomial approximations is not fully resolved. This is largely due to the ill-conditioned nature of the linear systems that must be solved to derive polynomial approximations of filter functions. To address this challenge, we propose a novel Arnoldi orthonormalization-based algorithm, along with a unifying approach, called G-Arnoldi-GCN that can efficiently and effectively approximate a given filter function with a polynomial. We evaluate G-Arnoldi-GCN in the context of multi-class node classification across ten datasets with diverse topological characteristics. Our experiments show that G-Arnoldi-GCN consistently outperforms state-of-the-art methods when suitable filter functions are employed. Overall, G-Arnoldi-GCN opens important new directions in graph machine learning by enabling the explicit design and application of diverse filter functions. Code link: https://github.com/mustafaCoskunAgu/GArnoldi-GCN

Astrophysical simulations are computation, memory, and thus energy intensive, thereby requiring new hardware advances for progress. Stony Brook University recently expanded its computing cluster "SeaWulf" with an addition of 94 new nodes featuring Intel Sapphire Rapids Xeon Max series CPUs. We present a performance and power efficiency study of this hardware performed with FLASH: a multi-scale, multi-physics, adaptive mesh-based software instrument. We extend this study to compare performance to that of Stony Brook's Ookami testbed which features ARM-based A64FX-700 processors, and SeaWulf's AMD EPYC Milan and Intel Skylake nodes. Our application is a stellar explosion known as a thermonuclear (Type Ia) supernova and for this 3D problem, FLASH includes operators for hydrodynamics, gravity, and nuclear burning, in addition to routines for the material equation of state. We perform a strong-scaling study with a 220 GB problem size to explore both single- and multi-node performance. Our study explores the performance of different MPI mappings and the distribution of processors across nodes. From these tests, we determined the optimal configuration to balance runtime and energy consumption for our application., Comment: Accepted to PEARC '24 (Practice and Experience in Advanced Research Computing)

We study the cohomology of a general stable sheaf on an abelian surface. We say that a moduli space satisfies weak Brill-Noether if the general sheaf has at most one non-zero cohomology group. Let $(X,H)$ be a polarized abelian surface and let $\mathbf{v}=(r,\xi,a)$ be a Mukai vector on $X$ with $\mathbf{v}^2\ge 0$,$r>0$, and $\xi\cdot H>0$. We show that if $\rho(X)=1$ or $\rho(X)=2$ and $X$ contains an elliptic curve, then all the moduli spaces $M_{X,H}(\mathbf{v})$ satisfy weak Brill-Noether. Conversely, if $\rho(X)>2$ or $\rho(X)=2$ and $X$ does not contain an elliptic curve, we show that there are infinitely many moduli spaces $M_{X,H}(\mathbf{v})$ that fail weak Brill-Noether. As a consequence, we classify Chern classes of Ulrich bundles on abelian surfaces., Comment: 22 pages. Comments Welcome!

We present time-dependent density functional theory (TDDFT) calculations of fluorescence emission energies for 284 distinct graphene quantum dots (GQDs) of varying shapes (square, hexagonal, and amorphous) and sizes ($\sim$1-2 nm). These GQDs are doped with one or two elements from B, N, O, S, and P at dopant percentages of 1.5%, 3%, 5%, and 7%. Our study systematically investigates the trends and patterns in emission energies as a function of shape, size, dopant type and dopant percentage. Twelve structures are identified to have emission wavelengths in the visible spectrum. The emission energies derived from our calculations can guide the formulation of specific GQD mixtures to achieve desired emission spectra within and beyond the visible range for industrial applications. Furthermore, the extensive dataset, including emission energies along with molecular structures generated in this work, creates a DFT dateset for further machine learning studies., Comment: v2

Sustainability-focused design research is witnessing a change in approach with the emergence of More-than-human Design (MTHD), which challenges human-centered thinking by incorporating nonhuman perspectives into the design process. However, implementing MTHD presents challenges for design researchers and practitioners, such as understanding non-verbal species. Despite the techniques developed to facilitate such an understanding (e.g. contact zone), the growing literature on MTHD lacks studies reflecting on how these techniques are utilized in the design process. In this paper, we present a case study on designing olive harvesting tools from a MTH lens, where designers used contact zone, plant interviews, plant persona, and experience map to explore the perspectives of olive trees and incorporate them into ideas in collaboration with farmers and agricultural engineers. The results indicate the significance of reconsidering decentralization in MTHD from the standpoint of entanglements among techniques and incorporating various knowledge types to manage tensions arising from perspective shifts., Comment: 35 pages, 8 figures, 3 tables, The Design Journal

Design practice traditionally focused on human concerns, either overseeing the various effects of climate issues on nonhuman stakeholders or considering them as resources to address these problems. The climate crisis's urgency demands a design shift towards sustainability and inclusivity. This shift was happening through an emerging theme in design, More-Than-Human (MTH), which expands the notion of the user to animals, things, nature, and microbes. Such an expansion creates a requirement for designers to consider nonhuman perspectives during the design process. This paper investigates the methods used in MTH Design studies to explore and synthesize the perspectives of nonhuman users. Reviewing 30 papers, it highlights a predominant focus on animals and things over plants and microbes in MTH studies, along with a scarcity of synthesis methods. It identifies the necessity of tools that represent nonhumans with their relationships within larger ecosystems, and calls for increased attention to plants and microbes, emphasizing their vital role in sustainable environments and urging researchers to develop methods for understanding these species. By highlighting method strengths and weaknesses, it aims to guide designers and design researchers who plan to work with nonhuman users in selecting appropriate methods., Comment: 20 pages, 1 figure, DTRS14 - Design Thinking Research Symposium

The advent of the sixth generation (6G) wireless networks heralds a transformative era for mobile communication, where the integration of cutting-edge technologies like Reconfigurable Intelligent Surfaces (RISs) is paramount in addressing the burgeoning demands for energy efficiency, high data rates, reliable connectivity, and enhanced coverage in densely populated areas. RISs have attracted the attention of academia as well as the industry and emerged as a beacon of innovation, offering a novel paradigm to reconfigure the wireless propagation environment beneficially, thereby enhancing the overall network performance. It is envisioned that the deployment of numerous RISs in a mobile network will serve user equipments (UEs) to boost various key performance indicators (KPIs), with lower energy consumption compared to alternative solutions (e.g., relays, and network-controlled repeaters). However, the knowledge of whether or not a UE is being served through an RIS and if so, which RIS it is being served by is crucial and beneficial for various network planning and operational reasons. In this paper, we address the importance and the benefits of RIS identification in a mobile network. Additionally, we guide the readers and researchers by introducing alternative methods to enable RIS identification. Through the lens of this research, we unveil and shed light to various benefits, the challenges, and future opportunities in the identification of RISs serving mobile users in complex network environments, highlighting the necessity for advanced identification strategies to fully harness the potential of RIS technology in next-generation wireless systems., Comment: 7 pages, 5 figures, 13 references, magazine

A novel method to reconstruct the energy of hadronic showers in the CMS High Granularity Calorimeter (HGCAL) is presented. The HGCAL is a sampling calorimeter with very fine transverse and longitudinal granularity. The active media are silicon sensors and scintillator tiles readout by SiPMs and the absorbers are a combination of lead and Cu/CuW in the electromagnetic section, and steel in the hadronic section. The shower reconstruction method is based on graph neural networks and it makes use of a dynamic reduction network architecture. It is shown that the algorithm is able to capture and mitigate the main effects that normally hinder the reconstruction of hadronic showers using classical reconstruction methods, by compensating for fluctuations in the multiplicity, energy, and spatial distributions of the shower's constituents. The performance of the algorithm is evaluated using test beam data collected in 2018 prototype of the CMS HGCAL accompanied by a section of the CALICE AHCAL prototype. The capability of the method to mitigate the impact of energy leakage from the calorimeter is also demonstrated.

Distributed tracing has become a fundamental tool for diagnosing performance issues in the cloud by recording causally ordered, end-to-end workflows of request executions. However, tracing in production workloads can introduce significant overheads due to the extensive instrumentation needed for identifying performance variations. This paper addresses the trade-off between the cost of tracing and the utility of the "spans" within that trace through Astraea, an online probabilistic distributed tracing system. Astraea is based on our technique that combines online Bayesian learning and multi-armed bandit frameworks. This formulation enables Astraea to effectively steer tracing towards the useful instrumentation needed for accurate performance diagnosis. Astraea localizes performance variations using only 10-28% of available instrumentation, markedly reducing tracing overhead, storage, compute costs, and trace analysis time.