Search

The Diffusion Transformer plays a pivotal role in advancing text-to-image and text-to-video generation, owing primarily to its inherent scalability. However, existing controlled diffusion transformer methods incur significant parameter and computational overheads and suffer from inefficient resource allocation due to their failure to account for the varying relevance of control information across different transformer layers. To address this, we propose the Relevance-Guided Efficient Controllable Generation framework, RelaCtrl, enabling efficient and resource-optimized integration of control signals into the Diffusion Transformer. First, we evaluate the relevance of each layer in the Diffusion Transformer to the control information by assessing the "ControlNet Relevance Score"-i.e., the impact of skipping each control layer on both the quality of generation and the control effectiveness during inference. Based on the strength of the relevance, we then tailor the positioning, parameter scale, and modeling capacity of the control layers to reduce unnecessary parameters and redundant computations. Additionally, to further improve efficiency, we replace the self-attention and FFN in the commonly used copy block with the carefully designed Two-Dimensional Shuffle Mixer (TDSM), enabling efficient implementation of both the token mixer and channel mixer. Both qualitative and quantitative experimental results demonstrate that our approach achieves superior performance with only 15% of the parameters and computational complexity compared to PixArt-delta., Comment: Homepage: https://360cvgroup.github.io/RelaCtrl/ Github: https://github.com/360CVGroup/RelaCtrl

The bottleneck in enhanced sampling lies in finding collective variables (CVs) that can effectively accelerate protein conformational changes. True reaction coordinates (tRCs) that can predict the committor are considered the optimal CVs, but identifying them requires unbiased natural reactive trajectories, which, paradoxically, depend on effective enhanced sampling. Using the generalized work functional method, we found that tRCs control both conformational changes and energy relaxation, enabling us to compute tRCs from energy relaxation simulations. Applying bias to tRCs accelerated conformational changes and ligand dissociation in HIV-1 protease and the PDZ2 domain by 10^5 to 10^15-fold. The resulting trajectories follow natural transition pathways, enabling efficient generation of natural reactive trajectories. In contrast, biased trajectories from empirical CVs often display non-physical features. Furthermore, by computing tRCs from a single protein structure, our method enables predictive sampling of conformational changes. These findings significantly broaden the range of protein functional processes accessible to molecular dynamics simulations.

The task of layout-to-image generation involves synthesizing images based on the captions of objects and their spatial positions. Existing methods still struggle in complex layout generation, where common bad cases include object missing, inconsistent lighting, conflicting view angles, etc. To effectively address these issues, we propose a \textbf{Hi}erarchical \textbf{Co}ntrollable (HiCo) diffusion model for layout-to-image generation, featuring object seperable conditioning branch structure. Our key insight is to achieve spatial disentanglement through hierarchical modeling of layouts. We use a multi branch structure to represent hierarchy and aggregate them in fusion module. To evaluate the performance of multi-objective controllable layout generation in natural scenes, we introduce the HiCo-7K benchmark, derived from the GRIT-20M dataset and manually cleaned. https://github.com/360CVGroup/HiCo_T2I., Comment: NeurIPS2024

The global self-attention mechanism in diffusion transformers involves redundant computation due to the sparse and redundant nature of visual information, and the attention map of tokens within a spatial window shows significant similarity. To address this redundancy, we propose the Proxy-Tokenized Diffusion Transformer (PT-DiT), which employs sparse representative token attention (where the number of representative tokens is much smaller than the total number of tokens) to model global visual information efficiently. Specifically, within each transformer block, we compute an averaging token from each spatial-temporal window to serve as a proxy token for that region. The global semantics are captured through the self-attention of these proxy tokens and then injected into all latent tokens via cross-attention. Simultaneously, we introduce window and shift window attention to address the limitations in detail modeling caused by the sparse attention mechanism. Building on the well-designed PT-DiT, we further develop the Qihoo-T2X family, which includes a variety of models for T2I, T2V, and T2MV tasks. Experimental results show that PT-DiT achieves competitive performance while reducing the computational complexity in both image and video generation tasks (e.g., a 49% reduction compared to DiT and a 34% reduction compared to PixArt-$\alpha$). The visual exhibition and source code of Qihoo-T2X is available at https://360cvgroup.github.io/Qihoo-T2X/.

Reaction coordinates (RCs) are the few essential coordinates of a protein that control its functional processes, such as allostery, enzymatic reaction, and conformational change. They are critical for understanding protein function and provide optimal enhanced sampling of protein conformational changes and states. Since the pioneering works in the late 1990s, identifying the correct and objectively provable RCs has been a central topic in molecular biophysics and chemical physics. This review summarizes the major advances in identifying RCs over the past 25 years, focusing on methods aimed at finding RCs that meet the rigorous committor criterion, widely accepted as the true RCs. Importantly, the newly developed physics-based energy flow theory and generalized work functional method provide a general and rigorous approach for identifying true RCs, revealing their physical nature as the optimal channels of energy flow in biomolecules.

Synthesizing motion-rich and temporally consistent videos remains a challenge in artificial intelligence, especially when dealing with extended durations. Existing text-to-video (T2V) models commonly employ spatial cross-attention for text control, equivalently guiding different frame generations without frame-specific textual guidance. Thus, the model's capacity to comprehend the temporal logic conveyed in prompts and generate videos with coherent motion is restricted. To tackle this limitation, we introduce FancyVideo, an innovative video generator that improves the existing text-control mechanism with the well-designed Cross-frame Textual Guidance Module (CTGM). Specifically, CTGM incorporates the Temporal Information Injector (TII), Temporal Affinity Refiner (TAR), and Temporal Feature Booster (TFB) at the beginning, middle, and end of cross-attention, respectively, to achieve frame-specific textual guidance. Firstly, TII injects frame-specific information from latent features into text conditions, thereby obtaining cross-frame textual conditions. Then, TAR refines the correlation matrix between cross-frame textual conditions and latent features along the time dimension. Lastly, TFB boosts the temporal consistency of latent features. Extensive experiments comprising both quantitative and qualitative evaluations demonstrate the effectiveness of FancyVideo. Our video demo, code and model are available at https://360cvgroup.github.io/FancyVideo/.

Large Language Models (LLMs) have showcased exceptional ability in causal reasoning from textual information. However, will these causalities remain straightforward for Vision Large Language Models (VLLMs) when only visual hints are provided? Motivated by this, we propose a novel Multimodal Causal Reasoning benchmark, namely MuCR, to challenge VLLMs to infer semantic cause-and-effect relationship when solely relying on visual cues such as action, appearance, clothing, and environment. Specifically, we introduce a prompt-driven image synthesis approach to create siamese images with embedded semantic causality and visual cues, which can effectively evaluate VLLMs' causal reasoning capabilities. Additionally, we develop tailored metrics from multiple perspectives, including image-level match, phrase-level understanding, and sentence-level explanation, to comprehensively assess VLLMs' comprehension abilities. Our extensive experiments reveal that the current state-of-the-art VLLMs are not as skilled at multimodal causal reasoning as we might have hoped. Furthermore, we perform a comprehensive analysis to understand these models' shortcomings from different views and suggest directions for future research. We hope MuCR can serve as a valuable resource and foundational benchmark in multimodal causal reasoning research. The project is available at: https://github.com/Zhiyuan-Li-John/MuCR, Comment: 20 pages 19 figures

The current state-of-the art techniques for image segmentation are often based on U-Net architectures, a U-shaped encoder-decoder networks with skip connections. Despite the powerful performance, the architecture often does not perform well when used on data which has different characteristics than the data it was trained on. Many techniques for improving performance in the presence of domain shift have been developed, however typically only have loose connections to the theory of domain adaption. In this work, we propose an unsupervised domain adaptation framework for U-Nets with theoretical guarantees based on the Margin Disparity Discrepancy [1] called the MDD-UNet. We evaluate the proposed technique on the task of hippocampus segmentation, and find that the MDD-UNet is able to learn features which are domain-invariant with no knowledge about the labels in the target domain. The MDD-UNet improves performance over the standard U-Net on 11 out of 12 combinations of datasets. This work serves as a proof of concept by demonstrating an improvement on the U-Net in it's standard form without modern enhancements, which opens up a new avenue of studying domain adaptation for models with very large hypothesis spaces from both methodological and practical perspectives. Code is available at https://github.com/asbjrnmunk/mdd-unet., Comment: Published at NLDL 2024

Parameter-efficient tuning has become a trend in transferring large-scale foundation models to downstream applications. Existing methods typically embed some light-weight tuners into the backbone, where both the design and the learning of the tuners are highly dependent on the base model. This work offers a new tuning paradigm, dubbed Res-Tuning, which intentionally unbinds tuners from the backbone. With both theoretical and empirical evidence, we show that popular tuning approaches have their equivalent counterparts under our unbinding formulation, and hence can be integrated into our framework effortlessly. Thanks to the structural disentanglement, we manage to free the design of tuners from the network architecture, facilitating flexible combination of various tuning strategies. We further propose a memory-efficient variant of Res-Tuning, where the bypass i.e., formed by a sequence of tuners) is effectively detached from the main branch, such that the gradients are back-propagated only to the tuners but not to the backbone. Such a detachment also allows one-time backbone forward for multi-task inference. Extensive experiments on both discriminative and generative tasks demonstrate the superiority of our method over existing alternatives from the perspectives of efficacy and efficiency. Project page: $\href{https://res-tuning.github.io/}{\textit{https://res-tuning.github.io/}}$., Comment: Accepted to NeurIPS 2023

Dynamics of reaction coordinates during barrier-crossing are key to understand activated processes in complex systems such as proteins. The default assumption from Kramers physical intuition is that of a diffusion process. However, the dynamics of barrier-crossing in natural complex molecules are largely unexplored. Here we investigate the transition dynamics of alanine-dipeptide isomerization, the simplest complex system with a large number of non-reaction coordinates that can serve as an adequate thermal bath feeding energy into the reaction coordinates. We separate conformations along the time axis and construct the dynamic probability surface of reaction. We quantify its topological structure and rotational flux using persistent homology and differential form. Our results uncovered a region with strong reactive vortex in the configuration-time space, where the highest probability peak and the transition state ensemble are located. This reactive region contains strong rotational fluxes: Most reactive trajectories swirl multiple times around this region in the subspace of the two most-important reaction coordinates. Furthermore, the rotational fluxes result from cooperative movement along the isocommitter surfaces and orthogonal barrier-crossing. Overall, our findings offer a first glimpse into the reactive vortex regions that characterize the non-diffusive dynamics of barrier-crossing of a naturally occurring activation process.

To gain insight into reaction mechanism of activated processes, we introduce an exact approach for quantifying the topology of high-dimensional probability surfaces of the underlying dynamic processes. Instead of Morse indexes, we study the homology groups of a sequence of superlevel sets of the probability surface over high-dimensional configuration spaces using persistent homology. For alanine-dipeptide isomerization, a prototype of activated processes, we identify locations of probability peaks and connecting-ridges, along with measures of their global prominence. Instead of a saddle-point, the transition state ensemble (TSE) of conformations are at the most prominent probability peak after reactants/products, when proper reaction coordinates are included. Intuition-based models, even those exhibiting a double-well, fail to capture the dynamics of the activated process. Peak occurrence, prominence, and locations can be distorted upon subspace projection. While principal component analysis account for conformational variance, it inflates the complexity of the surface topology and destroy dynamic properties of the topological features. In contrast, TSE emerges naturally as the most prominent peak beyond the reactant/product basins, when projected to a subspace of minimum dimension containing the reaction coordinates. Our approach is general and can be applied to investigate the topology of high-dimensional probability surfaces of other activated process., Comment: 45 pages, 13 figures

An activated process consists of energy activation and barrier crossing; the former is a prerequisite for the latter. Barrier crossing has been studied extensively, but energy activation has been overlooked due to a lack of means to gauge its progress. We define reaction stability as the probability that reactive trajectories pass a vicinity in phase space; it enabled us to analyze energy activation of a biomolecular isomerization. This process follows a mechanism fundamentally different from presumed mechanisms in standard reaction rate theories: it features accumulation of high kinetic energy in reaction coordinates, achieved by precise synergy between them coordinated by momentum space.

Protein conformational changes are activated processes essential for protein functions. Activation in a protein differs from activation in a small molecule in that it involves directed and systematic energy flows through preferred channels encoded in the protein structure. Understanding the nature of these energy flow channels and how energy flows through them during activation is critical for understanding protein conformational changes. We recently developed a rigorous statistical mechanical framework for understanding potential energy flows. Here we complete this theoretical framework with a rigorous theory for kinetic energy flows: potential and kinetic energy inter-convert when impressed forces oppose inertial forces whereas kinetic energy transfers directly from one coordinate to another when inertial forces oppose each other. This theory is applied to analyzing a prototypic system for biomolecular conformational dynamics: the isomerization of an alanine dipeptide. Among the two essential energy flow channels for this process, dihedral phi confronts the activation barrier, whereas dihedral theta receives energy from potential energy flows. Intriguingly, theta helps phi to cross the activation barrier by transferring to phi via direct kinetic energy flow all the energy it received: increase in theta caused by potential energy flow converts into increase in phi. As a compensation, theta receives kinetic energy from bond angle alpha via direct mechanism and bond angle beta via indirect mechanism.

In the study of rare events in complex systems with many degrees of freedom, a key element is to identify the reaction coordinates of a given process. Over recent years, a number of methods and protocols have been developed to extract the reaction coordinates based on limited information from molecular dynamics simulations. In this review, we provide a brief survey over a number of major methods developed in the past decade, some of which are discussed in greater detail, to provide an overview of the problems that are partially solved and challenges that still remain. A particular emphasis has been placed on methods for identifying reaction coordinates that are related to the committor., Comment: Review on a specific topic (15 pages)