Back to Search Start Over

MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing

Authors :
Cao, Mingdeng
Wang, Xintao
Qi, Zhongang
Shan, Ying
Qie, Xiaohu
Zheng, Yinqiang
Cao, Mingdeng
Wang, Xintao
Qi, Zhongang
Shan, Ying
Qie, Xiaohu
Zheng, Yinqiang
Publication Year :
2023

Abstract

Despite the success in large-scale text-to-image generation and text-conditioned image editing, existing methods still struggle to produce consistent generation and editing results. For example, generation approaches usually fail to synthesize multiple images of the same objects/characters but with different views or poses. Meanwhile, existing editing methods either fail to achieve effective complex non-rigid editing while maintaining the overall textures and identity, or require time-consuming fine-tuning to capture the image-specific appearance. In this paper, we develop MasaCtrl, a tuning-free method to achieve consistent image generation and complex non-rigid image editing simultaneously. Specifically, MasaCtrl converts existing self-attention in diffusion models into mutual self-attention, so that it can query correlated local contents and textures from source images for consistency. To further alleviate the query confusion between foreground and background, we propose a mask-guided mutual self-attention strategy, where the mask can be easily extracted from the cross-attention maps. Extensive experiments show that the proposed MasaCtrl can produce impressive results in both consistent image generation and complex non-rigid real image editing.<br />Comment: Project available at https://ljzycmd.github.io/projects/MasaCtrl

Details

Database :
OAIster
Publication Type :
Electronic Resource
Accession number :
edsoai.on1381618172
Document Type :
Electronic Resource