Search

Visual emotion analysis(VEA) aiming to detect the emotions behind images, has gained increasing attention with the development of online social media. Recent studies in prompt learning have significantly advanced visual emotion classification. However, these methods usually utilize random vectors or non-emotional texts as the initialization for prompt optimization. This restricts the emotional semantic representation of prompts and hinders the performance of the model. To tackle this problem, we leverage emotional prompts with multiple views to enhance the semantic emotional information. We first translate the image to caption as context prompt(COP) from the view of background information for the image. Additionally, we introduce hybrid emotion prompt(HEP) from the view of the interaction between the emotional visual and textual information, where different modalities are integrated with a novel Emotion Joint Congruity Learning module. Furthermore, we also provide label prompt(LP) to enhance the emotional association with labels, enabling better emotional information fusion. Extensive experiments conducted on five publicly visual emotion classification datasets, i.e. EmoSet, FI, have demonstrated the superiority of our MVP model over cutting-edge methods. • This work reveals a novel standpoint for visual emotion analysis via observing the semantic space changes and achieves an effective and stable emotional semantic learning process. • We design the Multi-Views Prompt Learning method (MVP) to capture the emotional cues, achieving a new state-of-the-art (SOTA) performance. • Our work provides an in-depth analysis and insights of the multi-views method for further exploration in semantic optimization. • This work validates the value of introducing multimodal approaches to visual emotion analysis. The validity of this method also provides the possibility to extend it to other visual understanding tasks. [ABSTRACT FROM AUTHOR]

Visual emotion analysis (VEA), a crucial task that identifies the emotional content conveyed in an image, has garnered increasing attention due to the rapid growth of image posts in modern social media. Prompted by the remarkable generalization ability of prompt learning, recent studies have endeavoured to develop adaptive contextual prompts to adjust the representation spaces in visual emotion analysis dynamically. However, the majority of these studies have focused on non-oriented prompting strategies, overlooking the sentiment information associated with emotion labels. In this paper, we propose a novel approach, Multi-Perspective Prompt Learning (MPP-CLIP), within the context of CLIP, for visual emotion analysis. Our approach not only involves non-oriented prompting using learnable contexts but also leverages emotion-oriented prompting based on emotion labels. Thus, our design facilitates a strong coupling between the vision-emotion features, ensuring mutual synergy and discouraging the adoption of solely non-oriented solutions. In addition, we introduce Residual-Enhanced Adapter, which employs residual-style feature mixing to achieve efficient transfer learning from pre-trained CLIP to visual emotion analysis. Extensive experimental evaluations demonstrate our approach outperforms previous state-of-the-art (SOTA) methods while maintaining significantly lower computational and memory costs. [ABSTRACT FROM AUTHOR]