Back to Search
Start Over
Cross-Modal Attention Preservation with Self-Contrastive Learning for Composed Query-Based Image Retrieval.
- Source :
- ACM Transactions on Multimedia Computing, Communications & Applications; Jun2024, Vol. 20 Issue 6, p1-22, 22p
- Publication Year :
- 2024
-
Abstract
- In this article, we study the challenging cross-modal image retrieval task, Composed Query-Based Image Retrieval (CQBIR), in which the query is not a single text query but a composed query, i.e., a reference image, and a modification text. Compared with the conventional cross-modal image-text retrieval task, the CQBIR is more challenging as it requires properly preserving and modifying the specific image region according to the multi-level semantic information learned from the multi-modal query. Most recent works focus on extracting preserved and modified information and compositing it into a unified representation. However, we observe that the preserved regions learned by the existing methods contain redundant modified information, inevitably degrading the overall retrieval performance. To this end, we propose a novel method termed Cross-Modal Attention Preservation (CMAP). Specifically, we first leverage the cross-level interaction to fully account for multi-granular semantic information, which aims to supplement the high-level semantics for effective image retrieval. Furthermore, different from conventional contrastive learning, our method introduces self-contrastive learning into learning preserved information, to prevent the model from confusing the attention for the preserved part with the modified part. Extensive experiments on three widely used CQBIR datasets, i.e., FashionIQ, Shoes, and Fashion200k, demonstrate that our proposed CMAP method significantly outperforms the current state-of-the-art methods on all the datasets. The anonymous implementation code of our CMAP method is available at https://github.com/CFM-MSG/Code_CMAP. [ABSTRACT FROM AUTHOR]
- Subjects :
- IMAGE retrieval
IMAGE databases
SEMANTICS
Subjects
Details
- Language :
- English
- ISSN :
- 15516857
- Volume :
- 20
- Issue :
- 6
- Database :
- Complementary Index
- Journal :
- ACM Transactions on Multimedia Computing, Communications & Applications
- Publication Type :
- Academic Journal
- Accession number :
- 176301548
- Full Text :
- https://doi.org/10.1145/3639469