Search

Referring expression grounding is an important and challenging task in computer vision. To avoid the laborious annotation in conventional referring grounding, unpaired referring grounding is introduced, where the training data only contains a number of images and queries without correspondences. The few existing solutions to unpaired referring grounding are still preliminary, due to the challenges of learning vision-language correlation and lack of the top-down guidance with unpaired data. Existing works are only able to learn vision-language correlation by modality conversion, where critical information are lost. They also heavily rely on pre-extracted object proposals and thus cannot generate correct predictions with defective proposals. In this paper, we propose a novel bidirectional cross-modal matching (BiCM) framework to address these challenges. Particularly, we design a query-aware attention map (QAM) module that introduces top-down perspective via generating query-specific visual attention maps to avoid the over-reliance on pre-extracted object proposals. A cross-modal object matching (COM) module is further introduced to predict the target objects from a bottom-up perspective. This module exploits the recently emerged image-text matching pretrained model, CLIP, to learn cross-modal correlation without modality conversion. The top-down and bottom-up predictions are then integrated via a similarity fusion (SF) module. We also propose a knowledge adaptation matching (KAM) module that leverages unpaired training data to adapt pretrained knowledge to the target dataset and task. Experiments show that our framework significantly outperforms previous works on five grounding datasets. [ABSTRACT FROM AUTHOR]

Open-vocabulary (OV) semantic segmentation has attracted increasing attention in recent years, which aims to recognize objects in an open class set for real-world applications. While prior OV semantic segmentation approaches have relied on additional semantic knowledge derived from vision-language (VL) pre-training, such as the popular CLIP model, this paper introduces a novel paradigm by harnessing the unprecedented capabilities of large language models (LLMs). Inspired by recent breakthroughs in LLMs that provide a richer knowledge base compared to traditional vision-language pre-training, our proposed methodology capitalizes on the vast knowledge embedded within LLMs for OV semantic segmentation. Particularly, we partition LLM knowledge into object, attribute, and relation priors, and propose three novel attention modules-semantic, scaled visual, and relation attentions, to utilize the LLM priors. Extensive experiments are conducted on common benchmarks including ADE20K (847 classes) and Pascal Context (459 classes). The results show that our model outperforms previous state-of-the-art (SoTA) methods by up to 7.2% absolute. Moreover, unlike previous VL-pre-training-based works, our method can even predict OV segmentation results without target candidate classes. [ABSTRACT FROM AUTHOR]

The generation of object proposals plays an important role in object detection. Most existing methods produce object proposals by using bottom-up cues, such as closed contour or superpixel. In this paper, we propose a novel method to improve the ranking of object proposals by combining bottom-up cues with top-down information of objectivity. Firstly, we utilize the bottom-up method to generate initial object proposals of the given test image. Then we retrieve its top-k similar images from training images set. Considering both appearance and spatial similarity between initial object proposals and the ground truth bounding boxes of these top-k similar images, we obtain the top-down guided scores of initial object proposals. Finally, the refined score of each initial object proposal is modeled as a fusion of the bottom-up score and the top-down score. Experiments show that our method achieves better performance compared with the state-of-art on the Pascal VOC2007 dataset. [ABSTRACT FROM AUTHOR]

Person re-identification is an important topic in video surveillance. We present a new feature representation for person re-identification based on multi-region-set ensembles (MRSE) by combining some semantic regions with their contextual information. The motivation of this paper is that people can recognize and identify whether it’s the same person by one or several local regions of the appearance. Our approach is divided into three steps: firstly, we segment the person into some semantic regions such as “hair”, “face”, “up-cloth” (upper clothes) and “lo-cloth” (lower clothes). After getting these regions, we form multiple sets of different combination by selecting a few regions and concatenating the features of them. We then combine the distance of all multiple sets for computing the similarity of a query-gallery image pair. Finally, we achieve higher rank-1 recognition rate and competitive performance compared with the state-of-the-art on iLIDS, VIPeR, CAVIAR4REID, 3DPeS four challenging datasets. [ABSTRACT FROM AUTHOR]

Object detection is a significant and challenging problem in the study area of remote sensing and image analysis. However, most existing methods are easy to miss or incorrectly locate objects due to the various sizes and aspect ratios of objects. In this paper, we propose a novel end-to-end Adaptively Aspect Ratio Multi-Scale Network (A 2 RMNet) to solve this problem. On the one hand, we design a multi-scale feature gate fusion network to adaptively integrate the multi-scale features of objects. This network is composed of gate fusion modules, refine blocks and region proposal networks. On the other hand, an aspect ratio attention network is leveraged to preserve the aspect ratios of objects, which alleviates the excessive shape distortions of objects caused by aspect ratio changes during training. Experiments show that the proposed A 2 RMNet significantly outperforms the previous state of the arts on the DOTA dataset, NWPU VHR-10 dataset, RSOD dataset and UCAS-AOD dataset by 5.73 % , 7.06 % , 3.27 % and 2.24 % , respectively. [ABSTRACT FROM AUTHOR]