51. Animal Pose Estimation Based on Contrastive Learning with Dynamic Conditional Prompts.
- Author
-
Hu, Xiaoling and Liu, Chang
- Subjects
- *
ANIMAL communication , *IDENTIFICATION of animals , *LEARNING , *TRAINING needs , *PRIOR learning , *STOCHASTIC learning models - Abstract
Simple Summary: Detecting animal key points and poses is crucial for recognizing behaviors and protecting species. Traditional methods based on images face challenges like limited training data and the need for extensive manual annotations. To address this, we utilized a language–image contrastive learning model that learns the relationship between text and images, developing a new approach for animal pose estimation that combines textual descriptions and visual data. Our experiments on the AP10K dataset, a benchmark for animal pose estimation with over 10,000 images of 23 species, showed this method to be more accurate than image-based techniques. Traditional animal pose estimation techniques based on images face significant hurdles, including scarce training data, costly data annotation, and challenges posed by non-rigid deformation. Addressing these issues, we proposed dynamic conditional prompts for the prior knowledge of animal poses in language modalities. Then, we utilized a multimodal (language–image) collaborative training and contrastive learning model to estimate animal poses. Our method leverages text prompt templates and image feature conditional tokens to construct dynamic conditional prompts that integrate rich linguistic prior knowledge in depth. The text prompts highlight key points and relevant descriptions of animal poses, enhancing their representation in the learning process. Meanwhile, transformed via a fully connected non-linear network, image feature conditional tokens efficiently embed the image features into these prompts. The resultant context vector, derived from the fusion of the text prompt template and the image feature conditional token, generates a dynamic conditional prompt for each input sample. By utilizing a contrastive language–image pre-training model, our approach effectively synchronizes and strengthens the training interactions between image and text features, resulting in an improvement to the precision of key-point localization and overall animal pose estimation accuracy. The experimental results show that language–image contrastive learning based on dynamic conditional prompts enhances the average accuracy of animal pose estimation on the AP-10K and Animal Pose datasets. [ABSTRACT FROM AUTHOR]
- Published
- 2024
- Full Text
- View/download PDF