Back to Search Start Over

Wav2CLIP: Learning Robust Audio Representations From CLIP

Wav2CLIP: Learning Robust Audio Representations From CLIP

Authors :
Ho-Hsiang Wu
Prem Seetharaman
Kundan Kumar
Juan Pablo Bello
Publication Year :
2021
Publisher :
arXiv, 2021.

Abstract

We propose Wav2CLIP, a robust audio representation learning method by distilling from Contrastive Language-Image Pre-training (CLIP). We systematically evaluate Wav2CLIP on a variety of audio tasks including classification, retrieval, and generation, and show that Wav2CLIP can outperform several publicly available pre-trained audio representation algorithms. Wav2CLIP projects audio into a shared embedding space with images and text, which enables multimodal applications such as zero-shot classification, and cross-modal retrieval. Furthermore, Wav2CLIP needs just ~10% of the data to achieve competitive performance on downstream tasks compared with fully supervised models, and is more efficient to pre-train than competing methods as it does not require learning a visual model in concert with an auditory model. Finally, we demonstrate image generation from Wav2CLIP as qualitative assessment of the shared embedding space. Our code and model weights are open sourced and made available for further applications.<br />Comment: Copyright 2022 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

Details

Database :
OpenAIRE
Accession number :
edsair.doi.dedup.....2348ef5479bb8a6275a489971cfef6c4
Full Text :
https://doi.org/10.48550/arxiv.2110.11499