Search

In this paper, we address the problem of lip-voice synchronisation in videos containing human face and voice. Our approach is based on determining if the lips motion and the voice in a video are synchronised or not, depending on their audio-visual correspondence score. We propose an audio-visual cross-modal transformer-based model that outperforms several baseline models in the audio-visual synchronisation task on the standard lip-reading speech benchmark dataset LRS2. While the existing methods focus mainly on lip synchronisation in speech videos, we also consider the special case of the singing voice. The singing voice is a more challenging use case for synchronisation due to sustained vowel sounds. We also investigate the relevance of lip synchronisation models trained on speech datasets in the context of singing voice. Finally, we use the frozen visual features learned by our lip synchronisation model in the singing voice separation task to outperform a baseline audio-visual model which was trained end-to-end. The demos, source code, and the pre-trained models are available on https://ipcv.github.io/VocaLiST/, Comment: Paper accepted to Interspeech 2022; Project Page: https://ipcv.github.io/VocaLiST/

Dogs can recognize conspecifics in cross-modal audio–video presentations. In this paper, we aimed at exploring if such capability extends to the recognition of cats, and whether it is influenced by exposure to these animals. To reach our aim, we enrolled 64 pet dogs. Half of the dogs were currently living with cats, while the rest had never been living with cats, nor were at the time of the experiment. All dogs underwent a cross-modal violation of expectancy experiment, where they were presented with either a cat or a dog vocalization, followed by a video of either species on a blank background. The result revealed that dogs did not exhibit a surprise reaction towards the incoherent stimuli of a cat vocalization and a dog video or vice-versa, implying that they had not recognized the stimuli portraying cats. The pattern of results did not differ between dogs living or not with cats, implying that exposure to a limited number of cats, however, prolonged, is not sufficient to grant dogs with the ability to recognize them on audio–video presentations. We propose that the lack of recognition could be due to the small number of individual cats the dogs are regularly exposed to, or to the possible lack of early exposure to cats during the socialization phase.

Cross-Modal Hashing (CMH) retrieval methods have garnered increasing attention within the information retrieval research community due to their capability to deal with large amounts of data thanks to the computational efficiency of hash-based methods. To date, the focus of cross-modal hashing methods has been on training with paired data. Paired data refers to samples with one-to-one correspondence across modalities, e.g., image and text pairs where the text sample describes the image. However, real-world applications produce unpaired data that cannot be utilised by most current CMH methods during the training process. Models that can learn from unpaired data are crucial for real-world applications such as cross-modal neural information retrieval where paired data is limited or not available to train the model. This paper provides (1) an overview of the CMH methods when applied to unpaired datasets, (2) proposes a framework that enables pairwise-constrained CMH methods to train with unpaired samples, and (3) evaluates the performance of state-of-the-art CMH methods across different pairing scenarios.