Search

Recently, there has been an increasing number of efforts to introduce models capable of generating natural language explanations (NLEs) for their predictions on vision-language (VL) tasks. Such models are appealing, because they can provide human-friendly and comprehensive explanations. However, there is a lack of comparison between existing methods, which is due to a lack of re-usable evaluation frameworks and a scarcity of datasets. In this work, we introduce e-ViL and e-SNLI-VE. e-ViL is a benchmark for explainable vision-language tasks that establishes a unified evaluation framework and provides the first comprehensive comparison of existing approaches that generate NLEs for VL tasks. It spans four models and three datasets and both automatic metrics and human evaluation are used to assess model-generated explanations. e-SNLI-VE is currently the largest existing VL dataset with NLEs (over 430k instances). We also propose a new model that combines UNITER, which learns joint embeddings of images and text, and GPT-2, a pre-trained language model that is well-suited for text generation. It surpasses the previous state of the art by a large margin across all datasets. Code and data are available here: https://github.com/maximek3/e-ViL., Comment: Accepted at ICCV 2021 (camera-ready version)

With the rapid growth of social tagging systems, many research efforts are being put intopersonalized search and recommendation using social tags (i.e., folksonomies). As users can freely choosetheir own vocabulary, social tags can be very ambiguous (for instance, due to the use of homonymsor synonyms). Machine learning techniques (such as clustering and deep neural networks) are usuallyapplied to overcome this tag ambiguity problem. However, the machine-learning-based solutions alwaysneed very powerful computing facilities to train recommendation models from a large amount of data,so they are inappropriate to be used in lightweight recommender systems. In this work, we propose anontological similarity to tackle the tag ambiguity problem without the need of model training by usingcontextual information. The novelty of this ontological similarity is that it first leverages external domainontologies to disambiguate tag information, and then semantically quantifies the relevance between userand item profiles according to the semantic similarity of the matching concepts of tags in the respectiveprofiles. Our experiments show that the proposed ontological similarity is semantically more accurate thanthe state-of-the-art similarity metrics, and can thus be applied to improve the performance of content-based tag-aware personalized recommendation on the Social Web. Consequently, as a model-training-freesolution, ontological similarity is a good disambiguation choice for lightweight recommender systems anda complement to machine-learning-based recommendation solutions. Fil: Xu, Zhenghua. University of Oxford; Reino Unido Fil: Tifrea-Marciuska, Oana. Bloomberg; Reino Unido Fil: Lukasiewicz, Thomas. University of Oxford; Reino Unido Fil: Martinez, Maria Vanina. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Bahía Blanca. Instituto de Ciencias e Ingeniería de la Computación. Universidad Nacional del Sur. Departamento de Ciencias e Ingeniería de la Computación. Instituto de Ciencias e Ingeniería de la Computación; Argentina Fil: Simari, Gerardo. Consejo Nacional de Investigaciones Científicas y Técnicas. Centro Científico Tecnológico Conicet - Bahía Blanca. Instituto de Ciencias e Ingeniería de la Computación. Universidad Nacional del Sur. Departamento de Ciencias e Ingeniería de la Computación. Instituto de Ciencias e Ingeniería de la Computación; Argentina Fil: Chen, Cheng. China Academy of Electronics and Information Technology; China