Search

Visual Question Answering (VQA) aims to answer questions according to the given image. However, current VQA models tend to rely solely on textual information from the questions and ignore the visual information in the images to get answers, which is caused by bias that is generated during the training phase. Previous studies have shown that bias in VQA is mainly caused by the text modality, and our analysis suggests that question type is a crucial factor in bias formation. To address this bias, we proposed a self-supervised method including the Against Biased Samples(ABS) module that performs targeted debiasing by selecting samples that are prone to bias, and the Shuffle Question types(SQT) module that constructs negative samples by randomly replacing the question types of the samples selected by the ABS, to interrupting the shortcuts from question type to answer. Our approach mitigates the question-to-answer bias without using external annotations, overcoming the prior language problem. Additionally, we designed a new objective function for negative samples. Experimental results indicate that our method outperforms both self-supervised-based and supervised-based state-of-the-art approaches, achieving 70.36% accuracy on the VQA-CP v2 dataset. • We propose a framework use image–question pairs to construct counterfactual samples. • SQT and ABS can target data that is prone to bias in the training set to de-bias. • We introduce a novel loss function base on the constructed negative samples. • Our method can achieve state-of-the-art performance on benchmark VQA-CP v2 dataset. [ABSTRACT FROM AUTHOR]

As a popular cross-modal reasoning task, Visual Question Answering (VQA) has achieved great progress in recent years. However, the issue of language bias has always affected the reliability of VQA models. To address this problem, counterfactual learning methods are proposed to learn more robust features to mitigate the bias problem. However, current counterfactual learning approaches mainly focus on generating synthesized samples and assigning answers to them, neglecting the relationship between factual and original data, which hinders robust feature learning for effective reasoning. To overcome this limitation, we propose a Self-supervised Knowledge Distillation approach in Counterfactual Learning for VQA, dubbed as VQA-SkdCL, which utilizes a self-supervised constraint to make good use of the hidden knowledge in the factual samples, enhancing the robustness of VQA models. We demonstrate the effectiveness of the proposed approach on VQA v2, VQA-CP v1, and VQA-CP v2 datasets and our approach achieves excellent performance. • We study the relations of factual and original samples in counterfactual learning. • We design a self-supervised knowledge distillation module to learn robust features. • Our method is model-agnostic, end-to-end trainable, and easy to implement. • Extensive experiments on three datasets show the effectiveness of our approach. [ABSTRACT FROM AUTHOR]

• A new VQA for Big-Five-Factors personality traits identification. • A deep model for integrating text and person/face recognition-based features. • A dynamic Text-Object graph and a clip Transformer encoder have been explored. • A New dataset has been created, which will be released to the public for research. Visual Question Answering (VQA) for personality trait images on social media is challenging because of multiple emotions and actions with complex backgrounds in social media images. This work aims at developing a new VQA model for different personality traits (VQAPT) identification in a single image. This work considers the Big Five Factors (BFF) for personality traits namely, Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism. VQA is proposed based on the observation that multiple personality traits can be seen in a single image. We propose a model integrating text recognition and person/face recognition to derive the unique relationship between the text and the person's action in the image. Furthermore, a dynamic text-object graph for personality traits identification is constructed according to the query. For understanding a query, we explore the Contrastive Language-Image Pre-trained (CLIP) transformer encoder in this work. Since it is the first work of its kind, we have created a new dataset under this work for evaluation and the dataset is available publicly as mentioned in Section 4. The effectiveness of the proposed method is also evaluated on two benchmark datasets, namely TextVQA for VQA and PTI for personality traits identification. [Display omitted] [ABSTRACT FROM AUTHOR]

• A novel transformer network is proposed to predict answers and coordinates simultaneously by two decoupled decoders. • A set of instance query embeddings and a sampling strategy are introduced to precisely locate the object. • By dividing images into bins, the coordinate regression is transformed into a simpler classification prediction. • The model parameters and computational costs can be dramatically reduced. Visual Question Answering (VQA) have made stunning advances by exploiting Transformer architecture and large-scale visual-linguistic pretraining. State-of-the-art methods generally require large amounts of data and devices to predict textualized answers and fail to provide visualized evidence of the answers. To mitigate these limitations, we propose a novel dual-decoder Transformer network (DDTN) for predicting the language answer and corresponding vision instance. Specifically, the linguistic features are first embedded by Long Short-Term Memory (LSTM) block and Transformer encoder, which are shared between the Transformer dual-decoder. Then, we introduce object detector to obtain vision region features and grid features for reducing the size and cost of DDTN. These visual features are combined with the linguistic features and are respectively fed into two decoders. Moreover, we design an instance query to guide the fused visual-linguistic features for outputting the instance mask or bounding box. The classification layers aggregate results from decoders and predict answer as well as corresponding instance coordinates at last. Without bells and whistles, DDTN achieves state-of-the-art performance and even competitive to pretraining models on VizWizGround and GQA dataset. The code is available at https://github.com/zlj63501/DDTN. [ABSTRACT FROM AUTHOR]

• The paper presents concrete applications of Visual Question Answering • Domains where VQA has been experimented are presented together with the exploited dataset • The paper suggests some challenging techniques that can be especially suited for specific domains • Some final considerations sketch future work in domain-related VQA Visual Question Answering (VQA) is an extremely stimulating and challenging research area where Computer Vision (CV) and Natural Language Processig (NLP) have recently met. In image captioning and video summarization, the semantic information is completely contained in still images or video dynamics, and it has only to be mined and expressed in a human-consistent way. Differently from this, in VQA semantic information in the same media must be compared with the semantics implied by a question expressed in natural language, doubling the artificial intelligence-related effort. Some recent surveys about VQA approaches have focused on methods underlying either the image-related processing or the verbal-related one, or on the way to consistently fuse the conveyed information. Possible applications are only suggested, and, in fact, most cited works rely on general-purpose datasets that are used to assess the building blocks of a VQA system. This paper rather considers the proposals that focus on real-world applications, possibly using as benchmarks suitable data bound to the application domain. The paper also reports about some recent challenges in VQA research. [ABSTRACT FROM AUTHOR]

• New model for scene text VQA that jointly reasons about textual and visual modalities. • Attending on multi-modal features is better than attending separately to each modality. • Grid features from an object detection backbone proves to work well in this task. • The proposed model is up to 5 times faster than current state of the art models. • Combined with a standard VQA model reaches state of the art on mixed datasets. This paper presents a new model for the task of scene text visual question answering. In this task questions about a given image can only be answered by reading and understanding scene text. Current state of the art models for this task make use of a dual attention mechanism in which one attention module attends to visual features while the other attends to textual features. A possible issue with this is that it makes difficult for the model to reason jointly about both modalities. To fix this problem we propose a new model that is based on an single attention mechanism that attends to multi-modal features conditioned to the question. The output weights of this attention module over a grid of multi-modal spatial features are interpreted as the probability that a certain spatial location of the image contains the answer text to the given question. Our experiments demonstrate competitive performance in two standard datasets with a model that is × 5 faster than previous methods at inference time. Furthermore, we also provide a novel analysis of the ST-VQA dataset based on a human performance study. Supplementary material, code, and data is made available through this link. [ABSTRACT FROM AUTHOR]

• Study of VQA systems for cross-modal retrieval tasks. • Baselines in several classification and cross-modal tasks for well know databases. • Evaluation of multiple cross-modal retrieval tasks from an end-to-end system. • Approaches that improve the search experience by using all data present in articles. • End-to-end approach to perform specialized unimodal and crossmodal retrieval. We investigate the effectiveness of a successful model in Visual-Question-Answering (VQA) problems as the core component in a cross-modal retrieval system that can accept images or text as queries, in order to retrieve relevant data from a multimodal document collection. To this end, we adapt the VQA model for deep multimodal learning to combine visual and textual representations for information search, and we call this model "Deep Multimodal Embeddings (DME)". Instead of training the model to answer questions, we supervise DME to classify semantic topics/concepts previously identified in the document collection of interest. In contrast to previous approaches, we found that this model can handle any multimodal query with a single architecture while producing improved or competitive results in all retrieval tasks. We evaluate the model performance with 3 widely known databases for cross-modal retrieval tasks: Wikipedia Retrieval Database, Pascal Sentences, and MIR-Flickr-25k. The results show that the DME model learns effective multimodal representations, resulting in strongly improved retrieval performance, specifically in the top results of the ranked list, which are the most important to users in the most-common scenarios of information retrieval. Our work represents a new baseline for a wide set of different methodologies for cross-modal retrieval. [ABSTRACT FROM AUTHOR]

• A pre-trained MobileNetV2 was leveraged to extract features from brain MRIs automatically. • The optimal feature layer in MobileNetV2 was obtained by experiment. • Three random neural networks: ELM, RVFL and SNN were trained for classification. • A novel chaotic bat algorithm was employed to improve the performance of classifiers. • The trained model can serve as a visual question answering system and produce accurate results. Computer aided diagnosis systems are playing an important part in clinical treatment. They can help the doctors and physicians to verify the diagnosis decisions. In this study, a new classification algorithm for the brain magnetic resonance image is proposed. Initially, we utilized a MobileNetV2 to extract features from the input brain images, which was pre-trained on ImageNet dataset. Instead of training the deep network, we simply calculate the output of its certain layer to form the feature vector. The optimal feature layer is obtained by the experiment. Then, three different feedforward networks: extreme learning machine, Schmidt neural network and random vector functional-link net, are trained for classification. Chaotic bat algorithm was proposed to optimize the weights and biases in the three randomized neural networks to boost their classification accuracy. The result from 5×hold-out validation reveals that our method can achieve good generalization performance which is comparable to state-of-the-art pathological brain detection methods. The trained model can serve as a visual question answering system and produce accurate results. [ABSTRACT FROM AUTHOR]

• A multi-step attention model for Visual Question Answering is proposed. • It relies on attention transfer and a cross-modal gating mechanism. • The attention transfer model adjusts attention with question guidance. • The cross-modal gating mechanism filters out irrelevant information. Visual question answering (VQA) is challenging since it requires to understand both language information and corresponding visual contents. A lot of efforts have been made to capture single-step language and visual interactions. However, answering complex questions requires multiple steps of reasoning which gradually adjusts the region of interest to the most relevant part of the given image, which has not been well investigated. To integrate question related object relations into attention mechanism, we propose a multi-step attention architecture to facilitate the modeling of multi-modal correlations. Firstly, an attention transfer mechanism is integrated to gradually adjust the region of interest considering reasoning representation of questions. Secondly, we propose a cross-modal gating strategy to filter out irrelevant information based on multi-modal correlations. Finally, we achieve the state-of-the-art performance on the VQA 1.0 dataset and favorable results on the VQA 2.0 dataset, which verifies the effectiveness of our proposed method. [ABSTRACT FROM AUTHOR]

• Novel biometric surveillance system based on Visual Question Answering. • Reuse of pre-trained components without fine tuning to demonstrate system robustness. • Introduction of novel models for biometric-focused question image relevance. • New datasets for biometric-based surveillance tasks. Surveillance of individuals using visual data requires human-level capabilities for understanding the characteristics that differentiate one person from another. However, because the influx of both video and imagery is increasing at a greater rate than humans can cope with, biometric-based surveillance systems are required to assist with the triage of information based on human-generated queries. Unfortunately, current systems are not robust enough to tackle new tasks, as they involve specialized models that do not leverage existing, pre-trained components. To mitigate these issues, we propose a novel system for biometric-based surveillance that utilizes models that are relevance-aware to triage images and videos based on interaction with single or multiple users. As the system is initially focused on detection of people via their appearance and clothing, we have named the system Context and Collaborative (C2) Visual Question Answering (VQA) for Biometric Object-Attribute Relevance and Surveillance (C2VQA-BOARS). To validate the usefulness of C2VQA-BOARS in real-world scenarios, we provide an implementation of two novel components (Relevance and Triage) and apply them in tasks against two datasets created for biometric surveillance. Our results outperform baseline approaches, proving that a system with a minimal amount of fine-tuned components can robustly handle new datasets and problems as needed. [ABSTRACT FROM AUTHOR]

Visual Question Answering (VQA) is among the most difficult multi-modal problems as it requires a machine to be able to properly understand a question about a reference image and then infer the correct answer. Providing reliable attention information is crucial for correctly answering the questions. However, existing methods usually only use implicitly trained attention models that are frequently unable to attend to the correct image regions. To this end, an explicitly trained attention model for VQA is proposed in this paper. The proposed method utilizes attention-oriented word embeddings that allows efficiently learning the common representation spaces. Furthermore, multiple attention models of varying complexity are employed as a way of realizing a mixture of experts attention model, further improving the VQA accuracy over a single attention model. The effectiveness of the proposed method is demonstrated using extensive experiments on the Visual7W dataset that provides visual attention ground truth information. [ABSTRACT FROM AUTHOR]