Search

Due to the fast retrieval speed and low storage cost, cross-modal hashing has become the primary method for cross-modal retrieval. Since the emergence of deep cross-modal hashing methods, cross-modal retrieval significantly improved. However, the existing cross-modal hash retrieval methods still need to effectively utilize the dataset's supervisory information and the lack of similarity expression ability. This means that the label information needs to be maximized, and the potential semantic relationship between two modalities cannot be fully explored, thus affecting the judgment of semantic similarity between two modalities. To address these problems, this paper proposes Tri-CMH, a cross-modal hash retrieval method with fused triples, which is an end-to-end modeling framework consisting of two parts: feature extraction and hash learning. Firstly, the multi-modal data are preprocessing into the form of triple groups. The data supervision matrix is constructed so that the samples with labels and their meanings are aggregated together. In contrast, the samples with labels and their opposite meanings are separated, thus avoiding the problem of the under-utilization of supervisory information in the data set and achieving the effect of efficiently utilizing the global supervisory information. Meanwhile, the loss function of the hash learning part is optimized by considering the Hamming distance loss, single-modality internal loss, cross-modality loss, and quantization loss to explicitly constrain semantically similar hash codes and semantically dissimilar hash codes and to improve the model's ability to judge cross-modality semantic similarity. The method is trained and tested on the IAPR-TC12, MIRFLICKR-25K, and NUS-WIDE datasets, and the experimental evaluation criteria are mAP and PR curve, and the experimental results show the effectiveness and practicality of the method. [ABSTRACT FROM AUTHOR]

Skeleton-based action recognition, renowned for its computational efficiency and indifference to lighting variations, has become a focal point in the realm of motion analysis. However, most current methods typically only extract global skeleton features, overlooking the potential semantic relationships among various partial limb motions. For instance, the subtle differences between actions such as "brush teeth" and "brush hair" are mainly distinguished by specific elements. Although combining limb movements provides a more holistic representation of an action, relying solely on skeleton points proves inadequate for capturing these nuances. Therefore, integrating detailed linguistic descriptions into the learning process of skeleton features is essential. This motivates us to explore integrating fine-grained language descriptions into the learning process of skeleton features to capture more discriminative skeleton behavior representations. To this end, we introduce a new Linguistic-Driven Partial Semantic Relevance Learning framework (LPSR) in this work. While using state-of-the-art large language models to generate linguistic descriptions of local limb motions and further constrain the learning of local motions, we also aggregate global skeleton point representations and textual representations (which generated from an LLM) to obtain a more generalized cross-modal behavioral representation. On this basis, we propose a cyclic attentional interaction module to model the implicit correlations between partial limb motions. Numerous ablation experiments demonstrate the effectiveness of the method proposed in this paper, and our method also obtains state-of-the-art results. [ABSTRACT FROM AUTHOR]

With the rapid development of multimedia technology, considerable achievement has been achieved in image retrieval technology. On the one hand, users' demand for cross-modal retrieval is increasing rapidly. On the other hand, deep hashing algorithms as one of the most prominent high-dimensional reduction methods have received extensive attention. Under such background, the cross-modal retrieval method based on deep hashing came into being. Although the cross-modal hashing learning method has moved a long way in recent years, it still has space of promotion. First, existing cross-modal based hashing algorithms can only map the feature vectors into binary codes with a fixed length. The length of hash codes cannot be modified according to the practical situation. Second, the saturation level of the probability function used in existing methods is too high to concentrate relevant samples very well, which results in low retrieval efficiency. To solve the above problem, a novel variable-length hash code based on adaptive weight was proposed in this paper. The length of hash codes could be adjusted according to the importance of each bit. And a novel probability function based on Cauchy distribution was proposed to generate compact binary codes and make hashing retrieval more efficient. The experiment shows that the proposed cross-modal image retrieval algorithm based on deep hashing outperforms existing related algorithms on the accuracy of retrieval. [ABSTRACT FROM AUTHOR]

Tomato leaf disease control in the field of smart agriculture urgently requires attention and reinforcement. This paper proposes a method called LAFANet for image-text retrieval, which integrates image and text information for joint analysis of multimodal data, helping agricultural practitioners to provide more comprehensive and in-depth diagnostic evidence to ensure the quality and yield of tomatoes. First, we focus on six common tomato leaf disease images and text descriptions, creating a Tomato Leaf Disease Image-Text Retrieval Dataset (TLDITRD), introducing image-text retrieval into the field of tomato leaf disease retrieval. Then, utilizing ViT and BERT models, we extract detailed image features and sequences of textual features, incorporating contextual information from image-text pairs. To address errors in image-text retrieval caused by complex backgrounds, we propose Learnable Fusion Attention (LFA) to amplify the fusion of textual and image features, thereby extracting substantial semantic insights from both modalities. To delve further into the semantic connections across various modalities, we propose a False Negative Elimination-Adversarial Negative Selection (FNE-ANS) approach. This method aims to identify adversarial negative instances that specifically target false negatives within the triplet function, thereby imposing constraints on the model. To bolster the model's capacity for generalization and precision, we propose Adversarial Regularization (AR). This approach involves incorporating adversarial perturbations during model training, thereby fortifying its resilience and adaptability to slight variations in input data. Experimental results show that, compared with existing ultramodern models, LAFANet outperformed existing models on TLDITRD dataset, with top1, top5, and top10 reaching 83.3% and 90.0%, and top1, top5, and top10 reaching 80.3%, 93.7%, and 96.3%. LAFANet offers fresh technical backing and algorithmic insights for the retrieval of tomato leaf disease through image-text correlation. [ABSTRACT FROM AUTHOR]

The generation of co-speech gestures for digital humans is an emerging area in the field of virtual human creation. Prior research has progressed by using acoustic and semantic information as input and adopting a classification method to identify the person's ID and emotion for driving co-speech gesture generation. However, this endeavor still faces significant challenges. These challenges go beyond the intricate interplay among co-speech gestures, speech acoustic, and semantics; they also encompass the complexities associated with personality, emotion, and other obscure but important factors. This paper introduces "DiT-Gestures", a speech-conditional diffusion-based and non-autoregressive transformer-based generative model with the WavLM pre-trained model and a dynamic mask attention network (DMAN). It can produce individual and stylized full-body co-speech gestures by only using raw speech audio, eliminating the need for complex multimodal processing and manual annotation. Firstly, considering that speech audio contains acoustic and semantic features and conveys personality traits, emotions, and more subtle information related to accompanying gestures, we pioneer the adaptation of WavLM, a large-scale pre-trained model, to extract the style from raw audio information. Secondly, we replace the causal mask by introducing a learnable dynamic mask for better local modeling in the neighborhood of the target frames. Extensive subjective evaluation experiments are conducted on the Trinity, ZEGGS, and BEAT datasets to confirm WavLM's and the model's ability to synthesize natural co-speech gestures with various styles. [ABSTRACT FROM AUTHOR]

Cross-modal visible-infrared person re-identification (VI-ReID) aims to retrieve images of the same pedestrians captured by visible (VIS) cameras and infrared (IR) cameras and it is a challenging task in intelligent security systems. The differences in imaging principles between visible and infrared images lead to large cross-modal differences and intra-class differences, and such cross-modal image differences can be considered as special image style differences, while several intra-class differences can be considered as differences in the form of content expression between visible and infrared images. Some state-of-the-art methods improve the performance of the VI-ReID model by using additional feature enhancement or feature generation modules, however, these methods also introduce additional parameters and increase the training cost. In this paper, to mitigate the differences in image style and content between VIS and IR images, we design two objective functions based on content and style, which are style loss and content loss for the VI-ReID task, respectively. Our model can effectively mitigate the differences between modes by optimizing the objective function to map VIS and IR features into the same feature space without additional auxiliary modules. After extensive experiments, our model achieves competitive performance on two challenging datasets. Notably, under the visible2infrared setting on the RegDB dataset, our model achieves the state-of-the-art (SOTA) Rank-1/mAP/mINP = 96.13%/91.35%/83.67%. [ABSTRACT FROM AUTHOR]

Historically, cross-modal plasticity following early blindness has been largely studied in the context of visual deprivation. However, more recently, there has been a shift in focus towards understanding cross-modal plasticity from the perspective of skill acquisition: the striking plasticity observed in early blind individuals reflects the extraordinary perceptual and cognitive challenges they solve. Here, inspired by two seminal papers on skill learning (the "cortical recycling" theory) and cross-modal plasticity (the "metamodal" hypothesis) respectively, we present a unified hypothesis of cortical specialization that describes how shared functional, algorithmic, and structural constraints might mediate both types of plasticity. [ABSTRACT FROM AUTHOR]

Cross-modal image-text retrieval has gained increasing attention due to its ability to combine computer vision with natural language processing. Previously, image and text features were extracted and concatenated to feed the transformer-based retrieval network. However, these approaches implicitly aligned the image and text modalities since the self-attention mechanism computes attention coefficients for all input features. In this paper, we propose cross-modal Semantic Alignments Module (SAM) to establish an explicit alignment through enhancing an inter-modal relationship. Firstly, visual and textual representations were extracted from an image and text pair. Secondly, we constructed a bipartite graph by representing the image regions and words in the sentence as nodes, and the relationship between them as edges. Then our proposed SAM allows the model to compute attention coefficients based on the edges in the graph. This process helps explicitly align the two modalities. Finally, a binary classifier was used to determine whether the given image-text pair is aligned. We reported extensive experiments on MS-COCO and Flickr30K test sets, showing that SAM could capture the joint representation between the two modalities and could be applied to the existing retrieval networks. [ABSTRACT FROM AUTHOR]

The goal of RGB-D salient object detection is to aggregate the information of the two modalities of RGB and depth to accurately detect and segment salient objects. Existing RGB-D SOD models can extract the multilevel features of single modality well and can also integrate cross-modal features, but it can rarely handle both at the same time. To tap into and make the most of the correlations of intra- and inter-modality information, in this paper, we proposed an attention-guided cross-modal multi-feature aggregation network for RGB-D SOD. Our motivation was that both cross-modal feature fusion and multilevel feature fusion are crucial for RGB-D SOD task. The main innovation of this work lies in two points: One is the cross-modal pyramid feature interaction (CPFI) module that integrates multilevel features from both RGB and depth modalities in a bottom-up manner, and the other is cross-modal feature decoder (CMFD) that aggregates the fused features to generate the final saliency map. Extensive experiments on six benchmark datasets showed that the proposed attention-guided cross-modal multiple feature aggregation network (ACFPA-Net) achieved competitive performance over 15 state of the art (SOTA) RGB-D SOD methods, both qualitatively and quantitatively. [ABSTRACT FROM AUTHOR]

Learning about appearance embedding is of great importance for a variety of different computer-vision applications, which has prompted a surge in person re-identification (Re-ID) papers. The aim of these papers has been to identify an individual over a set of non-overlapping cameras. Despite recent advances in RGB–RGB Re-ID approaches with deep-learning architectures, the approach fails to consistently work well when there are low resolutions in dark conditions. The introduction of different sensors (i.e., RGB–D and infrared (IR)) enables the capture of appearances even in dark conditions. Recently, a lot of research has been dedicated to addressing the issue of finding appearance embedding in dark conditions using different advanced camera sensors. In this paper, we give a comprehensive overview of existing Re-ID approaches that utilize the additional information from different sensor-based methods to address the constraints faced by RGB camera-based person Re-ID systems. Although there are a number of survey papers that consider either the RGB–RGB or Visible-IR scenarios, there are none that consider both RGB–D and RGB–IR. In this paper, we present a detailed taxonomy of the existing approaches along with the existing RGB–D and RGB–IR person Re-ID datasets. Then, we summarize the performance of state-of-the-art methods on several representative RGB–D and RGB–IR datasets. Finally, future directions and current issues are considered for improving the different sensor-based person Re-ID systems. [ABSTRACT FROM AUTHOR]

Biomedical image registration refers to aligning corresponding anatomical structures among different images, which is critical tomany tasks, such as brain atlas building, tumor growth monitoring, and image fusion-based medical diagnosis. However, high-throughput biomedical image registration remains challenging due to inherent variations in the intensity, texture, and anatomy resulting from different imaging modalities, different sample preparation methods, or different developmental stages of the imaged subject. Recently, Generative Adversarial Networks (GAN) have attracted increasing interest in both mono- and cross-modal biomedical image registrations due to their special ability to eliminate the modal variance and their adversarial training strategy. This paper provides a comprehensive survey of the GAN-based mono- and cross-modal biomedical image registration methods. According to the different implementation strategies, we organize the GAN-basedmono- and cross-modal biomedical image registration methods into four categories: modality translation, symmetric learning, adversarial strategies, and joint training. The key concepts, the main contributions, and the advantages and disadvantages of the different strategies are summarized and discussed. Finally, we analyze the statistics of all the cited works from different points of view and reveal future trends for GAN-based biomedical image registration studies. [ABSTRACT FROM AUTHOR]

We consider the problem of referring segmentation in images and videos with natural language. Given an input image (or video) and a referring expression, the goal is to segment the entity referred by the expression in the image or video. In this paper, we propose a cross-modal self-attention (CMSA) module to utilize fine details of individual words and the input image or video, which effectively captures the long-range dependencies between linguistic and visual features. Our model can adaptively focus on informative words in the referring expression and important regions in the visual input. We further propose a gated multi-level fusion (GMLF) module to selectively integrate self-attentive cross-modal features corresponding to different levels of visual features. This module controls the feature fusion of information flow of features at different levels with high-level and low-level semantic information related to different attentive words. Besides, we introduce cross-frame self-attention (CFSA) module to effectively integrate temporal information in consecutive frames which extends our method in the case of referring segmentation in videos. Experiments on benchmark datasets of four referring image datasets and two actor and action video segmentation datasets consistently demonstrate that our proposed approach outperforms existing state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Fat and sugar-reduced foods and beverages have become increasingly popular for a variety of reasons, mainly relating to health and wellbeing. Depending on the food or beverage, it may be difficult to reduce the fat and/or sugar content and still maintain optimal sensory properties for the specific product. One way of approaching the problem is to gain a better understanding of how a product is affected by a reduction in fat and/or sugar. This paper aims to investigate the sensory interactions between fat and sweetness perception in a chocolate-flavored milk beverage by using a descriptive analysis with a trained sensory panel. The reduction of fat significantly reduced the sweetness intensity of the chocolate milk, while the reduction of sucrose significantly decreased the cream flavor and the fruity and lactic flavor. The perception of acesulfame-K was affected by fat concentration, similarly to sucrose. These results highlight the importance of considering the effects of reducing either sugar and fat on product attributes that are not directly related to the sugar or fat. [ABSTRACT FROM AUTHOR]

Automatic diagnosis of various ophthalmic diseases from ocular medical images is vital to support clinical decisions. Most current methods employ a single imaging modality, especially 2D fundus images. Considering that the diagnosis of ophthalmic diseases can greatly benefit from multiple imaging modalities, this paper further improves the accuracy of diagnosis by effectively utilizing cross-modal data. In this paper, we propose Transformer-based cross-modal multi-contrast network for efficiently fusing color fundus photograph (CFP) and optical coherence tomography (OCT) modality to diagnose ophthalmic diseases. We design multi-contrast learning strategy to extract discriminate features from cross-modal data for diagnosis. Then channel fusion head captures the semantically shared information across different modalities and the similarity features between patients of the same category. Meanwhile, we use a class-balanced training strategy to cope with the situation that medical datasets are usually class-imbalanced. Our method is evaluated on public benchmark datasets for cross-modal ophthalmic disease diagnosis. The experimental results demonstrate that our method outperforms other approaches. The codes and models are available at https://github.com/ecustyy/tcmn. [ABSTRACT FROM AUTHOR]

Dogs can recognize conspecifics in cross-modal audio–video presentations. In this paper, we aimed at exploring if such capability extends to the recognition of cats, and whether it is influenced by exposure to these animals. To reach our aim, we enrolled 64 pet dogs. Half of the dogs were currently living with cats, while the rest had never been living with cats, nor were at the time of the experiment. All dogs underwent a cross-modal violation of expectancy experiment, where they were presented with either a cat or a dog vocalization, followed by a video of either species on a blank background. The result revealed that dogs did not exhibit a surprise reaction towards the incoherent stimuli of a cat vocalization and a dog video or vice-versa, implying that they had not recognized the stimuli portraying cats. The pattern of results did not differ between dogs living or not with cats, implying that exposure to a limited number of cats, however, prolonged, is not sufficient to grant dogs with the ability to recognize them on audio–video presentations. We propose that the lack of recognition could be due to the small number of individual cats the dogs are regularly exposed to, or to the possible lack of early exposure to cats during the socialization phase. [ABSTRACT FROM AUTHOR]

A growing body of evidence suggests that taste is not only an essential physical experience, but also an embodied cue of evaluation. The embodied gustatory experiences may affect ethical evaluation. However, it remains unclear how different forms of taste, such as the word sweet, the sweet food image and the actual sweet taste, are associated with ethical evaluation. Does the visual food taste have a priming effect on the moral-terms evaluation? Does the actual gustatory taste influence the ethical processing similarly to the taste images? All the questions will be answered by the three experiments introduced in this paper. Experiment 1 was to test the implicit association between taste words and ethical words. Experiments 2 and 3 were to test the prime-effect of the food image and the actual gustatory perception on ethical evaluation. The results supported our hypothesis that sweet taste words were strongly associated with moral words and bitter taste words with immoral words. Regarding the priming effect, food images and actual gustatory taste lead to different modifications of ethical evaluation. With food image priming, the bitter taste pictures up-regulated the ethical ratings, i.e., the moral phrases were rated more positive, and the immoral phrases were rated less negative, compared with sweet taste images. On the other hand, with actual taste priming, the sweet stimulation could indeed up-regulate the ethical ratings compared with the bitter stimulation. Such a seemingly conflicting influence of different forms of taste on moral evaluation is further considered. [ABSTRACT FROM AUTHOR]

Cross-Modal Hashing (CMH) retrieval methods have garnered increasing attention within the information retrieval research community due to their capability to deal with large amounts of data thanks to the computational efficiency of hash-based methods. To date, the focus of cross-modal hashing methods has been on training with paired data. Paired data refers to samples with one-to-one correspondence across modalities, e.g., image and text pairs where the text sample describes the image. However, real-world applications produce unpaired data that cannot be utilised by most current CMH methods during the training process. Models that can learn from unpaired data are crucial for real-world applications such as cross-modal neural information retrieval where paired data is limited or not available to train the model. This paper provides (1) an overview of the CMH methods when applied to unpaired datasets, (2) proposes a framework that enables pairwise-constrained CMH methods to train with unpaired samples, and (3) evaluates the performance of state-of-the-art CMH methods across different pairing scenarios. [ABSTRACT FROM AUTHOR]

Visible-thermal person recognition is a sub problem of image retrieval, which aims to find out the images belonging to the same pedestrian as the current image from the image set of another modality. In this paper, we propose a novel cross-modal identity correlation mining algorithm to mine potential correlation knowledge from the features of visible and thermal modalities. First, aiming at the huge visual differences caused by different imaging mechanisms, we build a correlation-enhanced knowledge transfer module based on cross-modal identity similarity to enhance the feature representation by exchanging identity knowledge between two modalities and then compress it into a shared subspace. Second, in view of different pedestrian posture and camera perspective, we design a symmetric modal-specific feature embedding module to improve the intra-modality feature discrimination, which maps the two modal images to a pair of independent feature subspaces by two fine-grained network branches. The whole algorithm can be trained in an end-to-end manner. Extensive experiments demonstrated that the proposed method outperforms the state-of-the-art methods on SYSU-MM01 and RegDB. [ABSTRACT FROM AUTHOR]

Salient object detection (SOD) aims at highlighting important foreground objects automatically from the background. Most existing SOD methods only employ visible images (RGB images) for salient detection, which limits the performance of real-life applications when encountering challenging scenarios such as low illumination, haze, and smog. In this paper, we take advantage of the RGB and thermal images and propose an Enhancement and Aggregation–Feedback Network (EAF-Net) for SOD. Specifically, to achieve effective complementation between modalities and prevent the interference from noises, we first treat RGB and thermal images equally in the Feature Enhancement Block (FEB), and further, the Global Context Module expands receptive field to obtain the global features and the Top-Feature Enhancement Module suppresses the redundant information that may destroy the original features from the top layer. Subsequently, we embed several Cross Feature Aggregation Modules (CFAMs) into the Aggregation-and-Feedback Decoder to fuse different level features and compensation features for further obtaining comprehensive feature expression. Moreover, a feedback mechanism is adopted to propagate these fused features back into previous layers for refinement and generate saliency maps to decode features in a progressive way. Comprehensive experiments on RGB-T datasets demonstrate that EAF-Net achieves outstanding performance against the state-of-the-art models. [ABSTRACT FROM AUTHOR]

Visual Speech Recognition (VSR) is a task that recognizes speech from external appearances of the face (${\it i}.{\it e}.$ , lips) into text. Since the information from the visual lip movements is not sufficient to fully represent the speech, VSR is considered as one of the challenging problems. One possible way to resolve this problem is additionally utilizing audio which contains rich information for speech recognition. However, the audio information could not be always available such as in crowded situations. Thus, it is necessary to find a way that successfully provides enough information for speech recognition with visual inputs only. In this paper, we alleviate the information insufficiency of visual lip movement by proposing a cross-modal memory augmented VSR with Visual-Audio Memory (VAM). The proposed framework tries to utilize the complementary information of audio even when the audio inputs are not provided at the inference time. Concretely, the proposed VAM learns to imprint audio features of short clip-level into a memory network using the corresponding visual features. To this end, the VAM contains two memories, lip-video key and audio value. We guide the audio value memory to imprint the audio feature and the lip-video key memory to memorize the location of the imprinted audio. By doing this, the VAM can exploit rich audio information by accessing the memory using visual inputs only. Experimental results show that the proposed method achieves state-of-the-art performance on both word- and sentence-level VSR. In addition, we verify the learned representations inside the VAM contain meaningful information for VSR. [ABSTRACT FROM AUTHOR]

Inspired by visual-tactile cross-modal bidirectional mapping of the human brain, this paper introduces a novel approach to bidirectional mapping between visual and tactile data, an area not fully explored in the predominantly unidirectional existing studies. First, we adopt separate Variational AutoEncoder (VAE) models for visual and tactile data. Furthermore, we introduce a conditional flow model built on the VAE latent feature space, enabling cross-modal bidirectional mapping between visual and tactile data using one model. The experimental results show that our method achieves excellent performance in terms of the similarity between the generated data and the original data (Structural Similarity Index (SSIM) of visual data: 0.58, SSIM of tactile data: 0.80), the classification accuracy on generated data (visual data: 91.60%, tactile data: 88.05%), and the zero-shot classification accuracy between generated data and language (visual data: 44.49%, tactile data: 45.03%). To the best of our knowledge, the method proposed in this paper is the first one to utilize a single model to achieve bidirectional mapping between visual and tactile data. Our model and code will be made public after the acceptance of the paper. [ABSTRACT FROM AUTHOR]

With the ever-increasing urbanization process, modeling people's spatiotemporal activities from their online traces has become a crucial task. State-of-the-art methods for this task rely on cross-modal embedding, which maps items from different modalities (e.g., location, time, text) into the same latent space. Despite their inspiring results, existing cross-modal embedding methods merely capture co-occurrences between items without modeling their high-order interactions. In this paper, we first construct two graphs from raw data records to represent the user interaction graph layer and activity graph layer and propose a hierarchical cross-modal embedding method that takes the high-order relationships into consideration. The key notion behind our method is a novel hierarchical embedding framework with meta-graphs connecting different layers. We introduce both inter-record and intra-record meta-graph structures, which enable learning distributed representations that preserve high-order proximities across graphs from different layers. Our empirical experiments on three real-world datasets demonstrate that our method not only outperforms state-of-the-art methods for spatiotemporal activity prediction, but also captures cross-modal proximity at a finer granularity. [ABSTRACT FROM AUTHOR]

Person re-identification with natural language description is a process of retrieving the corresponding person's image from an image dataset according to a text description of the person. The key challenge in this cross-modal task is to extract visual and text features and construct loss functions to achieve cross-modal matching between text and image. Firstly, we designed a two-branch network framework for person re-identification with natural language description. In this framework we include the following: a Bi-directional Long Short-Term Memory (Bi-LSTM) network is used to extract text features and a truncated attention mechanism is proposed to select the principal component of the text features; a MobileNet is used to extract image features. Secondly, we proposed a Cascade Loss Function (CLF), which includes cross-modal matching loss and single modal classification loss, both with relative entropy function, to fully exploit the identity-level information. The experimental results on the CUHK-PEDES dataset demonstrate that our method achieves better results in Top-5 and Top-10 than other current 10 state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

Visual-semantic embedding (VSE) networks create joint image-text representations to map images and texts in a shared embedding space to enable various information retrieval-related tasks, such as image-text retrieval, image captioning, and visual question answering. The most recent state-of-the-art VSE-based networks are: VSE++, SCAN, VSRN, and UNITER. This study evaluates the performance of those VSE networks for the task of image-to-text retrieval and identifies and analyses their strengths and limitations to guide future research on the topic. The experimental results on Flickr30K revealed that the pre-trained network, UNITER, achieved 61.5% on average Recall@5 for the task of retrieving all relevant descriptions. The traditional networks, VSRN, SCAN, and VSE++, achieved 50.3%, 47.1%, and 29.4% on average Recall@5, respectively, for the same task. An additional analysis was performed on image-text pairs from the top 25 worst-performing classes using a subset of the Flickr30K-based dataset to identify the limitations of the performance of the best-performing models, VSRN and UNITER. These limitations are discussed from the perspective of image scenes, image objects, image semantics, and basic functions of neural networks. This paper discusses the strengths and limitations of VSE networks to guide further research into the topic of using VSE networks for cross-modal information retrieval tasks. [ABSTRACT FROM AUTHOR]

The incorporation of thermal imaging data in RGB-T images has demonstrated its usefulness in cross-modal crowd counting by offering complementary information to RGB representations. Despite achieving satisfactory results in RGB-T crowd counting, many existing methods still face two significant limitations: (1) The oversight of the heterogeneous gap between modalities complicates the effective integration of multimodal features. (2) The absence of mining consistency hinders the full exploitation of the unique complementary strengths inherent in each modality. To this end, we present C4-MIM, a novel Consistency-constrained RGB-T Crowd Counting approach via Mutual Information Maximization. It effectively leverages multimodal information by learning the consistency between the RGB and thermal modalities, thereby enhancing the performance of cross-modal counting. Specifically, we first advocate extracting feature representations of different modalities in a shared encoder to moderate the heterogeneous gap since they obey the identical coding rules with shared parameters. Then, we intend to mine the consistent information of different modalities to better learn conducive information and improve the performance of feature representations. To this end, we formulate the complementarity of multimodality representations as a mutual information maximization regularizer to maximize the consistent information of different modalities, in which the consistency would be maximally attained before combining the multimodal information. Finally, we simply aggregate the feature representations of the different modalities and send them into a regressor to output the density maps. The proposed approach can be implemented by arbitrary backbone networks and is quite robust in the face of single modality unavailable or serious compromised. Extensively experiments have been conducted on the RGBT-CC and DroneRGBT benchmarks to evaluate the effectiveness and robustness of the proposed approach, demonstrating its superior performance compared to the SOTA approaches. [ABSTRACT FROM AUTHOR]

Learning about appearance embedding is of great importance for a variety of different computer-vision applications, which has prompted a surge in person re-identification (Re-ID) papers. The aim of these papers has been to identify an individual over a set of non-overlapping cameras. Despite recent advances in RGB–RGB Re-ID approaches with deep-learning architectures, the approach fails to consistently work well when there are low resolutions in dark conditions. The introduction of different sensors (i.e., RGB–D and infrared (IR)) enables the capture of appearances even in dark conditions. Recently, a lot of research has been dedicated to addressing the issue of finding appearance embedding in dark conditions using different advanced camera sensors. In this paper, we give a comprehensive overview of existing Re-ID approaches that utilize the additional information from different sensor-based methods to address the constraints faced by RGB camera-based person Re-ID systems. Although there are a number of survey papers that consider either the RGB–RGB or Visible-IR scenarios, there are none that consider both RGB–D and RGB–IR. In this paper, we present a detailed taxonomy of the existing approaches along with the existing RGB–D and RGB–IR person Re-ID datasets. Then, we summarize the performance of state-of-the-art methods on several representative RGB–D and RGB–IR datasets. Finally, future directions and current issues are considered for improving the different sensor-based person Re-ID systems.

Fine-grained representation is fundamental to species classification based on deep learning, and in this context, cross-modal contrastive learning is an effective method. The diversity of species coupled with the inherent contextual ambiguity of natural language poses a primary challenge in the cross-modal representation alignment of conservation area image data. Integrating cross-modal retrieval tasks with generation tasks contributes to cross-modal representation alignment based on contextual understanding. However, during the contrastive learning process, apart from learning the differences in the data itself, a pair of encoders inevitably learns the differences caused by encoder fluctuations. The latter leads to convergence shortcuts, resulting in poor representation quality and an inaccurate reflection of the similarity relationships between samples in the original dataset within the shared space of features. To achieve fine-grained cross-modal representation alignment, we first propose a residual attention network to enhance consistency during momentum updates in cross-modal encoders. Building upon this, we propose momentum encoding from a multi-task perspective as a bridge for cross-modal information, effectively improving cross-modal mutual information, representation quality, and optimizing the distribution of feature points within the cross-modal shared semantic space. By acquiring momentum encoding queues for cross-modal semantic understanding through multi-tasking, we align ambiguous natural language representations around the invariant image features of factual information, alleviating contextual ambiguity and enhancing model robustness. Experimental validation shows that our proposed multi-task perspective of cross-modal momentum encoders outperforms similar models on standardized image classification tasks and image–text cross-modal retrieval tasks on public datasets by up to 8% on the leaderboard, demonstrating the effectiveness of the proposed method. Qualitative experiments on our self-built conservation area image–text paired dataset show that our proposed method accurately performs cross-modal retrieval and generation tasks among 8142 species, proving its effectiveness on fine-grained cross-modal image–text conservation area image datasets. [ABSTRACT FROM AUTHOR]

In this study, we adopt a monolingual, comparable, and cross-modal approach to investigate the use of hedges in interpreted and non-interpreted English speeches. To achieve this, we compile a corpus comprising speech data sourced from the interpreter-mediated Chinese government press conferences and the White House press conferences, respectively. Two research questions are formulated to guide our examination of hedging-related features in the corpus: (1) How does the use of hedges differ between interpreted English produced in different modes of interpreting (i.e., consecutive and simultaneous)? (2) How does the use of hedges in interpreted English differ from non-interpreted English? The results of our study show that interpreters, regardless of the consecutive and simultaneous mode, tend to employ significantly fewer hedges than spontaneous speakers of English. In addition, a vast majority of hedges in interpreted English stem from a relatively narrow range of core hedging devices. We posit that the underuse of hedges in interpreted English may be linked to constraints such as high cognitive pressure, rigid interactive structure, and source-language-related speech rhetoric conventions. Preliminary findings also suggest that the interpreting mode appears to exert little influence on the hedging patterns in the interpreters' renditions. Potential factors underlying such a trend are briefly discussed. [ABSTRACT FROM AUTHOR]

Early diagnosis and intervention of depression promote complete recovery, with its traditional clinical assessments depending on the diagnostic scales, clinical experience of doctors and patient cooperation. Recent researches indicate that functional near-infrared spectroscopy (fNIRS) based on deep learning provides a promising approach to depression diagnosis. However, collecting large fNIRS datasets within a standard experimental paradigm remains challenging, limiting the applications of deep networks that require more data. To address these challenges, in this paper, we propose an fNIRS-driven depression recognition architecture based on cross-modal data augmentation (fCMDA), which converts fNIRS data into pseudo-sequence activation images. The approach incorporates a time-domain augmentation mechanism, including time warping and time masking, to generate diverse data. Additionally, we design a stimulation task-driven data pseudo-sequence method to map fNIRS data into pseudo-sequence activation images, facilitating the extraction of spatial-temporal, contextual and dynamic characteristics. Ultimately, we construct a depression recognition model based on deep classification networks using the imbalance loss function. Extensive experiments are performed on the two-class depression diagnosis and five-class depression severity recognition, which reveal impressive results with accuracy of 0.905 and 0.889, respectively. The fCMDA architecture provides a novel solution for effective depression recognition with limited data.

The goal of RGB-D salient object detection is to aggregate the information of the two modalities of RGB and depth to accurately detect and segment salient objects. Existing RGB-D SOD models can extract the multilevel features of single modality well and can also integrate cross-modal features, but it can rarely handle both at the same time. To tap into and make the most of the correlations of intra- and inter-modality information, in this paper, we proposed an attention-guided cross-modal multi-feature aggregation network for RGB-D SOD. Our motivation was that both cross-modal feature fusion and multilevel feature fusion are crucial for RGB-D SOD task. The main innovation of this work lies in two points: One is the cross-modal pyramid feature interaction (CPFI) module that integrates multilevel features from both RGB and depth modalities in a bottom-up manner, and the other is cross-modal feature decoder (CMFD) that aggregates the fused features to generate the final saliency map. Extensive experiments on six benchmark datasets showed that the proposed attention-guided cross-modal multiple feature aggregation network (ACFPA-Net) achieved competitive performance over 15 state of the art (SOTA) RGB-D SOD methods, both qualitatively and quantitatively.

Visible and infrared image fusion is an important image enhancement technique that aims to generate high-quality fused images with prominent targets and rich textures in extreme environments. However, most of the current image fusion methods have poor visual perception of the generated fused images due to severe degradation of texture details of the visible light images in scenes with extreme lighting, which seriously affects the application of subsequent advanced vision tasks such as target detection and tracking. To address these challenges, this paper bridges the gap between image fusion and advanced vision tasks by proposing an efficient fusion method for mutual enhancement of spatial and radiometric information. First, we design the gradient residual dense block (LGCnet) to improve the description of fine spatial details in the fusion network. Then, we developed a cross-modal perceptual fusion (CMPF) module to facilitate modal interactions in the network, which effectively enhances the fusion of complementary information between different modalities and reduces redundant learning. Finally, we designed an adaptive light-aware network (ALPnet) to guide the training of the fusion network to facilitate the fusion network to adaptively select more effective information for fusion under different lighting conditions. Extensive experiments show that the proposed fusion approach has competitive advantages over six current state-of-the-art deep-learning methods in highlighting target features and describing the global scene.

• We propose Cross-Modal Transformer (CMFormer), a Transformer-based cross-modal semantic segmentation model, which achieves better cross-modal information interaction by capturing long-range contextual dependencies. The CMFormer includes the MS-CAC module and the GFA module, and achieves the state-of-the-art results in RGB-D semantic segmentation task. • We design the MS-CAC module and the GFA module by exploiting the cross-modal properties of Transformer. The MS-CAC module preserves multi-scale information while realizing channel feature correction, while the GFA module realizes spatial feature fusion, and fully considers global and local features, which make the CMFormer achieve better cross-modal information interaction by capturing long-range contextual dependencies. • The effectiveness of the CMFormer is evaluated with extensive experiments on the SOP dataset and NYU Depth v2 dataset. The results show that the CMFormer achieves the state-of-the-art results on both datasets. In the experiment of SOP dataset, CMFormer can achieve 96.74% MPA and 92.98% mIoU. At the same time, the real-time performance of CMFormer can reach 43 FPS(Frames Per Second), which has good real-time performance while meeting the requirements of high precision. Scene understanding in a production workshop is an important technology to improve its intelligence level, semantic segmentation of production workshop objects is an effective method for realizing scene understanding. Since the varieties of information of production workshop, making full use of the complementary information of RGB image and depth image can effectively improve the semantic segmentation accuracy of production workshop objects. Aiming at solving the multi-scale and real-time problems of segmenting the production workshop objects, this paper proposes Cross-Modal Transformer (CMFormer), a Transformer-based cross-modal semantic segmentation model. Its key feature correction and feature fusion parts are composed of the Multi-Scale Channel Attention Correction(MS-CAC) module and the Global Feature Aggregation(GFA) module. By improving Multi-Head Self-Attention(MHSA) in Transformer, we design Cross-Modal Multi-Head Self-Attention(CM-MHSA) to build long-range interaction between RGB image and depth image, and further design the MS-CAC module and the GFA module on the basis of the CM-MHSA module, to achieve cross-modal information interaction in the channel and spatial dimensions. Among them, the MS-CAC module enriches the multi-scale features of each channel and achieve more accurate channel attention correction between the two modals; the GFA module interacts with RGB feature and depth feature in the spatial dimension and fuses global and local features at the same time. In the experiments on the NYU Depth v2 dataset, the CMFormer reached 68.00% MPA(Mean Pixel Accuracy) and 55.75% mIoU(Mean Intersection over Union), achieves the state-of-the-art results. While in the experiments on the Scene Objects for Production workshop dataset(SOP), t he CMFormer achieves 96.74% MPA, 92.98% mIoU and 43 FPS(Frames Per Second), which has high precision and good real-time performance. Code is available at: https://github.com/FutureIAI/CMFormer [ABSTRACT FROM AUTHOR]

Neuroimaging studies suggest cross-sensory visual influences in human auditory cortices (ACs). Whether these influences reflect active visual processing in human ACs, which drives neuronal firing and concurrent broadband high-frequency activity (BHFA; >70 Hz), or whether they merely modulate sound processing is still debatable. Here, we presented auditory, visual, and audiovisual stimuli to 16 participants (7 women, 9 men) with stereo-EEG depth electrodes implanted near ACs for presurgical monitoring. Anatomically normalized group analyses were facilitated by inverse modeling of intracranial source currents. Analyses of intracranial event-related potentials (iERPs) suggested cross-sensory responses to visual stimuli in ACs, which lagged the earliest auditory responses by several tens of milliseconds. Visual stimuli also modulated the phase of intrinsic low-frequency oscillations and triggered 15–30 Hz event-related desynchronization in ACs. However, BHFA, a putative correlate of neuronal firing, was not significantly increased in ACs after visual stimuli, not even when they coincided with auditory stimuli. Intracranial recordings demonstrate cross-sensory modulations, but no indication of active visual processing in human ACs. [ABSTRACT FROM AUTHOR]

Ultrasonic inspection of pipeline welds still uses the traditional visual inspection signal method to identify pipeline defects. The identification of defects relies entirely on the subjective judgment of practitioners and is highly dependent on their level of experience. Deep learning models have achieved very good results in classification tasks, but they rely on a large number of annotated data samples for each category. However, it is difficult to collect a large number of samples with different defects and annotate them for the classification of pipe welding defects. Based on the idea of zero-shot learning (ZSL), which makes full use of experts' semantic descriptions of defect categories, artificial semantic features are integrated cross-modally with ultrasonic inspection signal features. In this way, a common semantic space containing seen and unseen classes is constructed to achieve the detection of various defects. Meanwhile, to alleviate the problem of extreme imbalance of training data between the seen and unseen classes in ZSL model training, a ZSL model Feature-GAN-ZSL (FGZ) fused with a generative adversarial network (GAN) is proposed. The model utilizes a Feature-GAN network to generate unseen class features during training and adds a classifier to enhance the generation of features with stronger discriminative power. In the experiments, sample data for porosity, incomplete penetration, and cracks were used as visible classes, and samples for incomplete fusion and slag entrapment were used as unseen classes. Five state-of-the-art models in the ZSL domain were compared. The results show that the FGZ model has a good ability to recognize various defects, not only the types of defects that participated in the training but also the defects that did not participate in the training. This plays a perfect role in dealing with various pipeline welding defects. [ABSTRACT FROM AUTHOR]

At present, Domain generalization for semantic segmentation relying on deep neural networks has made little progress. Most of the current methods are mainly divided into domain randomization, standardization, and whitening. We propose a novel approach to achieve domain generalization for semantic segmentation: leveraging cross-modal information to supervise the model training and improve the generalization ability of the network. We align visual features with textual features in a subspace and enhance the contrast between categories. Our method enables the network to learn rich semantic knowledge from text features and clearer category boundaries. Our experiments also prove that our method can effectively improve the generalization ability of the network. We are the first to exploit multi-modal information for domain-generalized semantic segmentation. [ABSTRACT FROM AUTHOR]

Skeleton-based action recognition, renowned for its computational efficiency and indifference to lighting variations, has become a focal point in the realm of motion analysis. However, most current methods typically only extract global skeleton features, overlooking the potential semantic relationships among various partial limb motions. For instance, the subtle differences between actions such as “brush teeth” and “brush hair” are mainly distinguished by specific elements. Although combining limb movements provides a more holistic representation of an action, relying solely on skeleton points proves inadequate for capturing these nuances. Therefore, integrating detailed linguistic descriptions into the learning process of skeleton features is essential. This motivates us to explore integrating fine-grained language descriptions into the learning process of skeleton features to capture more discriminative skeleton behavior representations. To this end, we introduce a new Linguistic-Driven Partial Semantic Relevance Learning framework (LPSR) in this work. While using state-of-the-art large language models to generate linguistic descriptions of local limb motions and further constrain the learning of local motions, we also aggregate global skeleton point representations and textual representations (which generated from an LLM) to obtain a more generalized cross-modal behavioral representation. On this basis, we propose a cyclic attentional interaction module to model the implicit correlations between partial limb motions. Numerous ablation experiments demonstrate the effectiveness of the method proposed in this paper, and our method also obtains state-of-the-art results.

To probe the presence of mirror neurons in the human brain, cross-modal fMRI adaptation has been suggested as a suitable technique. The rationale behind this suggestion is that this technique allows making more accurate inferences about neural response properties underlying fMRI voxel activations, beyond merely showing shared voxels that are active during both action observation and execution. However, the validity of using cross-modal fMRI adaptation to demonstrate the presence of mirror neurons in parietal and premotor brain regions has been questioned given the inconsistent and weak results obtained in human studies. A better understanding of cross-modal fMRI adaptation effects in the macaque brain is required as the rationale for using this approach is based on several assumptions related to macaque mirror neuron response properties that still need validation. Here, we conducted a cross-modal fMRI adaptation study in macaque monkeys, using the same action execution and action observation tasks that successfully yielded mirror neuron region cross-modal action decoding in a previous monkey MVPA study. We scanned two male rhesus monkeys while they first executed a sequence of either reach-and-grasp or reach-and-touch hand actions and then observed a video of a human actor performing these motor acts. Both whole-brain and region-of-interest analyses failed to demonstrate cross-modal fMRI adaptation effects in parietal and premotor mirror neuron regions. Our results, in line with previous findings in non-human primates, show that cross-modal motor-to-visual fMRI adaptation is not easily detected in monkey brain regions known to house mirror neurons. Thus, our results advocate caution in using cross-modal fMRI adaptation as a method to infer whether mirror neurons can be found in the primate brain. [ABSTRACT FROM AUTHOR]

3D LiDAR semantic segmentation has significant applications in environmental perception, such as auto-driving and intelligent robotics. For autonomous cars equipped with cameras and LiDAR, it is essential to fuse complementary modal information, such as camera color and point cloud depth for environment perception. However, existing fusion algorithms may not achieve encouraging performance owing to the enormous difference between the two modalities. In this work, we propose a Transformer-based cross-modal information fusion network (TCIFNet) scheme to explore model discrepancies. To this end, we first project the point clouds onto the camera coordinates to provide spatial depth information. Then we use Transformer to extract features and fuse the features with effective residual-based self-attentive modules. For the camera stream, additional window-based masked images module improve by 1.8% in mIoU. Moreover, we propose a multimodal distillation loss to measure the difference between the two modalities. Extensive experiments on the benchmark data set show the superiority of our method. Specifically, compared to the state-of-the-art model, we achieve better completeness as well as better robustness in the category with more points. [ABSTRACT FROM AUTHOR]

It has repeatedly been shown that visually presented stimuli can gain additional relevance by their association with affective stimuli. Studies have shown effects of associated affect in event-related potentials (ERP) like the early posterior negativity (EPN), late positive complex (LPC), and even earlier components as the P1 or N170. However, findings are mixed as to the extent associated affect requires directed attention to the emotional quality of a stimulus and which ERP components are sensitive to task instructions during retrieval. In this preregistered study (https://osf.io/ts4pb), we tested cross-modal associations of vocal affect-bursts (positive, negative, neutral) to faces displaying neutral expressions in a flash-card-like learning task, in which participants studied face-voice pairs and learned to correctly assign them to each other. In the subsequent EEG test session, we applied both an implicit ("old-new") and explicit ("valence-classification") task to investigate whether the behavior at retrieval and neurophysiological activation of the affect-based associations were dependent on the type of motivated attention. We collected behavioral and neurophysiological data from 40 participants who reached the preregistered learning criterium. Results showed EPN effects of associated negative valence after learning and independent of the task. In contrast, modulations of later stages (LPC) by positive and negative associated valence were restricted to the explicit, i.e., valence-classification, task. These findings highlight the importance of the task at different processing stages and show that cross-modal affect can successfully be associated to faces. [ABSTRACT FROM AUTHOR]