Search

In recent years, technologies for presenting users with virtual spaces have been rapidly developing. Some applications allow users to observe three-dimensional spaces via screens and influence those virtual spaces using certain devices. In such cases, it is necessary not only to rely on the presentation of depth information but also to adapt to depth in the virtual space to completely execute tasks. In this study, we investigated the spatial perception characteristics of commonly used two-dimensional (2D) displays and advanced spatial reality (SR) displays with spatial presentation capabilities. The research participants performed tasks of tracing a trajectory on a wall surface that changes angles in a three-dimensional virtual space. In addition, we examined the effect of haptic feedback intervention on spatial perception during task execution and its persistence. For 2D displays, we realized that haptic feedback improves task accuracy and that the effect persists even after the feedback is removed. However, this applied only to tasks under feedback conditions, and no broad effect on spatial perception was observed. By contrast, for SR displays, we realized that haptic feedback may have a detrimental effect on spatial perception. Moreover, we quantitatively proved that the use of SR displays improves spatial perception accuracy compared with 2D displays and that the relationship between the line of sight and display angle is critical for spatial perception. In conclusion, the following two points are inferred from this study. (1) To improve spatial perception, it is necessary to consider methods that directly intervene in the body schema and the peripersonal space in the future. (2) Feedback by multiple modalities is not necessarily effective in presenting information on virtual space and obtaining spatial perception.

Drug discovery, especially virtual screening and drug repositioning, can be accelerated through deeper understanding and prediction of Drug Target Interactions (DTIs). The advancement of deep learning as well as the time and financial costs associated with conventional wet-lab experiments have made computational methods for DTI prediction more popular. However, the majority of these computational methods handle the DTI problem as a binary classification task, ignoring the quantitative binding affinity that determines the drug efficacy to their target proteins. Moreover, computational space as well as execution time of the model is often ignored over accuracy. To address these challenges, we introduce a novel method, called Time-efficient Multimodal Drug Target Binding Affinity (TeM-DTBA), which predicts the binding affinity between drugs and targets by fusing different modalities based on compound structures and target sequences. We employ the Lasso feature selection method, which lowers the dimensionality of feature vectors and speeds up the proposed model training time by more than 50%. The results from two benchmark datasets demonstrate that our method outperforms state-of-the-art methods in terms of performance. The mean squared errors of 18.8% and 23.19%, achieved on the KIBA and Davis datasets, respectively, suggest that our method is more accurate in predicting drug-target binding affinity. [ABSTRACT FROM AUTHOR]

Recently proposed state-of-the-art saliency detection models rely heavily on labeled datasets and rarely focus on perfect RGBD feature fusion, which lowers their generalization ability. In this paper, we propose a depth-based interaction and refinement network (DIR-Net) to fully leverage the depth information provided with RGB images to generate and refine the corresponding saliency segmentation maps. In total, three modules are included in our framework. A depth-based refinement module (DRM) and an RGB module work in parallel while coordinating via interactive spatial guidance modules (ISGMs), which utilize spatial and channel attention computed from both depth features and RGB features. In each layer, the features in both modules are refined and guided by the spatial information obtained from the other module through ISGMs. In the RGB module, before sending the depth-guided feature map to the decoder, a convolutional gated recurrent unit (ConvGRU)-based block is introduced to handle temporal information. Thinking about the clear movement information in RGB features, the block also guides temporal information in DRM. By merging the results from both the DRM and RGB modules, a segmentation map with distinct boundaries is generated. Considering the lack of depth images in popular public datasets, we utilize a depth estimation network that incorporates manual postprocessing-based correction to generate depth images on the DAVIS and UVSD datasets. The state-of-the-art performance achieved on both the original and new datasets illustrates the advantage of our RGBD feature fusion strategy, with a real-time speed of 19 fps on a single GPU. [ABSTRACT FROM AUTHOR]

Abstract Recently proposed state-of-the-art saliency detection models rely heavily on labeled datasets and rarely focus on perfect RGBD feature fusion, which lowers their generalization ability. In this paper, we propose a depth-based interaction and refinement network (DIR-Net) to fully leverage the depth information provided with RGB images to generate and refine the corresponding saliency segmentation maps. In total, three modules are included in our framework. A depth-based refinement module (DRM) and an RGB module work in parallel while coordinating via interactive spatial guidance modules (ISGMs), which utilize spatial and channel attention computed from both depth features and RGB features. In each layer, the features in both modules are refined and guided by the spatial information obtained from the other module through ISGMs. In the RGB module, before sending the depth-guided feature map to the decoder, a convolutional gated recurrent unit (ConvGRU)-based block is introduced to handle temporal information. Thinking about the clear movement information in RGB features, the block also guides temporal information in DRM. By merging the results from both the DRM and RGB modules, a segmentation map with distinct boundaries is generated. Considering the lack of depth images in popular public datasets, we utilize a depth estimation network that incorporates manual postprocessing-based correction to generate depth images on the DAVIS and UVSD datasets. The state-of-the-art performance achieved on both the original and new datasets illustrates the advantage of our RGBD feature fusion strategy, with a real-time speed of 19 fps on a single GPU.

Integration of Information, Communication, and Technology helps instructors meet the global need for substituting technology-based teaching and learning tools and services for conventional teaching techniques. English teachers can customise learning for students, and thus it is important to explore new subjects and provide goal-focused training to students. Thus, motivating teachers to use technology and develop their teaching styles to emphasise good teaching is important. Teachers should stop seeing technology as an obstacle to learning. Self-efficacy theory relates to the expectancy-value framework, and motivation is based on the expectation of outcomes through a given course of action. Fundamental psychological needs can also be met with the help of motivation, which eventually increases the desire to undertake competent activities. This study revealed the importance of teachers' technological skills to connect well with students and help them complete their tasks efficiently. Different theories, such as attribution and self-efficacy, have also been incorporated to understand the effectiveness of different tools and technologies. This method could improve students' ability to learn new languages and their communication skills. This study recommends using information tools, as they effectively improve teaching methods and strategies. Information tools are considered strong and constructive tools that allow teachers to improve their strategies and communicate with students. Besides, it is also effective to incorporate lesson plans that can enhance the scope of students' use of technologies to study and gather relevant ideas about various language and linguistic topics. [ABSTRACT FROM AUTHOR]

The paper starts with a theoretical discussion of the possible introduction of aesthetic linguistics into the research field of multimodal critical discourse analysis for analysing tourist news of two modalities: visual and text. Thanks to the successful stories of analysis concerning aesthetic linguistics, systemic functional grammar and visual grammar; first, a framework of multimodal aesthetic critical discourse analysis is carefully redesigned based on multimodal aesthetic critical discourse analysis framework of version 1, originally designed for the same purpose. Then, the validity of the reconstructed framework is verified via analysing a piece of Shannxi tourist news with it. The result indicated that the multimodal aesthetic critical discourse analysis framework re-designed in this paper is of practical significance and can be applied to the analysis of news of multiple modalities.

Quantifying neurological disorders from voice is a rapidly growing field of research and holds promise for unobtrusive and large-scale disorder monitoring. The data recording setup and data analysis pipelines are both crucial aspects to effectively obtain relevant information from participants. Therefore, we performed a systematic review to provide a high-level overview of practices across various neurological disorders and highlight emerging trends. PRISMA-based literature searches were conducted through PubMed, Web of Science, and IEEE Xplore to identify publications in which original (i.e., newly recorded) datasets were collected. Disorders of interest were psychiatric as well as neurodegenerative disorders, such as bipolar disorder, depression, and stress, as well as amyotrophic lateral sclerosis amyotrophic lateral sclerosis, Alzheimer's, and Parkinson's disease, and speech impairments (aphasia, dysarthria, and dysphonia). Of the 43 retrieved studies, Parkinson's disease is represented most prominently with 19 discovered datasets. Free speech and read speech tasks are most commonly used across disorders. Besides popular feature extraction toolkits, many studies utilise custom-built feature sets. Correlations of acoustic features with psychiatric and neurodegenerative disorders are presented. In terms of analysis, statistical analysis for significance of individual features is commonly used, as well as predictive modeling approaches, especially with support vector machines and a small number of artificial neural networks. An emerging trend and recommendation for future studies is to collect data in everyday life to facilitate longitudinal data collection and to capture the behavior of participants more naturally. Another emerging trend is to record additional modalities to voice, which can potentially increase analytical performance.

Multi-modality data is becoming readily available in remote sensing (RS) and can provide complementary information about the Earth's surface. Effective fusion of multi-modal information is thus important for various applications in RS, but also very challenging due to large domain differences, noise, and redundancies. There is a lack of effective and scalable fusion techniques for bridging multiple modality encoders and fully exploiting complementary information. To this end, we propose a new multi-modality network (MultiModNet) for land cover mapping of multi-modal remote sensing data based on a novel pyramid attention fusion (PAF) module and a gated fusion unit (GFU). The PAF module is designed to efficiently obtain rich fine-grained contextual representations from each modality with a built-in cross-level and cross-view attention fusion mechanism, and the GFU module utilizes a novel gating mechanism for early merging of features, thereby diminishing hidden redundancies and noise. This enables supplementary modalities to effectively extract the most valuable and complementary information for late feature fusion. Extensive experiments on two representative RS benchmark datasets demonstrate the effectiveness, robustness, and superiority of the MultiModNet for multi-modal land cover classification. [ABSTRACT FROM AUTHOR]

Classifying hard samples in the course of RGBT tracking is a quite challenging problem. Existing methods only focus on enlarging the boundary between positive and negative samples, but ignore the relations of multilevel hard samples, which are crucial for the robustness of hard sample classification. To handle this problem, we propose a novel Multi-Modal Multi-Margin Metric Learning framework named M5L for RGBT tracking. In particular, we divided all samples into four parts including normal positive, normal negative, hard positive and hard negative ones, and aim to leverage their relations to improve the robustness of feature embeddings, e.g., normal positive samples are closer to the ground truth than hard positive ones. To this end, we design a multi-modal multi-margin structural loss to preserve the relations of multilevel hard samples in the training stage. In addition, we introduce an attention-based fusion module to achieve quality-aware integration of different source data. Extensive experiments on large-scale datasets testify that our framework clearly improves the tracking performance and performs favorably the state-of-the-art RGBT trackers. [ABSTRACT FROM AUTHOR]

While much is known about how negative emotions are expressed in different modalities, our understanding of the nonverbal expressions of positive emotions remains limited. In the present research, we draw upon disparate lines of theoretical and empirical work on positive emotions, and systematically examine which channels are thought to be used for expressing four positive emotions: feeling moved, gratitude, interest, and triumph. Employing the intersubjective approach, an established method in cross-cultural psychology, we first explored how the four positive emotions were reported to be expressed in two North American community samples (Studies 1a and 1b: n = 1466). We next confirmed the cross-cultural generalizability of our findings by surveying respondents from ten countries that diverged on cultural values (Study 2: n = 1826). Feeling moved was thought to be signaled with facial expressions, gratitude with the use of words, interest with words, face and voice, and triumph with body posture, vocal cues, facial expressions, and words. These findings provide cross-culturally consistent findings of differential expressions across positive emotions. Notably, positive emotions were thought to be expressed via modalities that go beyond the face. [ABSTRACT FROM AUTHOR]

Recommending hashtags for micro-videos is a challenging task due to the following two reasons: 1) micro-video is a unity of multi-modalities, including the visual, acoustic, and textual modalities. Therefore, how to effectively extract features from multi-modalities and utilize them to express the micro-video is of great significance; 2) micro-videos usually include moods and feelings, which may provide crucial cues for recommending proper hashtags. However, most of the existing works have not considered the sentiment of media data for hashtag recommendation. In this paper, the senTiment enhanced multi-mOdal Attentive haShtag recommendaTion (TOAST) model is proposed for micro-video hashtag recommendation. Different from previous hashtag recommendation models, which merely consider content features, sentiment features of modalities are further incorporated in TOAST to improve the recommendation performance of the sentiment hashtags (e.g., #funny, #sad). Specifically, the multi-modal content features and the multi-modal sentiment features are modeled by a content common space learning branch based on self-attention and a sentiment common space learning branch, respectively. Furthermore, the varying importance of the multi-modal sentiment and content features are dynamically captured via an attention neural network according to their consistency with the hashtag semantic embedding by an attention neural network. Extensive experiments on a real-world dataset have demonstrated the effectiveness of the proposed method compared with the baseline methods. Meanwhile, the findings from the experiments may provide new insight for future developments of micro-video hashtag recommendation.

Classifying hard samples in the course of RGBT tracking is a quite challenging problem. Existing methods only focus on enlarging the boundary between positive and negative samples, but ignore the relations of multilevel hard samples, which are crucial for the robustness of hard sample classification. To handle this problem, we propose a novel Multi-Modal Multi-Margin Metric Learning framework named M5L for RGBT tracking. In particular, we divided all samples into four parts including normal positive, normal negative, hard positive and hard negative ones, and aim to leverage their relations to improve the robustness of feature embeddings, e.g., normal positive samples are closer to the ground truth than hard positive ones. To this end, we design a multi-modal multi-margin structural loss to preserve the relations of multilevel hard samples in the training stage. In addition, we introduce an attention-based fusion module to achieve quality-aware integration of different source data. Extensive experiments on large-scale datasets testify that our framework clearly improves the tracking performance and performs favorably the state-of-the-art RGBT trackers. [ABSTRACT FROM AUTHOR]

Sentiment analysis is a computational technique that analyses the subjective information conveyed within a given expression. This encompasses appraisals, opinions, attitudes or emotions towards a particular subject, individual, or entity. Conventional sentiment analysis solely considers the text modality and derives sentiment by identifying the semantic relationship between words within a sentence. Despite this, certain expressions, such as exaggeration, sarcasm and humor, pose a challenge for automated detection when conveyed only through text. Multimodal sentiment analysis incorporates various forms of data, such as visual and acoustic cues, in addition to text. By utilizing fusion analysis, this approach can more precisely determine the implied sentiment polarity, which includes positive, neutral, and negative sentiments. Thus, the recent advancements in deep learning have boosted the domain of multimodal sentiment analysis to new heights. The research community has also shown significant interest in this topic due to its potential for both practical application and educational research. In light of this fact, this paper aims to present a thorough analysis of recent ground-breaking research studies conducted in multimodal sentiment analysis, which employs deep learning models across various modalities such as text, audio, image, and video. Furthermore, the article dives into a discussion of the multiple categories of multimodal data, diverse domains in which multimodal sentiment analysis can be applied, a range of operations that are integral to multimodal sentiment analysis, deep learning architectures, a variety of fusion methods, challenges associated with multimodal sentiment analysis, and the benchmark datasets in addition to the state-of-the-art approaches. The ultimate goal of this survey is to indicate the success of deep learning architectures in tackling the complexities associated with multimodal sentiment analysis. • All the different types of modalities and their respective cutting-edge research work was analysed. • Demonstrated almost every possible application area of multimodal sentiment analysis. • Deep learning models and some state-of-the-art architectures for all publicly available multimodal datasets were studied. • Analysed various fusion methods to reveal their potential advantages and disadvantages. • Challenges pertaining to multimodal sentiment analysis were discussed. [ABSTRACT FROM AUTHOR]

Metaphors are a common linguistic phenomenon, and metaphor identification plays an essential role in metaphor processing. Most existing metaphor computing techniques use only texts to gain features, but we acquire additional knowledge from other modalities. At present, the multimodal model in the metaphor field is in the exploratory stage, and the few multimodal models available are still relatively crude. We propose a multimodal metaphor detection method according to the idea that different types of words are suitable for different modality calculations. First, our proposed framework uses a fine-grained concreteness calculation method based on part of speech to distinguish abstract and concrete words. We then choose a different appropriate modal feature and a different metaphor computational method for words with different concreteness. Additionally, we also improve the use of image features in the field of metaphor detection. [ABSTRACT FROM AUTHOR]

Copyright of Journal of Computer-Aided Design & Computer Graphics / Jisuanji Fuzhu Sheji Yu Tuxingxue Xuebao is the property of Gai Kan Bian Wei Hui and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)

Affective video content analysis is an important research topic in video content analysis and has extensive applications. Intuitively, multimodal features can depict elicited emotions, and the accumulation of temporal inputs influences the viewer's emotion. Although a number of research works have been proposed for this task, the adaptive weights of modalities and the correlation of temporal inputs are still not well studied. To address these issues, a novel framework is designed to learn the weights of modalities and temporal inputs from video data. Specifically, three network layers are designed, including statistical-data layer to improve the robustness of data, temporal-adaptive-fusion layer to fuse temporal inputs, and multimodal-adaptive-fusion layer to combine multiple modalities. In particular, the feature vectors of three input modalities are respectively extracted from three pre-trained convolutional neural networks and then fed to three statistical-data layers. Then, the output vectors of these three statistical-data layers are separately connected to three recurrent layers, and the corresponding outputs are fed to a fully-connected layer which shares parameters across modalities and temporal inputs. Finally, the outputs of the fully-connected layer are fused by the temporal-adaptive-fusion layer and then combined by the multimodal-adaptive-fusion layer. To discover the correlation of both multiple modalities and temporal inputs, adaptive weights of modalities and temporal inputs are introduced into loss functions for model training, and these weights are learned by an optimization algorithm. Extensive experiments are conducted on two challenging datasets, which demonstrate that the proposed method achieves better performances than baseline and other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

To effectively educate the future public health workforce, it is essential for public health faculty to engage in purposeful and thoughtful pedagogical practices in their teaching. In the post-COVID higher education environment, instructors teaching course sections offered both in-person and online concurrently must ensure that curricular continuity is maintained, while respecting modality-driven differences. Developing multiple course sections, parallel in outcomes and rigor, but bespoke in instructional approach requires both additional development time and faculty support structures. While time is always in short supply, it is troubling to note that there is a paucity of information directed toward faculty that provides guidance on how faculty engage in course design work that requires them to plan in two modalities simultaneously. In the absence of available best practices for course design in multiple modalities, this paper provides a description of a pedagogical curriculum reflection exercise created to ensure that course sections delivered across different modalities are comparable and meet course competencies, such as those of the Council on Education for Public Health, in equitable ways. [ABSTRACT FROM AUTHOR]

• Our new approach for continuous cross-modal hashing (CCMH) can handle arriving new classes (ANCs), which has not been explored before. • The proposed CT-CMH can be flexibly deployed in real-world applications by adopting a single-head setup. • The proposed CT-CMH consisting of HCD and AWIU can effectively address the problem of catastrophic forgetting and hash-code conflicts. • The experimental results and visualizations demonstrate the effectiveness of our approach for the CCMH task. Generally, multimodal data with new classes arrive continuously in the real world. While advanced cross-modal hashing (CMH) focuses primarily on batch-based data with previously observed classes (ASCs), it disregards the effect of newly arriving classes (ANCs) on hash-code conflicts. In addition, class-level continuous hashing scenarios do not suit themselves well with the generic CMH configuration. To solve the aforementioned issues, we propose a novel framework, called CT-CMH, for the new task of continuous cross-modal hashing. For dealing with ANCs, CMH models require the ability of continuous learning, i.e. they can preserve the knowledge of previously observed data and, more crucially, they can be adapted to unseen data with ANCs. Specifically, we introduce the adaptive weight importance updating (AWIU) mechanism to alleviate the catastrophic forgetting problem of CMH and a new hash-code divergence (HCD) method to eliminate hash-code conflicts between ASCs and ANCs. When CT-CMH is equipped with both AWIU and HCD, it can consistently achieve high retrieval performance. The experiment results and visualization analyses validate the effectiveness of our approach. To the best of our knowledge, we are the first to introduce and implement the task of CCMH for ANCs. [ABSTRACT FROM AUTHOR]

We developed a new flexible approach for a co-analysis of multi-modal brain imaging data using a non-parametric framework. In this approach, results from separate analyses on different modalities are combined using a combining function and assessed with a permutation test. This approach identifies several cross-modality relationships, such as concordance and dissociation, without explicitly modeling the correlation between modalities. We applied our approach to structural and perfusion MRI data from an Alzheimer's disease (AD) study. Our approach identified areas of concordance, where both gray matter (GM) density and perfusion decreased together, and areas of dissociation, where GM density and perfusion did not decrease together. In conclusion, these results demonstrate the utility of this new non-parametric method to quantitatively assess the relationships between multiple modalities. (c) 2005 Elsevier Inc. All rights reserved.

This commentary on the five articles in the Special Issue will first introduce research issues regarding the processing of multiple texts in order to outline the scenario in which the contributions are placed. Then, in the light of the need to move beyond multiple text processing in the digital era, the commentary will illustrate how the five articles contribute to finding answers to some important open questions, furthering our understanding of the processing of instructional materials which feature multiple sources, multiple symbol systems, and multiple modalities of presentation. Finally, the commentary will suggest some issues arising from the outcomes of the studies in this Special Issue, which are worthy of further exploration. [ABSTRACT FROM AUTHOR]

Alzheimer's disease (AD) is a neurodegenerative disorder, the most common cause of dementia, so the accurate diagnosis of AD and its prodromal stage mild cognitive impairment (MCI) is significant. Recent studies have demonstrated that multiple neuroimaging and biological measures contain complementary information for diagnosis. Many existing multi-modal models based on deep learning simply concatenate each modality's features despite substantial differences in representation spaces. In this paper, we propose a novel multi-modal cross-attention AD diagnosis (MCAD) framework to learn the interaction between modalities for better playing their complementary roles for AD diagnosis with multi-modal data including structural magnetic resonance imaging (sMRI), fluorodeoxyglucose-positron emission tomography (FDG-PET) and cerebrospinal fluid (CSF) biomarkers. Specifically, the imaging and non-imaging representations are learned by the image encoder based on cascaded dilated convolutions and CSF encoder, respectively. Then, a multi-modal interaction module is introduced, which takes advantage of cross-modal attention to integrate imaging and non-imaging information and reinforce relationships between these modalities. Moreover, an extensive objective function is designed to reduce the discrepancy between modalities for effectively fusing the features of multi-modal data, which could further improve the diagnosis performance. We evaluate the effectiveness of our proposed method on the ADNI dataset, and the extensive experiments demonstrate that our MCAD achieves superior performance for multiple AD-related classification tasks, compared to several competing methods. Also, we investigate the importance of cross-attention and the contribution of each modality to the diagnostics performance. The experimental results demonstrate that combining multi-modality data via cross-attention is helpful for accurate AD diagnosis., Competing Interests: Declaration of Competing Interest The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper., (Copyright © 2023 Elsevier Ltd. All rights reserved.)

Currently, there exists diverse modalities of 3D models, such as the view set representation of 3D models, the virtual 3D model designed by CAD tools, and the 2.5D model captured by Kinect sensors. To realize flexible access to 3D models in different modalities, this paper proposes the unified framework for cross-modality 3D model retrieval. First, we develop a toolbox with OpenGL to convert 3D models in multiple modalities into the view set representation of 3D models, which are usually represented by a set of characteristic views. Then, we extract discriminative visual feature for multi-view representation. These visual features can be utilized to construct the graphical model to represent the structural characteristics of individual 3D model. Finally, we leverage the graph matching algorithm for similarity measure between pairwise 3D models. We evaluate this unified framework on several well-known 3D model datasets. The comparison experiments demonstrate that this unified framework can achieve competing performances against the state of the arts. [ABSTRACT FROM AUTHOR]