Search

With the recent advancement in the medical diagnostic tools, multi-modality medical images are extensively utilized as a lifesaving tool. An efficient fusion of medical images can improve the performance of various medical diagnostic tools. But, gathering of all modalities for a given patient is defined as an ill-posed problem as medical images suffer from poor visibility and frequent patient dropout. Therefore, in this paper, a novel image fusion model is designed to fuse multi-modality medical images. The proposed model model suffers from hyper-parameters tuning issue, therefore, a multi-objective differential evolution (MDE) is used to optimize the initial parameters of the proposed model. The fusion factor and edge strength metrics are utilized to form a multi-objective fitness function. The performance of the proposed model is validated by comparing the proposed model models with nine competitive models over fifteen benchmark images. Performance analysis reveal that the proposed model outperforms the competitive fusion models.

In this article, a three-dimensional detection framework for detecting non-small cell lung cancer (NSCLC) in 18F-fluorodeoxyglucose (18F-FDG) positron emission tomography/computed tomography (PET/CT) images and guided by a multi-modality attention fusion is proposed. A total of 250 18F-FDG PET/CT scans between January 1, 2015 and December 31, 2019 of patients who had histopathologically proven NSCLC were acquired. A customized dual-path 3D CenterNet is used for NSCLC detection. Moreover, we propose a multi-modality attention module that adaptively refines the multi-modality feature map fusion. Since the 3D convolutional neural network (CNN) requires many graphic memories and sliding windows, a 384×384×32 patch size is designed and used for training and testing. Five-fold cross-validation is applied for this study. The sensitivity and false positive per scan (FPPS) obtained by our proposed method are 0.96 and 1.04, respectively. Our method significantly outperforms 3D CenterNet in terms of sensitivity (P =.031). This module demonstrates the potential to be implemented in other multi-modality applications. Our result performs competitively against other lung cancer detections. Furthermore, case studies show that the proposed method can detect difficult-to-diagnose NSCLCs. Our result shows the proposed method can help radiologists and medical physicists diagnose NSCLC.

Nowadays, with the development of various kinds of sensors in smartphones or wearable devices, human activity recognition (HAR) has been widely researched and has numerous applications in healthcare, smart city, etc. Many techniques based on hand-crafted feature engineering or deep neural network have been proposed for sensor based HAR. However, these existing methods usually recognize activities offline, which means the whole data should be collected before training, occupying large-capacity storage space. Moreover, once the offline model training finished, the trained model can't recognize new activities unless retraining from the start, thus with a high cost of time and space. In this paper, we propose a multi-modality incremental learning model, called HarMI, with continuous learning ability. The proposed HarMI model can start training quickly with little storage space and easily learn new activities without storing previous training data. In detail, we first adopt attention mechanism to align heterogeneous sensor data with different frequencies. In addition, to overcome catastrophic forgetting in incremental learning, HarMI utilizes the elastic weight consolidation and canonical correlation analysis from a multi-modality perspective. Extensive experiments based on two public datasets demonstrate that HarMI can achieve a superior performance compared with several state-of-the-arts.

Magnetic resonance imaging (MRI) can provide multi-modality MR images by setting task-specific scan parameters, and has been widely used in various disease diagnosis and planned treatments. However, in practical clinical applications, it is often difficult to obtain multi-modality MR images simultaneously due to patient discomfort, and scanning costs, etc. Therefore, how to effectively utilize the existing modality images to synthesize missing modality image has become a hot research topic. In this paper, we propose a novel confidence-guided aggregation and cross-modality refinement network (CACR-Net) for multi-modality MR image synthesis, which effectively utilizes complementary and correlative information of multiple modalities to synthesize high-quality target-modality images. Specifically, to effectively utilize the complementary modality-specific characteristics, a confidence-guided aggregation module is proposed to adaptively aggregate the multiple target-modality images generated from multiple source-modality images by using the corresponding confidence maps. Based on the aggregated target-modality image, a cross-modality refinement module is presented to further refine the target-modality image by mining correlative information among the multiple source-modality images and aggregated target-modality image. By training the proposed CACR-Net in an end-to-end manner, high-quality and sharp target-modality MR images are effectively synthesized. Experimental results on the widely used benchmark demonstrate that the proposed method outperforms state-of-the-art methods.

Big challenges are usually occurring in non-rigid 3D shape retrieval, for the shapes undergoing arbitrarily non-affine transformations. In this work, a novel design of feature learning approach is proposed for non-rigid 3D shape retrieval, dubbed Structured Sparsity Regularized Multi-Modality Method (SSR-MM). The shape signatures which capture the deformation-invariant characteristics are averaged and stacked for a multi-modality machine learning approach, and a transform matrix based on the structure sparsity regularization is utilized to map those signatures obtaining the discriminative features for retrieval. The proposed framework is evaluated on the publicly available non-rigid 3D human benchmarks, and the experimental results show the efficacy of our contributions and the advantages of our method over existing ones.

In this study, we focus on improving the efficiency and accuracy of nonrigid multi-modality registration of medical images. In this regard, we analyze the potentials of using the point similarity measurement approach as an alternative to global computation of mutual information (MI), which is still the most renown multi-modality similarity measure. The improvement capabilities are illustrated using the popular B-spline transformation model. The proposed solution is a combination of three related improvements of the most straightforward implementation, i.e., efficient computation of the voxel displacement field, local estimation of similarity and usage of a static image intensity dependence estimate. Five image registration prototypes were implemented to show contribution and dependence of the proposed improvements. When all the proposed improvements are applied, a significant reduction of computational cost and increased accuracy are obtained. The concept offers additional improvement opportunities by incorporating prior knowledge and machine learning techniques into the static intensity dependence estimation.

This paper proposes an automated registration method for multi‐modality retinal fundus photographs based on the directional vessel skeleton. The main purpose is to register two retinal fundus photographs with different modalities of the same scanning region, which can provide multi‐modality information for clinicians to diagnose retinal diseases or to make a treatment decision. The directional vessel skeleton of each fundus image is first detected by bias field correction and Gabor filter. The final registered fundus photographs are then obtained by the iterative affine registration between the detected directional vessel skeletons of two photographs. In this work, four kinds of fundus photographs in the macular regions of the patient with diseases, consisting of 20 optical coherence tomography fundus images, 20 colour fundus photographs, 20 fluorescein fundus angiography images and 20 indocyanine green angiography images, are utilised to quantitatively evaluate the proposed method. The root‐mean‐square errors show an advantageous performance in both registration success rate and accuracy.

In this paper, we propose a triple intersecting U-Nets (TIU-Nets) for brain glioma segmentation. First, the proposed TIU-Nets is composed of binary-class segmentation U-Net (BU-Net) and multi-class segmentation U-Net (MU-Net), in which MU-Net reuses multi-resolution features from BU-Net. Second, we introduce a segmentation soft-mask predicted by BU-Net, that is, candidate glioma region is generated by removing most of non-glioma backgrounds, which guides multi-category segmentation of MU-Net in a weighted manner. Third, an edge branch in MU-Net is leveraged to enhance boundary information of glioma substructure, which facilitates to locate glioma true boundaries and improve segmentation accuracy. Finally, we propose a sigmoid-evolution based polarized cross-entropy loss (S-CE) to resolve class unbalance problem, and apply S-CE loss to soft-mask prediction loss in BU-Net, multi-class segmentation loss in MU-Net and edge prediction loss in edge branch. Experimental results have demonstrated that the proposed 2D/3D TIU-Nets achieves a higher segmentation accuracy than corresponding 2D/3D state-of-the-art segmentation methods including FCN, U-Net, SegNet, CRDN, IVD-Net, FCDenseNet, DeepMedic, DMFNet, etc, evaluating on publicly available brain tumor segmentation challenge 2015 (BRATS2015) datasets. To show the universality of the proposed method, we also give a comparison of segmentation performance on BrainWeb dataset.

The automatic segmentation of brain tumors in magnetic resonance (MR) images is very important in the diagnosis, radiotherapy planning, surgical navigation and several other clinical processes. As the location, size, shape, boundary of gliomas are heterogeneous, segmenting gliomas and intratumoral structures is very difficult. Besides, the multi-center issue makes it more challenging that multimodal brain gliomas images (such as T1, T2, fluid-attenuated inversion recovery (FLAIR), and T1c images) are from different radiation centers. This paper presents a multimodal, multi-scale, double-pathway, 3D residual convolution neural network (CNN) for automatic gliomas segmentation. In the pre-processing step, a robust gray-level normalization method is proposed to solve the multi-center problem, that the intensity range from deferent centers varies a lot. Then, a doublepathway 3D architecture based on DeepMedic toolkit is trained using multi-modality information to fuse the local and context features. In the post-processing step, a fully connected conditional random field (CRF) is built to improve the performance, filling and connecting the isolated segmentations and holes. Experiments on the Multimodal Brain Tumor Segmentation (BRATS) 2017 and 2019 dataset showed that this methods can delineate the whole tumor with a Dice coefficient, a sensitivity and a positive predictive value (PPV) of 0.88, 0.89 and 0.88, respectively. As for the segmentation of the tumor core and the enhancing area, the sensitivity reached 0.80. The results indicated that this method can segment gliomas and intratumoral structures from multimodal MR images accurately, and it possesses a clinical practice value.

Neurophysiological signals are crucial intermediaries, through which brain activity can be quantitatively measured and brain mechanisms are able to be revealed. In particular, non‐invasive neurophysiological signals, such as electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI), are welcomed and frequently utilised in various studies since these signals can be non‐invasively recorded without harming the human brain while they convey abundant information pertaining to brain activity. The recorded neurophysiological signals are analysed to mine meaningful information for the understanding of brain mechanisms or are classified to distinguish different patterns (e.g., different cognitive states, brain diseases versus healthy controls). To date, remarkable progress has been made in both the analysis and classification of neurophysiological signals, but scholars are not feeling complacent. Consistent effort ought to be paid to advance the research of analysis and classification based on neurophysiological signals. In this paper, I express my thoughts regarding promising future directions in neurophysiological signal analysis and classification based on current developments and accomplishments. I will elucidate the thoughts after brief summaries of relevant backgrounds, accomplishments, and tendencies. According to my personal selection and preference, I mainly focus on brain connectivity, multidimensional array (tensor), multi‐modality, multiple task classification, deep learning, big data, and naturalistic experiment. Hopefully, my thoughts could give a little help to inspire new ideas and contribute to the research of the analysis and classification of neurophysiological signals in some way.

Segmentation of nasopharyngeal carcinoma (NPC) from Magnetic Resonance Images (MRI) is a crucial prerequisite for NPC radiotherapy. However, manual segmenting of NPC is time-consuming and labor-intensive. In addition, single-modality MRI generally cannot provide enough information for accurate delineation. Therefore, a novel multi-modality MRI fusion network (MMFNet) is proposed to complete accurate segmentation of NPC via utilizing T1, T2 and contrast-enhanced T1 of MRI. The backbone of MMFNet is designed as a multi-encoder-based network, consisting of several encoders and one decoder, where the encoders aim to capture modality-specific features and the decoder is to obtain fused features for NPC segmentation. A fusion block consisting of a 3D Convolutional Block Attention Module (3D-CBAM) and a residual fusion block (RFBlock) is presented. The 3D-CBAM recalibrates low-level features captured from modality-specific encoders to highlight both informative features and regions of interest (ROIs) whereas the RFBlock fuses re-weighted features to keep balance between fused ones and high-level features from decoder. Moreover, a training strategy named self-transfer is also proposed which utilizes pre-trained modality-specific encoders to initialize multi-encoder-based network in order to make full mining of individual information from multi-modality MRI. The proposed method based on multi-modality MRI can effectively segment NPC and its advantages are validated by extensive experiments.

Radiomics has great prospects in terms of tumour grading, diagnosis and prediction of prognosis by analysing multifaceted data from sources such as clinical treatments, medical images, and pathology. However, exploring an effective way to manage miscellaneous clinical information, as well as to select an appropriate classifier for prediction modelling, is still demanding in a practical clinical context. In this study, we propose a multi-criterion decision-making (MCDM) based classifier fusion (MCF) strategy to combine different classifiers within an MCDM framework. A hierarchical predictive scheme (H-MCF) based on the proposed MCF is also investigated to reliably link the multi-modality features and multi-classifiers. Ten public UCI datasets and two clinical datasets were used to validate the proposed MCF and H-MCF. The experimental results showed that H-MCF has superior predictive performance when compared with the traditional fusion strategies and other fusion architectures, thus demonstrating the feasibility of the proposed H-MCF in integrating information from features of diversified modalities and different classifiers.

Recent deep learning methods rely on a large amount of labeled data to achieve high performance. These methods may be impractical in some scenarios, where manual data annotation is costly or the samples of certain categories are scarce (e.g., tumor lesions, endangered animals and rare individual activities). When only limited annotated samples are available, these methods usually suffer from the overfitting problem severely, which degrades the performance significantly. In contrast, humans can recognize the objects in the images rapidly and correctly with their prior knowledge after exposed to only a few annotated samples. To simulate the learning schema of humans and relieve the reliance on the large-scale annotation benchmarks, researchers start shifting towards the few-shot learning problem: they try to learn a model to correctly recognize novel categories with only a few annotated samples.

In this talk, we present our experiences and applications of large-scale multi-modality pretrained models, developed at Alibaba and Ant Group. We first present a cross-modal pretraining method called M6 (Multi-Modality to Multi-Modality Multitask Mega-transformer) [1], for unified pretraining on the data of multiple modalities. We scale the model size up to 1 trillion parameters [2], and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding performance in comparison with strong baselines. Furthermore, we specifically design a downstream task of text-guided image generation [3], and show that the finetuned M6 can create high-quality images with high resolution and fidelity. We also present research and applications of image editing with pretrained Generative Adversarial Networks (GANs). A general principle between the underlying manifold and the generator is discovered. Based on our discovery, we propose an algorithm for GANs with low-rank factorization [4], which can be harnessed for image editing with pretrained GAN models.

The visualization and data mining of tumor multidimensional information may play a major role in the analysis of the growth, metastasis, and microenvironmental changes of tumors while challenging traditional imaging and data processing techniques. In this study, a general trans-scale and multi-modality measurement method was developed for the quantitative diagnosis of hepatocellular carcinoma (HCC) using a combination of propagation-based phase-contrast computed tomography (PPCT), scanning transmission soft X-ray microscopy (STXM), and Fourier transform infrared micro-spectroscopy (FTIR). Our experimental results reveal the trans-scale micro-morphological HCC pathology and facilitate quantitative data analysis and comprehensive assessment. These results include some visualization features of PPCT-based tissue microenvironments, STXM-based cellular fine structures, and FTIR-based bio-macromolecular spectral characteristics during HCC tumor differentiation and proliferation. The proposed method provides multidimensional feature data support for constructing a high-accuracy machine learning algorithm based on a gray-level histogram, gray-gradient co-occurrence matrix, gray-level co-occurrence matrix, and back-propagation neural network model. Multi-dimensional information analysis and diagnosis revealed the morphological pathways of HCC pathological evolution and we explored the relationships between HCC-related feature changes in inflammatory microenvironments, cellular metabolism, and the stretching vibration peaks of biomolecules of lipids, proteins, and nucleic acids. Therefore, the proposed methodology has strong potential for the visualization of complex tumors and assessing the risks of tumor differentiation and metastasis.

Multimodal pretraining has demonstrated success in the downstream tasks of cross-modal representation learning. However, it is limited to the English data, and there is still a lack of large-scale dataset for multimodal pretraining in Chinese. In this work, we propose the largest dataset for pretraining in Chinese, which consists of over 1.9TB images and 292GB texts. The dataset has large coverage over domains, including encyclopedia, question answering, forum discussion, etc. Besides, we propose a method called M6, referring to Multi-Modality-to-Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. The model is pretrained with our proposed tasks, including text-to-text transfer, image-to-text transfer, as well as multi-modality-to-text transfer. The tasks endow the model with strong capability of understanding and generation. We scale the model to 10 billion parameters, and build the largest pretrained model in Chinese. Experimental results show that our proposed M6 outperforms the baseline in a number of downstream tasks concerning both single modality and multiple modalities, and the 10B-parameter pretrained model demonstrates strong potential in the setting of zero-shot learning.

Locoregional recurrence (LRR) remains one of leading causes in head and neck (H&N) cancer treatment failure despite the advancement of multidisciplinary management. Accurately predicting LRR in early stage can help physicians make an optimal personalized treatment strategy. In this study, we propose an end-to-end multi-modality and multi-view convolutional neural network model (mMmV-CNN) for LRR prediction in H&N cancer. In mMmV, a dimension reduction operator is designed, projecting the 3D volume onto 2D images in different directions, and a multi-view strategy is used to replace the original 3D method, which reduces the complexity of the algorithm while preserving important 3D information. Meanwhile, multi-modal data is used for the classification by making full use of the complementary information from cross modality data. Furthermore, we design a multi-modality deep neural network which is trained in an end-to-end manner and jointly optimize the deep features of CT, PET and clinical features. A H&N dataset which consists of 206 patients was used to evaluate the performance. Experimental results demonstrated that mMm V-CNN can obtain an AUC value of 0.81 and outperform a state of the art CNN-based method.

State-of-the-art multi-stream methods for image manipulation detection suffer from gaps between features. Moreover, the rapid development of GANs makes it an emerging method of image tampering. However, existing natural scene image tampered datasets are limited to manual tampering. In this paper, we address these two issues. Firstly, we propose a novel two-stream multi-modality image manipulation detection model (MM-net) that abandons the way of fusion. The main idea of the proposed model is to exploit one stream to guide the learning of the other stream through attention mechanism. Therefore, our approach enjoys the benefits of the multi-stream methods while avoiding the semantic gaps caused by bridging gaps between different streams. Secondly, we build the first tampered dataset of natural images based on GANs, pushing manipulation detection toward more realistic and challenging scenarios. Extensive experimental results demonstrate that our model outperforms state-of-the-art approaches on both manually and GANs tampered images.

Unsupervised domain adaptation aims to transfer domain knowledge from existing well-defined tasks to new ones where labels are unavailable. In the real-world applications, domain discrepancy is usually uncontrollable especially for multi-modality data. Therefore, it is significantly motivated to deal with a multi-modality domain adaptation task. As labels are unavailable in a target domain, how to learn semantic multi-modality representations and successfully adapt the classifier from a source to the target domain remain open challenges in a multi-modality domain adaptation task. To deal with these issues, we propose a multi-modality adversarial network (MMAN), which applies stacked attention to learn semantic multi-modality representations and reduces domain discrepancy via adversarial training. Unlike the previous domain adaptation methods, which cannot make full use of source domain categories information, multi-channel constraint is employed to capture fine-grained categories of knowledge that could enhance the discrimination of target samples and boost target performance on single-modality and multi-modality domain adaptation problems. We apply the proposed MMAN to two applications including cross-domain object recognition and cross-domain social event recognition. The extensive experimental evaluations demonstrate the effectiveness of the proposed model for unsupervised domain adaptation.

Brain tissue segmentation from Magnetic resonance (MR) image is significant for assessing both neurologic conditions and brain disease. Manual brain tissue segmentation is time-consuming, tedious and subjective which indicates a need for more efficiently automated approaches. However, due to ambiguous boundaries, anatomically complex structure and individual differences, conventional automated segmentation methods performed poorly. Therefore, more effective feature extraction techniques and advanced segmentation models are in essential demand. Inspired by deep learning concepts, we propose a multi-modality aggregation network (MMAN), which is able to extract multi-scale features of brain tissues and harness complementary information from multi-modality MR images for fast and accurate segmentation. Extensive experiments on the well-known MRBrainS Challenge database corroborate the efficiency of the proposed model. Within approximately thirteen seconds, the MMAN can segment three different brain tissues from MRI data of each individual, that is faster than many existing methods. For the segmentation of gray matter, white matter, and cerebrospinal fluid, the MMAN achieved dice coefficients of 86.40%, 89.70% and 84.86%, respectively. Consequently, the proposed model outperformed many state-of-the-art methods and got the second place in the MRBrainS Challenge. Therefore, the proposed MMAN is promising for automated brain segmentation in clinical applications.

In modern scientific research, data are often collected from multiple modalities. Since different modalities could provide complementary information, statistical prediction methods using multi-modality data could deliver better prediction performance than using single modality data. However, one special challenge for using multi-modality data is related to block-missing data. In practice, due to dropouts or the high cost of measures, the observations of a certain modality can be missing completely for some subjects. In this paper, we propose a new DIrect Sparse regression procedure using COvariance from Multi-modality data (DISCOM). Our proposed DISCOM method includes two steps to find the optimal linear prediction of a continuous response variable using block-missing multi-modality predictors. In the first step, rather than deleting or imputing missing data, we make use of all available information to estimate the covariance matrix of the predictors and the cross-covariance vector between the predictors and the response variable. The proposed new estimate of the covariance matrix is a linear combination of the identity matrix, the estimates of the intra-modality covariance matrix and the cross-modality covariance matrix. Flexible estimates for both the sub-Gaussian and heavy-tailed cases are considered. In the second step, based on the estimated covariance matrix and the estimated cross-covariance vector, an extended Lasso-type estimator is used to deliver a sparse estimate of the coefficients in the optimal linear prediction. The number of samples that are effectively used by DISCOM is the minimum number of samples with available observations from two modalities, which can be much larger than the number of samples with complete observations from all modalities. The effectiveness of the proposed method is demonstrated by theoretical studies, simulated examples, and a real application from the Alzheimer’s Disease Neuroimaging Initiative. The comparison between DISCOM and some existing methods also indicates the advantages of our proposed method.

Automatic accurate segmentation of medical images has significant role in computer-aided diagnosis and disease treatment. The segmentation of cerebrospinal fluid (CSF), gray matter (GM), and white matter (WM) tissues plays an important role in infant brain structure for studying early brain development. However, this task is very challenging due to low contrast between GM and WM in isointense phase (approximately 6-8 months of age). In this study, we develop a hyper-densely connected convolutional neural network (CNN) for segmentation of volumetric infant brain. The proposed model provides dense connection between layers to improve the performance of flow information in the network. It also allows the multiscale contextual information by concatenating the feature maps of early, intermediate, and later layers. This architecture employs MR-T1 and T2 as input, which are processed in two separate independent paths, and then their low, intermediate, and high layer features are fused for final segmentation. An important change relative to earlier densely connected networks is the application of direct layer connections from the same and different paths. In this scenario, each modality is processed in an independent path, and dense connections occur not only between layers within the same path, but also between layers in different paths. Adopting such dense connectivity leads to benefits of deep supervision and improved gradient flow. Furthermore, by combining the feature maps of early, intermediate, and late convolutional layers, our architecture injects multiscale information into the final segmentation. This suggested approach is examined in the MICCAI Grand Challenge iSEG and obtains significant advantages over existing approaches in terms of parameter efficiency and segmentation accuracy on 6-month infant brain MRI segmentation.

In scene-level classification of remote sensing, fusion of multi-feature can significantly boost the performance. However, most methods directly fuse the features of different modalities wi...

When traditional 3D reconstruction techniques were used to reconstruct a scene with hidden and disguised heat source targets, the reconstructed scene could not contain these targets; thus, we could not recognize them. The disadvantages of such problems are particularly acute in the military and remote-sensing areas. For this application problem, the authors proposed a visible/infrared combined 3D reconstruction scheme. The 3D scene containing thermal radiation information could be reconstructed by fusing the data from RGB optical and infrared images combined with computer vision passive optical 3D scene reconstruction technique. Meanwhile, the authors proposed the nonrigid registration of multi-modality images with mixed features to solve the problem that the registration algorithm, which is widely used in traditional passive reconstruction technology, cannot accurately match visible and infrared images. Registration accuracy was improved by approximately 40%, experimental results showed that the visible/infrared combined 3D reconstruction scenes retain the visual reality of the traditional 3D reconstruction, and the hidden targets are highlighted in the scenes, which is conducive to the detection and recognition of interesting targets.

For updating the tracking models, most existing approaches have an assumption that the target changes smoothly over time. Despite their success in some cases, these approaches struggle in dealing with occlusion, illumination changes and abrupt motion which may break the temporal smoothness assumption. To tackle this problem, in this paper we propose a tree-guided visual tracking model based on the multimodality correlation filter which could estimate the target state according to the most reliable information in previous frames. We maintain a representative target state set in a tree model over the whole tracking process. Ideally, the tree model is able to capture all the landmark states of the target, and provides a confident template for the correlation filter. Therefore, we propose an optimal updating strategy to record the most recent stable and representative states for tree updating. By utilizing stable target-states for template training, the multi-modality correlation filter is able to output a more accurate target position than the baseline and the SOTA (state-of-the-art) methods. Tested on the OTB50 (object tracking benchmark) and OTB100 dataset, the proposed TGMCF has demonstrated outstanding performance on several typical tracking difficulties and overall comparative results with the SOTA trackers are obtained on several public tracking benchmarks.

In this paper, we build a multi-style generative model for stylish image captioning which uses multi-modality image features, ResNeXt features and text features generated by DenseCap. We propose the 3M model, a Multi-UPDOWN caption model that encodes multi-modality features and decode them to captions. We demonstrate the effectiveness of our model on generating human-like captions by examining its performance on two datasets, the PERSONALITY-CAPTIONS dataset and the FlickrStyle10K dataset. We compare against a variety of state-of-the-art baselines on various automatic NLP metrics such as BLEU, ROUGE-L, CIDEr, SPICE, etc. A qualitative study has also been done to verify our 3M model can be used for generating different stylized captions., To be published at FLAIRS-34

An integrated augmented reality (AR) surgical navigation system that potentially improves intra-operative visualization of concealed anatomical structures. Integration of real-time tracking technology with a laser pico-projector allows the surgical surface to be augmented by projecting virtual images of lesions and critical structures created by multimodality imaging. We aim to quantitatively and qualitatively evaluate the performance of a prototype interactive AR surgical navigation system through a series of pre-clinical studies. Four pre-clinical animal studies using xenograft mouse models were conducted to investigate system performance. A combination of CT, PET, SPECT, and MRI images were used to augment the mouse body during image-guided procedures to assess feasibility. A phantom with machined features was employed to quantitatively estimate the system accuracy. All the image-guided procedures were successfully performed. The tracked pico-projector correctly and reliably depicted virtual images on the animal body, highlighting the location of tumour and anatomical structures. The phantom study demonstrates the system was accurate to 0.55 ± 0.33mm. This paper presents a prototype real-time tracking AR surgical navigation system that improves visualization of underlying critical structures by overlaying virtual images onto the surgical site. This proof-of-concept pre-clinical study demonstrated both the clinical applicability and high precision of the system which was noted to be accurate to

Games has been considered as a benchmark for practicing computational models to analyze players interest as well as its involvement in the game. Though several aspects of game related research are carried out in different fields of research including development of game contents, avatar’s control in games, artificial intelligent competitions, analysis of games using professional gamer’s feedback, and advancements in different traditional and deep learning based computational models. However, affective video summarization of gamer’s behavior and experience are also important to develop innovative features, in-game attractions, synthesizing experience and player’s engagement in the game. Since it is difficult to review huge number of videos of experienced players for the affective analysis, this study is designed to generate video summarization for game players using multi-modal data analysis. Bedside’s physiological and peripheral data analysis, summary of recorded videos of gamers is also generated using attention model-based framework. The analysis of the results has shown effective performance of proposed method.

When learning concepts, cognitive psychology research has revealed that there are two types of concept representation in the human brain: language-derived codes and sensory-derived codes. Thus, in terms of computing, do the multimodal concept vectors and linguistic concept vectors have differences and similarities as human beings? Four experiments are presented in this work, all focused on multi-modality representations labeled by psychologists and text-derived representations generated by computer scientists for concept learning, and the results demonstrate that 1)for the same concept, both forms of representations can properly reflect the concept, but 2) the representational similarity analysis findings reveal that the two types of representations are significantly different, 3) as the concreteness of a concept grows larger, the multi-modality representation of the concept becomes closer to human beings than the text-derived representation, and 4) we verified that combining the two improves the concept representation.

Careful delineation of normal-tissue organs-at-risk is essential for brain tumor radiotherapy. However, this process is time-consuming and subject to variability. In this work, we propose a multi-modality framework that automatically segments eleven structures. Large structures used for defining the clinical target volume (CTV), such as the cerebellum, are directly segmented from T1-weighted and T2-weighted MR images. Smaller structures used in radiotherapy plan optimization are more difficult to segment, thus, a region of interest is first identified and cropped by a classification model, and then these structures are segmented from the new volume. This bi-directional framework allows for rapid model segmentation and good performance on a standardized challenge dataset when evaluated with volumetric and surface metrics.

Cerebral blood volume (CBV) refers to the blood volume of a certain brain tissue per unit time, which is the most useful parameter to evaluate intracranial mass lesions. However, the current CBV measurement methods rely on blood perfusion imaging technology which has obvious shortcomings, i.e., long imaging time, high cost, and great discomfort to the patients. To address this, we attempt to utilize some techniques to synthesize the CBV maps from multiple MRI sequences, which is the least harmful imaging technology currently, so as to reduce the time and cost of clinical diagnosis as well as the patients’ discomfort. Two image synthesis techniques are investigated to synthesize the CBV maps on our collection of 103 groups of multiple MRI modalities of 70 subjects. The experimental results on various modality combinations demonstrate that our redesigned algorithms are possible to synthesize promising CBV maps, which is a good start of developing efficient and cheaper CBV prediction system.

A prevalent goal of building energy management is to search for the optimal schedule of occupants’ actions by minimizing multiple objectives pertaining to occupants’ dissatisfaction. Existing approaches have ignored the multi-modal (different schedules having the same objectives) and the multi-view (different facets of occupants’ actions related to door, window and heater) nature of the action schedules. For addressing these problem characteristics, a recent multi-modal multi-objective evolutionary algorithm, known as LORD, is customized in this work. Moreover, a decision-making strategy is proposed to consider the user preference in the decision space. This strategy also complies with the existing decision-making strategies to avoid neglecting the preference in the objective space. The superior performance of the proposed strategies on a real-world dataset establishes their effectiveness.

In voice therapy, assessment of pathological voice widely depends on subjective evaluation by clinicians and practitioners. This approach of voice evaluation is apparently prone to bias, especially for mildly dysphonic voices in which features are hard to distinguish. As such, research has been focused on objective and automatic estimation of the mildly dysphonic voices. In the present study, a multi-modality approach using six kinematic features and six acoustic features were examined. The pathological voices GO and G1 on the GRBAS scale was classified, and the best result (0.84 using SVM classifier) was achieved when combining features from both modalities. The findings indicated that the proposed method can be used as a reliable way to assess the mildly pathological voice.

Airbnb is a short-term housing rental platform, which is gaining tremendous popularity among tourists. A reasonable prediction of Airbnb rentals could help customers make the best choices and provide homeowners with an indicator of prices to reach maximum benefits. In this paper, a variety of data are combined as inputs into machine learning algorithms and natural language processing framework to construct a reliable price prediction model for Airbnb rentals. This research shows: 1) customer reviews, house features, and geographical data can be used as predictive factors for Airbnb rentals; 2) using multimodality data in price forecast performs higher accuracy compared with single-type data.

The dynamic feature extracted by the 3D convolutional network and the static feature extracted by CNN are proved to be beneficial for video captioning. We adaptively fuse these two kinds of features in the X-Linear Attention Network Video and propose XlanV model for video captioning. However, we notice that the dynamic feature is not compatible with vision-language pre-training techniques when the frame length distribution and average pixel difference of training video and test video biases. Consequently, we directly train the XlanV model on the MSR-VTT dataset without pre-training on the GIF dataset in this challenge. The proposed XlanV model reaches the 1st place in the pre-training for video captioning challenge, which shows that substantially exploiting the dynamic feature is more effective than vision-language pre-training in this challenge.

Multi-modal brain data has been extensively used for improving the accuracy of disease diagnosis by providing complementary information. A problem using multi-modality data is that the data is commonly incomplete for many subjects in the ADNI dataset. A straightforward strategy to tackle this challenge is to simply discard subjects with missing data, but this will greatly reduce the number of training subjects for learning reliable diagnostic models. In this work, we first adopted the RevGAN model to complete missing data. After that, a 3D convolutional neural network was designed to perform AD diagnosis by all subjects (with both real images and synthetic PET images). We tested our method on the Alzheimer's Disease Neuroimaging Initiative (ADNI) database. The results have demonstrated that our synthesized PET images with 3D-RevGAN are reasonable, and our method is successful in Alzheimer's diagnosis.

This work details the combination of cooperative AUVs, accurate timing and clock synchronization, underwater sensors, and acoustic communication for the purposes of detection of various underwater targets for seismic surveys, surveillance, and marine mammals detection applications. Future work will include sensor fusion of passive sensor array of sensors of same kind or passive sensor array of sensors of different kind(magnetic and acoustic) similar to an underwater monitoring systems by introducing autonomous triggering based on acoustic presence indicators of marine species.