Search

• Review of papers on the diagnosis of brain diseases using the fusion of neuroimaging modalities is presented. • For each brain disease, the most appropriate neuroimaging modalities and best combinations are introduced. • Advantages and disadvantages of fusion levels and different modalities are introduced. • Details of various types of deep neural networks used for the task at hand are also examined. • Finally, future directions and challenges in the field are discussed. Brain diseases, including tumors and mental and neurological disorders, seriously threaten the health and well-being of millions of people worldwide. Structural and functional neuroimaging modalities are commonly used by physicians to aid the diagnosis of brain diseases. In clinical settings, specialist doctors typically fuse the magnetic resonance imaging (MRI) data with other neuroimaging modalities for brain disease detection. As these two approaches offer complementary information, fusing these neuroimaging modalities helps physicians accurately diagnose brain diseases. Typically, fusion is performed between a functional and a structural neuroimaging modality. Because the functional modality can complement the structural modality information, thus improving the performance for the diagnosis of brain diseases by specialists. However, analyzing the fusion of neuroimaging modalities is difficult for specialist doctors. Deep Learning (DL) is a branch of artificial intelligence that has shown superior performances compared to more conventional methods in tasks such as brain disease detection from neuroimaging modalities. This work presents a comprehensive review paper in the field of brain disease detection from the fusion of neuroimaging modalities using DL models like convolutional neural networks (CNNs), recurrent neural networks (RNNs), pretrained, generative adversarial networks (GANs), and Autoencoders (AEs). First, neuroimaging modalities and the need for fusion are discussed. Then, review papers published in the field of neuroimaging multimodalities using AI techniques are explored. Moreover, fusion levels based on DL methods, including input, layer, and decision, with related studies conducted on diagnosing brain diseases, are discussed. Other sections present the most important challenges for diagnosing brain diseases from the fusion of neuroimaging modalities. In the discussion section, the details of previous research on the fusion of neuroimaging modalities based on MRI and DL models are reported. In the following, the most important future directions include Datasets, DA, imbalanced data, DL models, explainable AI, and hardware resources are presented. Finally, the main findings of this study are presented in the conclusion section. [ABSTRACT FROM AUTHOR]

Multi-view learning (MVL) is a rapidly evolving direction in the field of machine learning. Despite the positive results, most algorithms that combine multi-view learning with twin support vector machines (TSVM) focus on the traditional machine learning domain. No method has been accomplished for combining MVL, TSVM, and deep learning. In this paper, we propose two novel multi-view deep models to solve the multiclass classification problem, namely deep multi-view twin support vector machines (DMvTSVM) based on deep neural network (DNN) and auto-encoder (AE) network. They find two non-parallel hyperplanes such that each hyperplane is as close to its own class as possible while being as far away from the other class as possible. Meanwhile, we apply similarity regularization to the output of the Deep TSVM classifier for each view to learn consensus information between views, and use this to refine the joint weights of the deep model and TSVM. Finally, the novel models employ the o n e − v s − r e s t strategy to allow the DMvTSVM classifier to solve the multiclass classification problems. In the experiments, the proposed methods are compared with existing state-of-the-art algorithms to prove their effectiveness. • The first combination of twin support vector machines (TSVM) and deep neural networks. • Nesting of more advanced auto encoder networks. • Incorporating multi-view learning capabilities. • Mutual negotiation and simultaneous learning between deep neural networks and TSVM. • The experiments prove the effectiveness of the algorithm in this paper. [ABSTRACT FROM AUTHOR]

Over the years, Machine Learning models have been successfully employed on neuroimaging data for accurately predicting brain age. Deviations from the healthy brain aging pattern are associated with the accelerated brain aging and brain abnormalities. Hence, efficient and accurate diagnosis techniques are required to elicit accurate brain age estimations. Several contributions have been reported in the past for this purpose, resorting to different data-driven modeling methods. Recently, deep neural networks (also referred to as deep learning) have become prevalent in manifold neuroimaging studies, including brain age estimation. In this review, we offer a comprehensive analysis of the literature related to the adoption of deep learning for brain age estimation with neuroimaging data. We detail and analyze different deep learning architectures used for this application, pausing at research works published to date quantitatively exploring their application. We also examine different brain age estimation frameworks, comparatively exposing their advantages and weaknesses. Finally, the review concludes with an outlook towards future directions that should be followed by prospective studies. The ultimate goal of this paper is to establish a common and informed reference for newcomers and experienced researchers willing to approach brain age estimation by using deep learning models. • We categorize brain age estimation models based on the deep learning architectures. • We classify deep learning architectures based on the slice/voxel-based input data. • We examine the modalities, size of data, performance and the datasets used. • We enumerate several challenges and trace directions that can be explored further. [ABSTRACT FROM AUTHOR]

Machine learning algorithms especially deep neural networks have seen tremendous growth in their real-world deployment. While these algorithms have to yield high performances for the task at hand, it is of equal importance that they are robust against the different kinds of singularities or adversarial anomalies. Researchers have generally addressed singularity detection independently for each anomaly, however, for the algorithms to be effective in the real world, the singularity detection algorithm must be able to handle a wide variety of attacks whether trained individually on them or not seen at the time of training. With this objective, in this paper, we propose a unique end-to-end transformation domain network to detect a variety of well-known attacks. The proposed architecture utilizes a combination of two different wavelet transformations to simultaneously learn low-level and high-level image features in a deep stacked layer fashion. The proposed method is generalized to handle a broad set of singularities of several computer vision algorithms whether operating in the object recognition domain or biometrics recognition. We showcase the results on a wide range of singularity points including (i) adversarial perturbations which are learned using deep learning networks, (ii) physical presentation attacks on face recognition which aim to produce fake data for the sensor for acquisition, and identification, (iii) synthetic images, and (iv) digital retouching of the images. Even with a zero-day or open-world attack setting, the results of the proposed algorithm show improvement over state-of-the-art results. The proposed architecture is agnostic to parameters, computationally efficient to provide a sustainable detector to resource-constrained institutions, provides robustness to attacks, and supports green computing. • A novel parameter-agnostic architecture is proposed to defend anomalies. • The proposed algorithm can handle adversaries in seen, unseen, and open-set settings. • The proposed algorithm is domain agnostic i.e., can handle vision and face datasets. • Our unified algorithm surpasses the SOTA works developed to handle specific attacks. • Bring a computationally feasible algorithm, it can be deployed on mobile devices. [ABSTRACT FROM AUTHOR]

An in-depth understanding of uncertainty is the first step to making effective decisions under uncertainty. Machine/deep learning (ML/DL) has been hugely leveraged to solve complex problems involved with processing high-dimensional data. However, reasoning and quantifying different uncertainties to achieve effective decision-making have been much less explored in ML/DL than in other Artificial Intelligence (AI) domains. In particular, belief/evidence theories have been studied in Knowledge representation and reasoning (KRR) since the 1960s to reason and measure uncertainties to enhance decision-making effectiveness. Based on our in-depth literature review, only a few studies have leveraged mature uncertainty research in belief/evidence theories in ML/DL to tackle complex problems under different types of uncertainty. Our present survey paper discusses major belief theories and their core ideas dealing with uncertainty causes and types and quantifying them, along with the discussions of their applicability in ML/DL. Particularly, we discuss three main approaches leveraging belief theories in Deep Neural Networks (DNNs), including Evidential DNNs, Fuzzy DNNs, and Rough DNNs, in terms of their uncertainty causes, types, and quantification methods along with their applicability in diverse problem domains. Through an in-depth understanding of the extensive survey on this topic, we discuss insights, lessons learned, limitations of the current state-of-the-art bridging belief theories and ML/DL, and future research directions. This paper conducts an extensive survey by bridging belief theories and deep learning in reasoning and quantifying uncertainty to help researchers initiate uncertainty and decision-making research. • Identifying the causes and types of uncertainty in various belief models. • Uncertainty reasoning and quantification of belief models for decision-making. • Applying uncertainty quantification of belief models into deep learning. • Comparing uncertainty consideration in belief models and deep learning. • Insights, lessons learned, limitations, and future research directions. [ABSTRACT FROM AUTHOR]

Knowledge resources, e.g. knowledge graphs, which formally represent essential semantics and information for logic inference and reasoning, can compensate for the unawareness nature of many natural language processing techniques based on deep neural networks. This paper provides a focused review of the emerging but intriguing topic that fuses quality external knowledge resources in improving the performance of natural language processing tasks. Existing methods and techniques are summarised in three main categories: (1) static word embeddings, (2) sentence-level deep learning models, and (3) contextualised language representation models, depending on when, how and where external knowledge is fused into the underlying learning models. We focus on the solutions to mitigate two issues: knowledge inclusion and inconsistency between language and knowledge. Details on the design of each representative method, as well as their strength and limitation, are discussed. We also point out some potential future directions in view of the latest trends in natural language processing research. • First comprehensive review on knowledge fusion into deep learning for NLP. • A taxonomy for the related research and extensive overview of the state-of-the-art. • A focus on the theoretical formulation and optimisation methods. • Harmonise the mathematical notations across a plethora of publications. [ABSTRACT FROM AUTHOR]

Studies in affective audio–visual correspondence learning require ground-truth data to train, validate, and test models. The number of available datasets together with benchmarks, however, is still limited. In this paper, we create a collection of three datasets (called EmoMV) for affective correspondence learning between music and video modalities. The first two datasets (called EmoMV-A, and EmoMV-B, respectively) are constructed by making use of music video segments from other available datasets. The third one called EmoMV-C is created from music videos that we self-collected from YouTube. The music-video pairs in our datasets are annotated as matched or mismatched in terms of the emotions they are conveying. The emotions are annotated by humans in the EmoMV-A dataset, while in the EmoMV-B and EmoMV-C datasets they are predicted using a pretrained deep neural network. A user study is carried out to evaluate the accuracy of the "matched" and "mismatched" labels offered in the EmoMV dataset collection. In addition to creating three new datasets, a benchmark deep neural network model for binary affective music-video correspondence classification is also proposed. This proposed benchmark model is then modified to adapt to affective music-video retrieval. Extensive experiments are carried out on all three datasets of the EmoMV collection. Experimental results demonstrate that our proposed model outperforms state-of-the-art approaches on both the binary classification and retrieval tasks. We envision that our newly created dataset collection together with the proposed benchmark models will facilitate advances in affective computing research. • We construct the EmoMV dataset collection for affective music-video correspondence learning. • Deep multi-task learning for affective music-video correspondence classification. • A deep learning approach is developed for affective music-video retrieval. • The new datasets and benchmark models can facilitate advances in affective computing. [ABSTRACT FROM AUTHOR]

• Thispaperfirstlyproposed a supervised-unsupervised combinednetwork for high-fidelitypansharpening. • Thispaperproposed a supervised networkby multiscale mechanisms, dilated convolution and skip connection. • This paperproposeda universalunsupervisedspatial-spectral joint compensationnetwork. Deep learning for pansharpening method has become a hot research topic in recent years due to the impressive performance, and the convolutional neural networks (CNN)-based pansharpening methods on Wald's protocol (i.e., the general adoption of the network learned at a coarser reduced resolution scale to the finer full resolution) have been dominating for a long time in this research area. However, the scale-invariant assumption may not be accurate enough to make full use of the spatial and spectral information of original panchromatic (PAN) and multispectral (MS) images at full resolution. In this paper, a Supervised-Unsupervised combined Fusion Network (SUFNet) for high-fidelity pansharpening is proposed to alleviate this problem. First, by comprehensively considering the robustness of the network with reference label images, a novel supervised network based on Wald's protocol is proposed by integrating the multiscale mechanisms, dilated convolution, and skip connection, termed SMDSNet. Then, an interesting Unsupervised Spatial-Spectral Compensation Network (USSCNet) without real high-spatial-resolution (HR) MS label image is proposed to enhance the spatial and spectral fidelity of the SMDSNet. The qualitative and quantitative results in reduced resolution and full resolution experiments on different satellite datasets demonstrate the competitive performance of the proposed method. Furthermore, the proposed USSCNet can be employed as a universal spatial-spectral compensation framework for other pansharpening methods. [ABSTRACT FROM AUTHOR]

Dropout is a popular regularisation tool for deep neural classifiers, but it is applied regardless of the nature of the classification task: nominal or ordinal. Consequently, the order relation between the class labels of ordinal problems is ignored. In this paper, we propose the fusion of standard dropout and a new dropout methodology for ordinal classification regularising deep neural networks to avoid overfitting and improve generalisation, but taking into account the extra information of the ordinal task, which is exploited to improve performance. The correlation between the outputs of every neuron and the target labels is used to guide the dropout process: the higher the neuron is correlated with the expected labels, the lower its probability of being dropped. Given that randomness also plays a crucial role in the regularisation process, a balancing factor (β) is also added to the training process to determine the influence of the ordinality with respect to a constant probability, providing a hybrid ordinal regularisation method. An extensive battery of experiments shows that the new hybrid ordinal dropout methodology perform better than standard dropout, obtaining improved results in most evaluation metrics, including not only ordinal metrics but also nominal ones. • Deep ordinal classification. • Hybrid ordinal dropout as regularisation methodology for deep neural networks. • Fusion of standard and ordinal dropout techniques. • Balance factor for the influence of the ordinality in the hybrid ordinal dropout. • Comparison of the new hybrid ordinal dropout using ordinal benchmark datasets. [ABSTRACT FROM AUTHOR]

Accurate forecasting of regional solar photovoltaic power (SPVP) generation is essential for efficient energy management and planning. Existing approaches have shown the effectiveness of decomposing the time series to model the stochastic variability in SPVP data. However, these approaches have limitations in extracting and exploiting both spatial and temporal information from complex and high-dimensional data from multiple sources with intricate relationships, which can impact the accuracy of predictions. In this paper, we propose a novel approach called multilevel data fusion and neural basis expansion analysis (MF-NBEA) for forecasting aggregated regional-level SPVP generation. MF-NBEA integrates exogenous data at multiple levels, uses supervised and unsupervised encoders to provide compact data representation, and enhances model learning from complex data by incorporating spatial information. It also includes a sequence analyser module based on a neural network decomposition mechanism to learn the variability in data and incorporates a residuals learner module to improve overall predictions. We evaluate MF-NBEA using two real-world datasets and find that it outperforms state-of-the-art deep learning methods in terms of forecast accuracy. Furthermore, MF-NBEA facilitates information fusion and knowledge extraction to provide interpretable predictions regarding trend, seasonality, and residual components. The insights gained from our approach inform decision-making for energy management and planning, and can lead to more efficient and sustainable resource utilisation. • MF-NBEA for multivariate solar power time series forecasting. • A 3D autoencoder encodes spatial and temporal information from heterogeneous sources. • Interpretable forecasting in terms of trend, seasonality, and residuals. [ABSTRACT FROM AUTHOR]

Multimodal learning has gained significant attention in recent years for combining information from different modalities using Deep Neural Networks (DNNs). However, existing approaches often overlook the varying importance of modalities and neglect uncertainty estimation, leading to limited generalization and unreliable predictions. In this paper, we propose a novel algorithm, Dual-level Deep Evidential Fusion (DDEF), to address these challenges by integrating multimodal information at both the Basic Belief Assignment (BBA) level and multimodal level, for enhancing accuracy, robustness, and reliability. The proposed DDEF approach utilizes the Dirichlet framework and BBA methods to connect neural network outputs with Dirichlet distribution parameters, enabling effective uncertainty estimation, and the Dempster-Shafer Theory (DST) is used for dual-level fusion, facilitating the fusion of evidence from two BBA methods and multiple modalities. It has been validated by two experiments on synthetic digit classification, and real-world medical prognosis after brain–computer interface (BCI) treatment, and by demonstrating superior performance compared to existing methods. Our findings emphasize the importance of considering multimodal integration and uncertainty estimation for reliable decision-making in deep learning. • We propose a novel dual-level fused evidential deep learning model. • The dual levels encompass basic belief assignment and multimodal aspects. • The model adeptly employs the Dempster–Shafer theory for information integration. • Our approach demonstrates enhanced accuracy and reliability in decision-making. • Evidence stems from theoretical analysis, numerical instances, and empirical experiments. [ABSTRACT FROM AUTHOR]

Robustness has become an important consideration in deep learning. With the help of explainable AI, mismatches between an explained model's decision strategy and the user's domain knowledge (e.g. Clever Hans effects) have been identified as a starting point for improving faulty models. However, it is less clear what to do when the user and the explanation agree. In this paper, we demonstrate that acceptance of explanations by the user is not a guarantee for a machine learning model to be robust against Clever Hans effects, which may remain undetected. Such hidden flaws of the model can nevertheless be mitigated, and we demonstrate this by contributing a new method, Explanation-Guided Exposure Minimization (EGEM), that preemptively prunes variations in the ML model that have not been the subject of positive explanation feedback. Experiments demonstrate that our approach leads to models that strongly reduce their reliance on hidden Clever Hans strategies, and consequently achieve higher accuracy on new data. • Explainable AI may not detect all Clever Hans effects in practical ML scenarios. • Hidden Clever Hans effects can be mitigated using Exposure Minimization. • We propose a solution in the form of a soft-pruning scheme on network activations. • We demonstrate the practical benefits of our approach on image and text data. [ABSTRACT FROM AUTHOR]

Establishing reliable correspondences by a deep neural network is an important task in computer vision, and it generally requires permutation-equivariant architecture and rich contextual information. In this paper, we design a Permutation-Equivariant Split Attention Network (called PESA-Net), to gather rich contextual information for the feature matching task. Specifically, we propose a novel "Split–Squeeze–Excitation–Union" (SSEU) module. The SSEU module not only generates multiple paths to exploit the geometrical context of putative correspondences from different aspects, but also adaptively captures channel-wise global information by explicitly modeling the interdependencies between the channels of features. In addition, we further construct a block by fusing the SSEU module, Multi-Layer Perceptron and some normalizations. The proposed PESA-Net is able to effectively infer the probabilities of correspondences being inliers or outliers and simultaneously recover the relative pose by essential matrix. Experimental results demonstrate that the proposed PESA-Net relative surpasses state-of-the-art approaches for pose estimation and outlier rejection on both outdoor scenes and indoor scenes (i.e., YFCC100M and SUN3D). Source codes: https://github.com/x-gb/PESA-Net. • We propose an iterative network to establish reliable correspondences. • We develop an attention mechanism to capture rich contextual information. • We construct a novel block to exploit the global context of unordered data. • The proposed network achieves the state-of-the-art performance. [ABSTRACT FROM AUTHOR]

Nowadays, Artificial intelligence (AI), combined with the digitalization of healthcare, can lead to substantial improvements in Patient Care, Disease Management, Hospital Administration, and supply chain effectiveness. Among predictive analytics tools, time series forecasting represents a central task to support healthcare management in terms of bookings and medical services predictions. In this context, the development of flexible frameworks to provide robust and reliable predictions became a central point in this healthcare innovation process. This paper presents and discusses a multi-source time series fusion and forecasting framework relying on Deep Learning. By combining weather, air-quality and medical bookings time series through a feature compression stage which preserves temporal patterns, the prediction is provided through a flexible ensemble technique based on machine learning models and a hybrid neural network. The proposed system is able to predict the number of bookings related to a specific medical examination for a 7-days horizon period. To assess the proposed approach's effectiveness, we rely on time series extracted from a real dataset of administrative e-health records provided by the Campania Region health department, in Italy. • We propose a multi-source time-series fusion methodology relying on Deep Learning. • We design a robust framework for daily 7-days-horizon predictions. • We assess the proposed approach on a real medical bookings dataset. [ABSTRACT FROM AUTHOR]