Search

• Five Machine Learning (ML) models compared for wave height time series prediction. • Complex ML models do not outperform simple AR in wave height time series prediction. • Comment to related papers: signal decomposition in test set series is WRONG. Significant Wave Height (SWH) is crucial in many aspect of ocean engineering. The accurate prediction of SWH has therefore been of immense practical value. Recently, Artificial Intelligence (AI) time series prediction methods have been widely used for single-point short-term SWH time-series forecasting, resulting in many AI-based models claiming to achieve good results. However, the extent to which these complex AI models can outperform traditional methods has largely been overlooked. This study compared five different models - AutoRegressive (AR), eXtreme Gradient Boosting (XGB), Artificial Neural Network (ANN), Long Short-Term Memory (LSTM), and WaveNet - for their performance on SWH time series prediction at 16 buoy locations. Surprisingly, the results suggest that the differences of performance among different models are negligible, indicating that all these AI models have only "learned" the linear auto-regression from the data. Additionally, we noticed that many recent studies used signal decomposition method for such time series prediction, and most of them decomposed the test sets, which is WRONG. [ABSTRACT FROM AUTHOR]

The accidental leakage of gaseous hydrogen from hydrogen refueling station facilities has the potential to result in a large fire and explosion catastrophe. The ability to forecast the spatiotemporal evolution of hydrogen plumes in real-time is crucial for monitoring the distribution of plume concentrations and mitigating the risk of fire and explosion. The utilization of deep learning has been extensively employed in a diverse range of spatiotemporal forecasting tasks. However, these approaches encounter limitations in accurately quantifying uncertainty when predicting spatiotemporal concentrations and plume boundaries. This research paper introduces a novel model called DPSTNN_H 2 Evolution, which is a deep probabilistic spatial-temporal neural network designed for forecasting the spatiotemporal evolution of hydrogen plumes. The benchmark dataset is constructed by the implementation of numerical simulations pertaining to the inadvertent leak of hydrogen within a hydrogen refueling station. The findings of the study indicate that incorporating quantified uncertainty information might enhance the precision of plume boundaries and the resilience of plume concentrations when predicting the spatiotemporal development of hydrogen. By employing Monte Carlo sampling with a sample size of m = 100 and a dropout rate of p = 0.1 , the model demonstrates the ability to provide real-time inference for 10 instances of plumes, while maintaining a high level of accuracy with R2 = 0.9829. In comparison to the current state-of-the-art model, the proposed model demonstrates superior accuracy and robustness in forecasting the spatiotemporal evolution of hydrogen plumes. In general, our proposed model has the potential to serve as a dependable option for the development of a digital twin for emergency management of hydrogen refueling stations. • Probabilistic spatial-temporal neural network model for hydrogen plume evolutions is proposed. • Normalized uncertainty contours and uncertainty intervals to improve accuracy and robustness. • Numerical experiment of hydrogen leakage and dispersion in refueling station are conducted. • Optimal hyper-parameters of the proposed model are determined • Comparison between the proposed model with point-estimation based model is conducted [ABSTRACT FROM AUTHOR]

While deep neural networks have shown impressive results in automatic speaker recognition and related tasks, it is dissatisfactory how little is understood about what exactly is responsible for these results. Part of the success has been attributed in prior work to their capability to model supra-segmental temporal information (SST), i.e., learn rhythmic-prosodic characteristics of speech in addition to spectral features. In this paper, we (i) present and apply a novel test to quantify to what extent the performance of state-of-the-art neural networks for speaker recognition can be explained by modeling SST; and (ii) present several means to force respective nets to focus more on SST and evaluate their merits. We find that a variety of CNN- and RNN-based neural network architectures for speaker recognition do not model SST to any sufficient degree, even when forced. The results provide a highly relevant basis for impactful future research into better exploitation of the full speech signal and give insights into the inner workings of such networks, enhancing explainability of deep learning for speech technologies. • Literature explains speaker recognition in neural nets by modeling of voice dynmaics. • Diagnostic: We quantify how well deep learning models actually capture dynamics. • Observation: State-of-the-art deep nets do not model speaker prosody but ignore it. • Interpretation as "cheating": Achieving high without putting in due effort. • Outlook: Increasing task difficulty biases models towards prosody, but not enough. [ABSTRACT FROM AUTHOR]

Large deep learning models are impressive, but they struggle when real-time data is not available. Few-shot class-incremental learning (FSCIL) poses a significant challenge for deep neural networks to learn new tasks from just a few labeled samples without forgetting the previously learned ones. This setup can easily leads to catastrophic forgetting and overfitting problems, severely affecting model performance. Studying FSCIL helps overcome deep learning model limitations on data volume and acquisition time, while improving practicality and adaptability of machine learning models. This paper provides a comprehensive survey on FSCIL. Unlike previous surveys, we aim to synthesize few-shot learning and incremental learning, focusing on introducing FSCIL from two perspectives, while reviewing over 30 theoretical research studies and more than 20 applied research studies. From the theoretical perspective, we provide a novel categorization approach that divides the field into five subcategories, including traditional machine learning methods, meta learning-based methods, feature and feature space-based methods, replay-based methods, and dynamic network structure-based methods. We also evaluate the performance of recent theoretical research on benchmark datasets of FSCIL. From the application perspective, FSCIL has achieved impressive achievements in various fields of computer vision such as image classification, object detection, and image segmentation, as well as in natural language processing and graph. We summarize the important applications. Finally, we point out potential future research directions, including applications, problem setups, and theory development. Overall, this paper offers a comprehensive analysis of the latest advances in FSCIL from a methodological, performance, and application perspective. • In-depth survey on FSCIL methods, applications, performance. • Categorizing FSCIL into five subcategories for clear analysis. • Evaluating FSCIL research on benchmarks for strengths, weaknesses. • FSCIL applications in computer vision, NLP, and graph. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved outstanding performance in computer vision tasks. Convolutional Neural Networks (CNNs) typically operate in the spatial domain with raw images, but in practice, images are usually stored and transmitted in their compressed representation where JPEG is one of the most widely used encoder. Also, these networks are computationally intensive and slow. This paper proposes performing the learning and inference processes in the compressed domain in order to reduce the computational complexity and improve the speed of popular CNNs. For this purpose, a novel graph-based frequency channel selection method is proposed to identify and select the most important frequency channels. The computational complexity is reduced by retaining the important frequency components and discarding the insignificant ones as well as eliminating the unnecessary layers of the network. Experimental results show that the modified ResNet-50 operating in the compressed domain is up to 70% faster than the spatial-based traditional ResNet-50 while resulting in similar classification accuracy. Moreover, this paper proposes a preprocessing step with partial encoding to improve the resilience to distortions caused by low-quality encoded images. Finally, we show that training a network with highly compressed data can achieve a good classification accuracy with up to 93% reduction in the storage requirements of the training data. [ABSTRACT FROM AUTHOR]

Despite the wide variety of methods developed for synthetic image attribution, most of them can only attribute images generated by models or architectures included in the training set and do not work with unknown architectures, hindering their applicability in real-world scenarios. In this paper, we propose a verification framework that relies on a Siamese Network to address the problem of open-set attribution of synthetic images to the architecture that generated them. We consider two different settings. In the first setting, the system determines whether two images have been produced by the same generative architecture or not. In the second setting, the system verifies a claim about the architecture used to generate a synthetic image, utilizing one or multiple reference images generated by the claimed architecture. The main strength of the proposed system is its ability to operate in both closed and open-set scenarios so that the input images, either the query and reference images, can belong to the architectures considered during training or not. Experimental evaluations encompassing various generative architectures such as GANs, diffusion models, and transformers, focusing on synthetic face image generation, confirm the excellent performance of our method in both closed and open-set settings, as well as its strong generalization capabilities. • New verification framework for open-set architecture attribution of synthetic images. • Tested with several types of generative architectures in closed and open set scenarios. • Generalization tests prove that the system can verify unknown models of the architecture. • Outperforms state-of-the-art methods for open-set architecture attribution with rejection. [ABSTRACT FROM AUTHOR]

Artificial Intelligence has found wide application, but also poses risks due to unintentional or malicious tampering during deployment. Regular checks are therefore necessary to detect and prevent such risks. Fragile watermarking is a technique used to identify tampering in AI models. However, previous methods have faced challenges including risks of omission, additional information transmission, and inability to locate tampering precisely. In this paper, we propose a method for detecting tampered parameters and bits, which can be used to detect, locate, and restore parameters that have been tampered with. We also propose an adaptive embedding method that maximizes information capacity while maintaining model accuracy. Our approach was tested on multiple neural networks subjected to attacks that modified weight parameters, and our results demonstrate that our method achieved great recovery performance when the modification rate was below 20%. Furthermore, for models where watermarking significantly affected accuracy, we utilized an adaptive bit technique to recover more than 15% of the accuracy loss of the model. • Fragile watermarking for rigorous neural network integrity detection. • Watermarking ensures 100% tampering detection, precise location, and recovery. • Watermarking fragility by combining Mutual- and Self-neural net parameters. • Incorporates watermarking with neural net feature, maintaining watermarked models' performance via gradient descent. • Bit-level training controls step size with carefully set hyperparameters. [ABSTRACT FROM AUTHOR]

• A Digital Cultural Heritage dataset of synthetic vaults exported as point cloud, based on shape grammar discussion. • A vaults synthetic dataset creation method for automatically generating training data when real 3D objects are insufficient. • A Deep Learning system able to segment and distinguish types of architectural vaults according to standard classifications. Historical heritage is demanding robust pipelines for preserving, enhancing, and disseminating its prominent value. Semantic segmentation of 3D Point Clouds has gained increasing attention over the years, since it might assist in automati-cally recognizing historical architectural elements, thus facilitating large dataset management. Nonetheless, semantic segmentation is particularly challenging in Cultural Heritage (CH) domain, due to the shapes complexity and the limited repeatability of elements across different architectures, which strengthens the difficulty to define common patterns within the same class of elements. Besides, as Deep Neural Networks demand an appreciably amount of labelled data to be trained, the lack of available annotated heritage point clouds prevent the research in this direction. To tackle these issues, in this paper it is proposed a Deep Learning system able to recognize historical building elements by lever-aging synthetic point cloud. The generation of the 3D models, vaults, is based on a procedural modeling approach that follows the ideal shapes, according to the rules of descriptive geometry for the main types of vaults. The approach has been applied to a newly synthetic dataset which is publicly available. This dataset comprises 6 labelled points clouds, derived from a comprehensive on-tological taxonomy in order to describe an univocal and robust architectural hierarchy: barrel vaults, groined vaults, mirror vaults, barrel vaults with clois-ter heads and lunettes, barrel vaults with lunettes, sail vaults. The experiments yield high accuracy, demonstrating the effectiveness and suitability of the pro-posed approach. [ABSTRACT FROM AUTHOR]

Recent years have witnessed increasing interest in adversarial attacks on images, while adversarial video attacks have seldom been explored. In this paper, we propose a sparse adversarial attack strategy on videos (DeepSAVA). Our model aims to add a small human-imperceptible perturbation to the key frame of the input video to fool the classifiers. To carry out an effective attack that mirrors real-world scenarios, our algorithm integrates spatial transformation perturbations into the frame. Instead of using the l p norm to gauge the disparity between the perturbed frame and the original frame, we employ the structural similarity index (SSIM), which has been established as a more suitable metric for quantifying image alterations resulting from spatial perturbations. We employ a unified optimisation framework to combine spatial transformation with additive perturbation, thereby attaining a more potent attack. We design an effective and novel optimisation scheme that alternatively utilises Bayesian Optimisation (BO) to identify the most critical frame in a video and stochastic gradient descent (SGD) based optimisation to produce both additive and spatial-transformed perturbations. Doing so enables DeepSAVA to perform a very sparse attack on videos for maintaining human imperceptibility while still achieving state-of-the-art performance in terms of both attack success rate and adversarial transferability. Furthermore, built upon the strong perturbations produced by DeepSAVA, we design a novel adversarial training framework to improve the robustness of video classification models. Our intensive experiments on various types of deep neural networks and video datasets confirm the superiority of DeepSAVA in terms of attacking performance and efficiency. When compared to the baseline techniques, DeepSAVA exhibits the highest level of performance in generating adversarial videos for three distinct video classifiers. Remarkably, it achieves an impressive fooling rate ranging from 99.5% to 100% for the I3D model, with the perturbation of just a single frame. Additionally, DeepSAVA demonstrates favourable transferability across various time series models. The proposed adversarial training strategy is also empirically demonstrated with better performance on training robust video classifiers compared with the state-of-the-art adversarial training with projected gradient descent (PGD) adversary. • Sparse attacks on video models: perturb fewer frames to gain high fooling rate. • Combining additive and spatial perturbations to enhance attacking performance. • Using SSIM instead of l p -norm to maintain the human perception. • Applying Bayesian Optimisation to identify the most critical frame to perturb. • A new adversarial training method based on combination of diverse perturbations. [ABSTRACT FROM AUTHOR]

We study how the topology of feature embedding space changes as it passes through the layers of a well-trained deep neural network (DNN) through Betti numbers. Motivated by existing studies using simplicial complexes on shallow fully connected networks (FCN), we present an extended analysis using Cubical homology instead, with a variety of popular deep architectures and real image datasets. We demonstrate that as depth increases, a topologically complicated dataset is transformed into a simple one, resulting in Betti numbers attaining their lowest possible value. The rate of decay in topological complexity (as a metric) helps quantify the impact of architectural choices on the generalization ability. Interestingly from a representation learning perspective, we highlight several invariances such as topological invariance of (1) an architecture on similar datasets; (2) embedding space of a dataset for architectures of variable depth; (3) embedding space to input resolution/size, and (4) data sub-sampling. In order to further demonstrate the link between expressivity & the generalization capability of a network, we consider the task of ranking pre-trained models for downstream classification task (transfer learning). Compared to existing approaches, the proposed metric has a better correlation to the actually achievable accuracy via fine-tuning the pre-trained model. • This paper categorizes the change in topological complexity through Betti-Numbers. • Attempts to characterize the learning ability of the DNNs through topological transformation of feature space. • Attempts to use the topological complexity of feature space for the task of ranking pre-trained models. [ABSTRACT FROM AUTHOR]

Deep neural networks are highly vulnerable to adversarial samples. Most existing adversarial defense methods do not consider the distribution of adversarial samples. We argue that very few adversarial samples in the natural sample set prevent the deep neural networks from learning a complete and effective representation of the adversarial samples. This causes the spatial structures between the natural and the adversarial samples to be vastly different from that of the input space, thus making the models vulnerable to adversarial attacks. Based on this viewpoint, this paper proposes an effective deep-learning adversarial defense method, which incorporates information about the spatial structures of the natural and the adversarial samples in the embedding space during the training process. This proposed approach improves the deep learning model's generalization to new adversarial samples and achieves the purpose of defending against adversarial attacks. Four deep neural networks with different scales are used and experimentally verified on four typical publicly available image data. The experimental results show that our method effectively improves the defense ability of deep learning models against adversarial attacks. • Development of an effective adversarial training method using spatial structure constraints in embedding space. • Spatial structure information of the natural and adversarial samples can help improve the adversarial robustness of DNNs. • The spatial structure constrain can be extended to adversarial training of semi-supervised and unsupervised learning. [ABSTRACT FROM AUTHOR]

Automatic identity recognition from ear images is an active research topic in the biometric community. The ability to secretly acquire images of the ear remotely and the stability of the ear shape over time make this technology a promising alternative for surveillance, authentication, and forensic applications. In recent years, significant research has been conducted in this area. Nevertheless, challenges remain that limit the commercial use of this technology. Several phases of the ear recognition system have been studied in the literature, from ear detection, normalization, and feature extraction to classification. This paper reviews the most recent methods used to describe and classify biometric features of the ear. We propose a first taxonomy to group existing approaches to ear recognition, including 2D, 3D, and combined 2D and 3D methods, as well as an overview of historical advances in this field. It is well known that data and algorithms are the essential components in biometrics, particularly in-ear recognition. However, early ear recognition datasets were very limited and collected in laboratory with controlled environments. With the wider use of deep neural networks, a considerable amount of training data has become necessary if acceptable ear recognition performance is to be achieved. As a consequence, current ear recognition datasets have increased significantly in size. This paper gives an overview of the chronological evolution of ear recognition datasets and compares the performance of conventional vs. deep learning methods on several datasets. We proposed a second taxonomy to classify the existing databases, including 2D, 3D, and video ear datasets. Finally, some open challenges and trends are debated for future research. [ABSTRACT FROM AUTHOR]

Deep neural networks are enjoying unprecedented attention and success in recent years. However, catastrophic forgetting undermines the performance of deep models when the training data are arrived sequentially in an online multi-task learning fashion. To address this issue, we propose a novel method named continual learning with declarative memory (CLDM) in this paper. Specifically, our idea is inspired by the structure of human memory. Declarative memory is a major component of long-term memory which helps human beings memorize past experiences and facts. In this paper, we propose to formulate declarative memory as task memory and instance memory in neural networks to overcome catastrophic forgetting. Intuitively, the instance memory recalls the input–output relations (fact) in previous tasks, which is implemented by jointly rehearsing previous samples and learning current tasks as replaying-based methods act. In addition, the task memory aims to capture long-term task correlation information across task sequences to regularize the learning of the current task, thus preserving task-specific weight realizations (experience) in high task-specific layers. In this work, we implement a concrete instantiation of the proposed task memory by leveraging a recurrent unit. Extensive experiments on seven continual learning benchmarks verify that our proposed method is able to outperform previous approaches with tremendous improvements by retaining the information of both samples and tasks. [ABSTRACT FROM AUTHOR]

• Review of papers on the diagnosis of brain diseases using the fusion of neuroimaging modalities is presented. • For each brain disease, the most appropriate neuroimaging modalities and best combinations are introduced. • Advantages and disadvantages of fusion levels and different modalities are introduced. • Details of various types of deep neural networks used for the task at hand are also examined. • Finally, future directions and challenges in the field are discussed. Brain diseases, including tumors and mental and neurological disorders, seriously threaten the health and well-being of millions of people worldwide. Structural and functional neuroimaging modalities are commonly used by physicians to aid the diagnosis of brain diseases. In clinical settings, specialist doctors typically fuse the magnetic resonance imaging (MRI) data with other neuroimaging modalities for brain disease detection. As these two approaches offer complementary information, fusing these neuroimaging modalities helps physicians accurately diagnose brain diseases. Typically, fusion is performed between a functional and a structural neuroimaging modality. Because the functional modality can complement the structural modality information, thus improving the performance for the diagnosis of brain diseases by specialists. However, analyzing the fusion of neuroimaging modalities is difficult for specialist doctors. Deep Learning (DL) is a branch of artificial intelligence that has shown superior performances compared to more conventional methods in tasks such as brain disease detection from neuroimaging modalities. This work presents a comprehensive review paper in the field of brain disease detection from the fusion of neuroimaging modalities using DL models like convolutional neural networks (CNNs), recurrent neural networks (RNNs), pretrained, generative adversarial networks (GANs), and Autoencoders (AEs). First, neuroimaging modalities and the need for fusion are discussed. Then, review papers published in the field of neuroimaging multimodalities using AI techniques are explored. Moreover, fusion levels based on DL methods, including input, layer, and decision, with related studies conducted on diagnosing brain diseases, are discussed. Other sections present the most important challenges for diagnosing brain diseases from the fusion of neuroimaging modalities. In the discussion section, the details of previous research on the fusion of neuroimaging modalities based on MRI and DL models are reported. In the following, the most important future directions include Datasets, DA, imbalanced data, DL models, explainable AI, and hardware resources are presented. Finally, the main findings of this study are presented in the conclusion section. [ABSTRACT FROM AUTHOR]

Supervised learning in deep neural networks is commonly performed using error backpropagation. However, the sequential propagation of errors during the backward pass limits its scalability and applicability to low-powered neuromorphic hardware. Therefore, there is growing interest in finding local alternatives to backpropagation. Recently proposed methods based on forward-mode automatic differentiation suffer from high variance in large deep neural networks, which affects convergence. In this paper, we propose the Forward Direct Feedback Alignment algorithm that combines Activity-Perturbed Forward Gradients with Direct Feedback Alignment and momentum. We provide both theoretical proofs and empirical evidence that our proposed method achieves lower variance than forward gradient techniques. In this way, our approach enables faster convergence and better performance when compared to other local alternatives to backpropagation and opens a new perspective for the development of online learning algorithms compatible with neuromorphic systems. • Forward Gradient methods suffer from high variance that hinders convergence. • Activity-Perturbed Forward Gradients can be used to learn derivatives as direct feedback connections. • Feedback learning acts as a momentum that reduces the gradient variance closer to backpropagation. • Feedback learning reduces the biasness of Direct Feedback Alignment. • Feedback learning enables learning in convolutional layers. [ABSTRACT FROM AUTHOR]

Sharpness aware minimization (SAM) optimizer has been extensively explored as it can generalize better for training deep neural networks via introducing extra perturbation steps to flatten the landscape of deep learning models. Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step. In this paper, we try to analyze the convergence rate of AdaSAM in the stochastic non-convex setting. We theoretically show that AdaSAM admits a O (1 / b T) convergence rate, which achieves linear speedup property with respect to mini-batch size b. Specifically, to decouple the stochastic gradient steps with the adaptive learning rate and perturbed gradient, we introduce the delayed second-order momentum term to decompose them to make them independent while taking an expectation during the analysis. Then we bound them by showing the adaptive learning rate has a limited range, which makes our analysis feasible. To the best of our knowledge, we are the first to provide the non-trivial convergence rate of SAM with an adaptive learning rate and momentum acceleration. At last, we conduct several experiments on several NLP tasks and the synthetic task, which show that AdaSAM could achieve superior performance compared with SGD, AMSGrad, and SAM optimizers. • Adaptive SAM with momentum acceleration ensures automatic learning rate adjustment. • Adaptive SAM shows linear speedup, converging at O (1 / b T). • Delaying 2nd-order momentum separates grad from adaptive rate & perturbed gradients. [ABSTRACT FROM AUTHOR]

In this paper, we investigate the forward problems on the data-driven rational solitons for the (2+1)-dimensional Kadomtsev–Petviashvili-I (KP-I) equation and spin-nonlinear Schrödinger (spin-NLS) equation via the deep neural networks learning. Moreover, the inverse problems of the (2+1)-dimensional KP-I equation and spin-NLS equation are studied via deep learning. The main idea of the data-driven forward and inverse problems is to use the deep neural networks with the activation function to approximate the solutions of the considered (2+1)-dimensional nonlinear wave equations by optimizing the chosen loss functions related to the considered nonlinear wave equations. • Forward problems of 2D KP-I and spin-NLS equations with rational solitons are studied via deep learning. • Inverse problems of 2D KP-I and spin-NLS equations are discussed via deep learning. • Inverse problems of the 2D combination of NLS and cmKdV equations are explored via deep learning. [ABSTRACT FROM AUTHOR]

Multi-view learning (MVL) is a rapidly evolving direction in the field of machine learning. Despite the positive results, most algorithms that combine multi-view learning with twin support vector machines (TSVM) focus on the traditional machine learning domain. No method has been accomplished for combining MVL, TSVM, and deep learning. In this paper, we propose two novel multi-view deep models to solve the multiclass classification problem, namely deep multi-view twin support vector machines (DMvTSVM) based on deep neural network (DNN) and auto-encoder (AE) network. They find two non-parallel hyperplanes such that each hyperplane is as close to its own class as possible while being as far away from the other class as possible. Meanwhile, we apply similarity regularization to the output of the Deep TSVM classifier for each view to learn consensus information between views, and use this to refine the joint weights of the deep model and TSVM. Finally, the novel models employ the o n e − v s − r e s t strategy to allow the DMvTSVM classifier to solve the multiclass classification problems. In the experiments, the proposed methods are compared with existing state-of-the-art algorithms to prove their effectiveness. • The first combination of twin support vector machines (TSVM) and deep neural networks. • Nesting of more advanced auto encoder networks. • Incorporating multi-view learning capabilities. • Mutual negotiation and simultaneous learning between deep neural networks and TSVM. • The experiments prove the effectiveness of the algorithm in this paper. [ABSTRACT FROM AUTHOR]

This paper explores the potential of self-supervised learning as an alternative to supervised learning in the context of geometry-based 3D object retrieval. With the ongoing digitalization of many industries, an exponentially increasing number of 3D objects are processed by retrieval systems. In order to support new shapes, modern deep learning-based retrieval systems require retraining. The dominant paradigm for optimizing neural networks in this field is supervised classification training. Supervised learning requires time-consuming and expensive data annotation. Moreover, training neural networks for classification introduces a bias towards the classes in the training data, which is undesirable for retrieval systems encountering unseen object types in the wild. Through extensive experiments, we make a direct comparison between supervised and self-supervised learning on four datasets from three different domains (household, manufacturing and medical). For object classes seen during training, self-supervised and supervised learning are competitive. For unseen classes, self-supervised learning outperforms supervised learning in many cases. We conclude that self-supervised learning provides a powerful tool for circumventing labeling costs and providing more robust retrieval systems. [Display omitted] • Self-supervised learning is on par with supervised learning for 3D object retrieval. • Self-supervised object retrieval outperforms supervised learning out-of-distribution. • Object alignment is highly impactful on the performance of retrieval systems. • Non-contrastive siamese networks are a practical option in 3D deep learning. [ABSTRACT FROM AUTHOR]

Over the years, Machine Learning models have been successfully employed on neuroimaging data for accurately predicting brain age. Deviations from the healthy brain aging pattern are associated with the accelerated brain aging and brain abnormalities. Hence, efficient and accurate diagnosis techniques are required to elicit accurate brain age estimations. Several contributions have been reported in the past for this purpose, resorting to different data-driven modeling methods. Recently, deep neural networks (also referred to as deep learning) have become prevalent in manifold neuroimaging studies, including brain age estimation. In this review, we offer a comprehensive analysis of the literature related to the adoption of deep learning for brain age estimation with neuroimaging data. We detail and analyze different deep learning architectures used for this application, pausing at research works published to date quantitatively exploring their application. We also examine different brain age estimation frameworks, comparatively exposing their advantages and weaknesses. Finally, the review concludes with an outlook towards future directions that should be followed by prospective studies. The ultimate goal of this paper is to establish a common and informed reference for newcomers and experienced researchers willing to approach brain age estimation by using deep learning models. • We categorize brain age estimation models based on the deep learning architectures. • We classify deep learning architectures based on the slice/voxel-based input data. • We examine the modalities, size of data, performance and the datasets used. • We enumerate several challenges and trace directions that can be explored further. [ABSTRACT FROM AUTHOR]

The backpropagation algorithm has promoted the rapid development of deep learning, but it relies on a large amount of labeled data and still has a large gap with how humans learn. The human brain can quickly learn various conceptual knowledge in a self-organized and unsupervised manner, accomplished through coordinating various learning rules and structures in the human brain. Spike-timing-dependent plasticity (STDP) is a general learning rule in the brain, but spiking neural networks (SNNs) trained with STDP alone is inefficient and perform poorly. In this paper, taking inspiration from short-term synaptic plasticity, we design an adaptive synaptic filter and introduce the adaptive spiking threshold as the neuron plasticity to enrich the representation ability of SNNs. We also introduce an adaptive lateral inhibitory connection to adjust the spikes balance dynamically to help the network learn richer features. To speed up and stabilize the training of unsupervised spiking neural networks, we design a samples temporal batch STDP (STB-STDP), which updates weights based on multiple samples and moments. By integrating the above three adaptive mechanisms and STB-STDP, our model greatly accelerates the training of unsupervised spiking neural networks and improves the performance of unsupervised SNNs on complex tasks. Our model achieves the current state-of-the-art performance of unsupervised STDP-based SNNs in the MNIST and FashionMNIST datasets. Further, we tested on the more complex CIFAR10 dataset, and the results fully illustrate the superiority of our algorithm. Our model is also the first work to apply unsupervised STDP-based SNNs to CIFAR10. At the same time, in the small-sample learning scenario, it will far exceed the supervised ANN using the same structure. [ABSTRACT FROM AUTHOR]

Weight-tied models have attracted attention in the modern development of neural networks. The deep equilibrium model (DEQ) represents infinitely deep neural networks with weight-tying, and recent studies have shown the potential of this type of approach. DEQs are needed to iteratively solve root-finding problems in training and are built on the assumption that the underlying dynamics determined by the models converge to a fixed point. In this paper, we present the stable invariant model (SIM) , a new class of deep models that in principle approximates DEQs under stability and extends the dynamics to more general ones converging to an invariant set (not restricted in a fixed point). The key ingredient in deriving SIMs is a representation of the dynamics with the spectra of the Koopman and Perron–Frobenius operators. This perspective approximately reveals stable dynamics with DEQs and then derives two variants of SIMs. We also propose an implementation of SIMs that can be learned in the same way as feedforward models. We illustrate the empirical performance of SIMs with experiments and demonstrate that SIMs achieve comparative or superior performance against DEQs in several learning tasks. • A Koopman spectral analysis is performed for deep equilibrium models. • The analysis identifies stable dynamics determined by deep equilibrium models. • We develop a class of deep models to approximate the stable dynamics. • Models in the class can represent dynamics converging to an invariant set. [ABSTRACT FROM AUTHOR]

In this paper, an approach that makes use of knuckle creases and fingernails for person identification is presented. It introduces a framework for automatic person identification that includes localisation of the region of interest (ROI) of many components within hand images, recognition and segmentation of the detected components using bounding boxes, and similarity matching between two different sets of segmented images. The following hand components are considered: i) the metacarpophalangeal (MCP) joint, commonly known as the base knuckle; ii) the proximal interphalangeal (PIP) joint, commonly known as the major knuckle; iii) the distal interphalangeal (DIP) joint, commonly known as the minor knuckle; iv) the interphalangeal (IP) joint, commonly known as the thumb knuckle, and v) the fingernails. Crucial elements of the proposed framework are the feature extraction and similarity matching. This paper exploits different deep learning neural networks (DLNNs), which are essential in extracting discriminative high-level abstract features. We further use various similarity measures for the matching process. We validate the proposed approach on well-known benchmarks, including the 11k Hands dataset and the Hong Kong Polytechnic University Contactless Hand Dorsal Images known as PolyU. The results indicate that knuckle patterns and fingernails play a significant role in the person identification framework. The 11K Hands dataset results indicate that the left-hand results are better than the right-hand results and the fingernails produce consistently higher identification results than other hand components, with a rank-1 score of 100 %. In addition, the PolyU dataset attains 100 % in the fingernail of the thumb finger. [ABSTRACT FROM AUTHOR]

Search and retrieval remains a major research topic in several domains, including computer graphics, computer vision, engineering design, etc. A search engine requires primarily an input search query and a database of items to search from. In engineering, which is the primary context of this paper, the database consists of 3D CAD models, such as washers, pistons, connecting rods, etc. A query from a user is typically in the form of a sketch, which attempts to capture the details of a 3D model. However, sketches have certain typical defects such as gaps, over-drawn portions (multi-strokes), etc. Since the retrieved results are only as good as the input query, sketches need cleaning-up and enhancement for better retrieval results. In this paper, a deep learning approach is proposed to improve or clean the query sketches. Initially, sketches from various categories are analysed in order to understand the many possible defects that may occur. A dataset of cleaned-up or enhanced query sketches is then created based on an understanding of these defects. Consequently, an end-to-end training of a deep neural network is carried out in order to provide a mapping between the defective and the clean sketches. This network takes the defective query sketch as the input and generates a clean or an enhanced query sketch. Qualitative and quantitative comparisons of the proposed approach with other state-of-the-art techniques show that the proposed approach is effective. The results of the search engine are reported using both the defective and enhanced query sketches, and it is shown that using the enhanced query sketches from the developed approach yields improved search results. [Display omitted] • The first learning-based strategy to clean rough query sketches of 3D CAD models • Introduces SketchCleanNet — an end-to-end image translation scheme • SketchCleanNet aims to understand the mapping between rough sketches and clean query images • A novel scheme to calculate the loss is introduced • Dataset Contribution: The resulting enhanced query sketch dataset is made available publicly. • This paper will significantly contribute to the research community and give researchers opportunities to develop new algorithms for search and retrieval of 3D mechanical components. [ABSTRACT FROM AUTHOR]

• An overview of deep learning algorithms used in medical image segmentation is presented. • More than 150 papers applying deep learning to different medical applications are summarised. • Challenges and future directions in medical image segmentation are discussed. Deep learning (DL) algorithms have rapidly become a robust tool for analyzing medical images. They have been used extensively for medical image segmentation as the first and significant components of the diagnosis and treatment pipeline. Medical image segmentation is efficiently addressed by many types of deep neural networks, such as convolutional neural networks, fully convolutional network recurrent networks, adversarial networks, and U-shaped networks. This paper reviews the major DL models and applications pertinent to medical image segmentation and summarizes over 150 contributions to the field. Brief overviews of articles are provided by application area: anatomical structures such as organs, bones, and vessels, and abnormalities such as lesions and calcification. Moreover, we discuss current challenges and suggest directions for future research. [ABSTRACT FROM AUTHOR]

The problem of global warming is getting more serious, and vehicle emission is the main cause. In recent years, the number of locomotives in China has been increasing at a rate of more than 20% per year, and the problem of automobile pollution is becoming more serious. The transportation industry is the main source of fossil fuel combustion and environmental pollution. Therefore, in this paper, we propose a multi-view deep neural network (MVDNN) to analyze the key factors affecting the fuel consumption of automobiles. The experiments show that the introduction of human input improves the prediction accuracy and the root mean square error (RMSE) achieves 0.993. In addition, this paper also finds that for drivers, driving habits, driving frequency, and safety awareness are the most important factors affecting the fuel consumption of vehicles by combining Lasso regression with MVDNN. Finally, by comparing the prediction accuracy of different experiments, relevant policy suggestions are made. [ABSTRACT FROM AUTHOR]

Two participants with competing objectives are considered in one-to-one orbital pursuit-evasion problem, one of which attempts to satisfy some certain terminal conditions with a minimum cost through its own control while the other player attempts to maximize the specified payoff and avoids entering this terminal range. This paper treats the problem of minimax time to a given final distance, within which the interception between pursuer and evader is defined to occur. To achieve the successful interception with an evader that adopts any unknown maneuvers, a near-optimal interception strategy for the orbital pursuit-evasion is presented in this paper. The pursuit-evasion problem is formulated as a differential game first, and a closed-form barrier solution is provided next. A near-optimal guidance law using deep learning is proposed to intercept the evader inside the capture zone. For the games that start outside the barrier, a learning algorithm for the capture zone embedding strategy is presented based on deep reinforcement learning to help the game state cross the barrier surfaces. The simulation cases of intercepting the evaders that adopt different maneuvering strategies are provided, and the results demonstrate the effectiveness and onboard feasibility of the proposed strategy in solving the orbital interception problem. • The closed-form barrier solution for a planar orbital pursuit-evasion game is provided. • A deep neural network is trained on a dataset consisting of plenty of barrier trajectories generated through the closed-form barrier formula. • The near-optimal guidance law inside capture zone is proposed using trained neural network. • A capture zone embedding strategy is presented based on deep reinforcement learning. • The closed loop interception strategy is proposed combining the strategies outside and inside capture zone. [ABSTRACT FROM AUTHOR]

Over the past decades, deep learning techniques have attracted increased attention from various research and industrial domains aligned with the development of Industry Internet-of-Things(IIoT). Specifically, with the advantage of data-driven methods, industrial organizations are seeking novel proactive strategies supported by analytic models to guarantee the quality of their production by observing degradation or predicting failure ahead of the occurrence of the component or asset. Predictive strategies are expected to promise the influence of unnecessary maintenance interruptions and mitigate the consequence of that, hence, extending the remaining useful life of products. This paper conducts a survey of the utilization of deep learning technologies on engineering applications where they provide satisfactory solutions with respect to specific data types or input signals. 106 primary papers are reviewed on deep learning–driven approaches which mainly explore five of the most popular architectures in the application of predictive maintenance. The main content of this paper summarizes the common advantages of each architecture and, accordingly, points out their limitations, as well as describes the application scopes of fully connected deep neural networks, convolutional neural networks, stacked autoencoders, deep belief networks, and deep recurrent neural networks. Based on the technique discussion for each of them, we intend to provide a comprehensive understanding and guidance of the appropriate usage of deep learning architectures to devise an effective predictive maintenance strategy for the scientific and industrial developers whose expertise lies in the prior domain knowledge of multi-source isomerization data. Moreover, the main content demonstrated the summarization of the decisive factor by which the incremental stages of the approaches were determined, fundamentally including the dataset specification, feature extraction, and the integration of deep learning approaches. [ABSTRACT FROM AUTHOR]

Electricity load exhibits daily and weekly cyclical patterns as well as random characteristics. At present, prevailing deep learning models cannot learn electricity load cyclical and stochastic features adequately. This results in insufficient prediction accuracy and the scalability of current methods. To tackle these difficulties, this paper proposes a framework for electrical load prediction based on an Attention Mechanism Time Series Depthwise Separable Convolutional Neural Network (ELPF-ATDSCN). The framework starts by using the Maximum Information Coefficient for exogenous variable selection. It then incorporates a seasonal decomposition algorithm with manual feature engineering to extract the cyclical and stochastic features of the electrical load. Subsequently, the framework employs the ATDSCN to learn the cyclical and stochastic features of the electrical load. In addition, the Bayesian algorithm optimizes model hyperparameters for optimal model performance. Experimental results of point and interval load prediction on datasets from the US and Nordic power markets reveal that the ATDSCN model proposed in this paper enhances load prediction accuracy compared with other models. It can provide more reliable predictions for power system operation and dispatch. • A new attention mechanism is designed to improve the model memory capability. • The ATDSCN model is proposed for learning periodic and stochastic features of electricity loads. • A framework for electricity load prediction is proposed to improve the accuracy of electricity load forecasting. • The effectiveness of the proposed method is validated on the real world dataset. [ABSTRACT FROM AUTHOR]

• A multi-physics 3D FEM is established for exciting electrodes of ECS installed around pipeline subjected to LTCTMF. • The LTCTMF behavior is studied at different levels of LTCTMF. • A theoretical model results of LTCTMF compliance over creep time are analyzed using the Modulus degradation Approach. • A deep learning model is designed to predict compliance for various LTCTMF levels not included in previous FEM evaluation. • The proposed method performance is estimated by computing the accuracy rate, regression rate, and F-score indexes. The composite pipeline is a relatively new and viable alternative pipeline to the more commonly used traditional one due to its good mechanical and fatigue properties and lower production cost. For this purpose, it is critical to assess the mechanical and fatigue performance of composite pipeline material under various working conditions, particularly for monitoring long-term creep thermo-mechanical fatigue behavior. In this paper, a long-term creep thermo-mechanical fatigue behavior in a basalt fiber reinforced polymer laminated composite pipeline is detected through an integrated expert system consisting of the electrical capacitance sensors and a deep learning algorithm. First, a multi-physics finite element model is established for the simulation of a long-term creep thermo-mechanical fatigue behavior in basalt fiber reinforced polymer composite pipelines subjected to long-term fatigue loading of internal pressure and thermal effect. Second, theoretical model results of long-term creep thermo-mechanical fatigue compliance (S f (t)) over the time of creep are analyzed in pipeline material using the modulus degradation approach. Finally, an electrical potential change between electrical capacitance sensors electrodes corresponding to S f (t) over the time of creep for some levels of long-term creep thermo-mechanical fatigue (R f %) is recorded and then used in these datasets for training of the novel deep neural network based on one of the most widely used of the deep neural network families is the convolutional neural network, to predict S f (t) in pipeline for various R f % not included in the previous finite element model evaluation (i.e. electrical capacitance sensors technique). In this paper first is detected the long-term creep thermo-mechanical fatigue behavior for R f % = 25 % , 50 % , and 75 % from a finite element model and modulus degradation approach, and then is predicted the long-term creep thermo-mechanical fatigue behavior for R f % = 15 % , 35 % , 60 % , and 85 % respectively via deep neural network. The proposed method results are in good agreement with the experimental results available in the literature, thus verifying the accuracy and reliability of the proposed technique and its applicability to other different composite structures. [ABSTRACT FROM AUTHOR]

• The proposed AP-GAN-DNN method is able to predict creep fracture life more accurately in the case of small samples, which solves the complex nonlinear problem and the problem of test data dispersion. • Compared with traditional physical methods and machine learning models, the AP-GAN-DNN model has better generalization and robustness for creep fracture life prediction. • AP-GAN data enhancement method can significantly reduce the "pattern collapse" effect of GAN and improve the performance of prediction algorithms. 7050 aluminum alloy has high strength and excellent fracture toughness. However, damage can be caused to the 7050 aluminum alloy by creep at elevated temperatures. The accuracy of conventional creep life prediction methods may fall short of meeting the desired standards. And the application of machine learning (ML) methods is limited due to the scarcity and dispersion of creep sample data. Generative Adversarial Networks (GAN) are often used for data enhancement, but suffer from "pattern collapse" under small sample conditions. To solve the above problems, this paper proposes a creep fracture life prediction method, which combines the affinity propagation (AP) clustering algorithm, GAN, and deep neural networks (DNN) for accurate creep fracture life prediction. The AP clustering algorithm for adaptive clustering was utilized to better reflect the creep sample distribution. Independent GAN models were trained for each cluster to better capture distribution characteristics of the data and generate synthetic data that was highly similar to the real data. The DNN models were trained using the synthetic data and then predicted using real creep fracture life data. The presented method was compared with traditional physical and machine learning methods, and the method combining K-Means, GAN, and DNN. The experimental results show that the method proposed in this paper has better prediction accuracy in a small sample creep fracture life dataset. [ABSTRACT FROM AUTHOR]

This paper introduces Elegans-AI models, a class of neural networks that leverage the connectome topology of the Caenorhabditis elegans to design deep and reservoir architectures. Utilizing deep learning models inspired by the connectome, this paper leverages the evolutionary selection process to consolidate the functional arrangement of biological neurons within their networks. The initial goal involves the conversion of natural connectomes into artificial representations. The second objective centers on embedding the complex circuitry topology of artificial connectomes into both deep learning and deep reservoir networks, highlighting their neural-dynamic short-term and long-term memory and learning capabilities. Lastly, our third objective aims to establish structural explainability by examining the heterophilic/homophilic properties within the connectome and their impact on learning capabilities. In our study, the Elegans-AI models demonstrate superior performance compared to similar models that utilize either randomly rewired artificial connectomes or simulated bio-plausible ones. Notably, these Elegans-AI models achieve a top-1 accuracy of 99.99% on both Cifar10 and Cifar100, and 99.84% on MNIST Unsup. They do this with significantly fewer learning parameters, particularly when reservoir configurations of the connectome are used. Our findings indicate a clear connection between bio-plausible network patterns, the small-world characteristic, and learning outcomes, emphasizing the significant role of evolutionary optimization in shaping the topology of artificial neural networks for improved learning performance. [ABSTRACT FROM AUTHOR]

The paper introduces a deep learning (DL) model as an alternative approach to predict the supercontinuum generation in waveguides. The traditional technique of using the numerical method for this task is computationally laborious and time-consuming. The Artificial Neural Network (ANN) model described in the paper is trained from scratch and demonstrates high precision in predicting the spectral bandwidth of supercontinuum phenomena. It achieves a mean squared error of 0.0032, which indicates its ability to accurately map the optical characteristics of the waveguide. Furthermore, the deep learning model significantly reduces the computational time required compared to simulative processes conducted in COMSOL Multiphysics and MATLAB software, approximately taking only 50% of the time. By leveraging deep learning techniques, the model offers a promising alternative to the numerical method for predicting supercontinuum generation in planar waveguides. It overcomes the limitations associated with computational resources and complexity, making it an attractive substitute for traditional numerical techniques. • Investigated chalcogenide planar waveguide, generating versatile supercontinuum spectra dataset. • Preprocessed dataset for Tanh activation function compatibility. • Developed multi-layered ANN model with decaying learning rate for predicting supercontinuum spectra. • Efficiently predicts spectral broadening at waveguide's end and different lengths. • Developed ANN model handles both linear and nonlinear waveguide dynamics effectively. [ABSTRACT FROM AUTHOR]

With the increasing complexity of the power grid, the precision of the thermal power units' execution of automatic generation control (AGC) commands is gradually increasing the impact on the online regulation of the power grid. The deviation between the actual output of thermal power units and the AGC command of the grid will not only affect the consumption of new energy output, but also endanger the safe operation of the grid. This paper introduces "deep learning" technology to solve the problem. Firstly, an AGC command execution effect identification and confidence evaluation algorithm (ACEEI-CEA) is proposed. The algorithm builds a neural network model to accurately predict the unit output and the confidence evaluation of the prediction results. Next, a high-dimensional input preprocessing strategy based on variational autoencoder (VAE) is proposed to reduce the dimensionality of the model input attributes, improving the convergence and accuracy of the model. Finally, an AGC optimal command fast inversion solution method (AOCFISM) is designed. This method transforms the unit output deviation problem into an objective optimisation problem. And improve the efficiency of solving the optimal AGC command value by constraining the unit command value. The calculation results show that the error of the prediction results of the model proposed in this paper is 5% lower than that of the traditional neural network. The difference between the output value of the optimal AGC command and the expected output value obtained is less than 0.5 MV, which can support AGC online decision-making.© 2017 Elsevier Inc. All rights reserved. • An AGC command execution effect identification and confidence evaluation algorithm (ACEEI-CEA) is proposed. • Results show the ACEEI-CEA performs well in command prediction compared to the state-of-the-art methods. • ACEEI-CEA achieves confidence assessment of predicted values. • The pre-processing strategy based on variational autoencoder shortens the training time of the model. • The AGC optimal command fast inversion solution method (AOCFISM) can obtain the AGC optimal command value. [ABSTRACT FROM AUTHOR]

In personal identity recognition systems, detecting a person's age, gender, and language using voice signal characteristics is a crucial issue, especially because of the importance of security considerations. Age, gender, and language classification problems are important in signal processing because they are used to analyze and understand human behavior, interactions, and preferences. This can be especially useful in the fields of human-computer interaction, psychology, and social science research. In this paper, a new system for detecting a speaker's age, gender, and language based on deep learning models is presented. Deep learning models have shown great efficacy in various fields of signal processing. For this paper, a range of deep models were tested, including convolutional neural networks (CNNs), recurrent neural network (RNN), and a fine-tuning ResNet34 architecture. Additionally, techniques such as transfer learning were applied to improve the efficiency of the proposed system. The input voice signals are preprocessed by applying the spectro-temporal transform to obtain additional features that can be fed to the ResNet34 model, which is designed specifically for the task of voice signal processing. The dataset used in this paper was sourced from the Mozilla common voice initiative, which is dedicated to advancing speech recognition and language identification technologies. The performance of the proposed algorithm was evaluated in the presence of Gaussian noise to determine its robustness. The experimental results demonstrated that the proposed algorithm significantly outperformed basic algorithms and other deep neural networks in terms of age and gender recognition from voice signals. [ABSTRACT FROM AUTHOR]

• This paper proposes an innovative approach that integrates neural networks with photoluminescence detection technology to address defects such as cracks, dirt, dark spots, and scratches in solar cells. • The YOLOv5 model undergoes optimization in three distinct stages, encompassing global optimization, neck network structure refinement, and head structure enhancement. These optimizations collectively elevate the model's capabilities in recognizing, detecting, and classifying defects in solar cells. • Advanced techniques are employed, including the introduction of structures such as Transformer Encoder, the incorporation of additional small target detection boxes, utilization of Bi-FPN structure, and insertion of CBAM structure blocks. These optimizations contribute to the improvement of the model's feature recognition and extraction capabilities, further enhancing its ability to identify defects. In the production process of solar cells, inevitable faults such as cracks, dirt, dark spots, and scratches may occur, which could potentially impact the lifespan and power generation efficiency of solar cells. Addressing this issue, this paper combines neural networks with photoluminescence detection technology and proposes a novel neural network model for the classification and grading of defects in solar cells. Firstly, the YOLOv5 model is optimized and adjusted for algorithm and network structure. The optimization process is divided into three parts: global optimization of the network structure, optimization of the neck network structure, and optimization of the head structure, each addressing specific issues in recognition, detection, and classification. The impact of the optimized network model on recognition and detection speed is analyzed, and solutions are proposed to address any observed effects. Additionally, an iterative update of neural network hyperparameter combinations is performed for solar cell defect identification. Finally, using the ultimately optimized model structure in conjunction with the optimal hyperparameter combination, comparative experiments are conducted on neural networks for different target identification using the photoluminescence characteristics dataset of solar cells. The recognition improvement of the optimized model and its differences from other models are analyzed. [ABSTRACT FROM AUTHOR]

Transformers are Deep Neural Networks (DNN) that utilize a self-attention mechanism to capture contextual relationships within sequential data. Unlike traditional neural networks and variants of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM), Transformer models excel at managing long dependencies among input sequence elements and facilitate parallel processing. Consequently, Transformer-based models have garnered significant attention from researchers in the field of artificial intelligence. This is due to their tremendous potential and impressive accomplishments, which extend beyond Natural Language Processing (NLP) tasks to encompass various domains, including Computer Vision (CV), audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published, spotlighting the Transformer's contributions in specific fields, architectural disparities, or performance assessments, there remains a notable absence of a comprehensive survey paper that encompasses its major applications across diverse domains. Therefore, this paper addresses this gap by conducting an extensive survey of proposed Transformer models spanning from 2017 to 2022. Our survey encompasses the identification of the top five application domains for Transformer-based models, namely: NLP, CV, multi-modality, audio and speech processing, and signal processing. We analyze the influence of highly impactful Transformer-based models within these domains and subsequently categorize them according to their respective tasks, employing a novel taxonomy. Our primary objective is to illuminate the existing potential and future prospects of Transformers for researchers who are passionate about this area, thereby contributing to a more comprehensive understanding of this groundbreaking technology. • The paper presents a comprehensive survey on transformers for deep learning tasks. • The paper conducts a thorough analysis on highly effective models in five domains. • The paper classifies the models based on respective tasks using a proposed taxonomy. • The characteristics of the surveyed models are deeply explored and analyzed. • Future directions and challenges for transformer-based models are deciphered. [ABSTRACT FROM AUTHOR]

Machine learning algorithms especially deep neural networks have seen tremendous growth in their real-world deployment. While these algorithms have to yield high performances for the task at hand, it is of equal importance that they are robust against the different kinds of singularities or adversarial anomalies. Researchers have generally addressed singularity detection independently for each anomaly, however, for the algorithms to be effective in the real world, the singularity detection algorithm must be able to handle a wide variety of attacks whether trained individually on them or not seen at the time of training. With this objective, in this paper, we propose a unique end-to-end transformation domain network to detect a variety of well-known attacks. The proposed architecture utilizes a combination of two different wavelet transformations to simultaneously learn low-level and high-level image features in a deep stacked layer fashion. The proposed method is generalized to handle a broad set of singularities of several computer vision algorithms whether operating in the object recognition domain or biometrics recognition. We showcase the results on a wide range of singularity points including (i) adversarial perturbations which are learned using deep learning networks, (ii) physical presentation attacks on face recognition which aim to produce fake data for the sensor for acquisition, and identification, (iii) synthetic images, and (iv) digital retouching of the images. Even with a zero-day or open-world attack setting, the results of the proposed algorithm show improvement over state-of-the-art results. The proposed architecture is agnostic to parameters, computationally efficient to provide a sustainable detector to resource-constrained institutions, provides robustness to attacks, and supports green computing. • A novel parameter-agnostic architecture is proposed to defend anomalies. • The proposed algorithm can handle adversaries in seen, unseen, and open-set settings. • The proposed algorithm is domain agnostic i.e., can handle vision and face datasets. • Our unified algorithm surpasses the SOTA works developed to handle specific attacks. • Bring a computationally feasible algorithm, it can be deployed on mobile devices. [ABSTRACT FROM AUTHOR]

The application of neural network models to solve combinatorial optimization has recently drawn much attention and shown promising results in dealing with similar problems, like Travelling Salesman Problem. The neural network allows to learn solutions based on given problem instances, using reinforcement learning or supervised learning. In this paper, we present a novel end-to-end method to solve routing problems. In specific, we propose a gated cosine-based attention model (GCAM) to train policies, which accelerates the training process and the convergence of policy. Extensive experiments on different scale of routing problems show that the proposed method can achieve faster convergence of the training process than the state-of-the-art deep learning models while achieving solutions of the same quality. • We proposed a cosine-based neural network model based on the transformer architecture to improve node embeddings. • We use cosine decomposability to accelerate training speed of the proposed model. • We use a gating connection to stabilize training and get faster convergence, which leads to better overall performance. • We apply the proposed gated cosine-based attention model to solve several Routing Problems. [ABSTRACT FROM AUTHOR]

• We analyzed older driver at-fault crashes at unsignalized intersections in Alabama. • We estimated random parameter models. • We introduced a deep learning approach based on Artificial Neural Networks. • We identified variables that increase the likelihood of injuries and fatalities. • We discussed possible countermeasures to address older driver at-fault crashes. Introduction: The research described in this paper explored the factors contributing to the injury severity resulting from the male and female older driver (65 years and older) at-fault crashes at unsignalized intersections in Alabama. Method: Random parameter logit models of injury severity were estimated. The estimated models identified a variety of statistically significant factors influencing the injury severities resulting from older driver at-fault crashes. Results : According to these models, some variables were found to be significant only in one model (male or female) but not in the other one. For example, variables such as driver under the influence of alcohol/drugs, horizontal curve, and stop sign were found significant only in the male model. On the other hand, variables such as intersection approaches on tangents with flat grade, and driver older than 75 years were found significant only in the female model. In addition, variables such as making turning maneuver, freeway-ramp junction, high speed approach, and so forth were found significant in both models. Estimation findings showed that two parameters in the male model and another two parameters in the female model could be modeled as random parameters, indicating their varying influences on the injury severity due to unobserved effects. In addition to the random parameter logit approach, a deep learning approach based on Artificial Neural Networks was introduced to predict the outcome of the crashes based on 164 variables that are listed in the crash database. The artificial intelligence (AI)-based method achieved an accuracy of 76% indicating the role of the variables in deciding the final outcome. Practical Applications: Future plans are set to study the use of AI on large sized datasets to achieve a relatively high-performance, and hence to be able to identify which variables contribute the most to the final outcome. [ABSTRACT FROM AUTHOR]

The paper presents an efficient and robust data-driven deep learning (DL) computational framework developed for linear continuum elasticity problems. The methodology is based on the fundamentals of the Physics Informed Neural Networks (PINNs). For an accurate representation of the field variables, a multi-objective loss function is proposed. It consists of terms corresponding to the residual of the governing partial differential equations (PDE), constitutive relations derived from the governing physics, various boundary conditions, and data-driven physical knowledge fitting terms across randomly selected collocation points in the problem domain. To this end, multiple densely connected independent artificial neural networks (ANNs), each approximating a field variable, are trained to obtain accurate solutions. Several benchmark problems including the Airy solution to elasticity and the Kirchhoff–Love plate problem are solved. Performance in terms of accuracy and robustness illustrates the superiority of the current framework showing excellent agreement with analytical solutions. The present work combines the benefits of the classical methods depending on the physical information available in analytical relations with the superior capabilities of the DL techniques in the data-driven construction of lightweight, yet accurate and robust neural networks. The models developed herein can significantly boost computational speed using minimal network parameters with easy adaptability in different computational platforms. • A physics-aware deep learning-based accelerated computational framework has been developed for solving linear elasticity problems. • A multi-objective loss functional is proposed, when minimized, can accurately predict elastic field variables. • Several benchmark problems are solved illustrating the usefulness of the model showing excellent agreement with analytical solutions. • Current study demonstrates the applicability of data-driven enhancement using the transfer learning-based approach in reducing training time, simultaneously improving the accuracy of the model. [ABSTRACT FROM AUTHOR]

Crude oil and natural gas are the primary energy sources, mainly transported by pipelines. Pipeline safety has to be seriously considered to ensure the continuous and stable transportation of these two types of energy sources. The burst pressure is an important indicator of pipeline safety. Accurate prediction of the burst pressure is of great significance to the design, construction, daily operation, and maintenance of the pipeline. This paper proposes a theory-guided neural network model-based method to predict burst pressure prediction of corroded pipelines, which can incorporate physical principles into the deep learning framework. First, higher-order features with physical meaning are constructed and coupled with the original features to form a new feature space. Then the traditional burst pressure prediction formula Pipeline Corrosion Criterion (PCORRC) is integrated into the model to make full use of the prior knowledge contained in the empirical formula. The designed loss function enables the network to have different weights for different samples and focuses on learning the PCORRC formula to predict samples with large deviations. Finally, the model was verified using a public dataset based on experiments and finite element simulations. The results show that the theory-guided neural network model proposed in this paper has the highest accuracy compared with other models. The correlation coefficient is 0.9945, the root mean square error is 0.562, and the mean absolute percentage error is 2.65%. Further tests have shown that the model is very robust and has good adaptability to different data. This work presented that integrating domain knowledge into the traditional neural network model can effectively improve the performance of burst pressure prediction of the corroded pipeline. [ABSTRACT FROM AUTHOR]

In this work, we examine the advantages of using multiple types of behaviours in recommendation systems. Intuitively, each user often takes some implicit actions (e.g., click) before making an explicit decision (e.g., purchase). Previous studies show that implicit and explicit feedback has different roles for a useful recommendation. However, these studies either exploit implicit and explicit behaviours separately or ignore the semantics of sequential interactions between users and items. In addition, we go from the hypothesis that a user's preferences at a time are combinations of long-term and short-term interests. In this paper, we propose some Deep Learning architectures. The first one is Implicit to Explicit (ITE) , to exploit users' interests through the sequence of their actions. The second and third ones are two versions of ITE with Bidirectional Encoder Representations from Transformers based (BERT-based) architecture called BERT-ITE and BERT-ITE-Si , which combine users' long- and short-term preferences without and with side information to enhance users' representations. The experimental results show that our models outperform previous state-of-the-art ones and also demonstrate our views on the effectiveness of exploiting the implicit to explicit order as well as combining long- and short-term preferences in three large-scale datasets. The source code of our paper is available at: https://github.com/tranquyenbk173/BERT_ITE. [ABSTRACT FROM AUTHOR]

This paper presents a federated learning approach based on utilizing computational resources of the IoT edge devices for training deep neural networks. In this approach, the edge devices and the cloud server collaborate in the training phase while preserving the privacy of the edge device data. Owing to the limited computational power and resources available to the edge devices, instead of the original neural network (NN), we suggest to use a smaller NN generated using a proposed heuristic method. In the proposed approach, the smaller model, which is trained on the edge device, is generated from the main NN model. By the exploiting Knowledge Distillation (K D) approach, the learned knowledge in the server and the edge devices can be exchanged, leading to lower required computation on the server and preserving data privacy of the edge devices. Also, to reduce the knowledge transfer overhead on the communication links between the server and the edge devices, a method for selecting the most valuable data to transfer the knowledge is introduced. The effectiveness of this method is assessed by comparing it to state-of-the-art methods. The results show that the proposed method lowers the communication traffic by up to 250 × and increases the learning accuracy by an average of 8.9 % in the cloud compared to the prior K D -based distributed training approaches in CIFAR-10 dataset. [ABSTRACT FROM AUTHOR]

As a powerful data modeling tool, Heterogeneous Information Network (HIN) has been successfully used in auxiliary information exploitation to boost recommendation performance. For HIN based recommendation, it is challenging to extract and fuse useful features of user preferences and item attributes under different semantic paths in HINs. Existing methods leverage a pre-defined fusion function to integrate different semantics for recommendation, which cannot characterize the complex nonlinear interactions between users and items. In this paper, we present a general framework named MNRec, short for Meta-path fusion based Neural Recommendation, to extract and fuse user and item embeddings under different meta-paths for recommendation. Under the framework, we propose an instantiation of MNRec with Multi-Layer Perceptron (MLP) structure. It consists of two major steps, i.e., meta-path based heterogeneous network embedding and deep learning based rating prediction. Concretely, appropriate meta-paths are first designed according to domain knowledge. Then the embeddings of users and items are obtained through a meta-path and commuting matrix based heterogeneous network embedding method. Finally, in light of the powerful nonlinear modeling capabilities of deep neural networks, the learned embeddings under different meta-paths are integrated into a two-pathway MLP structure for rating prediction. Experimental results on three real-world datasets demonstrate the superiority and effectiveness of MNRec compared with state-of-the-art baselines in rating prediction. [ABSTRACT FROM AUTHOR]

Knowledge resources, e.g. knowledge graphs, which formally represent essential semantics and information for logic inference and reasoning, can compensate for the unawareness nature of many natural language processing techniques based on deep neural networks. This paper provides a focused review of the emerging but intriguing topic that fuses quality external knowledge resources in improving the performance of natural language processing tasks. Existing methods and techniques are summarised in three main categories: (1) static word embeddings, (2) sentence-level deep learning models, and (3) contextualised language representation models, depending on when, how and where external knowledge is fused into the underlying learning models. We focus on the solutions to mitigate two issues: knowledge inclusion and inconsistency between language and knowledge. Details on the design of each representative method, as well as their strength and limitation, are discussed. We also point out some potential future directions in view of the latest trends in natural language processing research. • First comprehensive review on knowledge fusion into deep learning for NLP. • A taxonomy for the related research and extensive overview of the state-of-the-art. • A focus on the theoretical formulation and optimisation methods. • Harmonise the mathematical notations across a plethora of publications. [ABSTRACT FROM AUTHOR]