Search

• Five Machine Learning (ML) models compared for wave height time series prediction. • Complex ML models do not outperform simple AR in wave height time series prediction. • Comment to related papers: signal decomposition in test set series is WRONG. Significant Wave Height (SWH) is crucial in many aspect of ocean engineering. The accurate prediction of SWH has therefore been of immense practical value. Recently, Artificial Intelligence (AI) time series prediction methods have been widely used for single-point short-term SWH time-series forecasting, resulting in many AI-based models claiming to achieve good results. However, the extent to which these complex AI models can outperform traditional methods has largely been overlooked. This study compared five different models - AutoRegressive (AR), eXtreme Gradient Boosting (XGB), Artificial Neural Network (ANN), Long Short-Term Memory (LSTM), and WaveNet - for their performance on SWH time series prediction at 16 buoy locations. Surprisingly, the results suggest that the differences of performance among different models are negligible, indicating that all these AI models have only "learned" the linear auto-regression from the data. Additionally, we noticed that many recent studies used signal decomposition method for such time series prediction, and most of them decomposed the test sets, which is WRONG. [ABSTRACT FROM AUTHOR]

The accidental leakage of gaseous hydrogen from hydrogen refueling station facilities has the potential to result in a large fire and explosion catastrophe. The ability to forecast the spatiotemporal evolution of hydrogen plumes in real-time is crucial for monitoring the distribution of plume concentrations and mitigating the risk of fire and explosion. The utilization of deep learning has been extensively employed in a diverse range of spatiotemporal forecasting tasks. However, these approaches encounter limitations in accurately quantifying uncertainty when predicting spatiotemporal concentrations and plume boundaries. This research paper introduces a novel model called DPSTNN_H 2 Evolution, which is a deep probabilistic spatial-temporal neural network designed for forecasting the spatiotemporal evolution of hydrogen plumes. The benchmark dataset is constructed by the implementation of numerical simulations pertaining to the inadvertent leak of hydrogen within a hydrogen refueling station. The findings of the study indicate that incorporating quantified uncertainty information might enhance the precision of plume boundaries and the resilience of plume concentrations when predicting the spatiotemporal development of hydrogen. By employing Monte Carlo sampling with a sample size of m = 100 and a dropout rate of p = 0.1 , the model demonstrates the ability to provide real-time inference for 10 instances of plumes, while maintaining a high level of accuracy with R2 = 0.9829. In comparison to the current state-of-the-art model, the proposed model demonstrates superior accuracy and robustness in forecasting the spatiotemporal evolution of hydrogen plumes. In general, our proposed model has the potential to serve as a dependable option for the development of a digital twin for emergency management of hydrogen refueling stations. • Probabilistic spatial-temporal neural network model for hydrogen plume evolutions is proposed. • Normalized uncertainty contours and uncertainty intervals to improve accuracy and robustness. • Numerical experiment of hydrogen leakage and dispersion in refueling station are conducted. • Optimal hyper-parameters of the proposed model are determined • Comparison between the proposed model with point-estimation based model is conducted [ABSTRACT FROM AUTHOR]

While deep neural networks have shown impressive results in automatic speaker recognition and related tasks, it is dissatisfactory how little is understood about what exactly is responsible for these results. Part of the success has been attributed in prior work to their capability to model supra-segmental temporal information (SST), i.e., learn rhythmic-prosodic characteristics of speech in addition to spectral features. In this paper, we (i) present and apply a novel test to quantify to what extent the performance of state-of-the-art neural networks for speaker recognition can be explained by modeling SST; and (ii) present several means to force respective nets to focus more on SST and evaluate their merits. We find that a variety of CNN- and RNN-based neural network architectures for speaker recognition do not model SST to any sufficient degree, even when forced. The results provide a highly relevant basis for impactful future research into better exploitation of the full speech signal and give insights into the inner workings of such networks, enhancing explainability of deep learning for speech technologies. • Literature explains speaker recognition in neural nets by modeling of voice dynmaics. • Diagnostic: We quantify how well deep learning models actually capture dynamics. • Observation: State-of-the-art deep nets do not model speaker prosody but ignore it. • Interpretation as "cheating": Achieving high without putting in due effort. • Outlook: Increasing task difficulty biases models towards prosody, but not enough. [ABSTRACT FROM AUTHOR]

Large deep learning models are impressive, but they struggle when real-time data is not available. Few-shot class-incremental learning (FSCIL) poses a significant challenge for deep neural networks to learn new tasks from just a few labeled samples without forgetting the previously learned ones. This setup can easily leads to catastrophic forgetting and overfitting problems, severely affecting model performance. Studying FSCIL helps overcome deep learning model limitations on data volume and acquisition time, while improving practicality and adaptability of machine learning models. This paper provides a comprehensive survey on FSCIL. Unlike previous surveys, we aim to synthesize few-shot learning and incremental learning, focusing on introducing FSCIL from two perspectives, while reviewing over 30 theoretical research studies and more than 20 applied research studies. From the theoretical perspective, we provide a novel categorization approach that divides the field into five subcategories, including traditional machine learning methods, meta learning-based methods, feature and feature space-based methods, replay-based methods, and dynamic network structure-based methods. We also evaluate the performance of recent theoretical research on benchmark datasets of FSCIL. From the application perspective, FSCIL has achieved impressive achievements in various fields of computer vision such as image classification, object detection, and image segmentation, as well as in natural language processing and graph. We summarize the important applications. Finally, we point out potential future research directions, including applications, problem setups, and theory development. Overall, this paper offers a comprehensive analysis of the latest advances in FSCIL from a methodological, performance, and application perspective. • In-depth survey on FSCIL methods, applications, performance. • Categorizing FSCIL into five subcategories for clear analysis. • Evaluating FSCIL research on benchmarks for strengths, weaknesses. • FSCIL applications in computer vision, NLP, and graph. [ABSTRACT FROM AUTHOR]

Deep neural networks have achieved outstanding performance in computer vision tasks. Convolutional Neural Networks (CNNs) typically operate in the spatial domain with raw images, but in practice, images are usually stored and transmitted in their compressed representation where JPEG is one of the most widely used encoder. Also, these networks are computationally intensive and slow. This paper proposes performing the learning and inference processes in the compressed domain in order to reduce the computational complexity and improve the speed of popular CNNs. For this purpose, a novel graph-based frequency channel selection method is proposed to identify and select the most important frequency channels. The computational complexity is reduced by retaining the important frequency components and discarding the insignificant ones as well as eliminating the unnecessary layers of the network. Experimental results show that the modified ResNet-50 operating in the compressed domain is up to 70% faster than the spatial-based traditional ResNet-50 while resulting in similar classification accuracy. Moreover, this paper proposes a preprocessing step with partial encoding to improve the resilience to distortions caused by low-quality encoded images. Finally, we show that training a network with highly compressed data can achieve a good classification accuracy with up to 93% reduction in the storage requirements of the training data. [ABSTRACT FROM AUTHOR]

Despite the wide variety of methods developed for synthetic image attribution, most of them can only attribute images generated by models or architectures included in the training set and do not work with unknown architectures, hindering their applicability in real-world scenarios. In this paper, we propose a verification framework that relies on a Siamese Network to address the problem of open-set attribution of synthetic images to the architecture that generated them. We consider two different settings. In the first setting, the system determines whether two images have been produced by the same generative architecture or not. In the second setting, the system verifies a claim about the architecture used to generate a synthetic image, utilizing one or multiple reference images generated by the claimed architecture. The main strength of the proposed system is its ability to operate in both closed and open-set scenarios so that the input images, either the query and reference images, can belong to the architectures considered during training or not. Experimental evaluations encompassing various generative architectures such as GANs, diffusion models, and transformers, focusing on synthetic face image generation, confirm the excellent performance of our method in both closed and open-set settings, as well as its strong generalization capabilities. • New verification framework for open-set architecture attribution of synthetic images. • Tested with several types of generative architectures in closed and open set scenarios. • Generalization tests prove that the system can verify unknown models of the architecture. • Outperforms state-of-the-art methods for open-set architecture attribution with rejection. [ABSTRACT FROM AUTHOR]

Artificial Intelligence has found wide application, but also poses risks due to unintentional or malicious tampering during deployment. Regular checks are therefore necessary to detect and prevent such risks. Fragile watermarking is a technique used to identify tampering in AI models. However, previous methods have faced challenges including risks of omission, additional information transmission, and inability to locate tampering precisely. In this paper, we propose a method for detecting tampered parameters and bits, which can be used to detect, locate, and restore parameters that have been tampered with. We also propose an adaptive embedding method that maximizes information capacity while maintaining model accuracy. Our approach was tested on multiple neural networks subjected to attacks that modified weight parameters, and our results demonstrate that our method achieved great recovery performance when the modification rate was below 20%. Furthermore, for models where watermarking significantly affected accuracy, we utilized an adaptive bit technique to recover more than 15% of the accuracy loss of the model. • Fragile watermarking for rigorous neural network integrity detection. • Watermarking ensures 100% tampering detection, precise location, and recovery. • Watermarking fragility by combining Mutual- and Self-neural net parameters. • Incorporates watermarking with neural net feature, maintaining watermarked models' performance via gradient descent. • Bit-level training controls step size with carefully set hyperparameters. [ABSTRACT FROM AUTHOR]

• A Digital Cultural Heritage dataset of synthetic vaults exported as point cloud, based on shape grammar discussion. • A vaults synthetic dataset creation method for automatically generating training data when real 3D objects are insufficient. • A Deep Learning system able to segment and distinguish types of architectural vaults according to standard classifications. Historical heritage is demanding robust pipelines for preserving, enhancing, and disseminating its prominent value. Semantic segmentation of 3D Point Clouds has gained increasing attention over the years, since it might assist in automati-cally recognizing historical architectural elements, thus facilitating large dataset management. Nonetheless, semantic segmentation is particularly challenging in Cultural Heritage (CH) domain, due to the shapes complexity and the limited repeatability of elements across different architectures, which strengthens the difficulty to define common patterns within the same class of elements. Besides, as Deep Neural Networks demand an appreciably amount of labelled data to be trained, the lack of available annotated heritage point clouds prevent the research in this direction. To tackle these issues, in this paper it is proposed a Deep Learning system able to recognize historical building elements by lever-aging synthetic point cloud. The generation of the 3D models, vaults, is based on a procedural modeling approach that follows the ideal shapes, according to the rules of descriptive geometry for the main types of vaults. The approach has been applied to a newly synthetic dataset which is publicly available. This dataset comprises 6 labelled points clouds, derived from a comprehensive on-tological taxonomy in order to describe an univocal and robust architectural hierarchy: barrel vaults, groined vaults, mirror vaults, barrel vaults with clois-ter heads and lunettes, barrel vaults with lunettes, sail vaults. The experiments yield high accuracy, demonstrating the effectiveness and suitability of the pro-posed approach. [ABSTRACT FROM AUTHOR]

Recent years have witnessed increasing interest in adversarial attacks on images, while adversarial video attacks have seldom been explored. In this paper, we propose a sparse adversarial attack strategy on videos (DeepSAVA). Our model aims to add a small human-imperceptible perturbation to the key frame of the input video to fool the classifiers. To carry out an effective attack that mirrors real-world scenarios, our algorithm integrates spatial transformation perturbations into the frame. Instead of using the l p norm to gauge the disparity between the perturbed frame and the original frame, we employ the structural similarity index (SSIM), which has been established as a more suitable metric for quantifying image alterations resulting from spatial perturbations. We employ a unified optimisation framework to combine spatial transformation with additive perturbation, thereby attaining a more potent attack. We design an effective and novel optimisation scheme that alternatively utilises Bayesian Optimisation (BO) to identify the most critical frame in a video and stochastic gradient descent (SGD) based optimisation to produce both additive and spatial-transformed perturbations. Doing so enables DeepSAVA to perform a very sparse attack on videos for maintaining human imperceptibility while still achieving state-of-the-art performance in terms of both attack success rate and adversarial transferability. Furthermore, built upon the strong perturbations produced by DeepSAVA, we design a novel adversarial training framework to improve the robustness of video classification models. Our intensive experiments on various types of deep neural networks and video datasets confirm the superiority of DeepSAVA in terms of attacking performance and efficiency. When compared to the baseline techniques, DeepSAVA exhibits the highest level of performance in generating adversarial videos for three distinct video classifiers. Remarkably, it achieves an impressive fooling rate ranging from 99.5% to 100% for the I3D model, with the perturbation of just a single frame. Additionally, DeepSAVA demonstrates favourable transferability across various time series models. The proposed adversarial training strategy is also empirically demonstrated with better performance on training robust video classifiers compared with the state-of-the-art adversarial training with projected gradient descent (PGD) adversary. • Sparse attacks on video models: perturb fewer frames to gain high fooling rate. • Combining additive and spatial perturbations to enhance attacking performance. • Using SSIM instead of l p -norm to maintain the human perception. • Applying Bayesian Optimisation to identify the most critical frame to perturb. • A new adversarial training method based on combination of diverse perturbations. [ABSTRACT FROM AUTHOR]

We study how the topology of feature embedding space changes as it passes through the layers of a well-trained deep neural network (DNN) through Betti numbers. Motivated by existing studies using simplicial complexes on shallow fully connected networks (FCN), we present an extended analysis using Cubical homology instead, with a variety of popular deep architectures and real image datasets. We demonstrate that as depth increases, a topologically complicated dataset is transformed into a simple one, resulting in Betti numbers attaining their lowest possible value. The rate of decay in topological complexity (as a metric) helps quantify the impact of architectural choices on the generalization ability. Interestingly from a representation learning perspective, we highlight several invariances such as topological invariance of (1) an architecture on similar datasets; (2) embedding space of a dataset for architectures of variable depth; (3) embedding space to input resolution/size, and (4) data sub-sampling. In order to further demonstrate the link between expressivity & the generalization capability of a network, we consider the task of ranking pre-trained models for downstream classification task (transfer learning). Compared to existing approaches, the proposed metric has a better correlation to the actually achievable accuracy via fine-tuning the pre-trained model. • This paper categorizes the change in topological complexity through Betti-Numbers. • Attempts to characterize the learning ability of the DNNs through topological transformation of feature space. • Attempts to use the topological complexity of feature space for the task of ranking pre-trained models. [ABSTRACT FROM AUTHOR]

Deep neural networks are highly vulnerable to adversarial samples. Most existing adversarial defense methods do not consider the distribution of adversarial samples. We argue that very few adversarial samples in the natural sample set prevent the deep neural networks from learning a complete and effective representation of the adversarial samples. This causes the spatial structures between the natural and the adversarial samples to be vastly different from that of the input space, thus making the models vulnerable to adversarial attacks. Based on this viewpoint, this paper proposes an effective deep-learning adversarial defense method, which incorporates information about the spatial structures of the natural and the adversarial samples in the embedding space during the training process. This proposed approach improves the deep learning model's generalization to new adversarial samples and achieves the purpose of defending against adversarial attacks. Four deep neural networks with different scales are used and experimentally verified on four typical publicly available image data. The experimental results show that our method effectively improves the defense ability of deep learning models against adversarial attacks. • Development of an effective adversarial training method using spatial structure constraints in embedding space. • Spatial structure information of the natural and adversarial samples can help improve the adversarial robustness of DNNs. • The spatial structure constrain can be extended to adversarial training of semi-supervised and unsupervised learning. [ABSTRACT FROM AUTHOR]

Supervised learning in deep neural networks is commonly performed using error backpropagation. However, the sequential propagation of errors during the backward pass limits its scalability and applicability to low-powered neuromorphic hardware. Therefore, there is growing interest in finding local alternatives to backpropagation. Recently proposed methods based on forward-mode automatic differentiation suffer from high variance in large deep neural networks, which affects convergence. In this paper, we propose the Forward Direct Feedback Alignment algorithm that combines Activity-Perturbed Forward Gradients with Direct Feedback Alignment and momentum. We provide both theoretical proofs and empirical evidence that our proposed method achieves lower variance than forward gradient techniques. In this way, our approach enables faster convergence and better performance when compared to other local alternatives to backpropagation and opens a new perspective for the development of online learning algorithms compatible with neuromorphic systems. • Forward Gradient methods suffer from high variance that hinders convergence. • Activity-Perturbed Forward Gradients can be used to learn derivatives as direct feedback connections. • Feedback learning acts as a momentum that reduces the gradient variance closer to backpropagation. • Feedback learning reduces the biasness of Direct Feedback Alignment. • Feedback learning enables learning in convolutional layers. [ABSTRACT FROM AUTHOR]

Sharpness aware minimization (SAM) optimizer has been extensively explored as it can generalize better for training deep neural networks via introducing extra perturbation steps to flatten the landscape of deep learning models. Integrating SAM with adaptive learning rate and momentum acceleration, dubbed AdaSAM, has already been explored empirically to train large-scale deep neural networks without theoretical guarantee due to the triple difficulties in analyzing the coupled perturbation step, adaptive learning rate and momentum step. In this paper, we try to analyze the convergence rate of AdaSAM in the stochastic non-convex setting. We theoretically show that AdaSAM admits a O (1 / b T) convergence rate, which achieves linear speedup property with respect to mini-batch size b. Specifically, to decouple the stochastic gradient steps with the adaptive learning rate and perturbed gradient, we introduce the delayed second-order momentum term to decompose them to make them independent while taking an expectation during the analysis. Then we bound them by showing the adaptive learning rate has a limited range, which makes our analysis feasible. To the best of our knowledge, we are the first to provide the non-trivial convergence rate of SAM with an adaptive learning rate and momentum acceleration. At last, we conduct several experiments on several NLP tasks and the synthetic task, which show that AdaSAM could achieve superior performance compared with SGD, AMSGrad, and SAM optimizers. • Adaptive SAM with momentum acceleration ensures automatic learning rate adjustment. • Adaptive SAM shows linear speedup, converging at O (1 / b T). • Delaying 2nd-order momentum separates grad from adaptive rate & perturbed gradients. [ABSTRACT FROM AUTHOR]

Over the past decades, deep learning techniques have attracted increased attention from various research and industrial domains aligned with the development of Industry Internet-of-Things(IIoT). Specifically, with the advantage of data-driven methods, industrial organizations are seeking novel proactive strategies supported by analytic models to guarantee the quality of their production by observing degradation or predicting failure ahead of the occurrence of the component or asset. Predictive strategies are expected to promise the influence of unnecessary maintenance interruptions and mitigate the consequence of that, hence, extending the remaining useful life of products. This paper conducts a survey of the utilization of deep learning technologies on engineering applications where they provide satisfactory solutions with respect to specific data types or input signals. 106 primary papers are reviewed on deep learning–driven approaches which mainly explore five of the most popular architectures in the application of predictive maintenance. The main content of this paper summarizes the common advantages of each architecture and, accordingly, points out their limitations, as well as describes the application scopes of fully connected deep neural networks, convolutional neural networks, stacked autoencoders, deep belief networks, and deep recurrent neural networks. Based on the technique discussion for each of them, we intend to provide a comprehensive understanding and guidance of the appropriate usage of deep learning architectures to devise an effective predictive maintenance strategy for the scientific and industrial developers whose expertise lies in the prior domain knowledge of multi-source isomerization data. Moreover, the main content demonstrated the summarization of the decisive factor by which the incremental stages of the approaches were determined, fundamentally including the dataset specification, feature extraction, and the integration of deep learning approaches. [ABSTRACT FROM AUTHOR]

Electricity load exhibits daily and weekly cyclical patterns as well as random characteristics. At present, prevailing deep learning models cannot learn electricity load cyclical and stochastic features adequately. This results in insufficient prediction accuracy and the scalability of current methods. To tackle these difficulties, this paper proposes a framework for electrical load prediction based on an Attention Mechanism Time Series Depthwise Separable Convolutional Neural Network (ELPF-ATDSCN). The framework starts by using the Maximum Information Coefficient for exogenous variable selection. It then incorporates a seasonal decomposition algorithm with manual feature engineering to extract the cyclical and stochastic features of the electrical load. Subsequently, the framework employs the ATDSCN to learn the cyclical and stochastic features of the electrical load. In addition, the Bayesian algorithm optimizes model hyperparameters for optimal model performance. Experimental results of point and interval load prediction on datasets from the US and Nordic power markets reveal that the ATDSCN model proposed in this paper enhances load prediction accuracy compared with other models. It can provide more reliable predictions for power system operation and dispatch. • A new attention mechanism is designed to improve the model memory capability. • The ATDSCN model is proposed for learning periodic and stochastic features of electricity loads. • A framework for electricity load prediction is proposed to improve the accuracy of electricity load forecasting. • The effectiveness of the proposed method is validated on the real world dataset. [ABSTRACT FROM AUTHOR]

• A multi-physics 3D FEM is established for exciting electrodes of ECS installed around pipeline subjected to LTCTMF. • The LTCTMF behavior is studied at different levels of LTCTMF. • A theoretical model results of LTCTMF compliance over creep time are analyzed using the Modulus degradation Approach. • A deep learning model is designed to predict compliance for various LTCTMF levels not included in previous FEM evaluation. • The proposed method performance is estimated by computing the accuracy rate, regression rate, and F-score indexes. The composite pipeline is a relatively new and viable alternative pipeline to the more commonly used traditional one due to its good mechanical and fatigue properties and lower production cost. For this purpose, it is critical to assess the mechanical and fatigue performance of composite pipeline material under various working conditions, particularly for monitoring long-term creep thermo-mechanical fatigue behavior. In this paper, a long-term creep thermo-mechanical fatigue behavior in a basalt fiber reinforced polymer laminated composite pipeline is detected through an integrated expert system consisting of the electrical capacitance sensors and a deep learning algorithm. First, a multi-physics finite element model is established for the simulation of a long-term creep thermo-mechanical fatigue behavior in basalt fiber reinforced polymer composite pipelines subjected to long-term fatigue loading of internal pressure and thermal effect. Second, theoretical model results of long-term creep thermo-mechanical fatigue compliance (S f (t)) over the time of creep are analyzed in pipeline material using the modulus degradation approach. Finally, an electrical potential change between electrical capacitance sensors electrodes corresponding to S f (t) over the time of creep for some levels of long-term creep thermo-mechanical fatigue (R f %) is recorded and then used in these datasets for training of the novel deep neural network based on one of the most widely used of the deep neural network families is the convolutional neural network, to predict S f (t) in pipeline for various R f % not included in the previous finite element model evaluation (i.e. electrical capacitance sensors technique). In this paper first is detected the long-term creep thermo-mechanical fatigue behavior for R f % = 25 % , 50 % , and 75 % from a finite element model and modulus degradation approach, and then is predicted the long-term creep thermo-mechanical fatigue behavior for R f % = 15 % , 35 % , 60 % , and 85 % respectively via deep neural network. The proposed method results are in good agreement with the experimental results available in the literature, thus verifying the accuracy and reliability of the proposed technique and its applicability to other different composite structures. [ABSTRACT FROM AUTHOR]

• The proposed AP-GAN-DNN method is able to predict creep fracture life more accurately in the case of small samples, which solves the complex nonlinear problem and the problem of test data dispersion. • Compared with traditional physical methods and machine learning models, the AP-GAN-DNN model has better generalization and robustness for creep fracture life prediction. • AP-GAN data enhancement method can significantly reduce the "pattern collapse" effect of GAN and improve the performance of prediction algorithms. 7050 aluminum alloy has high strength and excellent fracture toughness. However, damage can be caused to the 7050 aluminum alloy by creep at elevated temperatures. The accuracy of conventional creep life prediction methods may fall short of meeting the desired standards. And the application of machine learning (ML) methods is limited due to the scarcity and dispersion of creep sample data. Generative Adversarial Networks (GAN) are often used for data enhancement, but suffer from "pattern collapse" under small sample conditions. To solve the above problems, this paper proposes a creep fracture life prediction method, which combines the affinity propagation (AP) clustering algorithm, GAN, and deep neural networks (DNN) for accurate creep fracture life prediction. The AP clustering algorithm for adaptive clustering was utilized to better reflect the creep sample distribution. Independent GAN models were trained for each cluster to better capture distribution characteristics of the data and generate synthetic data that was highly similar to the real data. The DNN models were trained using the synthetic data and then predicted using real creep fracture life data. The presented method was compared with traditional physical and machine learning methods, and the method combining K-Means, GAN, and DNN. The experimental results show that the method proposed in this paper has better prediction accuracy in a small sample creep fracture life dataset. [ABSTRACT FROM AUTHOR]

This paper introduces Elegans-AI models, a class of neural networks that leverage the connectome topology of the Caenorhabditis elegans to design deep and reservoir architectures. Utilizing deep learning models inspired by the connectome, this paper leverages the evolutionary selection process to consolidate the functional arrangement of biological neurons within their networks. The initial goal involves the conversion of natural connectomes into artificial representations. The second objective centers on embedding the complex circuitry topology of artificial connectomes into both deep learning and deep reservoir networks, highlighting their neural-dynamic short-term and long-term memory and learning capabilities. Lastly, our third objective aims to establish structural explainability by examining the heterophilic/homophilic properties within the connectome and their impact on learning capabilities. In our study, the Elegans-AI models demonstrate superior performance compared to similar models that utilize either randomly rewired artificial connectomes or simulated bio-plausible ones. Notably, these Elegans-AI models achieve a top-1 accuracy of 99.99% on both Cifar10 and Cifar100, and 99.84% on MNIST Unsup. They do this with significantly fewer learning parameters, particularly when reservoir configurations of the connectome are used. Our findings indicate a clear connection between bio-plausible network patterns, the small-world characteristic, and learning outcomes, emphasizing the significant role of evolutionary optimization in shaping the topology of artificial neural networks for improved learning performance. [ABSTRACT FROM AUTHOR]

The paper introduces a deep learning (DL) model as an alternative approach to predict the supercontinuum generation in waveguides. The traditional technique of using the numerical method for this task is computationally laborious and time-consuming. The Artificial Neural Network (ANN) model described in the paper is trained from scratch and demonstrates high precision in predicting the spectral bandwidth of supercontinuum phenomena. It achieves a mean squared error of 0.0032, which indicates its ability to accurately map the optical characteristics of the waveguide. Furthermore, the deep learning model significantly reduces the computational time required compared to simulative processes conducted in COMSOL Multiphysics and MATLAB software, approximately taking only 50% of the time. By leveraging deep learning techniques, the model offers a promising alternative to the numerical method for predicting supercontinuum generation in planar waveguides. It overcomes the limitations associated with computational resources and complexity, making it an attractive substitute for traditional numerical techniques. • Investigated chalcogenide planar waveguide, generating versatile supercontinuum spectra dataset. • Preprocessed dataset for Tanh activation function compatibility. • Developed multi-layered ANN model with decaying learning rate for predicting supercontinuum spectra. • Efficiently predicts spectral broadening at waveguide's end and different lengths. • Developed ANN model handles both linear and nonlinear waveguide dynamics effectively. [ABSTRACT FROM AUTHOR]

With the increasing complexity of the power grid, the precision of the thermal power units' execution of automatic generation control (AGC) commands is gradually increasing the impact on the online regulation of the power grid. The deviation between the actual output of thermal power units and the AGC command of the grid will not only affect the consumption of new energy output, but also endanger the safe operation of the grid. This paper introduces "deep learning" technology to solve the problem. Firstly, an AGC command execution effect identification and confidence evaluation algorithm (ACEEI-CEA) is proposed. The algorithm builds a neural network model to accurately predict the unit output and the confidence evaluation of the prediction results. Next, a high-dimensional input preprocessing strategy based on variational autoencoder (VAE) is proposed to reduce the dimensionality of the model input attributes, improving the convergence and accuracy of the model. Finally, an AGC optimal command fast inversion solution method (AOCFISM) is designed. This method transforms the unit output deviation problem into an objective optimisation problem. And improve the efficiency of solving the optimal AGC command value by constraining the unit command value. The calculation results show that the error of the prediction results of the model proposed in this paper is 5% lower than that of the traditional neural network. The difference between the output value of the optimal AGC command and the expected output value obtained is less than 0.5 MV, which can support AGC online decision-making.© 2017 Elsevier Inc. All rights reserved. • An AGC command execution effect identification and confidence evaluation algorithm (ACEEI-CEA) is proposed. • Results show the ACEEI-CEA performs well in command prediction compared to the state-of-the-art methods. • ACEEI-CEA achieves confidence assessment of predicted values. • The pre-processing strategy based on variational autoencoder shortens the training time of the model. • The AGC optimal command fast inversion solution method (AOCFISM) can obtain the AGC optimal command value. [ABSTRACT FROM AUTHOR]

In personal identity recognition systems, detecting a person's age, gender, and language using voice signal characteristics is a crucial issue, especially because of the importance of security considerations. Age, gender, and language classification problems are important in signal processing because they are used to analyze and understand human behavior, interactions, and preferences. This can be especially useful in the fields of human-computer interaction, psychology, and social science research. In this paper, a new system for detecting a speaker's age, gender, and language based on deep learning models is presented. Deep learning models have shown great efficacy in various fields of signal processing. For this paper, a range of deep models were tested, including convolutional neural networks (CNNs), recurrent neural network (RNN), and a fine-tuning ResNet34 architecture. Additionally, techniques such as transfer learning were applied to improve the efficiency of the proposed system. The input voice signals are preprocessed by applying the spectro-temporal transform to obtain additional features that can be fed to the ResNet34 model, which is designed specifically for the task of voice signal processing. The dataset used in this paper was sourced from the Mozilla common voice initiative, which is dedicated to advancing speech recognition and language identification technologies. The performance of the proposed algorithm was evaluated in the presence of Gaussian noise to determine its robustness. The experimental results demonstrated that the proposed algorithm significantly outperformed basic algorithms and other deep neural networks in terms of age and gender recognition from voice signals. [ABSTRACT FROM AUTHOR]

• This paper proposes an innovative approach that integrates neural networks with photoluminescence detection technology to address defects such as cracks, dirt, dark spots, and scratches in solar cells. • The YOLOv5 model undergoes optimization in three distinct stages, encompassing global optimization, neck network structure refinement, and head structure enhancement. These optimizations collectively elevate the model's capabilities in recognizing, detecting, and classifying defects in solar cells. • Advanced techniques are employed, including the introduction of structures such as Transformer Encoder, the incorporation of additional small target detection boxes, utilization of Bi-FPN structure, and insertion of CBAM structure blocks. These optimizations contribute to the improvement of the model's feature recognition and extraction capabilities, further enhancing its ability to identify defects. In the production process of solar cells, inevitable faults such as cracks, dirt, dark spots, and scratches may occur, which could potentially impact the lifespan and power generation efficiency of solar cells. Addressing this issue, this paper combines neural networks with photoluminescence detection technology and proposes a novel neural network model for the classification and grading of defects in solar cells. Firstly, the YOLOv5 model is optimized and adjusted for algorithm and network structure. The optimization process is divided into three parts: global optimization of the network structure, optimization of the neck network structure, and optimization of the head structure, each addressing specific issues in recognition, detection, and classification. The impact of the optimized network model on recognition and detection speed is analyzed, and solutions are proposed to address any observed effects. Additionally, an iterative update of neural network hyperparameter combinations is performed for solar cell defect identification. Finally, using the ultimately optimized model structure in conjunction with the optimal hyperparameter combination, comparative experiments are conducted on neural networks for different target identification using the photoluminescence characteristics dataset of solar cells. The recognition improvement of the optimized model and its differences from other models are analyzed. [ABSTRACT FROM AUTHOR]

Transformers are Deep Neural Networks (DNN) that utilize a self-attention mechanism to capture contextual relationships within sequential data. Unlike traditional neural networks and variants of Recurrent Neural Networks (RNNs), such as Long Short-Term Memory (LSTM), Transformer models excel at managing long dependencies among input sequence elements and facilitate parallel processing. Consequently, Transformer-based models have garnered significant attention from researchers in the field of artificial intelligence. This is due to their tremendous potential and impressive accomplishments, which extend beyond Natural Language Processing (NLP) tasks to encompass various domains, including Computer Vision (CV), audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published, spotlighting the Transformer's contributions in specific fields, architectural disparities, or performance assessments, there remains a notable absence of a comprehensive survey paper that encompasses its major applications across diverse domains. Therefore, this paper addresses this gap by conducting an extensive survey of proposed Transformer models spanning from 2017 to 2022. Our survey encompasses the identification of the top five application domains for Transformer-based models, namely: NLP, CV, multi-modality, audio and speech processing, and signal processing. We analyze the influence of highly impactful Transformer-based models within these domains and subsequently categorize them according to their respective tasks, employing a novel taxonomy. Our primary objective is to illuminate the existing potential and future prospects of Transformers for researchers who are passionate about this area, thereby contributing to a more comprehensive understanding of this groundbreaking technology. • The paper presents a comprehensive survey on transformers for deep learning tasks. • The paper conducts a thorough analysis on highly effective models in five domains. • The paper classifies the models based on respective tasks using a proposed taxonomy. • The characteristics of the surveyed models are deeply explored and analyzed. • Future directions and challenges for transformer-based models are deciphered. [ABSTRACT FROM AUTHOR]

Wind power is a critical source of renewable energy, but the variability of wind speed can impact the efficiency and reliability of wind power generation. To improve the accuracy of wind speed prediction, this paper proposes a novel system that combines deep learning and decomposition algorithms based on ensemble optimization methods. Firstly, the paper proposes an innovative hybrid modal decomposition (HMD) method that extracts accurate features from wind speed data using Singular Value Decomposition (SVD), modal number selection method, and Variational Mode Decomposition (VMD). Secondly, the variable screening method identifies relevant factors affecting wind speed as fixed inputs. Then, the stationary test method is used to identify the sequences of different features in the mode and establish different prediction models: a bidirectional long-term short-time neural network model (Bi-LSTM) model optimized by artificial hummingbird algorithm (AHA) is used for non-stationary sequences; an ARIMA model is used for stationary sequences. Data from Sotavento and Changma wind farms are used to test this forecasting system. Simulation experiments are carried out from multiple dimensions. The results show that all error indicators are significantly improved, and the proposed prediction system is better than other comparative forecasting schemes. • The research can be practically applied in the field of wind power generation. • The research can help improve the energy utilisation of wind power generation. • A new data decomposition algorithm called HMD is proposed. • A new prediction system based on ensemble optimization idea is proposed. • Combining AHA-optimized neural networks and ARIMA to predict wind speed. [ABSTRACT FROM AUTHOR]

This paper introduces an innovative decision support system for improving ship energy efficiency. It combines an engine optimization model and an artificial neural network (ANN). Real-time data regarding the ship and engine performance is obtained along a specific ship navigation, and an engine optimization model is built using Ricardo Wave software. This model is validated with high accuracy. Ship-specific parameters are derived, and fuel consumption is estimated using ANN models with different structures. The study identifies the most suitable ANN model with 1 hidden layer and 5 neurons based on error metric evaluation where the 0.99697 R2, 0.00035 RMSE, and 0.06470 MAPE scores are calculated. This approach offers a cost-effective solution for shipping companies to monitor critical engine parameters in real-time without investing in sensors or data collection systems. Thus, it contributes to dealing with the problem of the scarcity of analyzable data in the maritime literature, which is one of the significant issues in the papers related to energy efficiency, machine learning, and deep learning. It is an innovative approach since the parameters that are related to real-time operations rather than considering only instruction book information are aggregated, and the produced data is analyzed by an intelligent tool. • A system is proposed to control ship energy efficiency level by combining an optimization model and a neural network • A simulation model is created in the Ricardo Wave tool and the engine model is validated by actual data. • The ship fuel consumption is estimated by an artificial neural network based on the derived input parameters. • The prediction process is evaluated by error metrics to determine its success. [ABSTRACT FROM AUTHOR]

Marine ecosystems and their fish habitats are becoming increasingly important due to their integral role in providing a valuable food source and conservation outcomes. Due to their remote and difficult to access nature, marine environments and fish habitats are often monitored using underwater cameras to record videos and images for understanding fish life and ecology, as well as for preserve the environment. There are currently many permanent underwater camera systems deployed at different places around the globe. In addition, there exists numerous studies that use temporary cameras to survey fish habitats. These cameras generate a massive volume of digital data, which cannot be efficiently analysed by current manual processing methods, which involve a human observer. Deep Learning (DL) is a cutting-edge Artificial Intelligence (AI) technology that has demonstrated unprecedented performance in analysing visual data. Despite its application to a myriad of domains, its use in underwater fish habitat monitoring remains under explored. In this paper, we provide a tutorial that covers the key concepts of DL, which help the reader grasp a high-level understanding of how DL works. The tutorial also explains a step-by-step procedure on how DL algorithms should be developed for challenging applications such as underwater fish monitoring. In addition, we provide a comprehensive survey of key deep learning techniques for fish habitat monitoring including classification, counting, localisation, and segmentation. Furthermore, we survey publicly available underwater fish datasets, and compare various DL techniques in the underwater fish monitoring domains. We also discuss some challenges and opportunities in the emerging field of deep learning for fish habitat processing. This paper is written to serve as a tutorial for marine scientists who would like to grasp a high-level understanding of DL, develop it for their applications by following our step-by-step tutorial, and see how it is evolving to facilitate their research efforts. At the same time, it is suitable for computer scientists who would like to survey state-of-the-art DL-based methodologies for fish habitat monitoring. • Marine environments and fish habitats are widely surveyed using underwater cameras. • Deep learning (DL) offers an effective solution for processing underwater videos. • We write a tutorial on the key concepts and steps of DL for underwater monitoring. • We provide a survey of key DL-based visual fish monitoring methods/datasets. • We shed light on challenges/opportunities of DL for underwater fish monitoring. [ABSTRACT FROM AUTHOR]

The increasing integration of distributed energy resources has led to heightened complexity in distribution network models, posing challenges of uncertainty and volatility to the operation and control of distribution networks. Simultaneously, the widespread deployment of metering and sensing devices has furnished data-driven approaches with copious data sources. As a potent tool for data mining, artificial intelligence (AI) algorithms have the capacity to harness extensive datasets, thus furnishing comprehensive energy services for distribution grids. This paper provides a comprehensive exposition of AI techniques applied in the realm of distribution network voltage control (DNVC). Initially, a brief overview is provided for the four voltage control frameworks and the data calibration of DNVC, serving as the foundation for various voltage control algorithms. Subsequently, in ascending order of algorithmic advancement, the application of improved heuristic algorithms, deep learning (DL) algorithms structured around artificial neural networks (ANNs), and deep reinforcement learning (DRL) algorithms in the DNVC domain are sequentially introduced, and the relationships among the three AI technologies in framework, control principles, and performance are systematically analyzed. Additionally, attention is directed toward the algorithmic progress made in addressing specific challenges within the DNVC process. Lastly, building upon extant research, the paper summarizes four prominent developmental directions for AI in DNVC: safety exploration, information privacy, multi-feeder collaboration, and heightened uncertainty. Corresponding solutions to each of these directions are individually discussed. • The research status and applications of mainly used AI methods in distribution network voltage control is reviewed. • The relationships among the three artificial intelligence methods are systematically analyzed. • The existing algorithms, along with their strengths, limitations, and relationshipsare highlighted. • The research directions of artificial intelligence methods in distribution network voltage control are discussed in detail. [ABSTRACT FROM AUTHOR]

• The paper proposes a deep learning classifier based on a deep neural network (DNN) to improve the recognition of hand motion gestures with different force variations for amputees using EMG signals. • The paper shows that the DNN classifier is robust against the effect of force variation and can recognize 18 combinations of six gestures and three forces with an average accuracy of 92.0 % using time domain features. • The paper identifies the EMG channels with the highest contribution to the classifier's accuracy and suggests that the proposed method can be applied to developing prosthetic hands for amputees. Variations in muscular contraction are known to significantly impact the quality of the generated EMG signal and the output decision of a proposed classifier. This is an issue when the classifier is further implemented in prosthetic hand design. Therefore, this study aims to develop a deep learning classifier to improve the classification of hand motion gestures and investigate the effect of force variations on their accuracy on amputees. The contribution of this study showed that the resulting deep learning architecture based on DNN (deep neural network) could recognize the six gestures and robust against different force levels (18 combinations). Additionally, this study recommended several channels that most contribute to the classifier's accuracy. Also, the selected time domain features were used for a classifier to recognize 18 combinations of EMG signal patterns (6 gestures and three forces). The average accuracy of the proposed method (DNN) was also observed at 92.0 ± 6.1 %. Moreover, several other classifiers were used as comparisons, such as support vector machine (SVM), decision tree (DT), K-nearest neighbors, and Linear Discriminant Analysis (LDA). The increase in the mean accuracy of the proposed method compared to other conventional classifiers (SVM, DT, KNN, and LDA), was 17.86 %. Also, the study's implication stated that the proposed method should be applied to developing prosthetic hands for amputees that recognize multi-force gestures. [ABSTRACT FROM AUTHOR]

With the development of modern animal husbandry, especially pig farming, large-scale, intensive, and automated farming has become the trend in the industry. The realization of accurate recognition and warning of dangerous actions in feeding scenarios for sows and piglets holds high research and practical value in this field. This paper addresses this issue by proposing a Transformer-based Neural Network (TNN) model. This model emphasizes the extraction of global features and handling of long-distance dependencies, significantly improving the accuracy of behavior recognition. Compared with traditional neural network models, TNN demonstrates superiority in dealing with animal behavior recognition in farming scenarios. Furthermore, an in-depth study of the attention mechanism in the TNN model was conducted in this paper. By visualizing the attention heat map of the TNN model, it was found that TNN could effectively focus on key areas in the image, thereby accurately identifying the behavior of piglets. Finally, this paper proposes a unique model lightweighting strategy that allows the TNN model to run efficiently on edge devices. In the experimental part, the performance of the TNN model on five behavior recognition tasks was evaluated first. The results showed that the TNN model achieved high scores in both Precision and Recall, far exceeding traditional neural network models. Then, by visualizing the attention heat map of the TNN model, the superiority of the TNN model in focusing on key image areas was further confirmed. In the end, the effect of the model lightweighting strategy was demonstrated, and even with a significant reduction in parameters and computational complexity, the performance of the TNN model remained excellent. This research not only promotes the development of animal behavior recognition technology but also provides new insights and tools for the precise management and efficient operation of the animal husbandry industry. • A comprehensive dataset of sow nursing scenarios was collected and annotated. • A variety of data augmentation methods were proposed to rebalance it, ultimately leading to a model accuracy of 0.91 mAP. • A time-series neural network (TNN) model is proposed, tailored to the characteristics of the actual farm collection equipment. • An alarm triggering mechanism was designed, enabling the model's application in actual agricultural scenarios. [ABSTRACT FROM AUTHOR]

The current digital speech deletion and insertion tampering detection methods mainly employes the extraction of phase and frequency features of the Electrical Network Frequency (ENF). However, there are some problems with the existing approaches, such as the alignment problem for speech samples with different durations, the sparsity of ENF features, and the small number of tampered speech samples for training, which lead to low accuracy of deletion and insertion tampering detection. Therefore, this paper proposes a tampering detection method for digital speech deletion and insertion based on ENF Fluctuation Super-vector (ENF-FSV) and deep feature learning representation. By extracting the parameters of ENF phase and frequency fitting curves, feature alignment and dimensionality reduction are achieved, and the alignment and sparsity problems are avoided while the fluctuation information of phase and frequency is extracted. To solve the problem of the insufficient sample size of tampered speech for training, the ENF Universal Background Model (ENF-UBM) is built by a large number of untampered speech samples, and the mean vector is updated to extract ENF-FSV. Considering the shallow representation of ENF features with not highlighting important features, we construct an end-to-end deep neural network to strengthen the attention to the abrupt fluctuation information by the attention mechanism to enhance the representational power of the ENF-FSV features, and then the deep ENF-FSV features extracted by the Residual Network (ResNet) module are fed to the designed classification network for tampering detection. The experimental results show that the method in this paper exhibits higher accuracy and better robustness in the Carioca, New Spanish, and ENF High-sampling Group (ENF-HG) databases when compared with the state-of-the-art methods. [Display omitted] • The ENF fluctuation super vector is extracted for audio tampering detection. • An end-to-end DNN is designed for deletion and insertion tampering detection. • Experiments have verified that the proposed method outperforms the SOTA methods. [ABSTRACT FROM AUTHOR]

Due to the inevitable light absorption and scattering, underwater images always suffer from severe quality degradation, leading to significant performance decline for various maritime engineering-related applications. In recent years, deep neural networks (DNNs) have been proven to achieve high-quality enhancement of underwater images, which aims to extract and learn abstract features to exhibit superior performance. However, existing solutions often stack multilayer processing units to enrich features, which not only significantly burdens computational load but also encounters difficulties in fully utilizing and interacting between features at different depths. To address the above problems, this paper presents a cost-effective underwater image enhancement network via cascaded feature extraction, termed CFENet. Specifically, we develop a cost-effective cascaded structure for sufficient feature extraction and multiscale establishment of pixel-level long-range dependencies. In addition, we construct a dual-branch structure for effective feature fusion, so that the detailed texture and semantic information in the image can be simultaneously enhanced. Extensive experiments reveal the superiority of our proposed CFENet in both underwater image enhancement effects and computational complexity. Sufficient ablation study is conducted to demonstrate the effectiveness of each component in the network. [ABSTRACT FROM AUTHOR]

Owing to the development of deep learning, indoor personnel detection methods based on deep neural networks have been extensively investigated in recent years. However, more complex and deeper network structures may consume more computational resources, which seriously limits the deployment of large-scale deep neural networks on lightweight devices. In view of this, a lightweight anchor-free one-level feature indoor personnel detection method based on transformer (LAOF-IPDT) is proposed in this paper, which is deployed on two embedded devices and achieves good detection accuracy. In the feature extraction backbone, an enhanced cross-stage partial ghost convolution block with ghost convolution and channel shuffle is designed to extract shallow features. Additionally, to obtain more comprehensive global features in the high-level semantic structure, an embedded vision transformer cross-stage partial block is constructed by embedding a lightweight mobile-friendly vision transformer. For the path-aggregation neck, a lightweight feature pyramid network is proposed, which integrates multi-scale feature maps to obtain richer one-level feature representations. Subsequently, a dilated convolution group block is applied to expand the one-level feature receptive field and detection is accomplished using a one-level feature map. For the detection head, an anchor-free mechanism is applied to reduce the hyper-parameter interference of anchor boxes. Extensive experiments on four datasets indicated that the LAOF-IPDT outperforms other lightweight networks in terms of accuracy, speed, model parameters, and model size. For example, the frames per second of the LAOF-IPDT are 8.39 and 14.67 on CPU devices and Jetson Nano devices, respectively, and the mean average precision is 84.49%. [ABSTRACT FROM AUTHOR]

Accurate prediction of air pollutant concentrations can be effective in controlling and preventing air pollution, which is crucial for both the government's policy response and the public's reasonable ability to avoid air pollution. In this study, we propose a model based on the principle of big data correlation and deep learning techniques for predicting the concentration of air pollutant concentrations. The proposed model, named STARes-SaLSTM, consists of a residual neural network with spatiotemporal attention (STAResNet) and a temporal self-attention based on long short-term memory network (SaLSTM). The spatiotemporal attention module in STAResNet enhances the original input information by weighting it in both spatial and temporal dimensions, meanwhile the features of the spatial distribution of pollutant and meteorological information across many cities are deeply extracted using ResNet. A novel self-attention method known as temporal self-attention is developed to extract the temporal dependency of air pollutants. LSTM is used as a sequence encoder in this temporal attention method to calculate queries, keys, and values to acquire a more comprehensive temporal dependence than is possible with normal self-attention. The STARes-SaLSTM model successfully predicts air pollution concentrations in the target city for the future by extracting spatiotemporal correlation of feature sequence. In comparison to previous neural network models and conventional methods, the model increases the accuracy of pollutant concentration prediction. The suggested prediction model works well for the one to 3 h prediction job, with a root mean square error (RMSE) ranging from 6.716 to 11.648. Furthermore, even for the 1–24 h prediction assignment, we executed multi-scale prediction in the target city and obtained a reasonable performance, with an average RMSE value of 20.576. • This paper solves the problem of difficult extraction of spatiotemporal features. • ResNet allows for the extraction of spatially relevant features; • The spatiotemporal attention mechanism is used to spatiotemporal extract features; • The temporal self-attention mechanism obtain temporal dependence of data. [ABSTRACT FROM AUTHOR]

Research on automatic translation of sign language to verbal languages has been progressively explored in recent years to assist speech and hearing-impaired people in communicating with non-signers. In this paper, a tiny machine learning (TinyML) solution is proposed for sign language recognition using a low-cost, wearable, internet-of things (IoT) device. A lightweight deep neural network is deployed on the edge device to interpret isolated signs from the Indian sign language using the time-series data collected from the motion sensors of the device. The scarcity of labeled training data is addressed by employing the deep transfer learning approach. Here, the knowledge gained from the data collected using the motion sensors of a different device is used to initialize the model parameters. The performance of the model is assessed in terms of classification accuracy and prediction time for different sampling rates and transferring schemes. The model achieves an average accuracy of 87.18% when all the parameters are retrained with just 4 observations of each sign recorded from the motion sensors of the proposed IoT device. The recognized sign is transmitted to a cloud platform in real-time. A mobile application, SignTalk , is also developed, which wirelessly receives the predicted signs from the cloud and displays it as text. Additionally, text-to-speech conversion is also provided on SignTalk to vocalize the predicted sign for better communication. [ABSTRACT FROM AUTHOR]

Skin diseases are reported to contribute 1.79% of the global burden of disease. The accurate diagnosis of specific skin diseases is known to be a challenging task due, in part, to variations in skin tone, texture, body hair, etc. Classification of skin lesions using machine learning is a demanding task, due to the varying shapes, sizes, colors, and vague boundaries of some lesions. The use of deep learning for the classification of skin lesion images has been shown to help diagnose the disease at its early stages. Recent studies have demonstrated that these models perform well in skin detection tasks, with high accuracy and efficiency. Our paper proposes an end-to-end framework for skin lesion classification, and our contributions are two-fold. Firstly, two fundamentally different algorithms are proposed for segmenting and extracting features from images during image preprocessing. Secondly, we present a deep convolutional neural network model, S-MobileNet that aims to classify 7 different types of skin lesions. We used the HAM10000 dataset, which consists of 10000 dermatoscopic images from different populations and is publicly available through the International Skin Imaging Collaboration (ISIC) Archive. The image data was preprocessed to make it suitable for modeling. Exploratory data analysis (EDA) was performed to understand various attributes and their relationships within the dataset. A modified version of a Gaussian filtering algorithm and SFTA was applied for image segmentation and feature extraction. The processed dataset was then fed into the S-MobileNet model. This model was designed to be lightweight and was analyzed in three dimensions: using the Relu Activation function, the Mish activation function, and applying compression at intermediary layers. In addition, an alternative approach for compressing layers in the S-MobileNet architecture was applied to ensure a lightweight model that does not compromise on performance. The model was trained using several experiments and assessed using various performance measures, including, loss, accuracy, precision, and the F1-score. Our results demonstrate an improvement in model performance when applying a preprocessing technique. The Mish activation function was shown to outperform Relu. Further, the classification accuracy of the compressed S-MobileNet was shown to outperform S-MobileNet. To conclude, our findings have shown that our proposed deep learning-based S-MobileNet model is the optimal approach for classifying skin lesion images in the HAM10000 dataset. In the future, our approach could be adapted and applied to other datasets, and validated to develop a skin lesion framework that can be utilized in real-time. [ABSTRACT FROM AUTHOR]

This paper introduces a novel parallel deep neural network for fault diagnosis of drilling pumps. It integrates the Convolutional Block Attention Module with the AlexNet and synchronizes with the Anomaly Transformer model to delve meticulously into both the time and time-frequency domains of signals. The method prioritizes the singular extraction and subsequent amalgamation of features, facilitating a detailed view of diagnostic data and mitigating the risk of interference and overfitting. The integration of the anomaly attention of the Anomaly Transformer with the features of the Convolutional Block Attention Module results in a distinctive dual attention mechanism that is critical to the methodology. This mechanism emphasizes essential features in both the time domain and the time-frequency domain, improving the accuracy of fault diagnosis. Verification with on-site data underscores the preeminence of the approach over existing models, signaling improved reliability and accuracy in diagnosing faults in drilling pumps. This meticulous approach offers promising advances in the study and application of fault diagnosis in energy equipment, demonstrating increased efficiency and refined accuracy. • A novel Parallel Deep Neural Network-Based Fault Diagnosis Framework is proposed. • Time and time-frequency domains of signals are extracted and fused for Fault Diagnosis. • Integrating the Transformer with CBAM to consider the essential features of signals. [ABSTRACT FROM AUTHOR]

To mitigate the high cost of integrated ship design, in this paper, we propose an innovative integrated hull design methodology utilizing artificial intelligence-aided design technology. The AIAD methodology leverages a deep neural network model constructed from multiple fidelities datasets to curtail the exponential growth in data requirements resulting from the proliferation of design variables caused by integration. Combining the proposed bump-free deformation method and optimization algorithms with the multi-fidelity deep network, the AIAD-based design approach facilitates the achievement of optimal hull design solutions with remarkable fidelity while minimizing design workload. Utilizing the developed design method, we performed a numerical test on integrated-hull design for container ships. Through a comparative analysis and sensitive analysis of design results and workload, the optimized results obtained demonstrated the effectiveness of the proposed AIAA integrated-hull design methodology in achieving optimal designs while preserving high fidelity and low workload. The proposed new approach has shown promises to expedite ship integrated-hull design, addressing the longstanding issue of its high cost. • An artificial intelligence-based design method is proposed, reducing the workload brought by the integrated-hull design. • Balanced the conflicting relationship between fidelity and cost of the deep learning model by using multi-fidelity datasets. • A bump-free deformation method is provided, which solves the local concave-convex after deformation. • Carried out the integrated-hull design of a container ship hull, achieving a 3% reduction in resistance. [ABSTRACT FROM AUTHOR]

Extensive research has been conducted on analyzing human movements, driven by its diverse practical applications such as human–robot interaction, human learning, and clinical diagnosis. However, the current state-of-the-art still encounters scientific challenges when it comes to modeling human movements. There are two key aspects that need to be addressed. Firstly, new models should consider the stochastic nature of human movement and the physical structure of the human body to accurately predict the patterns in full-body motion descriptors over time. Secondly, while deep learning algorithms have been utilized, they lack explainability in terms of predicting body posture sequences, making it essential to improve their comprehensible representation of human movement. This paper aims to tackle these challenges by presenting three innovative methods for creating explainable representations of human movement. The study formulates human body movement as a state-space model based on the Gesture Operational Model (GOM). Model parameters are estimated through either one-shot training employing Kalman Filters or data-intensive training utilizing artificial neural networks. The trained models are utilized for analyzing the dexterity of expert professionals in full-body movements, enabling the identification of dynamic associations between body joints and gesture recognition. Additionally, these models are employed to generate artificial professional movements. • Three innovative methods for explainable human movement models. • Body dexterity analysis: Models reveal how artisans and operators perform tasks. • Minimal motion descriptors to enhance task recognition. • Artificial generation of full-body professional movements. • Extensive experimentation with real-world datasets. [ABSTRACT FROM AUTHOR]

Accurate diagnosis of Lithium-ion batteries (Li-ion batteries) degradation plays a critical role in improving the maintenance of energy storage technology. This paper presents a method based on a novel deep network model combined with a data transformation technique to diagnose Li-ion battery degradation modes. Different from conventional studies based on specific experimental and numerical methods to estimate and predict the degradation, the proposed method is based on data-driven approach, by leveraging datasets consisting of voltage/capacity curves, these were converted into incremental capacity (IC) curves and then transformed into images using the gramian angular summation field (GASF) technique. The study adopted two models: Inception-v3 and the proposed model, both underwent fine-tuning and a subsequent transfer learning process. Degradation modes, namely loss of lithium inventory (LLI) and the loss of active materials in both the positive (LAMPE) and negative electrodes (LAMNE), were diagnosed in relation to IC curves. Finally, the model was tested using two different datasets, and the results showed that the proposed method achieved high performance, especially across three Li-ion batteries, three degradation modes, three cells, and various cycles (totaling 378 cases) the proposed method outperformed in 233 cases, thereby outperforming other methods in comparison. Our method provides a flexible data-driven approach that accurately predicts various degradation modes across different cell chemistries throughout their lifespan. [ABSTRACT FROM AUTHOR]

Distributed generation sources (e.g. photovoltaic, wind power) are raised to fit the energy demands recently in proficient way. Further, adoption of renewable energy sources to the smart grid causes severe issues in quality of power signals. These problems can be overcome with correct identification and classification of power quality disturbances (PQDs) efficiently by the both producer and consumer. In this work, a new method for classification of PQDs based on signal processing technique and deep convolutional neural network (DCNN) is proposed. The proposed method compress the raw data of PQDs by using discrete wavelet transform (DWT) and multi-resolution analysis (MRA). Moreover, CNN has the ability to automatic feature selection and classification in a single stage instead of conventional methods i.e. artificial neural networks (ANNs). In this paper, DWT 1D-CNN is designed to extract and optimize multi-scale characteristics of disturbances. Further, 1-D convolutional, pooling and batch-normalization layers are added to facilitate normalize data, better generalization ability and, effective classification process against noisy data. A typical 11Kv distribution network is simulated in MATLAB to prove the validity of the proposed method in an essential part of smart grid. [ABSTRACT FROM AUTHOR]

This paper presents a scheme for retrieving wind direction without 180° ambiguity over the sea solely based on Synthetic Aperture Radar (SAR) images. The dataset utilized for training, validating, and testing wind direction estimation deep neural network consists of 19,210 spatiotemporal match-ups of Sentinel-1 sub-images from Google Earth Engine (GEE) platform and wind direction records from National Data Buoy Center (NDBC) stations, from 2014 to 2023. Evaluation metrics, namely Line Angle Error (LAE) and Wind Direction Ambiguity accuracy (WDA accuracy), are proposed to facilitate modelling and assessing the performance of model in various conditions, whether open or not, and under different marine atmospheric boundary layer (MABL) stratification states. Post-processing method is devised to derive final robust wind direction field. Stratification can be quantified by a bulk Richardson number, Ri. With a wide spatiotemporal distribution, 58 full-frame Sentinel-1 images that satisfy −0.016 < Ri < +0.001 constitute an independent dataset for validating the entire wind direction retrieval process. Statistical analysis was conducted to compare the SAR-derived wind directions with those from COSMO-REA6, ASCAT and ERA5. It reports bias <1.4° and centered root mean square difference (CRMSD) < 17.3°. Wind streaks (WS) and shallow convective cells (MC) are common coherent structures in these images, both of which contain wind direction information. The scheme has a certain ability to counteract the contamination from rain cells, internal waves and atmospheric gravity waves, and also demonstrates great potential for high-resolution wind direction retrieval of tropical cyclones (TC). Additionally, we introduced Gradient-weighted Class Activation Mapping (Grad-CAM) to emphasize the significance of land-shadow features to wind direction inference. • The deep neural network can remove 180° wind direction ambiguity to some extent. • Surface layer stratification affects the retrieval of wind direction. • A scheme for retrieving unambiguous wind direction from SAR imagery is presented. • The results have been compared with wind data from COSMO-REA6, ASCAT and ERA5. [ABSTRACT FROM AUTHOR]

Prostate cancer is the second most frequent cancer in men worldwide after lung cancer. Its diagnosis is based on the identification of the Gleason score that evaluates the abnormality of cells in glands through the analysis of the different Gleason patterns within tissue samples. The recent advancements in computational pathology, a domain aiming at developing algorithms to automatically analyze digitized histopathology images, lead to a large variety and availability of datasets and algorithms for Gleason grading and scoring. However, there is no clear consensus on which methods are best suited for each problem in relation to the characteristics of data and labels. This paper provides a systematic comparison on nine datasets with state-of-the-art training approaches for deep neural networks (including fully-supervised learning, weakly-supervised learning, semi-supervised learning, Additive-MIL, Attention-Based MIL, Dual-Stream MIL, TransMIL and CLAM) applied to Gleason grading and scoring tasks. The nine datasets are collected from pathology institutes and openly accessible repositories. The results show that the best methods for Gleason grading and Gleason scoring tasks are fully supervised learning and CLAM, respectively, guiding researchers to the best practice to adopt depending on the task to solve and the labels that are available. • We perform a systematic comparison of 12 training approaches on Gleason grading and scoring. • 9 highly-heterogeneous datasets were used, allowing evaluating the performance and the generalization of the methods over many datasets. • Self-supervision improves performance compared to using pre-trained weights. • Full supervision shows the highest performance in patch-level classification tasks. • CLAM obtains the highest performance in image-level classification tasks. [ABSTRACT FROM AUTHOR]

Computational modeling plays an increasingly important role in neuroscience, highlighting the philosophical question of how computational models explain. In the particular case of neural network models, concerns have been raised about their intelligibility, and how these models relate (if at all) to what is found in the brain. We claim that what makes a system intelligible is an understanding of the dependencies between its behavior and the factors that are responsible for that behavior. In biology, many of these dependencies are naturally "top-down", as ethological imperatives interact with evolutionary and developmental constraints under natural selection to produce systems with capabilities and behaviors appropriate to their evolutionary needs. We describe how the optimization techniques used to construct neural network models capture some key aspects of these dependencies, and thus help explain why brain systems are as they are — because when a challenging ecologically-relevant goal is shared by a neural network and the brain, it places constraints on the possible mechanisms exhibited in both kinds of systems. The presence and strength of these constraints explain why some outcomes are more likely than others. By combining two familiar modes of explanation — one based on bottom-up mechanistic description (whose relation to neural network models we address in a companion paper) and the other based on top-down constraints, these models have the potential to illuminate brain function. [ABSTRACT FROM AUTHOR]

Alternating current optimal power flow (OPF) analysis is critical for efficient and reliable operation of power systems. For large systems or repetitive computations, the traditional methods such as the direct and gradient methods, or non-traditional methods, such as the genetic algorithm and simulating annealing, are time-consuming and unsuitable for real-time computing. The work in this paper proposes a novel framework to obtain the optimal solution of power flow in real-time using a combination of convolutional neural networks and a self-attention mechanism. All parameters of the power networks are rearranged in an image-like shape of a multi-channel image where each channel is a two-dimensional matrix. The proposed approach is adaptive with every input size of power systems as well as frequent variations of network topologies without intervention to the framework core. The encompassment of all power system contexts in which all parameters of internal elements, generation costs, and topology information are included, contributes to the higher accuracy of inference compared to other current machine-learning-based OPF-solving methods. Besides, the proposed framework established on ubiquitous platforms is effortlessly integrated into current infrastructures of power systems, and the great efficiency along with the computation speed may serve as a critical point for practical implications, such as enabling faster decision-making during real-time operations, predicting system contingencies, and remedial actions based on an offline pre-trained model. This supervised learning process is applied to the dataset of four case studies of meshed power systems: the IEEE 5-bus system (IEEE-5), the IEEE 30-bus system (IEEE-30), the IEEE 39-bus system (IEEE-39), and the IEEE 57-bus system (IEEE-57) to prove the efficacy of the proposed method. • Developing a novel approach to solve the OPF problems in real-time. • Wrap power system characteristics under a mold of an image-like shape. • Overcome the outburst of trainable parameters when training with large-scale systems. • Do not require looking up a new set of hyperparameters or re-coding a framework. • All time-consuming computations are shifted to the offline training process. [ABSTRACT FROM AUTHOR]

Metastatic femoral tumors may lead to pathological fractures during daily activities. A CT-based finite element analysis of a patient's femurs was shown to assist orthopedic surgeons in making informed decisions about the risk of fracture and the need for a prophylactic fixation. Improving the accuracy of such analyses ruqires an automatic and accurate segmentation of the tumors and their automatic inclusion in the finite element model. We present herein a deep learning algorithm (nnU-Net) to automatically segment lytic tumors within the femur. A dataset consisting of fifty CT scans of patients with manually annotated femoral tumors was created. Forty of them, chosen randomly, were used for training the nnU-Net, while the remaining ten CT scans were used for testing. The deep learning model's performance was compared to two experienced radiologists. The proposed algorithm outperformed the current state-of-the-art solutions, achieving dice similarity scores of 0.67 and 0.68 on the test data when compared to two experienced radiologists, while the dice similarity score for inter-individual variability between the radiologists was 0.73. The automatic algorithm may segment lytic femoral tumors in CT scans as accurately as experienced radiologists with similar dice similarity scores. The influence of the realistic tumors inclusion in an autonomous finite element algorithm is presented in (Rachmil et al., "The Influence of Femoral Lytic Tumors Segmentation on Autonomous Finite Element Analyses", Clinical Biomechanics, 112, paper 106192, (2024)). • A deep learning algorithm for automatic segmentation of lytic femoral tumors is developed. • The deep learning dice similarity score of ∼0.67 is obtained, higher than any former study. • Inter-individual dice similarity score variability between two radiologists was 0.73. • The automatic segmentation of lytic tumors is as good as a radiologist. [ABSTRACT FROM AUTHOR]

Dropout is a popular regularisation tool for deep neural classifiers, but it is applied regardless of the nature of the classification task: nominal or ordinal. Consequently, the order relation between the class labels of ordinal problems is ignored. In this paper, we propose the fusion of standard dropout and a new dropout methodology for ordinal classification regularising deep neural networks to avoid overfitting and improve generalisation, but taking into account the extra information of the ordinal task, which is exploited to improve performance. The correlation between the outputs of every neuron and the target labels is used to guide the dropout process: the higher the neuron is correlated with the expected labels, the lower its probability of being dropped. Given that randomness also plays a crucial role in the regularisation process, a balancing factor (β) is also added to the training process to determine the influence of the ordinality with respect to a constant probability, providing a hybrid ordinal regularisation method. An extensive battery of experiments shows that the new hybrid ordinal dropout methodology perform better than standard dropout, obtaining improved results in most evaluation metrics, including not only ordinal metrics but also nominal ones. • Deep ordinal classification. • Hybrid ordinal dropout as regularisation methodology for deep neural networks. • Fusion of standard and ordinal dropout techniques. • Balance factor for the influence of the ordinality in the hybrid ordinal dropout. • Comparison of the new hybrid ordinal dropout using ordinal benchmark datasets. [ABSTRACT FROM AUTHOR]

• A dynamic SPECT kidney image database was established by manually labeling clinical renal dynamic images. • A multi-scale spatial perception module was proposed to enhance the semantic information of high-level feature maps. • Decoding modules with attention connection were proposed to refine the details gradually. • RASNet achieved the optimal performance in most of the indicators in automatic kidney segmentation. For the treatment of renal disease, the application of radioactive equipment has become one of the important methods. Accurate segmentation of renal contour plays an important role in clinical diagnosis. However, manual renal contour drawing is not only inefficient but also prone to inaccurate outlining results due to different manual proficiency and fatigue caused by long-term work. There is little research on automatic renal segmentation with renal dynamic imaging. To address this issue, an improved model based on a deep neural network called Renal Automatic Segmentation Network (RASNet) is proposed, to aid in the automatic segmentation of renal contours. Besides, a multi-scale spatial perception module and a decoding module with attention connection are introduced to enrich the semantic information and further improve the accuracy of network segmentation. Extensive experiments were conducted on a renal dynamic medical image database established in this paper. Analysis results show the superiority of the proposed RASNet to several existing segmentation frameworks. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) have gained immense popularity in recent years, finding their utility in diverse fields such as image recognition, natural language processing, and bio-informatics. Despite the remarkable progress made in deep learning theory, most studies on CNNs, especially in regression tasks, tend to heavily rely on the least squares loss function. However, there are situations where such learning algorithms may not suffice, particularly in the presence of heavy-tailed noises or outliers. This predicament emphasizes the necessity of exploring alternative loss functions that can handle such scenarios more effectively, thereby unleashing the true potential of CNNs. In this paper, we investigate the generalization error of deep CNNs with the rectified linear unit (ReLU) activation function for robust regression problems within an information-theoretic learning framework. Our study demonstrates that when the regression function exhibits an additive ridge structure and the noise possesses a finite p th moment, the empirical risk minimization scheme, generated by the maximum correntropy criterion and deep CNNs, achieves fast convergence rates. Notably, these rates align with the mini-max optimal convergence rates attained by fully connected neural network model with the Huber loss function up to a logarithmic factor. Additionally, we further establish the convergence rates of deep CNNs under the maximum correntropy criterion when the regression function resides in a Sobolev space on the sphere. [ABSTRACT FROM AUTHOR]