Search

The paper discusses algorithmic priority inversion in mission-critical machine inference pipelines used in modern neural-network-based perception subsystems and describes a solution to mitigate its effect. In general, priority inversion occurs in computing systems when computations that are "less important" are performed together with or ahead of those that are "more important." Significant priority inversion occurs in existing machine inference pipelines when they do not differentiate between critical and less critical data. We describe a framework to resolve this problem and demonstrate that it improves a perception system's ability to react to critical inputs, while at the same time reducing platform cost. [ABSTRACT FROM AUTHOR]

Purpose: With the growth of organizations and businesses, customer acquisition and retention processes have become more complex in the long run. That is why customer lifetime value (CLV) has become crucial to sales managers. Predicting the CLV is a strategic weapon and competitive advantage in increasing profitability and identifying customers with more splendid profitability and is one of the essential key performance indicators (KPI) used in customer segmentation. Thus, this paper proposes a stacked ensemble learning method, a combination of multiple machine learning methods, for CLV prediction. Design/methodology/approach: In order to utilize customers' behavioral features for predicting the value of each customer's CLV, the data of a textile sales company was used as a case study. The proposed stacked ensemble learning method is compared with several popular predictive methods named deep neural networks, bagging support vector regression, light gradient boosting machine, random forest and extreme gradient boosting. Findings: Empirical results indicate that the regression performance of the stacked ensemble learning method outperformed other methods in terms of normalized rooted mean squared error, normalized mean absolute error and coefficient of determination, at 0.248, 0.364 and 0.848, respectively. In addition, the prediction capability of the proposed method improved significantly after optimizing its hyperparameters. Originality/value: This paper proposes a stacked ensemble learning method as a new method for accurate CLV prediction. The results and comparisons support the robustness and efficiency of the proposed method for CLV prediction. [ABSTRACT FROM AUTHOR]

• Two-phase flow regime identification using ML with enhanced interpretability. • Evaluation of various dimensionality reduction techniques to optimize feature representation. • Kernel Fisher discriminant analysis is the best performing dimensionality reduction technique. • AutoML can impartially select the most effective ML classifier. • Our workflow can develop a virtual flow regime map to automate flow regime recognition. Accurate identification of flow regimes is paramount in several industries, especially in chemical and hydrocarbon sectors. This paper describes a comprehensive data-driven workflow for flow regime identification. The workflow encompasses: i) the collection of dynamic pressure signals using an experimentally verified numerical two-phase flow model for three different flow regimes: stratified, slug and annular flow, ii) feature extraction from pressure signals using Discrete Wavelet Transformation (DWT), iii) Evaluation and testing of 12 different Dimensionality Reduction (DR) techniques, iv) the application of an AutoML framework for automated Machine Learning classifier selection among K-Nearest Neighbors, Artificial Neural Networks, Support Vector Machines, Gradient Boosting, Random Forest, and Logistic Regression, with hyper-parameter tuning. Kernel Fisher Discriminant Analysis (KFDA) is the best DR technique, exhibiting superior goodness of clustering, while KNN proved to be the top classifier with an accuracy of 92.5 % and excellent repeatability. The combination of DWT, KFDA and KNN was used to produce a virtual flow regime map. The proposed workflow represents a significant step forward in automating flow regime identification and enhancing the interpretability of ML classifiers, allowing its application to opaque pipes fitted with pressure sensors for achieving flow assurance and automatic monitoring of two-phase flow in various process industries. [ABSTRACT FROM AUTHOR]

In this paper, we propose a variance reduction approach for Markov chains based on additive control variates and the minimization of an appropriate estimate for the asymptotic variance. We focus on the particular case when control variates are represented as deep neural networks. We derive the optimal convergence rate of the asymptotic variance under various ergodicity assumptions on the underlying Markov chain. The proposed approach relies upon recent results on the stochastic errors of variance reduction algorithms and function approximation theory. [ABSTRACT FROM AUTHOR]

This paper proposes a novel L1-based predictor–corrector method for the fractional differential equations involving generalized-Caputo type derivative. A decomposition scheme is used to obtain the three-point predictor–corrector formula. The error and stability of the proposed method are given in detail. A computer virus and a five-dimensional Hopfield neural network models are solved using the proposed approach. [ABSTRACT FROM AUTHOR]

In this paper, stochastic predictive computing networks are exploited to investigate the dynamics of the SIS with vaccination impact based epidemic model (SISV-EM) represented by nonlinear systems of stochastic differential equations (SDEs) by exploitation of artificial neural networks (ANNs) with the backpropagated Levenberg-Marquardt technique (BLMT) i.e., (ANNs-BLMT) to approximate the solution behavior. The stochastic nonlinear SISV-EM is governed with three classes: susceptible, infectious, and vaccinated populations. The referenced or target datasets for ANNs-BLMT are constructed by employing Euler-Maruyama (EM) scheme for solving stochastic differential systems in case of sufficiently various nonlinear SISV-EM scenarios by varying the percentage of vaccination for newly born, the coefficient of transmission, the natural mortality rates, the infectious rates of recovery, the rate at which vaccinated people lose their immunity, the rate of death caused by disease, the proportion of vaccinated against susceptible and the white noise in the environment. Based on arbitrary training, testing, and validation samples from the referenced dataset, the ANNs-BLMT provides an approximate solution for the stochastic nonlinear SISV-EM, with significant correlations to the referenced results. Exhaustive simulation-based results using error histograms, mean square errors, and regression analyses further demonstrate that the proposed ANNs-BLMT is efficient, consistent, and accurate for solving SISV-EM. • A two-layer framework of ANNs-BLMT is proposed as an innovative technique based on a stochastic computing paradigm to investigate the dynamics of stochastic nonlinear SISV-EM. • Innovative technique based on a stochastic computing ANNs-BLMT to investigate the dynamics of stochastic nonlinear SISV-EM Reference dataset for ANNs -BLMT is developed with Euler-Maruyama (EM) scheme nonlinear SISV-EM with varying parametersMean square error criteria is used for training of ANNs-BLMT to find approximate solutions to a variety of SISV-EM scenarios Computation of error histogram illustration, regression metrics, and MSE learning curves prove the performance of ANNs-BLMT. • The mean square error criteria are effectively utilized in approximation theory to develop an objective function for training the ANNs-BLMT to determine approximate solutions to a variety of stochastic nonlinear SISV-EM scenarios. • The computation of error histogram illustration, regression metrics, and MSE learning curves significantly improve the performance, precision, and consistency of the ANNs-BLMT for solving the stochastic nonlinear SISV-EM. [ABSTRACT FROM AUTHOR]

This article explores the question of whether artificial intelligence (AI) should be considered a coauthor of scientific papers. It provides a historical overview of AI, from its early beginnings to its current capabilities. The author argues that while AI can generate text, it lacks the ability to make new discoveries or truly understand concepts. The article also raises ethical concerns about the accuracy and truthfulness of AI-generated content. It concludes that while AI can assist in research, the responsibility for its use lies with human researchers, as AI tools are not yet capable of taking ethical responsibility for the research. [Extracted from the article]

In response to nonrenewable energy consumption and environmental pollution, the development of electric vehicles has accelerated. The safety of electric vehicles has garnered considerable attention. There are numerous incalculable foreign objects on the road that may collide with or scratch the battery-pack's frontal, resulting in battery-pack system damage or even explosion. This poses a significant risk to the safety of passengers and drivers. In this paper, a diversity of mechanical safety prediction models for battery-pack systems are proposed. These models support data-driven structural optimization of the battery-pack system by utilizing the numerical results of the bottom shell deformation. These simulation-based prediction models combine response surface method and machine learning algorithms. First, a nonlinear finite element model of the battery-pack system is established. The efficacy of the model is verified using constrained modal analysis in a variety of commercial software packages. Second, the collision simulations are executed and the data in various collision conditions are collected. Different sample sizes are used to develop various response surface models and machine learning models. The machine learning algorithms adopt support vector machine, Gaussian process regression, and neural network models. Third, the prediction accuracy of multiple prediction models is investigated according to error functions. The results show that the neural network model can predict the most accurate deformation under the condition of low speed frontal impact. The prediction average absolute percentage error of the neural network model is only 0.34% within the design domain, and only 2.54% outside the design domain. The proposed prediction model can be used for reliable design of the battery-pack system in electric vehicles. It also can be employed to design the early warning system of the battery-packs. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) for numerical solutions to partial differential equations (PDEs) have exhibited their remarkable merits of meshless methods, dimensionless features, and nonlinear approximation powers. The majority of conventional DNNs are mainly concentrated on the low- or high-dimensional PDE with smooth solutions. In singular problems the DNN does not behave so efficiently as in the smooth problem because of low regularities of the solution. In this paper, we propose a novel adaptive DNN for the PDE with corner singular solutions. In the corner singular problem gradients of solution vary rapidly around some particular vertices or lines. This poses a difficulty in development of DNN solvers. We design adaptive DNN techniques for the corner singularity, which incorporates (a) an adaptive loss function that assigns radial weights to local losses, (b) an adaptive activation function that varies the choice of activation functions from neuron to neuron, and (c) an adaptive sampling strategy that selectively focuses on sampling points near the singular locations. The adaptive DNN with these adaptive schemes greatly enhances approximation accuracy and computation efficiency. This is verified by a comprehensive series of numerical experiments of 2D and 3D corner singular problems. The comparison analyzes with the existing DNN solvers are also made to demonstrate effectiveness of the new method. The L 2 relative errors of the proposed adaptive DNN are reduced by an order of magnitude with respect to several conventional DNN methods. [ABSTRACT FROM AUTHOR]

The path planning and tracking control problem of autonomous vehicle with model matched uncertainty and external disturbance is studied in this paper. A safe lane change zone is designed by considering lane change time, distance, and driving speed. Meanwhile, the candidate paths are generated using cubic quasi-uniform B-spline curve, and the optimal path is selected based on feasibility, comfort, and efficiency criteria. The proposed planning method ensures an efficient and comfortable lane change path without collision with surrounding vehicles. Model uncertainty and external disturbance may affect lane change tracking control of autonomous vehicle. Hence, a fault-tolerant path tracking controller is implemented by using deep neural networks and adjustable zonotope-tube model predictive control. A deep neural network-based controller is proposed to compensate for model matched uncertainty, thereby enhancing the fault tolerance of the control system. Then, an adjustable zonotope-tube model predictive controller is designed to enclose the actual trajectory within a zonotope-tube with the nominal state as the center and the disturbance set as the radius by using the feedback control to achieve the asymptotic stabilization of the closed-loop system. The effectiveness of this path planning and tracking control method is finally verified by numerical simulation. • A lane change path planning method suitable for structured roads is proposed, which is based on the utilization of a safe lane change zone and cubic quasi-uniform B-spline curves. The proposed method ensures an efficient and comfortable lane change path. • A fault-tolerant path tracking controller is presented for autonomous vehicle, which incorporates a deep neural network and adjustable zonotope-tube model predictive control. The controller effectively handles model uncertainty and external disturbances. • The computation of robust positive invariant set is performed using the adjustable zonotope-tube method, which enhances scalability and reduces computational cost. [ABSTRACT FROM AUTHOR]

Explainable Artificial Intelligence (XAI) is targeted at understanding how models perform feature selection and derive their classification decisions. This paper explores post-hoc explanations for deep neural networks in the audio domain. Notably, we present a novel Open Source audio dataset consisting of 30,000 audio samples of English spoken digits which we use for classification tasks on spoken digits and speakers' biological sex. We use the popular XAI technique Layer-wise Relevance Propagation LRP to identify relevant features for two neural network architectures that process either waveform or spectrogram representations of the data. Based on the relevance scores obtained from LRP, hypotheses about the neural networks' feature selection are derived and subsequently tested through systematic manipulations of the input data. Further, we take a step beyond visual explanations and introduce audible heatmaps. We demonstrate the superior interpretability of audible explanations over visual ones in a human user study. • We present a novel audio dataset consisting of 30,000 audio samples of spoken digits. • We use LRP to explain predictions of two different models in the audio domain. • We confirm hypotheses about the neural networks' use of features from explanations. • We present audible explanations and demonstrate their superior interpretability. [ABSTRACT FROM AUTHOR]

Deep learning, with its powerful multilayer nonlinear representation of deep neural networks, enables models trained based on deep learning to describe the true distribution of data more accurately and thus have better generalization capabilities. However, the training data are usually assumed to obey independent identical distribution. Industrial process data nowadays usually have complex characteristics such as non-Gaussian, nonlinear, dynamic, strong correlation, and high-dimensional, which make it difficult to meet the assumptions of deep learning. Using industrial process data directly would result in low model accuracy and bias in the model output. This paper proposes a fault diagnosis method based on independent slow feature analysis (ISFA) and stacked sparse autoencoder (SSAE) network. The statistical properties possessed by ISFA are used to perform preliminary feature extraction of industrial process data, and then the extracted preliminary features are input to the SSAE network for fine-grained feature extraction. The proposed two-step feature extraction method not only makes the data input to the SSAE network as close as possible to obey independent identical distribution and provides convincing fault diagnosis results, but also makes full use of the powerful feature extraction capability of the SSAE network, which makes the proposed method also significantly improve the detection accuracy of incipient faults. The feasibility and superiority of the proposed method is verified through the TE process. • A two-step feature extraction fault diagnosis method is proposed. • Higher-order information and finer-grained features can be extracted. • The performance of both fault detection and classification has been validated. [ABSTRACT FROM AUTHOR]

The implications of neural synchronization extend beyond brain function, and can impact the development of artificial neural networks. This paper explores the synchronization of multiple fractional-order fuzzy complex-valued spatiotemporal neural networks (MFOFCVSNNs), which is novel and characterized using fuzzy logic and fractional-order partial differential equations, making it more adaptable and versatile. We first establish a new fractional-order complex-valued partial differential inequality, an integer-order complex-valued partial differential inequality, and an equation. Then, by combining the Lyapunov method with fuzzy set theory, employing newly established inequalities and equations, along with a newly designed fuzzy pinning controller, we derive two linear matrix inequality (LMI) formulations of synchronization criteria for MFOFCVSNNs using a direct non-complex decomposition approach. These criteria exhibit different dependencies on the membership function, with one being independent and the other dependent. Importantly, the criterion based on the membership function demonstrates reduced conservatism compared to its independent counterpart. By leveraging M -matrix theory, we present the synchronization criteria in a concise low-dimensional form. Moreover, this paper extends and enhances previous findings, resulting in reduced conservatism. Finally, we validate our theoretical analysis through numerical simulations. • A new model is proposed-multiple fractional-order fuzzy complex-valued spatiotemporal neural networks, which is novel and characterized using fuzzy logic and fractional-order partial differential equations, making it more adaptable and versatile. • Proposing a novel fuzzy pinning controller for large-scale networks, it streamlines implementation complexity and enhances design flexibility by not requiring identical fuzzy parameters as the model. • Exploring synchronization among multiple fractional-order fuzzy complex-valued delayed spatiotemporal neural networks without utilizing the complex-valued decomposition method. • The membership-function-independent synchronization criterion and the membership-function-dependent synchronization criterion are established. [ABSTRACT FROM AUTHOR]

This paper studies the fixed-time synchronization of a class of quaternion-valued neural networks (QVNNs) with time delays and impulses. In contrast to some existing decomposition methods for studying this problem, this paper presents several strategies to simplify the implicit Lyapunov function method from both controller and implicit function equation perspectives. The benefits of these strategies are, in the first strategy, the controller does not contain the Lyapunov function so that the controller will be more general, and in the second strategy, the implicit function equation is simpler so that the restriction on the bounded Lyapunov function can be dispensed with, which also makes the technique of handling time delays simpler. Furthermore, three sufficient conditions for fixed-time synchronization of the above QVNN are presented, despite the impulsive influence. Finally, we verify the feasibility of our methods with three examples. • Using a flexible implicit function approach to study fixed-time synchronization of QVNNs. • Fixed-time synchronization is studied without sign functions in the controller. • Non-decomposition method is utilized. [ABSTRACT FROM AUTHOR]

Lip reading is a visual alternative to enhance the intelligence of traditional speech recognition, which can benefit from retina-like event cameras that focus on dynamic movements. Spiking Neural Networks (SNNs) are inherently well-suited to cooperate with event cameras. However, employing SNN for event-based lip reading presents significant challenges, particularly in effectively extracting spatio-temporal features from input events and distinguishing between phonetically similar elements. To address these challenges, this paper proposes a novel event-based lip-reading model that leverages the SNN framework, enriched by a designed Spatial-Temporal Attention Block (STAB) and a triplet loss. Specifically, STAB comprises both spatial and temporal attention branches, dynamically emphasizing those spatial and temporal characteristics most reflective of lip movements. STAB also includes a fusion mechanism to integrate these spatial and temporal insights, providing a comprehensive and focused representation of lip movements. In addition, we enhance the model by incorporating triplet loss into the SNN training process, further improving the model's ability to distinguish between visually similar words. Experimental results show the superior performance of our SNN and validate the effect of STAB and triplet loss. We also conduct the energy consumption analysis to affirm the model's energy efficiency. [ABSTRACT FROM AUTHOR]

Dynamical rewiring widely exists in complex systems, however the impact of dynamical rewiring in the synchronization of neural systems is currently unknown. In this paper, we use memristive FitzHugh-Nagumo neurons to construct random, small-world and scale-free networks in which the connections between neurons can be rewired, and investigate the influence of rewiring on the synchronization of neural networks in with/without Gaussian white noise, and comparing it to the corresponding static networks. We found that dynamical rewiring enhances the synchronization of the network, and the degree of synchronization will be higher when the rewiring period is shorter and the rewiring proportion is larger. In addition, the synchronization of the network gradually diminishes as the coupling strength decreases and the noise intensity increases, and rewiring networks always exhibit superior synchronization to static networks since the dynamical rewiring enhances the interaction between neurons. Our study shows that neural network models with dynamically changing topology are more suitable and realistic network models, which may reveal the profound significance of dynamic rewiring for the multifaceted dynamic flexibility and adaptability of neural systems. • Memristive FitzHugh-Nagumo model is employed to construct random, small-world and scale-free networks. • Dynamical rewiring of connections between nodes is considered into the study of complex networks. • The shorter the rewiring period and the larger the rewiring proportion are, the better synchronized the network is. • Regardless of noise intensity and coupling strength, rewiring networks are always better synchronized than static networks. [ABSTRACT FROM AUTHOR]

In recent years, memristors have shown great potential in building artificial neural networks. Many researchers have worked on improving and innovating the design of memristors. The bifunctional memristor proposed in this paper provides new ideas for establishing artificial neural networks. And a novel second-order memristor Hindmarsh–Rose neuron (SOM-HR) is constructed on this basis. The SOM-HR model can well simulate the dynamical behavior of the memristive autapse HR neuron under electromagnetic radiation (EMR), and many types of firing patterns, coexisting firing patterns and state transition are found. In addition, the correctness of the numerical simulation is verified by circuit simulation and digital signal processor (DSP) implementation. Finally, the chaotic sequences generated by this model are used for image encryption, which guarantees the security of the encryption scheme. The results of the study provide a reference for construction and application of complex neural network. • A second-order flux-controlled memristor is designed. • An analog circuit for the memristor was constructed. • Biological neuron modeling based on bifunctional memristor • The dynamical behavior of the new neural model is analyzed. • The neuron model was implemented by analog circuit and DSP platform respectively. [ABSTRACT FROM AUTHOR]

This paper presents a novel approach to advancing artificial intelligence (AI) through the development of the Complex Recurrent Spectral Network (ℂ -RSN), an innovative variant of the Recurrent Spectral Network (RSN) model. The ℂ -RSN model introduces localized non-linearity, complex fixed eigenvalues, and a distinct separation of memory and input processing functionalities. These features enable the ℂ -RSN to evolve towards a dynamic, oscillating final state that bear some degree of similarity with biological cognition. The model's ability to classify data through a time-dependent function, and the localization of information processing, is demonstrated by using the MNIST dataset. Remarkably, distinct items supplied as a sequential input yield patterns in time which bear the indirect imprint of the insertion order (and of the separation in time between contiguous insertions). • The C-RSN overcomes some limitations in existing neural network models. • C-RSN introduces localized non-linearity, complex eigenvalues and a separated memory. • The network evolves towards a dynamic, oscillating final state. • The model's efficacy is demonstrated through empirical evaluation. • C-RSN is able to process sequential inputs keeping track of the insertion order. [ABSTRACT FROM AUTHOR]

In this paper, a fuzzy-dependent adaptive event-triggered mechanism (FAETM) for synchronizing Takagi–Sugeno (T–S) fuzzy reaction–diffusion neural networks (RDNNs) is developed while considering deception attacks. Firstly, a general neural network model considering both fuzzy logic rules and reaction–diffusion terms is established. Secondly, a FAETM based on an aperiodic sampling period is presented under deception attacks to alleviate the communication burden, wherein the adaptive threshold function is dependent on membership functions. Moreover, a membership-function-dependent asymmetric Lyapunov functional (LF) is constructed, and some positive-definite and symmetric constraints of matrices are removed as a result. Based on the LF method and integral inequality techniques, the H ∞ synchronization criteria for the T–S fuzzy RDNNs are presented by linear matrix inequalities (LMIs). Finally, one simulation example is exploited to demonstrate the feasibility and validity of the established method, and the outcome is applied to image secure communication. • A T–S fuzzy reaction diffusion neural networks is developed. • A fuzzy adaptive event-triggered mechanism under deception attacks is designed. • A membership-functions-dependent asymmetric Lyapunov functional is constructed. • A less conservative stability criterion is obtained. • An application to image encryption is proposed. [ABSTRACT FROM AUTHOR]

The continuous development of machine learning algorithms has stimulated the technological revolution on turbulence modeling for Reynolds-averaged Navier–Stokes (RANS) simulations. In this paper, a computable transition-enabled turbulence model is developed with the aid of an artificial neural network (ANN), which maps the relation between the mean flow variables from the shear stress transport (SST) model on coarse grid and the turbulent viscosity from SST- γ model on fine grid. It turns out that the ANN model can predict the aerodynamic integral quantities, transition onset location, laminar separation bubble structure, streamwise velocity distribution, etc. with superior accuracy and robustness for subsonic and moderate transonic airfoil flows. Meanwhile, the ANN model can significantly improve the convergence property with a much higher convergence speed of the model equation in comparison with traditional transition-enabled RANS model. Moreover, due to the lack of solving RANS model equations and use of grid interpolation technology, the computation speed of this ANN model is improved by a factor of about six over the conventional benchmark model. Therefore, the present computable ANN model provides a new perspective for high-efficiency simulation of transition-characterized flows in engineering applications. • A new transition-turbulence modeling strategy is suggested based on artificial neural network (ANN). • The ANN algorithm provides a corrector for the eddy viscosity in full-turbulence model. • The ANN model is intended to mimic the eddy viscosity in transition-enabled model. • No turbulence model equations need to be solved while retaining a good generalization property. • Grid interpolation and frozen correction help improve the computation efficiency while ensuring the simulation accuracy. [ABSTRACT FROM AUTHOR]

This paper introduces a novel machine learning approach, called BBATDD (Boosting-Based Algorithm Trained with Drift Detector), that departs from traditional epoch-based methods by adopting a data stream generation strategy from the original dataset. The proposed method is versatile, applicable to a broad range of deep structures and other machine learning architectures. The key to its effectiveness is the conversion of original data into a streaming sequence, which is then fed into the neural network structure. To monitor the learning process and prevent overfitting, a drift detector, commonly used in stream data mining, is employed. Additionally, two methods, ELB and ENT, are introduced to prioritize problematic data elements during the learning process, enhancing overall model performance. Extensive simulations on benchmark datasets (MNIST and CIFAR) validate that BBATDD achieves higher accuracy with significantly fewer batches, without sacrificing generalization capability. For instance, when applied to the MNIST dataset and processing 10 000 batches while comparing loss function values, our approach demonstrates a remarkable 45.6% improvement. [ABSTRACT FROM AUTHOR]

The k Nearest Neighbors (k NN) search in high-dimensional space is a fundamental problem with various applications. In this paper, we try to solve this problem by using deep neural networks (DNNs). We apply DNNs to represent complex correlations between high-dimensional objects, enabling us to project similar objects into the same class, thus reducing the search cost. Based on DNNs, we propose two novel techniques to improve query efficiency while keeping accuracy. First, traditional DNNs typically demand extensive training data for achieving high accuracy. To decrease the training size, we design a multi-module DNN framework comprising several small modules. Each module learns to capture part of knowledge for the given query. The collective output of these sub-modules is then seamlessly integrated to form the final result. Second, with machine learning models, the size of candidates are unbounded. Thus, we design a linear-time data layout refinement algorithm, aiming to limit the number of candidates to a small constant. Empirically we find that our approach significantly outperforms the state-of-the-art methods in terms of both time efficiency and space efficiency while still attaining comparable or better accuracy. [ABSTRACT FROM AUTHOR]

This paper presents a comprehensive empirical investigation into the interactions between various randomization techniques in Deep Neural Networks (DNNs) and their impact on learning performance. It is well-established that injecting randomness into the training process of DNNs, through various approaches, at different stages, is often beneficial for reducing overfitting and improving generalization. Nonetheless, the interactions between randomness techniques such as weight noise, dropout, and many others remain poorly understood. Consequently, it is challenging to determine which methods can be effectively combined to optimize DNN performance. To address this issue, we categorize the existing randomness techniques into four key types: injection of noise/randomness at the data, model structure, optimization or learning stage. We use this classification to identify gaps in the current coverage of potential mechanisms for the introduction of randomness, leading to proposing two new techniques: adding noise to the loss function and random masking of the gradient updates. In our empirical study, we employ a Particle Swarm Optimizer (PSO) for hyperparameter optimization (HPO) to explore the space of possible configurations to determine where and how much randomness should be injected to maximize DNN performance. We assess the impact of various types and levels of randomness for DNN architectures across standard computer vision benchmarks: MNIST, FASHION-MNIST, CIFAR10, and CIFAR100. Across more than 30 000 evaluated configurations, we perform a detailed examination of the interactions between randomness techniques and their combined impact on DNN performance. Our findings reveal that randomness through data augmentation and in weight initialization are the main contributors to performance improvement. Additionally, correlation analysis demonstrates that different optimizers, such as Adam and Gradient Descent with Momentum, prefer distinct types of randomization during the training process. A GitHub repository with the complete implementation and generated dataset is available.2 [ABSTRACT FROM AUTHOR]

Correlation filter-based video object-tracking algorithms have gained widespread attention due to their efficiency and excellent tracking performance. However, traditional correlation filtering tracking algorithms possess several limitations. (1) They extract image features only by using rule sampling, which ignores the shape information of the target, resulting in insufficient discriminative power of the features. (2) They focus only on the difference between the background and the object, ignoring the effects of more challenging intra-class interference. (3) They do not further evaluate the reliability of the optimal candidate samples, resulting in easy failure in occlusion scenarios. This paper proposes an asymmetric background-aware correlation filter method for object tracking with deep features to solve the above limitations. First, an asymmetrical background-aware sampling method based on the shape information of the object is proposed. This sampling method significantly differs from the symmetrical sampling method in the traditional correlation filter framework. By exploring the shape information of the object, improving the otherness between the background and object samples is easy, thus suppressing the intra-class interferences. Second, deep neural networks are introduced in the correlation filter framework to extract the deep object features and a spatio-temporal regularization factor is adaptively assigned to suppress the boundary effects, intra-class distractors and aberrance between frames. Finally, a multi-modal object pool is constructed to evaluate the optimal candidate sample in each frame. This template pool fully exploits object diversity and solves the tracking drift and failure caused by invalid appearance changes in scenes such as occlusion and vigorous motion scenarios. To validate the effectiveness of the proposed method, it was compared with the state-of-the-art methods on public datasets. The experimental results show that the tracking performance of the proposed method is competitive. [ABSTRACT FROM AUTHOR]

Activation functions play a crucial in emulating biological neurons within artificial neural networks. However, the exploration of neural networks composed of various activation functions and their associated dynamics have not been noticed yet. This paper proposes a novel method by introducing heterogeneous activation functions into a memristive Hopfield neural network for the first time. The special feature of the proposed model lies not only in its ability to mimic the diversity of brain neurons, providing a more realistic and adaptable frame for artificial neural networks but also in its rich dynamic properties suitable for engineering applications. Theoretical and experimental investigations into the dynamics of the memristive Hopfield neural network are conducted, employing phase portraits, bifurcation diagrams, Lyapunov exponent spectra, 0–1 tests, and bi-parameter dynamic maps. Complex dynamical behaviors, including periodic bursting, chaotic bursting, and chaotic state jump are revealed by the numerical simulations. Furthermore, a hardware implementation of the proposed neural network is designed and validated through circuit simulation software, which is consistent with the numerical simulation and confirms the validity of the proposed model. Finally, an encryption scheme based on the chaotic bursting is also proposed and evaluated. Results demonstrate that the chaotic bursting attractor exhibits excellent randomness, making it well-suited for image encryption applications. The novel exploration of heterogeneous activation neuronal networks in this paper may pave the way for further research in the field of more bionic networks with complex dynamical behaviors and their applications. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) have recently been found to be vulnerable to adversarial examples, which raises concerns about their reliability and poses potential security threats. Adversarial training has been extensively studied to counter adversarial attacks. However, the limited attack types incorporated during the training phase will restrict the defense performance of models against unknown attacks and impact their standard accuracies. Furthermore, we discover that adversarial training models tend to overfit redundant noisy features, which hinders their generalization. To alleviate these issues, this paper proposes the attention information bottleneck-guided knowledge distillation (AIB-KD) method to enhance models' adversarial robustness. We integrate adversarial training with attention information bottleneck as the defense framework to achieve an optimal trade-off between information compression and classification performance. Simultaneously, we specifically employ knowledge distillation to guide the adversarial training models in learning both the standard attention information and valuable deep feature distributions to enhance their defense generalization capability. Experimental results demonstrate that AIB-KD can effectively classify adversarial examples in multiple attack settings. The average white-box and black-box classification accuracies for the WideResNet-28-10 model on the CIFAR-10 dataset are 56.59% and 85.49%, respectively, and the average accuracies on the SVHN dataset are 61.71% and 88.96%. When applied to unknown attack scenarios, AIB-KD is more effective and interpretable than state-of-the-art methods. [ABSTRACT FROM AUTHOR]

Deep learning-based recommendations have demonstrated impressive performance in improving recommendation accuracy. However, such approaches mainly utilize implicit feedback to predict user preferences and neglect the adverse impact of explicit preference noise, which affects the robustness and reliability of model training. To consider the reliability of both rating input and output, we propose a novel joint deep neural recommendation framework that incorporates rating reliability derived solely from ratings to provide reliable recommendations for active users. Firstly, we introduce a noise detection method based on intuitionistic fuzzy sets to identify incorrect ratings from the perspective of fuzzy preferences and label them to generate a binary rating reliability matrix. Subsequently, we propose a joint deep neural framework that integrates rating reliability to simultaneously capture the high-order features of users and items, yielding predictions with their corresponding reliability probabilities. Finally, to achieve a balance between accuracy and reliability for recommendations, we design a reliability threshold selection strategy based on K-means clustering to find an appropriate threshold. Experimental results on three widely used datasets show that our model achieves an average improvement of 9.4% and 8.0% in the metrics Recall and NDCG, respectively, compared with the closest competitor. This paper provides new insights for integrating rating reliability into a deep neural network to enhance the performance of recommender systems. [ABSTRACT FROM AUTHOR]

Sugar crystal growth is a non-linear process dependent on the complex interaction of several variables. This causes the development of first-principal models of the process to be exceedingly difficult and often inaccurate. Louisiana Sugar Refining has partnered with SysCAD to develop a machine learning model of crystal growth to incorporate into the digital twin infrastructure being developed at LSR. The collaboration has successfully developed a supervised training machine learning model based on Deep Feed Forward (DFF) and Convolutional Neural Network (ConvNet) models to simulate the crystal growth in vacuum pans. Initially, a proof-of-concept model showed promising results using both DFF and ConvNet architectures, which lead to the expansion of these models to include the entire white and recovery boiling schemes. In this paper we present the concept of using neural network dynamic models, their analysis, and results of predicted particle size distribution parameters compared to real field observations. These fully optimized models will be implemented as standard unit operation models in SysCAD to enhance the larger process simulation model used by LSR to monitor and predict process conditions in real-time. This machine learning technique will also be expanded to develop a concept model for other non-linear phenomena in the sugar industry including colour. [ABSTRACT FROM AUTHOR]

The firing dynamics of excitable systems are critical to understand organized responses in cortical networks. In this paper, we examine a fractional-order Wilson–Cowan (W–C) neural network model composed of excitatory and inhibitory neuron populations, utilizing Caputo's fractional-order derivative formalism to explore the influence of fractional-order dynamics on firing behavior. The significance of extending to the fractional-order domain lies in the model's theoretical framework, which inherently retains memory and hereditary characteristics. We investigate memory-dependent response functions and average neuronal characteristics, enabling us to formulate a fractional-order model that incorporates past dynamics into the neuronal populations' features. This generalized model is capable of producing alternations between spiking and bursting phenomena, including mixed-mode oscillations (MMOs). Using stability and bifurcation analyses, we delineate the parameter space within which variations in firing patterns emerge. One notable impact of the model's memory property is the potential stabilization of neuronal activity. We demonstrate that our numerical findings are in alignment with the analytical predictions and that the memory trace depends on the fractional-order dynamics. • We explore memory effects in neuronal dynamics using fractional-order Wilson–Cowan model. • Stability and bifurcation are investigated in memory-dependent neuronal firing. • Mixed-mode oscillations in fractional-order neuronal systems are demonstrated. [ABSTRACT FROM AUTHOR]

Neural network solvers represent an innovative and promising approach for tackling time-fractional partial differential equations by utilizing deep learning techniques. L1 interpolation approximation serves as the standard method for addressing time-fractional derivatives within neural network solvers. However, we have discovered that neural network solvers based on L1 interpolation approximation are unable to fully exploit the benefits of neural networks, and the accuracy of these models is constrained to interpolation errors. In this paper, we present the high-precision Hermite Neural Solver (HNS) for solving time-fractional partial differential equations. Specifically, we first construct a high-order explicit approximation scheme for fractional derivatives using Hermite interpolation techniques, and rigorously analyze its approximation accuracy. Afterward, taking into account the infinitely differentiable properties of deep neural networks, we integrate the high-order Hermite interpolation explicit approximation scheme with deep neural networks to propose the HNS. The experimental results show that HNS achieves higher accuracy than methods based on the L1 scheme for both forward and inverse problems, as well as in high-dimensional scenarios. This indicates that HNS has significantly improved accuracy and flexibility compared to existing L1-based methods, and has overcome the limitations of explicit finite difference approximation methods that are often constrained to function value interpolation. As a result, the HNS is not a simple combination of numerical computing methods and neural networks, but rather achieves a complementary and mutually reinforcing advantages of both approaches. The data and code can be found at https://github.com/hsbhc/HNS. • A high-order explicit approximation scheme for the Caputo fractional derivative is constructed. • An efficient Hermite Neural Solver (HNS) is designed for solving time fractional partial differential equations. • The impact of Hermite interpolation order on the accuracy of the HNS is analyzed. • The performance of HNS has been thoroughly verified across various problem scenarios. [ABSTRACT FROM AUTHOR]

• The proposed FE-RNN learns incrementally and inferences on unseen time-series data. • The FE-RNN is assessed on chaotic time-series datasets for its forecasting ability. • An embedded fuzzy network provides interpretability of the neural network structure. • The FE-RNN based GA-fMACDH strategy performs well on returns and maximum drawdown. Deep learning enables effective predictions. But deep structures face some challenges on human interpretability compared to conventional techniques, e.g., fuzzy inference systems. It motivates more research works to alleviate the black box nature of deep structures with performance maintained. This paper proposes a fuzzy-embedded recurrent neural network (FE-RNN) to improve interpretability of the underlying neural networks. It is a parallel deep structure comprising an RNN and a Pseudo Outer-Product based Fuzzy Neural Network (POPFNN) that share a common set of input and output linguistic concepts. The inference processes undertaken are associated by RNN using fuzzy rules in the embedded POPFNN. Fuzzy IF-THEN rules provide better interpretability of the inference process of the hybrid networks. It allows an effective realisation of a data driven implication using RNN in the modelling of fuzzy entailment within a fuzzy neural networks (FNN) structure. FE-RNN obtains more consistent results than other FNN in the experiment using the Mackey-Glass dataset. FE-RNN achieves about 99% correlation for forecasting prices of market indexes. Its interpretability is also discussed. FE-RNN then acts as a prediction tool in a financial trading system using forecast-assisted technical indicators optimised with Genetic Algorithms. It outperforms the benchmark trading strategies in the trading experiments. [ABSTRACT FROM AUTHOR]

The 12-lead electrocardiogram (ECG) is a reliable diagnostic tool for detecting and treating severe cardiovascular conditions like arrhythmia and heart attack. Deep neural networks (DNNs) have achieved higher accuracy in recent years than traditional ECG signal classification task methods. Convolutional neural network (CNN) and Transformer are the two mainstream architectures of DNN, respectively good at extracting local and global features from input data. This paper proposes the multi-scale convolutional Transformer network (MCTnet), an efficient combination of Transformer encoder and CNN for ECG signal classification. MCTnet utilizes the advantages of CNN and self-attention mechanisms to capture potential features in ECG signal accurately. The dual-branch Transformer encoder extracts different-scale feature representations, enabling the capture of both local and global information. Additionally, an information bottleneck method eliminates redundant information and enhances task-relevant information in the learned representations. To evaluate the performance of MCTnet, comprehensive experiments are conducted on three commonly used ECG datasets. The results demonstrate that MCTnet outperforms current deep learning-based models, highlighting its effectiveness in ECG signal classification. It also shows that the performance of the model can be effectively improved by utilizing multi-scale representation learning and information bottleneck. [ABSTRACT FROM AUTHOR]

Turing dynamics mainly focuses on non-homogeneous self-adaptive spatial patterns of reaction diffusion systems in a continuous space. If defining the diffusion environment as network structure, then network patterns are available under Turing instability conditions. In allusion to parameter estimation for network Turing-instable systems, this paper proposes a new recognition method using artificial neural networks. When diffusion occurs on quadrilateral lattice networks, a deep convolutional neural network (CNN) is built to invert the unknown parameters. The results on training set and validation set indicate that the CNN model exhibits great robustness without overfitting and gradient explosion. Meanwhile, the prediction performance on two test sets is excellent, with average relative errors of the estimators ending up at 0.68% and 1.04%, respectively. When diffusion occurs on non-regular networks, a spatial domain graph convolutional network (GCN) is established, which is slightly less robust than the CNN. The average relative error of the estimators on both test sets ends up between 1.1% and 2.8%, which is still in the ideal range. • The parameter estimation method for network Turing systems is investigated. • A 14-layer deep neural network is designed for systems on lattice networks. • A 6-layer graph neural network is designed for systems on irregular networks. [ABSTRACT FROM AUTHOR]

In this paper, a neural network approach is suggested to the product design analysis. Namely, fractional-order neural network models are proposed as more flexible mechanism to study product form design. Since control and stability methods are fundamental in the construction and practical significance of a neural network model, appropriate controllers are designed and practical stability criteria are proposed for the fractional-order model under consideration. The stability and control analysis are based on the Lyapunov function method. Examples are elaborated to demonstrate the established results. The proposed modeling approach and the stability results are also applicable to numerous industrial design tasks. • Fractional-order neural network modeling approach is introduced to the product form design. • Appropriate controllers are utilized. • The practical stability notion is adopted to the introduced model. • Practical stability criteria are established using the Lyapunov function technique. • Examples and discussion are also offered to verify and justify the proposed results. [ABSTRACT FROM AUTHOR]

Prediction results in big data analysis can vary greatly depending on the data preprocessing methods used. Time series-based processing methods are particularly advantageous for prediction. While popular neural network models such as Back Propagation (BP), Recurrent Neural Network (RNN), and Long Short-Term Memory (LSTM) are based on weight, loss function, and other factors, their training efficiency is still relatively low. In this paper, we propose an efficient Time-series Recurrent Neural Network (TRNN) for stock price prediction. In the proposed model, trading volume is established and sliding windows are used to process the time series data. The trends and turning points of the data are extracted according to financial market features, and data compression is achieved. To improve the impact of recent trading volume on the current stock price, the price-volume relationship is upgraded from one dimension to two dimensions based on RNN. The information about trading volume is processed and compressed to establish the TRNN model, which guarantees both accuracy and efficiency. We compare our TRNN model with the original RNN and LSTM models in terms of efficiency and accuracy. We further discuss the feasibility of related expanded schemes of our TRNN model, as well as the extendability of the time-series compression and TRNN model to other fields. [ABSTRACT FROM AUTHOR]

Deep neural networks (DNNs) have tremendously succeeded in several computer vision-related fields. Nevertheless, previous research demonstrates that DNNs are vulnerable to adversarial sample attacks. Attackers add carefully designed perturbation noise to clean samples to form adversarial samples, which may lead to errors in the DNNs' predictions. Consequently, the safety of deep learning has attracted much attention, and researchers have commenced exploring adversarial samples from different perspectives. In this paper, a method based on class gradient networks (CGN) is proposed, which can generate high-quality adversarial samples by designing multiple objective functions. Specifically, the adversarial sample's high-level features are guided to change by introducing a high-level class gradient matrix, and the classification loss and perturbation loss are combined to jointly train a generator to fit the distribution of adversarial noises. We conducted experiments on two standard datasets, Fashion-MNIST and CIFAR-10. The results demonstrate the superiority of our method in the transferability of adversarial samples on targeted attacks and indicate the approach outperforms the baseline method. [ABSTRACT FROM AUTHOR]

Quantile regression is widely applied in various fields such as economy, energy, meteorological prediction research in recent years since it does not require distribution assumptions, has relatively loose conditions, and can effectively estimate the uncertainty of time series forecasting. In this paper, a monotone quantile regression neural network (MQRNN) framework is constructed for time series quantile forecasting. The proposed approach takes the monotonicity of quantile into consideration and handles the quantile crossing problem by adding the quantile information into the input structure and using the gradient based point-wise loss function. Aiming at the complex characteristics of time series, such as time-varying and asymmetric heavy-tailed features, a new quantile function is utilized to describe the complete conditional distribution information of data. Under this model framework, non-crossing multiple quantiles can be predicted simultaneously. The proposed approach is implemented based on artificial neural networks, and the constructed model is applied to actual data in different fields. The experimental results demonstrate that the proposed method combined with long short-term memory (LSTM) can provide accurate and reliable multi-quantile prediction, and alleviate the problem of quantile crossing. [ABSTRACT FROM AUTHOR]

Recent work has proposed novel data augmentation methods to improve the adversarial robustness of deep neural networks. In this paper, we re-evaluate such methods through the lens of different metrics that characterize the augmented manifold, finding contradictory evidence. Our extensive empirical analysis involving 5 data augmentation methods, all tested with an increasing probability of augmentation, shows that: (i) novel data augmentation methods proposed to improve adversarial robustness only improve it when combined with classical augmentations (like image flipping and rotation), and even worsen adversarial robustness if used in isolation; and (ii) adversarial robustness is significantly affected by the augmentation probability, conversely to what is claimed in recent work. We conclude by discussing how to rethink the development and evaluation of novel data augmentation methods for adversarial robustness. Our open-source code is available at https://github.com/eghbalz/rethink_da_for_ar. • Augmentation methods for adversarial robustness are often not tested in isolation. • They are often tested on one single value of augmentation probability. • They improve robustness only when combined with classical augmentations. • The augmentation probability significantly affects adversarial robustness. [ABSTRACT FROM AUTHOR]

This paper proposes a flexible probabilistic activation function that enhances the training and operation of artificial neural networks by intentionally injecting noise to gain additional control over the response of each neuron. During the learning phase, the level of injected noise is iteratively optimized by gradient-descent, realizing a form of adaptive stochastic resonance. From simple hard-threshold non-differentiable neuronal responses, controlled injection of noise gives access to a wide range of useful activation functions, with sufficient differentiability to enable gradient-descent learning for both the neuron and the injected-noise levels. Experimental results on function approximation demonstrate injected noise generally converging to non-vanishing optimal levels associated with improved generalization abilities in the neural networks. A theoretical explanation of the generalization improvement based on the path norm bound is presented. With injected noise in the deep neural network, experimental results on classifying images also obtain non-vanishing optimal noise levels to achieve better testing accuracies. The proposed probabilistic activation functions show the potential of adaptive stochastic resonance for useful applications in machine learning. • Establishing a flexible probabilistic activation function model on noise injection. • Explanation of the generalization improvement of the designed neural network. • Achieving better testing accuracies at non-vanishing optimal noise levels. [ABSTRACT FROM AUTHOR]

The multi-state memristor is a type of memristor capable of memorizing multiple conductance states. This special feature is successfully used to mimic the behavior of neural synapses and simulate the influences of electromagnetic induction and radiation in artificial neural networks such as Hopfield neural networks. This paper introduces and examines a novel fractional-order Hopfield neural network model consisting of two non-identical sub-Hopfield neural networks coupled by a multi-stable flux controlled memristor (MCFHNN). The pinched hysteresis loops of the multi-stable flux controlled memristor are analyzed via numerical simulations. The equilibrium points of MCFHNN model and their stability are investigated. The introduced MCFHNN model is solved by using Adomian Decomposition Method (ADM). By using some examination methods such as spectral entropy complexity, bifurcation diagrams, phase portraits, maximum Lyapunov exponent and basins of attraction, we analyze the dynamic behaviors of MCFHNN model associated with coupling strength, fractional-order and initial states. Numerical results demonstrate that the proposed MCFHNN model is capable to develop abundant and complicated dynamics including, chaos, multiple transient transition behaviors, double bubble bifurcations, initial-boosted behavior and symmetric multi-scroll chaotic attractors. It is found that the complexity results are in agreement with those obtained from the bifurcation diagrams. This proves that the complexity can well reflect the dynamic behaviors of MCFHNN model. In order to support theoretical and numerical results, the designed MCFHNN model is implemented with the help of an Arduino-Due microcontroller board based on the Atmel SAM3X8E ARM Cortex-M3 processor. The results are in very good agreement with those obtained from computational simulations. • Design a fractional-order neural network model consisting of two non-identical sub-Hopfield neural networks coupled by a multistable flux-controlled memristor (MCFHNN); • The Adomian Decomposition Method (ADM) is applied to the MCFHNN model to guarantee accurate approximation of the fractional derivatives and rapid convergence of the optimization procedure; • The MCFHNN model exhibits complex features and abundant dynamics, similar to those of the human brain. To our knowledge, these unique features have not yet been observed in other fractional-order neural network architectures; • The MCFHNN model is implemented using an Arduino-Due board. The results are in very good agreement with those obtained from computational simulations. [ABSTRACT FROM AUTHOR]