Search

Nowadays, neural networks have been widely applied in many fields such as pattern recognition, signal and image processing and control theory. Over the past two decades or so, the analysis and synthesis for neural networks have received significant research attention. This paper provides a survey on the analysis and synthesis for neural networks, which is mainly concerned with the recent advances on stability analysis, state estimation and synchronization control for neural networks. First of all, the paper summarizes the recent results on the stability analysis for delayed neural networks, especially for neural networks with multiple discrete delays, neural networks with distributed delays, and discrete-time delayed neural networks. Then, the paper reviews the recent advances regarding the state estimation for neural networks with the emphasis on the network-based state estimation. Subsequently, the paper provides an overview on the synchronization control for neural networks. Finally, the conclusions and further research directions are given. [ABSTRACT FROM AUTHOR]

This paper focusses on the optimal implementation of a Mean Variance Estimation network (MVE network) (Nix and Weigend, 1994). This type of network is often used as a building block for uncertainty estimation methods in a regression setting, for instance Concrete dropout (Gal et al., 2017) and Deep Ensembles (Lakshminarayanan et al., 2017). Specifically, an MVE network assumes that the data is produced from a normal distribution with a mean function and variance function. The MVE network outputs a mean and variance estimate and optimizes the network parameters by minimizing the negative loglikelihood. In our paper, we present two significant insights. Firstly, the convergence difficulties reported in recent work can be relatively easily prevented by following the simple yet often overlooked recommendation from the original authors that a warm-up period should be used. During this period, only the mean is optimized with a fixed variance. We demonstrate the effectiveness of this step through experimentation, highlighting that it should be standard practice. As a sidenote, we examine whether, after the warm-up, it is beneficial to fix the mean while optimizing the variance or to optimize both simultaneously. Here, we do not observe a substantial difference. Secondly, we introduce a novel improvement of the MVE network: separate regularization of the mean and the variance estimate. We demonstrate, both on toy examples, multiple benchmark UCI regression data sets, and on the UTKFace data set, that following the original recommendations and the novel separate regularization can lead to significant improvements. [ABSTRACT FROM AUTHOR]

This paper presents a novel safe integral reinforcement learning (IRL)-based optimal trajectory tracking scheme for nonlinear systems with uncertain dynamics that is subject to constraints. We leverage multilayer neural networks (MNNs) for actor-critic MNNs along with an NN identifier in the backstepping process for minimizing a discounted value function. A time-varying barrier Lyapunov function (TVBLF) is utilized for handling constraints and to provide safety assurances. Online weight update laws for the actor and critic MNNs are derived that are driven by Bellman error and control input error. We introduce an online lifelong learning (LL) method in the critic NN, utilizing the Bellman error in MNNs to address catastrophic forgetting. The method's effectiveness is demonstrated through simulations on mobile robot multitask tracking. The paper concludes with a stability analysis of the closed-loop system. [ABSTRACT FROM AUTHOR]

This paper investigates the problem of adaptive tracking control for multi-agent systems subject to output constraint and unknown hysteresis. Different from most existing results that use the barrier Lyapunov function and integral barrier Lyapunov function to solve the output constraint problem, the transformation technique is introduced in this paper, which can convert the original system to a constrained or completely unconstrained system. The difficulty caused by unknown Bouc-Wen hysteresis is solved via the Nussbaum gain technique. The continuous unknown function is estimated by using the radial basis function neural networks. The tracking differentiator is utilized to overcome the difficulty of "explosion of complexity". By using the Lyapunov stability theory, it is indicated that all signals of the closed-loop system are semi-globally uniformly ultimately bounded, and the tracking error converges to a small neighborhood of the origin. Eventually, the feasibility of the designed approach is demonstrated by some simulation results. [ABSTRACT FROM AUTHOR]

This paper considers the leader-following consensus problem for delayed neural networks with multi-layer signed digraph topologies. Both structurally balanced and unbalanced topologies are considered. By using the structural balance theory and the pinning control strategy, some sufficient conditions are given to guarantee that the bipartite leader-following consensus can be reached in structurally balanced multi-layer signed neural networks, which depend on the node dynamics and the topology of the multi-layer signed network. In addition, a scheme is designed for selecting the pinning nodes and gains of each layer to achieve bipartite leader-following consensus. When the multi-layer signed network is structurally unbalanced, the asymptotic stability of the neural network is investigated by decomposing the structurally unbalanced network into a structurally balanced subgraph and some special edges. The conditions proposed in this paper are also suitable for addressing the consensus problem of traditional single layer signed neural networks. Numerical examples are given to illustrate the correctness and effectiveness of the proposed theorems and algorithms. [ABSTRACT FROM AUTHOR]

• This paper addresses the problem of depth completion, which plays important roles in a myriad of vision and multimedia applications. • It proposes a novel multi-cue guidance network model, which introduces multi-cue guidance and seeks to predict residual depth values. • It tests the proposed model on challenging indoor and outdoor scene datasets. The results outperform the comparison approaches and extensive ablation studies prove the effectiveness of different modules in the proposed model. Depth completion is to predict a dense depth image from a raw sparse depth image with missing values, which is an important yet challenging problem in a myriad of vision, robotics, and multimedia applications. While previous studies have made substantial progress in this issue, most of them directly fuse RGB and depth features without considering the multiple latent cues in data. In this paper, we propose a multi-cue guidance network model for depth completion, which introduces multi-cue features to guide the regression of residual values. This network includes two major parts: multi-cue guidance structure and residual regression structure. The multi-cue guidance structure composed of two parallel convolutional sampling streams extracts features from the raw depth image and the corresponding RGB image, respectively. The extracted features are used as multi-cue guidance to combine with the depth image to feed the residual regression structure, where the estimated residual values are combined with the raw depth values to output the final dense depth values. By virtue of the multi-cue guidance and residual regression, the proposed model can leverage the multiple latent cues in the data to predict more accurate depth values. The proposed method was tested on two challenging datasets of outdoor and indoor scenes: KITTI benchmark and NYUv2 dataset. The experimental results show that the proposed method outperforms the comparison methods. Extensive ablation studies demonstrate the effectiveness of different modules in the proposed method. [ABSTRACT FROM AUTHOR]

• ANN, CART, and RF regression trees have shown the potential in modeling energy cost. • Three different strategies regarding variable selection were tested and compared. • Machine learning and Boruta method have produced the highest accuracy of prediction. • The model has extracted heating and occupational data as the most important. • The created model could be used to assess the concept of smart buildings and cities. The paper deals with modeling the cost of energy consumed in public buildings by leveraging three machine learning methods: artificial neural networks, CART, and random forest regression trees. Energy consumption is one of the major issues in global and national policies, therefore scientific efforts in creating prediction models of energy consumption and cost are highly important. One of the largest energy consumers in every state is its public sector, consisting of educational, health, public administration, military, and other types of public buildings. Recent technologies based on sensor networks and Big data platforms enable collection of large amounts of data that could be used to analyze energy consumption and cost. A real data from Croatian public sector is used in this paper including a large number of constructional, energetic, occupational, climate and other attributes. The algorithms for data pre-processing and modeling by optimizing parameters are suggested. Three strategies were tested: (1) with all available variables, (2) with a filter-based variable selection, and (3) with a wrapper-based variable selection which integrates Boruta algorithm and random forest. Prediction models of energy cost are created using two approaches: (a) comparative usage of artificial neural networks and two types of regression trees, CART and random forest, and (b) integration of RF-Boruta variable selection and machine learning methods for prediction. A cross-validation procedure was used to optimize the artificial neural network and regression tree topology, as well to select the most appropriate activation function. Along with creating a prediction model, the aim of the paper was also to extract the relevant predictors of energy cost in public buildings which are important in planning the construction or renovation of buildings. The results have shown that the second approach which integrates machine learning with Boruta method, where the random forest algorithm is used for both variable reduction and prediction modeling, has produced a higher accuracy of prediction than the individual usage of three machine learning methods. Such findings confirm the potential of hybrid machine learning methods which are suggested in previous research, but in favor of random forest method over CART and artificial neural networks. Regarding the variable selection, the model has extracted heating and occupational data as the most important, followed by constructional, cooling, electricity, and lighting attributes. The model could be implemented in public buildings information systems and their IoT networks within the concept of smart buildings and smart cities. [ABSTRACT FROM AUTHOR]

This paper presents the theoretical results on the multistability of state-dependent switching neural networks with discontinuous nonmonotonic piecewise linear activation functions. For n -neurons switching model, this paper shows that neural networks have 7 n equilibrium points, 6 n of which are located at the continuous points of activation functions and others are located at the discontinuous points of activation functions. Among these equilibrium points, 4 n or 5 n are stable and others are unstable, which depend on the relationship between the switching threshold and the discontinuous points of the activation functions. Compared with existing results, this paper reveals that switching threshold and discontinuous character are crucial in increasing the number of equilibrium points. Two examples are presented to verify the theoretical results. [ABSTRACT FROM AUTHOR]

[Display omitted] • This paper addresses the problem of 3D object detection in RGB-D images. It is an important and challenging problem in many vision, robotics, and human–machine interaction applications. • It proposes a multilevel fusion network (MFN) model. It presents a new loss function to describe the geometric attribute difference in 3D object bounding boxes and an adaptive depth image filtering algorithm to restore and correct noisy depth images. • It tests the proposed model on challenging datasets and the results outperform the comparison approaches. It also carried out the ablation studies which prove the effectiveness of different modules in the proposed model. 3D object detection is an important yet challenging problem in a myriad of vision, robotics, and human–machine interaction applications. Given an RGB-D image, the task is to infer the class labels and the 3D bounding boxes of the objects in the image. While the previous studies have made remarkable progress over the past decade, how to effectively exploit the feature fusion with neural networks for boosting 3D object detection performance remains an open problem. This paper proposes a multilevel fusion network (MFN) model to detect 3D objects in RGB-D images. The MFN model contains two streams of neural networks which respectively extracts the RGB and depth features with cascaded convolutional modules. To effectively exploit the information of 3D objects, a multilevel fusion mechanism is adopted to fuse the convolutional RGB and depth features at multiple levels. To train the network, we propose a new weighted loss function by encoding the difference of geometric attributes on 3D bounding box regression. Since the original depth data is full of noisy holes, we also develop an adaptive filtering algorithm to restore and correct the depth images. We test the proposed model on challenging RGB-D datasets. The experimental results on the datasets prove the strength and advantage of the proposed model. [ABSTRACT FROM AUTHOR]

As a powerful data modeling tool, Heterogeneous Information Network (HIN) has been successfully used in auxiliary information exploitation to boost recommendation performance. For HIN based recommendation, it is challenging to extract and fuse useful features of user preferences and item attributes under different semantic paths in HINs. Existing methods leverage a pre-defined fusion function to integrate different semantics for recommendation, which cannot characterize the complex nonlinear interactions between users and items. In this paper, we present a general framework named MNRec, short for Meta-path fusion based Neural Recommendation, to extract and fuse user and item embeddings under different meta-paths for recommendation. Under the framework, we propose an instantiation of MNRec with Multi-Layer Perceptron (MLP) structure. It consists of two major steps, i.e., meta-path based heterogeneous network embedding and deep learning based rating prediction. Concretely, appropriate meta-paths are first designed according to domain knowledge. Then the embeddings of users and items are obtained through a meta-path and commuting matrix based heterogeneous network embedding method. Finally, in light of the powerful nonlinear modeling capabilities of deep neural networks, the learned embeddings under different meta-paths are integrated into a two-pathway MLP structure for rating prediction. Experimental results on three real-world datasets demonstrate the superiority and effectiveness of MNRec compared with state-of-the-art baselines in rating prediction. [ABSTRACT FROM AUTHOR]

This paper investigates the multistability on a class of delayed neural networks with monotonically nondecreasing linear activation function. For n state neurons with 2 m + 1 piecewise linear activation function, we prove the neural networks have (2 m + 1) n equilibrium points, (m + 1) n of which are locally exponentially stable. Different from the traditional multistability analysis method such as fixed point theory, this paper only uses the character of activation functions, and can also handle the neural networks with disturbance term. The research results of this paper generalize the previous related research works and are easy to test. A numerical example is given to shown the effectiveness of our theoretical results. [ABSTRACT FROM AUTHOR]

In this paper, the stability of a type of neural network with time-varying delay is studied. Two general (α 2 , β 2) and (α 2 , β 2 , γ 2) -dependent reciprocally convex inequalities with two and three terms are derived to introduce some quadratic terms into the estimate for the derivative of the Lyapunov–Krasovskii functionals (LKFs). The LKF method is used to develop a stability criterion for time-delay neural networks by using the new inequalities. To demonstrate the improvement of the new criterion, numerical results are provided. [ABSTRACT FROM AUTHOR]

Learning and synthesizing stabilizing controllers for unknown nonlinear control systems is a challenging problem for real-world and industrial applications. Koopman operator theory allows one to analyze nonlinear systems through the lens of linear systems and nonlinear control systems through the lens of bilinear control systems. The key idea of these methods lies in the transformation of the coordinates of the nonlinear system into the Koopman observables, which are coordinates that allow the representation of the original system (control system) as a higher dimensional linear (bilinear control) system. However, for nonlinear control systems, the bilinear control model obtained by applying Koopman operator based learning methods is not necessarily stabilizable. Simultaneous identification of stabilizable lifted bilinear control systems as well as the associated Koopman observables is still an open problem. In this paper, we propose a framework to construct such stabilizable bilinear models and identify their associated observables from data by simultaneously learning a bilinear Koopman embedding for the underlying unknown control affine nonlinear system as well as a Control Lyapunov Function (CLF) for the Koopman based bilinear model using a learner and falsifier. Our proposed approach thereby provides provable guarantees of asymptotic stability for the Koopman based representation of the unknown control affine nonlinear control system as a bilinear system. Numerical simulations are provided to validate the efficacy of our proposed class of stabilizing feedback controllers for unknown control-affine nonlinear systems. [ABSTRACT FROM AUTHOR]

Dialogue reasoning is a new task beyond the traditional dialogue system, because it requires recognizing not only semantic relevance but also logical consistency between the candidate response and the dialogue history. Like "all happy families are happy alike, all unhappy families are unhappy in their own way", various illogical patterns exist in the data. For example, some candidate responses use many similar words but with contradicted meanings with history; while some candidates may employ totally different words but convey consistent meanings. Therefore, an ideal dialogue reasoning model should gather clues from both coarse-grained utterance-level and fine-grained word-level to determine the logical relation between candidates and the dialogue history. However, traditional models mainly rely on the widely used BERT to read all the history and candidates word by word but ignore the utterance-level signals, which cannot well capture various illogical patterns in this task. To tackle this problem, we propose a novel Hierarchical BERT (HiBERT) to recognize both utterance-level and word-level illogical patterns in this paper. Specifically, BERT is firstly utilized to encode the dialogue history and each candidate response as the contextualized representation. Secondly, hierarchical reasoning architecture is conducted with this contextualized representation to obtain the word-level and the utterance-level attention distributions, respectively. In detail, we utilize the word-grained attention mechanism to obtain the word-level representation, and propose two different types of attention function, i.e, hard attention and soft attention, to obtain the utterance-grained representation. Finally, we fuse both the word-grained representation and the utterance-grained representation to calculate the logical ranking scores for the given candidate. Experimental results on two public dialogue datasets show that our model obtains higher ranking measures than the widely used BERT model, validating the effectiveness of hierarchical reading of HiBERT. Further analysis on the impact of context length and attention weights shows that the HiBERT actually has the ability to recognize different illogical patterns. [ABSTRACT FROM AUTHOR]

Deep learning methods are criticized for the requirement of large set of labeled training data. Therefore, few-shot learning (FSL), which enables fast learning with only a few labeled examples, draws increasing attention. In FSL, the model is trained on base classes to learn the meta-knowledge, which is then applied to novel classes. The generalization from base classes to novel classes in FSL suffers from the class-level over-fitting problem. Generally, in traditional classification tasks, training samples and test samples are from the same class set, where over-fitting happens at the sample level. However, in FSL, the model is evaluated with classification tasks on novel classes. Thus, good fitting on base classes does not guarantee a good generalization to novel classes. In this paper, we reveal the class-level over-fitting problem in FSL and provide an explanation of the cause of this problem. Based on the explanation, we argue that simply scaling the logits of classifier during training can alleviate the class-level over-fitting problem, and analyze how scaling logits (SL) alleviates class-level over-fitting based on gradient back-propagation. Extensive experiments show that SL boosts the performance to the extent of 14% on four popular benchmark datasets. Further, SL also demonstrates its effectiveness on confidence calibration. [ABSTRACT FROM AUTHOR]

In this paper, a novel neural-network-based backstepping control method is designed for the post-capture tethered space combination system subjected to multi-source disturbances and actuator saturation. To cope with the multi-source disturbances, the so-called HDO is put forward which is capable of addressing both modeled and unmodeled disturbances. These disturbances are not necessarily matched disturbances and hence the backstepping design procedure is utilized. In virtue of the radial basis function neural networks (RBFNNs), the unknown nonlinearities is estimated. Additionally, the anti-windup technique is employed to handle the actuator saturation phenomenon which is unavoidable in the post-capture tethered space combination system. Sufficient conditions are derived to guarantee that the post-capture tethered space combination system is stabilized while carrying out the non-cooperative target capture mission. Finally, a number of numerical simulations are conducted on the post-capture tethered space combination to validate the proposed methodology. [ABSTRACT FROM AUTHOR]

Finding the optimal solution to a linear programming (LP) problem is a long-standing computational problem in Operations Research. This paper proposes a deep learning approach in the form of feed-forward neural networks to solve the LP problem. The latter is first modeled by an ordinary differential equations (ODE) system, the state solution of which globally converges to the optimal solution of the LP problem. A neural network model is constructed as an approximate state solution to the ODE system, such that the neural network model contains the prediction of the LP problem. Furthermore, we extend the capability of the neural network by taking the parameter of LP problems as an input variable so that one neural network can solve multiple LP instances in a one-shot manner. Finally, we validate the proposed method through a collection of specific LP examples and show concretely how the proposed method solves the example. [ABSTRACT FROM AUTHOR]

This paper presents an event-triggered-based optimal tracking control scheme for the unknown nonlinear system with unmeasurable state, constrained input, and limited communication. Firstly, an event-triggered Hamilton–Jacobi-Bellman (HJB) equation is derived based on an augmented system and a discounted cost function. Next, a novel online event-triggered-based neural networks (NNs) observer (NNsO) is constructed to obtain the unmeasurable state and the unknown system dynamics. Then, a single critic neural network (NN) based on the adaptive dynamic programming (ADP) framework is established to solve the event-triggered HJB equation, and the designed critic NN weight updating law can relax the restrictions of initial admissible control and persistence of excitation condition. Furthermore, a static triggering rule with an exponential term is designed to reduce communication and computing burden, and based on this, a dynamic triggering rule is established by introducing an auxiliary dynamic variable to obtain efficient communication and computing. It is proved that the tracking error and all the NN weight estimation errors are uniformly ultimately bounded (UUB), and the Zeno behavior is excluded. Finally, two simulation examples are presented to verify the effectiveness and superiority of the proposed control scheme. [ABSTRACT FROM AUTHOR]

This paper studies the exponential stability problem of neural networks with a time-varying delay. Firstly, an augmented Lyapunov-Krasovskii functional (LKF) containing a single integral state is constructed. Then a generalized free-matrix-based integral inequality and the auxiliary function-based integral inequality combined with an extended reciprocally convex matrix inequality are used to estimate the derivative of the LKF. The novelty of this paper is that some state-dependent zero equations are introduced into before and after bounding the LKF's derivative so as to increase the freedom and reduce the conservatism. As a result, a less conservative stability criterion is derived in the form of linear matrix inequality, whose superiority is illustrated with three numerical examples. [ABSTRACT FROM AUTHOR]

Two new hybrid neural architectures combining morphological neurons and perceptrons are introduced in this paper. The first architecture, called Morphological - Linear Neural Network (MLNN) consists of a hidden layer of morphological neurons and an output layer of classical perceptrons has the capability of extracting features. The second architecture, called Linear-Morphological Neural Network (LMNN) is composed of one or several perceptron layers as a feature extractor, it is then followed by an output layer of morphological neurons for non-linear classification. Both architectures are trained by stochastic gradient descent. One of the main contributions of this paper is to show that the morphological layer offers a greater capacity to extract features than the perceptron layer. This claim is supported both theoretically and experimentally. We prove that the morphological layer possesses a greater capacity per computation unit to segment the 2D input space than the perceptron layer. In other words, adding more hyper-boxes produces more response regions than adding hyperplanes. From an empirical point of view, we test the two new models on 25 standard datasets at low dimensionality and one big data dataset. The result is that MLNN requires a lesser number of learning parameters than the other tested architectures while achieving better accuracies. [ABSTRACT FROM AUTHOR]

This paper is concerned about barrier Lyapunov function(BLF)-based adaptive dynamic surface control(DSC) for a class of stochastic nonlinear systems of full state constraints with hysteresis input and time delay. All states of the system are guaranteed to be constrained in bounded compact set. The pure feedback nonlinear system is transformed into a strictly feedback nonlinear system with nonaffine nonlinear terms by using the mean value theorem. By using backstepping design of BLF combine with DSC technique, the explosion of complexity is avoided. Based on the neural network with the finite covering lemma, the time-delay functions are solved. Simultaneously, the backlash-like hysteresis input control in this paper is considered. The two cases of symmetric and asymmetric BLFs are discussed separately. An adaptive controller is designed to ensure that the output tracking error converges on a small region of the origin. Finally, the control scheme ensures that all signals in the closed-loop systems are semi-global uniformly ultimately bounded. Results of two simulation cases are presented to prove the effectivity of the theoretical analysis. [ABSTRACT FROM AUTHOR]

This paper deals with the global quasi-synchronization and global anti-synchronization issue of delayed neural networks with discontinuous activations (DNNDA) by means of the non-fragile control strategy. In this paper, a class of DNNDA model is presented. Furthermore, applying the Lyapunov-Krasovskii function with matrix measure, the sufficient conditions are put forward to ensure that such DNNDA drive system and DNNDA response system are global quasi-synchronized. In addition, based on a novel Lur′e-Postnikov Lyapunov function, differential inclusion theory and inequality analysis technique, the sufficient conditions to achieve the global anti-synchronization of the DNNDA drive system and DNNDA response system are proposed in term of the Linear matrix inequalities (LMIs). Finally, two typical numerical simulations are provided to show the effectiveness of the obtained results. [ABSTRACT FROM AUTHOR]

Since input constraints and external disturbances are unavoidable in tracking control problems, how to obtain a controller in this case to save communication and data resources at the same time is very challenging. Aiming at these challenges, this paper develops a novel neural network (NN)-based event-triggered integral reinforcement learning (IRL) algorithm for constrained H ∞ tracking control problems. First, the constrained H ∞ tracking control problem is transformed into a regulation problem. Second, an event-triggered optimal controller is designed to reduce network transmission burden and improve resource utilization, where a novel threshold is proposed and its non-negativity can be guaranteed. Third, for implementation purpose, a novel NN-based event-triggered IRL algorithm is developed. In order to improve data utilization, the experience replay technique with an easy-to-verify condition is employed in the learning process. Theoretical analysis proves that the tracking error and weight estimation error are uniformly ultimately bounded. Finally, simulation verification shows the effectiveness of the present method. [ABSTRACT FROM AUTHOR]

This paper studies prescribed-time stabilization for a class of unknown strict-feedback nonlinear systems. Adaptive time-varying feedback control method and non-scaling transformation strategies are employed to reduce the computation burden caused by the scaling functions. Besides, the proposed method uses a constrained function to avoid the possibility of an oversized signal when time approaches to the prescribed settling time. Based on the backstepping method, a neural prescribed-time controller is constructed to achieve finite-time regulation. Under the action of the proposed strategy, all the closed-loop signals are bounded and the system is stable in prescribed finite time. Finally, two simulation examples illustrate the effectiveness of the results. [ABSTRACT FROM AUTHOR]

This paper is concerned with the set-membership filtering issue for a class of artificial neural networks subject to mixed attacks. The encoding–decoding communication mechanism is adopted in the processing of data sharing between neurons. During the information exchanges among neurons, both injection and DoS attacks are concurrently considered to reflect the practical operating conditions of the investigated neural networks. The purpose of the addressed problem is to present an algorithm to estimate the neurons' states in the presence of mixed attacks, while guaranteeing the estimation errors at each neuron are confined within certain prescribed ellipsoidal region. Sufficient conditions are derived, in terms of convex optimization approach, to ensure the existence of desired filter, and the explicit filtering parameters are obtained via solving the provided set of matrix inequalities. Finally, a numerical simulation example is proposed to show the validity of the obtained theoretical results. [ABSTRACT FROM AUTHOR]

In this paper, a new data-driven reinforcement learning method based on Monte Carlo simulation is developed to solve the optimal control problem of unmanned aerial vehicle (UAV) systems. Based on the data which are generated by Monte Carlo simulation, neural network (NN) is used to construct the dynamics of the UAV system with unknown disturbances, where the mathematical model of the UAV system is unnecessary. An effective iterative framework of action and critic is constructed to obtain the optimal control law. The convergence property is developed to guarantee that the iterative performance cost function converges to a finite neighborhood of the optimal performance cost function. Finally, numerical results are given to illustrate the effectiveness of the developed method. [ABSTRACT FROM AUTHOR]

Artificial intelligence models encounter significant challenges due to their black-box nature, particularly in safety-critical domains such as healthcare, finance, and autonomous vehicles. Explainable Artificial Intelligence (XAI) addresses these challenges by providing explanations for how these models make decisions and predictions, ensuring transparency, accountability, and fairness. Existing studies have examined the fundamental concepts of XAI, its general principles, and the scope of XAI techniques. However, there remains a gap in the literature as there are no comprehensive reviews that delve into the detailed mathematical representations, design methodologies of XAI models, and other associated aspects. This paper provides a comprehensive literature review encompassing common terminologies and definitions, the need for XAI, beneficiaries of XAI, a taxonomy of XAI methods, and the application of XAI methods in different application areas. The survey is aimed at XAI researchers, XAI practitioners, AI model developers, and XAI beneficiaries who are interested in enhancing the trustworthiness, transparency, accountability, and fairness of their AI models. [ABSTRACT FROM AUTHOR]

This paper investigates an adaptive neural network (NN) optimal control problem for a permanent magnet synchronous motor (PMSM) system. The addressed PMSM system contains unknown nonlinear dynamics and constraint states. The neural networks (NNs) are employed to identify the unknown nonlinear dynamics and by constructing barrier performance index functions and barrier Lyapunov functions, an adaptive NN optimal control method is developed under the framework of the actor-critic architecture and backstepping control technique. It is proved that the presented optimal control strategy can not only ensure the closed-loop system stable, but also guarantee the angular velocity, stator current and other state variables of PMSM are within given bounds. Furthermore, it can minimize the performance index functions. The effectiveness of the developed NN adaptive optimal controller is verified by computer simulation results. [ABSTRACT FROM AUTHOR]

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: https://github.com/shivram1987/ActivationFunctions. [ABSTRACT FROM AUTHOR]

This paper proposes a generalized impulsive Halanay inequality from the impulsive control standpoints, where the impulse action is event-triggered. Namely, the control task is executed when an external event generated by the state-dependent event-triggered mechanism (ETM) is activated, rather than at a fixed time. It is shown that under the proposed event-triggered impulsive control (ETIC) strategy, the solution of the underlying inequality is guaranteed to converge to zero asymptotically. Moreover, the key point is the introduction of monitoring value to ensure a positive minimum inter-execution time, which exactly assists ETIC with reducing the pressure of information transmission. Then, the generalized impulsive Halanay inequality with appropriate ETIC scheme is utilized to the analysis of delayed neural networks (DNNs). In particular, some synchronization and asymptotic stability criteria of DNNs are derived, where the design of impulsive controller is based on ETIC strategy. At last, some numerical examples are provided to illustrate the validity of the obtained results. [ABSTRACT FROM AUTHOR]

This paper considers the reliable state estimation issue for discrete-time neural networks with the try-once-discard (TOD) scheduling protocol and mixed compensation strategy. For the phenomenon of medium access constraint, the measurement transmitted from sensors to the estimator is subjected to the TOD scheduling protocol. The mixed compensation is proposed to flexibly compensate those missing measurements caused by the TOD protocol. By using a novel polytopic uncertain model, a reliable state estimator is designed, where the gain matrix is determined by two vertex matrices. Then sufficient conditions are established, which ensure the error system meets the stochastic stability and the l 2 - l ∞ performance. Finally, an illustrative example shows the validity of the proposed reliable state estimator. [ABSTRACT FROM AUTHOR]

• The feedback information about the leader is only the relative distance and bearing angle that are measured by the on-board vision sensors mounted on the follower. • No communication is required among the vehicles, and the follower's acceleration is also not required. • The proposed formation controller not only guarantees the collision and connectivity constraints, but also achieves satisfaction of prescribed performance specifications on the formation errors. Limited sensing capability is one of features of onboard vision sensors. This paper addresses the formation tracking control problem for multiple nonholonomic autonomous vehicles with modeling uncertainties under limited sensing capabilities. Within the leader–follower formation control framework, the desired formation geometry of the vehicular team can be achieved by controlling the relative distance and angle between each pair of leader–follower vehicles. Each vehicle in the group is equipped with limited field-of-view onboard vision sensors to measure only the relative distance and angle with respect to its leader. For this scenario, each vehicle must remain within the visibility region relative to the leader in order to maintain the connectivity of the multi-vehicle system over time, while avoiding the possible collision with its leader. By incorporating the prescribed performance control methodology into the formation control design, the boundedness of the closed-loop system signals as well as the predefined performance indexes are achieved, where the neural networks (NNs) are employed to compensate for the uncertain dynamics. Visibility maintenance and collision avoidance between the leaders and followers are also proven mathematically. The proposed control strategy is decentralized in the sense that each vehicle uses only local relative information with respect to its leader to calculate its own control signals, as well as robust with respect to modeling uncertainties. Simulations and experiments on Pioneer-3AT robots are conducted to demonstrate the effectiveness of the proposed control strategy. [ABSTRACT FROM AUTHOR]

This paper proposes finite-horizon optimal tracking control approach based on data for completely unknown discrete-time nonlinear affine systems. First, the identifier is designed by input and output data, which is used to identify system function and system model. And based on tracking error, the system function is transformed to the augmentation system with finite-time optimal performance. In finite time, by minimizing the performance index function, the iterative approximate dynamic programming (ADP) is utilized to solve Hamilton–Jacobi–Bellman (HJB) equation. The idea is carried by the policy iterative (PI) based on the model neural network, which makes the iterative control of the augmentation system available at the each step. At the same time, the action neural network is utilized to acquire the approximate optimal tracking control law and the critic neural network is used for approximating the optimal performance index function for the augmentation system. Afterwards, the paper show the analysis process that the convergence and stability for the iterative ADP algorithm and the weight estimation errors based on the PI, respectively. The end of the paper, a simulation example is applied to show the theoretical results and proposed approach. [ABSTRACT FROM AUTHOR]

This paper investigates the consensus asymptotic convergence problem for a class of n th-order strict-feedback multiagent systems, which include the input quantization, actuator faults, unknown nonlinear functions and directed communication topology. Because the upper bounds of the time-varying stuck faults and external disturbance are commonly difficult to accurately determine, it is also assumed that these upper bounds are unknown in this paper. First, a group of first-order filters are designed to estimate the bounds of the reference signal for each agent. Second, smooth functions are introduced to compensate the effect of quantization and bounded stuck faults. Meanwhile, a new back-stepping method is used to propose an intermediate control law and an adaptive design procedure, and the final distributed control protocols are established. All closed-loop signals are uniformly bounded, and the tracking errors asymptotically converge to zero. Finally, a practical example simulation is provided to demonstrate the effectiveness of the proposed scheme. [ABSTRACT FROM AUTHOR]

Abstract In this paper, neural networks (NNs)-based adaptive dynamic surface control is investigated for a class of pure-feedback nonlinear systems with unknown time delay and input hysteresis nonlinearities. The design difficulty appeared in this paper due to time delay is handled by combining neural networks with finite covering lemma instead of using Krasovskii functionals, and the assumptions on the time-delay functions are removed based the finite covering lemma and NNs. Meanwhile, completely unknown backlash-like hysteresis control input that frequently exists in practice is also considered. By combining the adaptive backstepping recursive design technique with the universal approximation ability of neural networks, an adaptive neural control algorithm is systemically designed for the system under consideration, and the explosion of complexity exists in traditional backstepping design is avoided by using dynamic surface technique. It is shown that the designed adaptive controller can guarantee that all the signals are ultimately uniformly bounded and the desired signal can be tracked with a small domain of the origin. A numerical example and an example of a real plant for adjustable metal cutting system are provided to show the feasibility of the proposed method. [ABSTRACT FROM AUTHOR]

This paper investigates an adaptive asymptotic tracking controller for uncertain nonstrict-feedback nonlinear systems(NFNSs) with full-state constraints and unknown actuator failures. By constructing a class of new switched high-order barrier Lyapunov functions(BLFs) candidates to the backstepping design, the asymptotic tracking performance can be guaranteed, all states do not violate constraints. The neural networks(NNs) is used to handle the difficulties caused by the unknown nonlinearities, the event-triggered control law based on relative threshold method for reducing the communication load between the controller and actuator. The control scheme guarantees that all closed-loop signals of the system are bounded and the tracking error asymptotically approaches a pre-defined bound, no matter whether the actuators operate in normal or faulty modes and the number of actuator failures can be infinite. Finally, simulation results including numerical simulation and practical application simulation verify the feasibility of the control scheme. [ABSTRACT FROM AUTHOR]

Due to the current high availability of omics, data-driven biology has greatly expanded, and several papers have reviewed state-of-the-art technologies. Nowadays, two main types of investigation are available for a multi-omics dataset: extraction of relevant features for a meaningful biological interpretation and clustering of the samples. In the latter case, a few reviews refer to some outdated or no longer available methods, whereas others lack the description of relevant clustering metrics to compare the main approaches. This work provides a general overview of the major techniques in this area, divided into four groups: graph, dimensionality reduction, statistical and neural-based. Besides, eight tools have been tested both on a synthetic and a real biological dataset. An extensive performance comparison has been provided using four clustering evaluation scores: Peak Signal-to-Noise Ratio (PSNR), Davies-Bouldin(DB) index, Silhouette value and the harmonic mean of cluster purity and efficiency. The best results were obtained by using the dimensionality reduction, either explicitly or implicitly, as in the neural architecture. [ABSTRACT FROM AUTHOR]

In this paper, we give an upper bound on the variance of scalar multilayer perceptrons. The distribution of the input is assumed to be the class of log-concave distributions, which includes the well-known Gaussian distribution. The activation functions of the scalar multilayer perceptrons are assumed to be differentiable and Lipschitz continuous. [ABSTRACT FROM AUTHOR]

This paper investigates the asymptotic stability of neural network with time-varying delay. A new Lyapunov–Krasovskii functional (LKF) is constructed with some non-integral delay-product functional terms. Some parameter matrices in the LKF are not required to be positive definite. The convex method and the non-convex method are proposed to deal with the square of the time-varying delay appearing in the derivative of the LKF, respectively. Two less conservative delay-dependent stability criteria in the form of linear matrix inequalities (LMIs) are established. Finally, two widely used numerical examples are given to illustrate the validity and superiority of the obtained stability criteria. [ABSTRACT FROM AUTHOR]

In this paper, a new infinite horizon optimal tracking control method for continuous-time nonlinear systems is given using an actor-critic structure. This present integral reinforcement learning (IRL) method is a novelty method in adaptive dynamic programming (ADP) algorithms and an online policy iteration algorithm. For the optimal tracking problem, the cost function is defined by tracking errors. Consequently, the goal is to minimize tracking errors toward desired trajectories. Since it is hard to solve the Hamilton-Jacobi-Bellman (HJB) equation for continuous-time nonlinear systems control problems, leveraging the actor-critic architecture with neural networks (NNs) to approximate the tracking error performance index and error control law is necessary. Instead of using conventional neural networks, we employ higher-order polynomials in the whole actor-critic architecture. Finally, we apply this new neuro-optimal tracking method to the 2500MW pressurized water reactor (PWR) nuclear power plant, and simulation results are given to demonstrate the effectiveness of the developed method. [ABSTRACT FROM AUTHOR]

Deep learning has achieved impressive performance on many tasks in recent years. However, it has been found that it is still not enough for deep neural networks to provide only point estimates. For high-risk tasks, we need to assess the reliability of the model predictions. This requires us to quantify the uncertainty of model prediction and construct prediction intervals. One of the significant advantages of the proposed method is that it simultaneously implements point estimation and uncertainty quantification. In this paper, we explore the uncertainty in regression neural networks to construct the prediction intervals. In general, we comprehensively consider two categories of uncertainties: aleatory uncertainty and epistemic uncertainty. We design a novel loss function, which enables us to learn uncertainty without uncertainty labels. We only need to supervise the learning of regression tasks. In the process of training, the model implicitly learns aleatory uncertainty under the guidance of loss function. And that epistemic uncertainty is accounted for in the ensembled form. Our method correlates the construction of prediction intervals with uncertainty estimation. Experimental results on some publicly available datasets show that the performance of our method is competitive with other state-of-the-art methods. [ABSTRACT FROM AUTHOR]

This paper investigates the distributed fixed-time formation problem of unmanned surface vehicles (USVs) in the presence of external disturbances, model uncertainties, input saturation and quantization constraints. To deal with the problem, a fixed-time sliding-mode control algorithm is proposed, where multi-layer neural networks (MNNs) are designed to approximate the unknown dynamics and composite disturbances of the system. The proposed MNNs combine the advantages of fuzzy neural networks (FNNs) and radial basis function neural networks (RBFNNs), exhibiting robust dynamic characteristics. Furthermore, the non-singular fast terminal sliding mode (NFTSM) is integrated into the fixed-time control framework to improve the robustness and speed of convergence for uncertain USV systems. Comparative simulations conducted with USVs demonstrate the superiority and effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

This paper presents novel iterative learning methods along with the neural network implementation for multi-player zero-sum games. Solving zero-sum games depends on the solutions of Hamilton–Jacobi-Isaacs equations, which are nonlinear partial differential equations. These solutions are generally difficult or even impossible to be obtained analytically. To overcome this difficulty, iterative adaptive dynamic programming algorithms are utilized. In the related research works, three-network architecture, i.e., critic-actor-disturbance structure, is used to approximate the value function, control policies and disturbance policies. Different from the previous works, this paper employs single-network architecture, i.e., critic-only structure, to implement the proposed algorithms, which reduces the computation burden and the complexity of design procedure. Finally, two simulation examples are provided to illustrate the effectiveness of our proposed methods. [ABSTRACT FROM AUTHOR]

Neural Networks and Support Vector Machines (SVMs) are two of the most popular and efficient supervised classification models. However, in the context of large datasets many complexity issues arise due to high memory requirements and high computational cost. In the context of the application of Data Mining algorithms, data reduction techniques attempt to reduce the size of training datasets in terms of the number of instances by selecting some of the existing instances or by generating new training instances. The idea is to speed up the application of the data mining algorithm with minimum or no sacrifice in performance. Data reduction techniques have been extensively used in the context of k -Nearest Neighbor classification, a lazy classifier that works by directly using a training dataset rather than building a model. This paper explores the application of data reduction techniques as a preprocessing step before the training step of Neural Networks and SVMs. Furthermore, the paper proposes a new data reduction technique that is based on k-median clustering algorithm. Our experimental results illustrate that, in the case of SVMs, data reduction techniques can effectively reduce the dataset size incurring small performance degradation. In the case of Neural Networks, the performance loss is somewhat greater, for the same data reduction rate, but both SVM and Neural Network models outperform the k -NN approach that is typically used in Data Mining applications. [ABSTRACT FROM AUTHOR]

This paper proposes the design and implementation of a model-free tire slip control for a fast and highly nonlinear Anti-lock Braking System (ABS). A reinforcement Q-learning optimal control approach is inserted in a batch neural fitted scheme using two neural networks to approximate the value function and the controller, respectively. The transition samples required for learning high performance control can be collected by interacting with the process either by online exploiting the current iteration controller (or policy) under an ε-greedy exploration strategy, or by using data collected under any other controller that is capable of ensuring efficient exploration of the action-state space. Both approaches are highlighted in the paper. Fortunately, the ABS process fits this type of learning-by-interaction because it does not need an initial stabilizing controller. The validation case studies conducted on a real laboratory setup reveal that high control system performance can be achieved using the proposed approaches. Insightful comments on the observed control behavior are offered along with performance comparisons with several types of model-based and model-free controllers including relay, model-based optimal PI, an original model-free neural network state-feedback VRFT controller and a model-free neural network adaptive actor-critic one. With the ability to improve control performance starting from different supervisory controllers or to learn high performance controllers from scratch, the proposed Q-learning optimal control approach proves its performance in a wide operating range and is therefore recommended to its industrial application on ABS. [ABSTRACT FROM AUTHOR]

This paper investigates the L 2 – L ∞ state estimation problem for a class of delayed neural networks. Attention is focused on the design of a full-order state estimator such that the prescribed L 2 – L ∞ performance constraint can be ensured. By utilizing the time-delay information sufficiently, a novel L 2 – L ∞ performance analysis approach is proposed in this paper for the first time. Based on such an approach, the less conservative sufficient conditions are established in terms of linear matrix inequalities under which the L 2 – L ∞ performance level can be achieved for the estimation error dynamics. Several numerical examples show that the proposed approach in this paper is explicitly effective in reducing the possible conservatism. [ABSTRACT FROM AUTHOR]

Neural networks and deep learning algorithms are designed to function similarly to biological synaptic structures. However, classical deep learning algorithms fail to fully capture the need for continuous learning; this has led to the advent of incremental learning. Incremental learning adds new challenges that are handled differently by modern state-of-the-art approaches. Some of these include: utilization of network memory as additional knowledge increases the size of the network, open-set recognition to be able to identify unrecognized information, and efficient knowledge distillation as most incremental learning algorithms are prone to catastrophic forgetting of previously learned knowledge. Recent advancements achieve incremental learning through a multitude of methods. Most methods are characterized by augmenting the normal algorithm of neural network training by both directly modifying the neural network structure and by adding additional learning steps. This paper analyzes and provides a comprehensive survey of existing methods and various techniques used for incremental learning. A novel categorization of the methods is also introduced based on recent trends of the state-of-the-art solutions. The study focuses on methods that provide incremental learning success as well as discusses emerging patterns in new research. [ABSTRACT FROM AUTHOR]

This paper studies the adaptive distributed event-triggered fault-tolerant consensus problem for a class of multi-agent systems with time delays and external disturbance. The communication faults between the agent and its neighbours are considered, and a fault-tolerant control mechanism is designed to overcome the communication faults. Moreover, a distributed event-triggered mechanism is constructed based on the distributed errors, and hence the communication burden can be reduced. Combining the Lyapunov stability theorem with the Lyapunov-Krasovskii function, it verifies that the consensus tracking errors are bounded with the guaranteed security performance, and the time delays phenomenon can also be compensated. Finally, the effectiveness and utility of the proposed control method are fully validated by two simulation examples. [ABSTRACT FROM AUTHOR]

Exponential stability for neural networks with time-varying delay is considered in this paper. By extending the generalized free-weighting-matrix integral inequality, a novel integral inequality is derived by using weighted orthogonal functions. Then the new inequality is applied to investigate the exponential stability of time delay neural networks via an improved Lyapunov–Krasovskii functional. Numerical examples are given to verify the advantages of the proposed criterion. [ABSTRACT FROM AUTHOR]

This paper empirically analyses the impact of changes to the set of training examples on the neural network error surface. Specific quantitative characteristics of the error surface related to properties such as ruggedness, modality and structure are measured and visualized as the set of training examples changes. Both the case of random subsampling and active learning from a fixed dataset are examined, producing ten different training scenarios. For each scenario eleven error surface characteristics are calculated for five common benchmark problems. The results demonstrate that the error surface characteristics calculated using only a subsample of the available data commonly do not generalize to that of the full dataset. The observed error surface characteristics are significantly impacted by the particular set of examples used to calculate error. Some error surface characteristics are significantly altered by small changes in the set of examples used to calculate error. The main finding from this study is that when the set of training examples may change during training the training of a neural network is in essence a dynamic optimization problem, suggesting that optimization algorithms developed specifically to solve dynamic optimization problems may be more efficient at training neural networks under such conditions. [ABSTRACT FROM AUTHOR]