Search

Nonlinear activation functions play a crucial role in artificial neural networks. However, digital implementations of sigmoidal functions, the commonly used activation functions, are facing challenges related to energy consumption and area requirements. To address these issues, we develop a proof-of-concept computing system that utilizes magnetic tunnel junctions as the key element for implementing sigmoidal activation functions. Using this system, we train a neural network for speech separation. When compared to state-of-the-art digital implementations, our scalable circuit has the potential to consume up to 383 times less energy and occupy 7354 times smaller area. These results pave the way for more efficient computing systems in the future. [ABSTRACT FROM AUTHOR]

Hybrid neural network models are effective in analyzing time-series data by combining the strengths of neural networks and differential equation models. Although most studies have focused on linear hybrid models, few have examined nonlinear problems. This work explores the potential of a hybrid nonlinear epidemic neural network in predicting the correct infection function of an epidemic model. We design a novel loss function by combining bifurcation theory and mean-squared error loss to ensure the trainability of the hybrid model. Additionally, we identify unique existence conditions that support ordinary differential equations for estimating the correct infection function. Moreover, numerical experiments using the Runge–Kutta method confirm our proposed model's soundness both on our synthetic data and the real COVID-19 data. [ABSTRACT FROM AUTHOR]

Generalized structured component analysis (GSCA) is a multivariate method for specifying and examining interrelationships between observed variables and components. Despite its data-analytic flexibility honed over the decade, GSCA always defines every component as a linear function of observed variables, which can be less optimal when observed variables for a component are nonlinearly related, often reducing the component's predictive power. To address this issue, we combine deep learning and GSCA into a single framework to allow a component to be a nonlinear function of observed variables without specifying the exact functional form in advance. This new method, termed deep learning generalized structured component analysis (DL-GSCA), aims to maximize the predictive power of components while their directed or undirected network remains interpretable. Our real and simulated data analyses show that DL-GSCA produces components with greater predictive power than those from GSCA in the presence of nonlinear associations between observed variables per component. [ABSTRACT FROM AUTHOR]

Nonlinearity manipulation based on electromagnetic (EM) metasurfaces has always been a hot topic, which is an important part of communication and radar systems. However, most of the existing transmission metasurfaces are mainly focused on linear processing on the incident EM waves. In this article, a reprogrammable nonlinear transmission metasurface is proposed for spatial waves using active power amplifiers. An ultrathin and compact metasurface is designed, fabricated, and measured, to achieve reprogrammable functions on the nonlinear transmissions. For EM incidence, the metasurface can not only amplify the magnification of small signals, but also control high‐energy signals nonlinearly, whose magnification is determined by different bias voltages. The simulation and experimental results show good consistency, which verifies the feasibility of the proposed method. This work can find potential applications in signal gain compression and construction of nonlinear units of diffractive deep neural networks. [ABSTRACT FROM AUTHOR]

Nowadays, the accurate prediction of strength properties of cementitious materials containing nano‐ and micro‐silica (NS–MS) remains an open question because of the highly nonlinear function of its constituents on the porosity. In the present study, a combined framework is developed by integrating ant colony optimization (ACO), particle swarm optimization (PSO), and biogeography‐based optimization (BBO) with the artificial neural network (ANN) to predict compressive and flexural strengths of cement mortar in two different forms of ignoring (ANNII) and considering (ANNIII) the porosity as an input parameter. This procedure is accomplished considering the porosity effect on the strengths and implementing an experimental program containing 32 mixes (960 specimens) with different NS–MS contents at various ages. Macro‐ and micro‐structural analyses showed that NS–MS caused more decreased pore structure, and thus this situation increases strength properties compared to their separate use. Also, MBBO‐MOANNIII results indicated an improvement in convergence speed and model accuracy compared to other models. This improvement is because of considering the porosity. [ABSTRACT FROM AUTHOR]

Deep learning has achieved unprecedented success in recent years. This approach essentially uses the composition of nonlinear functions to model the complex relationship between input features and output labels. However, a comprehensive theoretical understanding of why the hierarchical layered structure can exhibit superior expressive power is still lacking. In this paper, we provide an explanation for this phenomenon by measuring the approximation efficiency of neural networks with respect to discontinuous target functions. We focus on deep neural networks with rectified linear unit (ReLU) activation functions. We find that to achieve the same degree of approximation accuracy, the number of neurons required by a single‐hidden‐layer (SHL) network is exponentially greater than that required by a multi‐hidden‐layer (MHL) network. In practice, discontinuous points tend to contain highly valuable information (i.e., edges in image classification). We argue that this may be a very important reason accounting for the impressive performance of deep neural networks. We validate our theory in extensive experiments. [ABSTRACT FROM AUTHOR]

We propose a new method for learning deep neural network models, which is based on a greedy learning approach: we add one basis function at a time, and a new basis function is generated as a non-linear activation function applied to a linear combination of the previous basis functions. Such a method (growing deep neural network by one neuron at a time) allows us to compute much more accurate approximants for several model problems in function approximation. [ABSTRACT FROM AUTHOR]

In single image super-resolution (SISR), deep neural networks learn mainly nonlinear functions to obtain promising high-resolution (HR) images. However, there are usually two undesired limitations to recovered images of existing SR methods. First, since this task is typically an ill-posed problem, recovering the structural information for the SR process is usually sharp edges but distorted. Second, since this task generally has an extremely large space for the mapping function, learning this mapping from low-resolution (LR) to HR images is typically difficult. For the most part, SR models usually suffer from the lack of structural information for the objects in images and poor performance. To address the above issues, we propose a dual gradient regression scheme in the first stage of the network to recover the structure information of objects in images and the corresponding loss function to learn the difference between structural maps. In the second stage of the network, we restore HR gradient maps by the first stage to provide additional structural information for the second stage, further constraining the structure of the SR image and reducing the space of the possible functions. Extensive experiments demonstrate our results outperform the state-of-the-art SR methods in the SSIM, and achieve better visual effects than the state-of-the-art SR methods. [Display omitted] • In this manuscript, we propose a dual gradient regression scheme and corresponding loss function for image super-resolution. • First, since this task is typically an ill-posed problem, recovering the structural information for the SR process is usually sharp edges but distorted. • Second, since this task generally has an extremely large space for the mapping function, learning this mapping from low-resolution (LR) to HR images is typically difficult. • To address the above issues, we propose a dual gradient regression scheme in the first stage of the network to recover the structure information of objects in images and the corresponding loss function to learn the difference between structural maps. • In the second stage of the network, we restore HR gradient maps by the first stage to provide additional structural information for the second stage, further constraining the structure of the SR image and reducing the space of the possible functions. • Extensive experiments substantiate the superiority of the proposed network and prove that our method outperforms the current state-of-the-art method. [ABSTRACT FROM AUTHOR]

Neural networks -- specifically, deep neural networks -- are, at present, the most effective machine learning techniques. There are reasonable explanations of why deep neural networks work better than traditional "shallow" ones, but the question remains: why neural networks in the first place? why not networks consisting of non-linear functions from some other family of functions? In this paper, we provide a possible theoretical answer to this question: namely, we show that of all families with the smallest possible number of parameters, families corresponding to neurons are indeed optimal -- for all optimality criteria that satisfy some reasonable requirements: namely, for all optimality criteria which are final and invariant with respect to coordinate changes, changes of measuring units, and similar linear transformations. [ABSTRACT FROM AUTHOR]

When describing the behaviour and modelling of real systems, which are characterized by considerable complexity, great difficulty, and often the impossibility of their formal mathematical description, and whose operational monitoring and measurement are difficult, conventional analytical–statistical models run into the limits of their use. The application of these models leads to necessary simplifications, which cause insufficient adequacy of the resulting mathematical description. In such cases, it is appropriate for modelling to use the methods brought by a new scientific discipline—artificial intelligence. Artificial intelligence provides very promising tools for describing and controlling complex systems. The method of neural networks was chosen for the analysis of the lifetime of the teeming ladle. Artificial neural networks are mathematical models that approximate non-linear functions of an arbitrary waveform. The advantage of neural networks is their ability to generalize the dependencies between individual quantities by learning the presented patterns. This property of a neural network is referred to as generalization. Their use is suitable for processing complex problems where the dependencies between individual quantities are not exactly known. [ABSTRACT FROM AUTHOR]

Object tracking has achieved impressive performance in computer vision. However, there are many challenges due to complex scenarios in reality. The mainstream trackers mostly locate the object in form of two branches, which limits the ability of trackers to fully mine similarity between template and search region. In this paper, we propose a multi-branch and multi-scale perception object tracking framework based on Siamese Convolutional Neural Networks (MultiBSP), in which the multi-branch tracking framework is established based on the idea of relation mining, and a tower-structured relation network is designed for each branch to learn the non-linear relation function between template and search region. By branch combination, multiple branches can verify their predictions with each other, which is beneficial to robust tracking. Besides, in order to sense the scale and aspect ratio of object in advance, a multi-scale perception module is designed by utilizing the dilated convolutions in five scales, which contributes to the ability of tracker to deal with scale variation. In addition, we propose an information enhancement module that focuses on important features and suppresses unnecessary ones along spatial and channel dimensions. Extensive experiments on six visual tracking benchmarks including OTB100, VOT2018, VOT2019, UAV123, GOT-10k, and LaSOT demonstrate that our MultiBSP can achieve robust tracking and have state-of-the-art performance. Finally, ablation experiments verify the effectiveness of each module and the tracking stability is proved by qualitative and quantitative analyses. [ABSTRACT FROM AUTHOR]

The sigmoid function is a widely used nonlinear activation function in neural networks. In this article, we present a modular approximation methodology for efficient fixed-point hardware implementation of the sigmoid function. Our design consists of three modules: piecewise linear (PWL) approximation as the initial solution, Taylor series approximation of the exponential function, and Newton–Raphson method-based approximation as the final solution. Its modularity enables the designer to flexibly choose the most appropriate approximation method for each module separately. Performance evaluation results indicate that our work strikes an appropriate balance among the objectives of approximation accuracy, hardware resource utilization, and performance. [ABSTRACT FROM AUTHOR]

The application of remote sensing observations in estimating ocean sub-surface temperatures has been widely adopted. Machine learning-based methods in particular are gaining more and more interest. While there is promising relevant progress, most temperature profile reconstruction models are still built upon the gridded Argo data regardless of the impacts of mesoscale oceanic processes. As a follow-on to the previous study that demonstrates the influence of ocean fronts is negligible, we focus on the improvement of temperature profile reconstruction by introducing the sea surface temperature (SST) gradient into the neural network model. The model sensitivity assessments reveal that the normalization of the input variables achieves a higher estimation accuracy than the original scale. Five experiments are then designed to examine the model performances with or without the SST gradient input. Our results confirm that, for a given model configuration, the one with the input of the SST gradient has the lowest reconstruction bias in comparison to the in situ Argo measurements. Such improvement is particularly pronounced below 200 m depth. We also found that the non-linear activation functions and deeper network structures facilitate the performance of reconstruction models. Results of this work open new insights and challenges to refine the mapping of upper ocean temperature structures. While more relevant machine learning methods are worth further exploitation, how to better characterize the mesoscale oceanic processes from surface observations and bring them into the reconstruction models is the key and needs much attention. [ABSTRACT FROM AUTHOR]

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: https://github.com/shivram1987/ActivationFunctions. [ABSTRACT FROM AUTHOR]

In this article, we demonstrate how a generative neural network enables the inverse design of a real-time beamforming structure that consists of simply eight rotatable dielectric slabs but involves high nonlinearity. The determination of the optimal architecture and the detailed training process is discussed, and the advantage of the proposed method over some conventional numerical methods is shown. The system is realized and measured with a platform controlled by a low-cost system-on-a-chip (SoC) computer to verify its real-time design capability. We showcase the proposed neural network as a competitive candidate for complex real-time reconfigurable EM designs. [ABSTRACT FROM AUTHOR]

Disordered nanoclusters with multielectrode input–output functionality had recently been experimentally realized with energy-efficient and emergent computational capacity, and thus an interconnected network of several such nanoclusters had been proposed to realize artificial neural networks. To aid that end, here we show that nanocluster functionality can be fit to the simplest dendritic neuron model (DNM), where the only form of nonlinearity is due to multiplicative interactions. This work brings into the spotlight higher-order neural networks (known for their efficient encoding of geometric invariances) to serve as an explainable baseline model of nano-networks against which experimentalists can compare more sophisticated models (deep neural networks or physics-based models such as the lin-min network introduced here) and provides ground for designing novel approximate hardware and a statistical mechanics analysis of the learning performance of interconnected nanoclusters versus perceptrons (where neurons output a nonlinear function of the weighted sum of their inputs). A network with just ten higher-order neurons is shown to achieve a classification accuracy of more than 96% on the MNIST benchmark for handwritten digit recognition (which required 100 times more neurons in three-layer perceptrons). [ABSTRACT FROM AUTHOR]

Deep convolutional neural networks (DCNNs) are currently the method of choice both for generative, as well as for discriminative learning in computer vision and machine learning. The success of DCNNs can be attributed to the careful selection of their building blocks (e.g., residual blocks, rectifiers, sophisticated normalization schemes, to mention but a few). In this paper, we propose $\Pi$ Π -Nets, a new class of function approximators based on polynomial expansions. $\Pi$ Π -Nets are polynomial neural networks, i.e., the output is a high-order polynomial of the input. The unknown parameters, which are naturally represented by high-order tensors, are estimated through a collective tensor factorization with factors sharing. We introduce three tensor decompositions that significantly reduce the number of parameters and show how they can be efficiently implemented by hierarchical neural networks. We empirically demonstrate that $\Pi$ Π -Nets are very expressive and they even produce good results without the use of non-linear activation functions in a large battery of tasks and signals, i.e., images, graphs, and audio. When used in conjunction with activation functions, $\Pi$ Π -Nets produce state-of-the-art results in three challenging tasks, i.e., image generation, face verification and 3D mesh representation learning. The source code is available at https://github.com/grigorisg9gr/polynomial_nets. [ABSTRACT FROM AUTHOR]

The main objective of this study was to conduct multi-stage and multi-variant prognostic research to assess the impact of e-mobility development on the Polish power system for the period 2022–2027. The research steps were as follows: forecast the number of electric vehicles (using seven methods), forecast annual power demand arising solely out of the operation of the forecast number of electric vehicles, forecast annual power demand with and without the impact of e-mobility growth (using six methods), forecast daily profiles of typical days with and without the impact of e-mobility growth (using three methods). For the purpose of this research, we developed a unique Growth Dynamics Model to forecast the number of electric vehicles in Poland. The application of Multi-Layer Perceptron (MLP) to the extrapolation of non-linear functions (to the forecast number of electric vehicles and forecast annual power demand without the impact of e-mobility growth) is our original, unique proposal to use the Artificial Neural Network (ANN). Another unique, innovative proposal is to include Artificial Neural Networks (Multi-Layer Perceptron and Long short-term memory (LSTM)) in an Ensemble Model for simultaneous extrapolation of 24 non-linear functions to forecast daily profiles of typical days without taking e-mobility into account. This research determined the impact of e-mobility development on the Polish power system, both in terms of annual growth of demand for power and within particular days (hourly distribution) for two typical days (summer and winter). Under the (most likely) balanced growth variant of annual demand for power, due to e-mobility, such demand would grow by more than 4%, and almost 7% under the optimistic variant. Percentage growth of power demand in terms of variation according to time of day was determined. For instance, for the balanced variant, the largest percentage share of e-mobility was in the evening "peak" time (about 6%), and the smallest percentage was in the night "valley" (about 2%). [ABSTRACT FROM AUTHOR]

Recently, Physics-Informed Neural Networks (PINNs) have gained significant attention for their versatile interpolation capabilities in solving partial differential equations (PDEs). Despite their potential, the training can be computationally demanding, especially for intricate functions like wavefields. This is primarily due to the neural-based (learned) basis functions, biased toward low frequencies, as they are dominated by polynomial calculations, which are not inherently wavefield-friendly. In response, we propose an approach to enhance the efficiency and accuracy of neural network wavefield solutions by modeling them as linear combinations of Gabor basis functions that satisfy the wave equation. Specifically, for the Helmholtz equation, we augment the fully connected neural network model with an adaptable Gabor layer constituting the final hidden layer, employing a weighted summation of these Gabor neurons to compute the predictions (output). These weights/coefficients of the Gabor functions are learned from the previous hidden layers that include nonlinear activation functions. To ensure the Gabor layer's utilization across the model space, we incorporate a smaller auxiliary network to forecast the center of each Gabor function based on input coordinates. Realistic assessments showcase the efficacy of this novel implementation compared to the vanilla PINN, particularly in scenarios involving high-frequencies and realistic models that are often challenging for PINNs. [ABSTRACT FROM AUTHOR]

This paper focuses on deriving and validating the fractional-order form of rectified linear unit activation function and its linear and nonlinear variants. The linear variants include the leaky and parametric, whereas the nonlinear variants include the exponential, sigmoid-weighted, and Gaussian error functions. Besides, a standard formula has been created and used while developing the fractional form of linear variants. Moreover, different expansion series such as Maclaurin and Taylor have been used while designing the fractional version of nonlinear variants. A simulation study has been conducted to validate the performance of all the developed fractional activation functions utilizing a single and multilayer neural network model and to compare them with their conventional counterparts. In this simulation study, a neural network model has been created to predict the system-generated power of a Texas wind turbine. The performance has been evaluated by varying the activation function in the hidden and output layers with the developed functions for single and multilayer networks. [ABSTRACT FROM AUTHOR]

The rapid growth of artificial intelligence technology has been deployed in art teaching and learning. Radial basis function (RBF) networks have a completely different design compared to most neural network architectures. Most neural networks consist of multiple layers that can introduce nonlinearity by repetitive application of nonlinear activation functions. In this research, people will study the application of the RBF neural network model based on deep learning in flower pattern design in art teaching. The image classification process is finding and labeling groups of pixels or vectors inside an image based on rules. Deep learning is a type of machine learning that uses artificial neural networks to replicate the structure and function of the human brain. The proposed model uses the RBF neural network-based deep learning model in flower pattern design in art teaching and provides efficient results. [ABSTRACT FROM AUTHOR]

Shallow models have limited ability to express high-dimensional nonlinear complex functions. Based on deep learning, a Gaussian radial basis function neural network (RBFNN) is proposed, which is an analysis method of nonlinear complex function approximation based on the Gaussian-RBFNN model. The proposed method can approximate single-variable and binary complex nonlinear functions, and the approximation error is less than 0.1, which can achieve the ideal approximation effect. Finally, the proposed method is applied to analyze the nonlinear complex stock prediction approximation problem, and the effectiveness and practicality of the method are further verified. It can be seen that compared with the BP neural network model, this model performs better in average training time, training standard error, Ratt, RATR, and trmse For RBFNN; the training only needs 50 iterations, while the traditional BP needs 7100 training to gradually converge. In the actual function test, after 275 times of training, the convergence speed gradually tends to be stable, and the predicted value is closer to the actual value. Therefore, the model can be used to analyze practical nonlinear complex function approximation problems. [ABSTRACT FROM AUTHOR]

This article investigates the coexistence of multistable equilibrium points (EPs) for n -neuron delayed quaternion-valued neural networks (DQVNNs) in which the continuous and piecewise nonlinear functions are used as the activation functions. Using the state space decomposition technique, Brouwer's fixed point theorem, and Lagrange's mean value theorem, some novel sufficient conditions have been derived to ensure the coexistence of 5 4 n EPs in which 3 4 n of them are locally stable. Furthermore, in this scientific contribution, positively invariant sets have been estimated which actually turned out to be quite challenging for the activation functions of the designed DQVNNs. Finally, some comparisons and convincing simulations with the application to associative memory of DQVNNs are given to verify the theoretical results at the end of this article. [ABSTRACT FROM AUTHOR]

Nonlinear activation is a crucial building block of most machine-learning systems. However, unlike in the digital electrical domain, applying a saturating nonlinear function in a neural network in the analog optical domain is not as easy, especially in integrated systems. In this paper, we first investigate in detail the photodetector nonlinearity in two main readout schemes: electrical readout and optical readout. On a 3-bit-delayed XOR task, we show that optical readout trained with backpropagation gives the best performance. Furthermore, we propose an additional saturating nonlinearity coming from a deliberately non-ideal voltage amplifier after the detector. Compared to an all-optical nonlinearity, these two kinds of nonlinearities are extremely easy to obtain at no additional cost, since photodiodes and voltage amplifiers are present in any system. Moreover, not having to design ideal linear amplifiers could relax their design requirements. We show through simulation that for long-distance nonlinear fiber distortion compensation, using only the photodiode nonlinearity in an optical readout delivers BER improvements over three orders of magnitude. Combined with the amplifier saturation nonlinearity, we obtain another three orders of magnitude improvement of the BER. [ABSTRACT FROM AUTHOR]

• We propose fuzzy reinforced polynomial neural networks based on PNN architecture. • We propose fuzzy reinforced polynomial neurons (FRPNs) in the design of FRPNNs. • The proposed FRPNNs is a generalization of hybrid predictors (HPs). In the field of dynamic system identification and prediction, linear models (e.g., autoregressive models), nonlinear models (namely, neural networks models), and hybrid predictors (HPs) that are a hybridization of linear and nonlinear models have been proposed in the past. However, they are not completely free from limitations: they exhibit difficulties to describe high-order nonlinear relations between input and output variables. In this study, we propose fuzzy reinforced polynomial neural networks (FRPNNs), which are polynomial neural network architecture-based on fuzzy reinforced polynomial neurons (FRPNs) to overcome this limitation. The proposed FRPNs that consist of approximation part (AP) and compensation part (CP) arise as novel HPs. Here the CP for modeling nonlinear patterns can be regarded as forming the reinforced part for the AP that aims at capturing linear patterns, while AP is the linear polynomial neuron used in the conventional polynomial neural networks. In some sense, the overall FRPNNs are essentially generalized polynomial neural network architecture with novel HPs. The parameters considered in the design of the proposed fuzzy reinforced polynomial neural networks are optimized with the aid of the particle swarm optimization (PSO). The performance of FRPNNs is discussed involving time series and system identification datasets. Experimental results demonstrate that the proposed FRPNNs achieve at most the accuracy of 43.6% higher in comparison with the accuracy produced by some classical models reported in the literature. [ABSTRACT FROM AUTHOR]

An artificial neural network consists of neurons and synapses. Neuron gives output based on its input according to non-linear activation functions such as the Sigmoid, Hyperbolic Tangent (Tanh), or Rectified Linear Unit (ReLU) functions, etc.. Synapses connect the neuron outputs to their inputs with tunable real-valued weights. The most resource-demanding operations in realizing such neural networks are the multiplication and accumulate (MAC) operations that compute the dot product between real-valued outputs from neurons and the synapses weights. The efficiency of neural networks can be drastically enhanced if the neuron outputs and/or the weights can be trained to take binary values ± 1 only, for which the MAC can be replaced by the simple XNOR operations. In this paper, we demonstrate an adiabatic training method that can binarize the fully-connected neural networks and the convolutional neural networks without modifying the network structure and size. This adiabatic training method only requires very minimal changes in training algorithms, and is tested in the following four tasks: the recognition of hand-writing numbers using a usual fully-connected network, the cat-dog recognition and the audio recognition using convolutional neural networks, the image recognition with 10 classes (CIFAR-10) using ResNet-20 and VGG-Small networks. In all tasks, the performance of the binary neural networks trained by the adiabatic method are almost identical to the networks trained using the conventional ReLU or Sigmoid activations with real-valued activations and weights. This adiabatic method can be easily applied to binarize different types of networks, and will increase the computational efficiency considerably and greatly simplify the deployment of neural networks. [ABSTRACT FROM AUTHOR]

Milling by mechanical means is vital unit operation in pharmaceutical processing which can be used for controlling particle size reduction. This approach can be also used for mechanochemical synthesis of pharmaceutical cocrystals. However, controlling the particle size of cocrystal during the milling process is challenging due to complexity of the process and different mechanisms involved. In this study, artificial neural network (ANN) approach was performed to predict the size of particles during mechanochemical synthesis of pharmaceutical cocrystals in ball milling operation. Theophylline was used as active pharmaceutical ingredient (API) and 4-aminobenzoic acid as co-former in the experiments. Different types of excipients including hydroxypropylmethylcellulose (HPMC), lactose, microcrystalline cellulose (MCC) polyvinylpyrrolidone (PVP) and polyethylene glycol (PEG) were used to see the effect of excipients on the cocrystals particle size. ANN was developed considering excipient type, excipient percentage, jar size, milling time as inputs, while median particle size (d50) was considered as the response, and representative of particle size. Two hidden layers were considered in developing ANN, and a combination of linear and nonlinear functions was used as transfer function. ANN was trained and validated with measured data, and R2 greater than 0.99 was obtained for the training and validation, with RMSE values of 1.06 and 3.89 for training and validation, respectively. The results were used to provide a design space for understanding the cocrystal particle size variation during ball milling process. It was indicated that the largest particles were formed using PEG as excipient, and increase in particle size over the milling time. The latter was attributed to the electrostatic attraction forces between the particles, thus aggregation and agglomeration of particles as the milling time exceeds a threshold. Furthermore, larger particles were obtained with the larger jar size. It turned out that faster cocrystallization rate occurred after only 5 min of milling in 25 mL jar compared to 25 min in 10 mL milling jar. The developed methodology has been shown to be robust and can be used as predictive tool for designing pharmaceutical cocrystallization using ball milling, and controlling the particle size during size reduction process. [ABSTRACT FROM AUTHOR]

This article investigates the sampled-data stabilization problem of a class of switched nonlinear systems. All subsystems of the considered system are allowed to be unstabilizable. To relax the restrictions on unknown nonlinear functions in some existing results, we use the nonlinear approximation ability of radial basis function neural networks. Novel mode-dependent adaptive laws and sampled-data control laws are constructed by only using the system states’ information at sampling instants. A novel sampled-data switching condition is derived, which can avoid Zeno behavior effectively. To guarantee that all states of the closed-loop system (CLS) are bounded, a new allowable sampling period is deduced. Finally, we demonstrate the proposed method’s effectiveness through two examples. [ABSTRACT FROM AUTHOR]

Sylvester equation is often applied to various fields, such as mathematics and control systems due to its importance. Zeroing neural network (ZNN), as a systematic design method for time-variant problems, has been proved to be effective on solving Sylvester equation in the ideal conditions. In this paper, in order to realize the predefined-time convergence of the ZNN model and modify its robustness, two new noise-tolerant ZNNs (NNTZNNs) are established by devising two novelly constructed nonlinear activation functions (AFs) to find the accurate solution of the time-variant Sylvester equation in the presence of various noises. Unlike the original ZNN models activated by known AFs, the proposed two NNTZNN models are activated by two novel AFs, therefore, possessing the excellent predefined-time convergence and strong robustness even in the presence of various noises. Besides, the detailed theoretical analyses of the predefined-time convergence and robustness ability for the NNTZNN models are given by considering different kinds of noises. Simulation comparative results further verify the excellent performance of the proposed NNTZNN models, when applied to online solution of the time-variant Sylvester equation. [ABSTRACT FROM AUTHOR]

In this paper, a neural network (NN)-based distributed cooperative identification strategy with event-triggered communication is studied for a group of coupled identical nonlinear systems. We develop a distributed cooperative learning law in the context of event-triggered communication, where an agent will transmit its NN weights to its neighbors only when its weight trigger error norm exceeds an exponentially decreasing threshold. It is proven that the estimated weights of all radial basis function NNs converge to a small neighborhood of their optimal values. Therefore, the unknown nonlinear function is approximated along the union of all the system trajectories. It is further proven that there exists a positive minimum interevent interval and Zeno behavior can be avoided. Finally, we give a simulation example to demonstrate these features. [ABSTRACT FROM AUTHOR]

Deep network in network (DNIN) model is an efficient instance and an important extension of the convolutional neural network (CNN) consisting of alternating convolutional layers and pooling layers. In this model, a multilayer perceptron (MLP), a nonlinear function, is exploited to replace the linear filter for convolution. Increasing the depth of DNIN can also help improve classification accuracy while its formation becomes more difficult, learning time gets slower, and accuracy becomes saturated and then degrades. This paper presents a new deep residual network in network (DrNIN) model that represents a deeper model of DNIN. This model represents an interesting architecture for on-chip implementations on FPGAs. In fact, it can be applied to a variety of image recognition applications. This model has a homogeneous and multilength architecture with the hyperparameter "L" ("L" defines the model length). In this paper, we will apply the residual learning framework to DNIN and we will explicitly reformulate convolutional layers as residual learning functions to solve the vanishing gradient problem and facilitate and speed up the learning process. We will provide a comprehensive study showing that DrNIN models can gain accuracy from a significantly increased depth. On the CIFAR-10 dataset, we evaluate the proposed models with a depth of up to L = 5 DrMLPconv layers, 1.66x deeper than DNIN. The experimental results demonstrate the efficiency of the proposed method and its role in providing the model with a greater capacity to represent features and thus leading to better recognition performance. [ABSTRACT FROM AUTHOR]

For turbulent reacting flow systems, identification of low-dimensional representations of the thermo-chemical state space is vitally important, primarily to significantly reduce the computational cost of device-scale simulations. Principal component analysis (PCA), and its variants, are a widely employed class of methods. Recently, an alternative technique that focuses on higher-order statistical interactions, co-kurtosis PCA (CoK-PCA), has been shown to effectively provide a low-dimensional representation by capturing the stiff chemical dynamics associated with spatiotemporally localized reaction zones. While its effectiveness has only been demonstrated based on a priori analyses with linear reconstruction, in this work, we employ nonlinear techniques to reconstruct the full thermo-chemical state and evaluate the efficacy of CoK-PCA compared to PCA. Specifically, we combine a CoK-PCA-/PCA-based dimensionality reduction (encoding) with an artificial neural network (ANN) based reconstruction (decoding) and examine, a priori , the reconstruction errors of the thermo-chemical state. In addition, we evaluate the errors in species production rates and heat release rates, which are nonlinear functions of the reconstructed state, as a measure of the overall accuracy of the dimensionality reduction technique. We employ four datasets to assess CoK-PCA/PCA coupled with ANN-based reconstruction: zero-dimensional (homogeneous) reactor for autoignition of an ethylene/air mixture that has conventional single-stage ignition kinetics, a dimethyl ether (DME)/air mixture which has two-stage (low and high temperature) ignition kinetics, a one-dimensional freely propagating premixed ethylene/air laminar flame, and a two-dimensional dataset representing turbulent autoignition of ethanol in a homogeneous charge compression ignition (HCCI) engine. Results from the analyses demonstrate the robustness of the CoK-PCA based low-dimensional manifold with ANN reconstruction in accurately capturing the data, specifically from the reaction zones. [ABSTRACT FROM AUTHOR]

• The BF-DNN model can be used to estimate high-dimensional and non-linear functions. • Hyperparameter optimization is efficacious for multi-fidelity neural networks. • BF-DNN and subset simulation methods are correlated to consider rare events. • Examples include high-dimensional benchmarks and shipbuilding problems. The present study proposes a two-stage Bi-Fidelity Deep Neural Network surrogate model to quantify the uncertainty of structural analysis using low-fidelity data samples added to the model to predict high-fidelity responses. This multi-fidelity surrogate model efficiently reduces the high computational cost for highly non-linear and high dimensional structural reliability problems. The framework is demonstrated using three different representative examples. First, it demonstrates the multi-fidelity model's accuracy for approximating a high non-linear 20-dimensional standard benchmark function that is hard to approximate with other methods and compared with another multi-fidelity neural network framework. In the other examples, the multi-fidelity framework is associated with Subset Simulation to efficiently estimate rare events considering a benchmark case study including high-dimensional scenarios, and the third example considers finite element model case studies. The proposed framework is also compared with a multi-fidelity Co-Kriging method. The results show that the proposed multi-fidelity framework, with its optimized hyperparameters using Bayesian Optimization, is an excellent strategy for reducing the number of samples used to construct the performance function's surrogate model. Moreover, the proposed framework can provide an accurate failure probability estimation with a lower computational cost in high non-linear, high dimensional, and rare events. [ABSTRACT FROM AUTHOR]

Context. Neutron stars are currently studied with an rising number of electromagnetic and gravitational-wave observations, which will ultimately allow us to constrain the dense matter equation of state and understand the physical processes at work within these compact objects. Neutron star global parameters, such as the mass and radius, can be used to obtain the equation of state by directly inverting the Tolman-Oppenheimer-Volkoff equations. Here, we investigate an alternative approach to this procedure. Aims. The aim of this work is to study the application of the artificial neural networks guided by the autoencoder architecture as a method for precisely reconstructing the neutron star equation of state, using their observable parameters: masses, radii, and tidal deformabilities. In addition, we study how well the neutron star radius can be reconstructed using only the gravitational-wave observations of tidal deformability, that is, using quantities that are not related in any straightforward way. Methods. The application of an artificial neural network in the equation-of-state reconstruction exploits the non-linear potential of this machine learning model. Since each neuron in the network is basically a non-linear function, it is possible to create a complex mapping between the input sets of observations and the output equation-of-state table. Within the supervised training paradigm, we construct a few hidden-layer deep neural networks on a generated data set, consisting of a realistic equation of state for the neutron star crust connected with a piecewise relativistic polytropes dense core, with its parameters representative of state-of-the art realistic equations of state. Results. We demonstrate the performance of our machine-learning implementation with respect to the simulated cases with a varying number of observations and measurement uncertainties. Furthermore, we study the impact of the neutron star mass distributions on the results. Finally, we test the reconstruction of the equation of state trained on parametric polytropic training set using the simulated mass–radius and mass–tidal-deformability sequences based on realistic equations of state. Neural networks trained with a limited data set are capable of generalising the mapping between global parameters and equation-of-state input tables for realistic models. [ABSTRACT FROM AUTHOR]

In this paper, we study the design of a fast finite-time consensus protocol for multiple nonlinear mechanical systems with uncertainties and disturbance. Combining the theories of finite-time control and algebraic graph, a distributed adaptive fast finite-time consensus protocol is proposed in a recursive design way. In the designed control scheme, the unknown nonlinear functions are modelled by neural networks. With the help of fast finite-time boundedness lemma, the practical finite-time consensus of the multiple mechanical systems is proved. In the end, the validity of the proposed protocol is confirmed by a numerical example of multiple manipulators. [ABSTRACT FROM AUTHOR]

Gait recognition has been considered as the emerging biometric technology for identifying the walking behaviors of humans. The major challenges addressed in this article is significant variation caused by covariate factors such as clothing, carrying conditions and view angle variations will undesirably affect the recognition performance of gait. In recent years, deep learning technique has produced a phenomenal performance accuracy on various challenging problems based on classification. Due to an enormous amount of data in the real world, convolutional neural network will approximate complex nonlinear functions in models to develop a generalized deep convolutional neural network (DCNN) architecture for gait recognition. DCNN can handle relatively large multiview datasets with or without using any data augmentation and fine‐tuning techniques. This article proposes a color‐mapped contour gait image as gait feature for addressing the variations caused by the cofactors and gait recognition across views. We have also compared the various edge detection algorithms for gait template generation and chosen the best from among them. The databases considered for our work includes the most widely used CASIA‐B dataset and OULP database. Our experiments show significant improvement in the gait recognition for fixed‐view, crossview, and multiview compared with the recent methodologies. [ABSTRACT FROM AUTHOR]

As one of standard approaches to train deep neural networks, dropout has been applied to regularize large models to avoid overfitting, and the improvement in performance by dropout has been explained as avoiding co-adaptation between nodes. However, when correlations between nodes are compared after training the networks with or without dropout, one question arises if co-adaptation avoidance explains the dropout effect completely. In this paper, we propose an additional explanation of why dropout works and propose a new technique to design better activation functions. First, we show that dropout can be explained as an optimization technique to push the input towards the saturation area of nonlinear activation function by accelerating gradient information flowing even in the saturation area in backpropagation. Based on this explanation, we propose a new technique for activation functions, gradient acceleration in activation function (GAAF) , that accelerates gradients to flow even in the saturation area. Then, input to the activation function can climb onto the saturation area which makes the network more robust because the model converges on a flat region. Experiment results support our explanation of dropout and confirm that the proposed GAAF technique improves image classification performance with expected properties. [ABSTRACT FROM AUTHOR]

Purpose: In this paper, we propose a convolutional neural network (CNN)‐based efficient model observer for breast computed tomography (CT) images. Methods: We first showed that the CNN‐based model observer provided similar detection performance to the ideal observer (IO) for signal‐known‐exactly and background‐known‐exactly detection tasks with an uncorrelated Gaussian background noise image. We then demonstrated that a single‐layer CNN without a nonlinear activation function provided similar detection performance in breast CT images to the Hotelling observer (HO). To train the CNN‐based model observer, we generated simulated breast CT images to produce a training dataset in which different background noise structures were generated using filtered back projection with a ramp, or a Hanning weighted ramp, filter. Circular, elliptical, and spiculated signals were used for the detection tasks. The optimal depth and the number of channels for the CNN‐based model observer were determined for each task. The detection performances of the HO and a channelized Hotelling observer (CHO) with Laguerre‐Gauss (LG) and partial least squares (PLS) channels were also estimated for comparison. Results: The results showed that the CNN‐based model observer provided higher detection performance than the HO, LG‐CHO, and PLS‐CHO for all tasks. In addition, it was shown that the proposed CNN‐based model observer provided higher detection performance than the HO using a smaller training dataset. Conclusions: In the presence of nonlinearity in the CNN, the proposed CNN‐based model observer showed better performance than other linear observers. [ABSTRACT FROM AUTHOR]

• Black-box decision-making algorithms proposals can be counter-intuitive. • Existing interpretability tools only extract global statistics. • Focus on interpretability for singular cases needed for data-driven decision making. • The algorithm is easy to implement, efficient and model-agnostic. With the widespread adoption of data mining models to solve real-world problems, the scientific community is facing the need of increasing their interpretability and comprehensibility. This is especially relevant in the case of black box models, in which inputs and outputs are usually connected by highly complex and nonlinear functions; in applications requiring an interaction between the user and the model; and when the machine's solution disagrees with the human experience. In this contribution we present a new methodology that allows to simplify the process of understanding the rules behind a classification model, even in the case of black box ones. It is based on the perturbation of the features describing one instance, and on finding the minimal variation required to change the forecasted class. It thus yields simplified rules describing under which circumstances would the solution have been different, and allows to compare these with the human expectation. We show how such methodology is well defined, model-agnostic, easy to implement and modular; and demonstrate its usefulness with several synthetic and real-world data sets. [ABSTRACT FROM AUTHOR]

In this work, the capability of bees-inspired algorithm with the aid of artificial neural networks (ANN-BA) for optimization of acid red 27 dye removal was studied using polypyrrole/SrFe12O19/graphene oxide (GO) as a novel nanocomposite. The nanocomposite fabricated through in situ polymerization and its characteristics were investigated by means of several instrumental techniques. The prepared polypyrrole/SrFe12O19/GO nanocomposite was employed to adsorb acid red 27 from aqueous solution. Based on the cost function as a nonlinear relation between some factors influencing the adsorption efficiency including adsorbent dosage, initial concentration, pH, shaking rate and contact time with percentage removal which were obtained by a multilayer perceptron artificial neural networks, bees metaheuristic algorithm was utilized to optimize of the batch sorption process. In addition, D-optimal response surface methodology (RSM) was also employed as a comparative study. In comparison with the D-optimal RSM, the ANN-BA model gave higher percentage removal (99%) about 4%. Under optimal conditions obtained by ANN-BA, equilibrium isotherms, kinetic behaviors and thermodynamics of the dye adsorption were thoroughly investigated. The pseudo-second-order model and the Langmuir adsorption model [with maximum capacity (qmax) of 294.11 mg g−1] fitted the experimental results with the determination coefficients (R2) of 0.94 and 0.99, respectively. The thermodynamic parameters have also been evaluated which showed the sorption procedure was endothermic and spontaneous. The findings obtained from the sorbent usage in wastewater treatment and its regeneration investigation revealed that the nanocomposite can be applied as an effective dye sorbent for removing acid red 27 dye in real samples with a reusable property. [ABSTRACT FROM AUTHOR]

The wide utility provided by the computational algorithms known as “deep learning” (or artificial neural networks) has prompted the development of specialized (electronic) hardware that can efficiently meet the speed and energy requirements that they demand. In the last few years, several proposals have also been made to build integrated photonic platforms to implement these algorithms by harnessing the light’s ability to efficiently perform several kinds of computational operations with low latency times and low power consumption. In this work, we present a proposal of a photonic device that is designed to implement one basic deep learning architecture, the multilayer perceptron, which consists of the sequential application of matrix-vector multiplications and element-wise nonlinear functions. We analyze how the speed and energy efficiency performance metrics, as well as system requirements, scale with the number of nodes in the layers of the network. Finally, we discuss the possible routes that may enable the implementation of larger networks, which opens a wide variety of research paths that can contribute to the creation of more scalable and applicable photonic devices. [ABSTRACT FROM AUTHOR]