Search

• A novel global optimization algorithm, rime optimization algorithm (RIME), is proposed. • RIME simulates the growth and crossover behavior of the rime-particle population. • The performance of RIME is adequately discussed based on benchmark functions. • RIME has better convergence accuracy and speed compared to other methods. • RIME is applied to solving five practical engineering optimization problems. This paper proposes an efficient optimization algorithm based on the physical phenomenon of rime-ice, called the RIME. The RIME algorithm implements the exploration and exploitation behaviors in the optimization methods by simulating the soft-rime and hard-rime growth process of rime-ice and constructing a soft-rime search strategy and a hard-rime puncture mechanism. Meanwhile, the greedy selection mechanism in the algorithm is improved, and the population is updated in the stage of selecting the optimal solution to enhance the exploitation capability of the RIME. In the experimental, this paper conducts qualitative analysis experiments on the RIME to clarify the characteristics of the algorithm in the process of finding the optimal solution. The performance of RIME is then tested on a total of 42 functions in the classic IEEE CEC2017 and the latest IEEE CEC2022 test sets. The proposed algorithm is compared with 10 well-established algorithms and 10 latest improved algorithms to verify its performance advantage. In addition, this paper designs experiments for the parametric analysis of RIME to discuss the potential of the algorithm in running different parameters and handling different problems. Finally, this paper applies RIME to five practical engineering problems to verify its effectiveness and superiority in real-world problems. The statistical and comparison results show that the RIME is a strong and competitive algorithm. The source code of the RIME 1 1 Source codes of RIME algorithm are also available at: https://codeocean.com/capsule/4472937/tree and https://www.mathworks.com/matlabcentral/fileexchange/124610-rime-a-physics-based-optimization and https://github.com/aliasgharheidaricom/RIME-A-physics-based-optimization. algorithm and associated files are publicly accessible at https://aliasgharheidari.com/RIME.html. [ABSTRACT FROM AUTHOR]

This paper proposes an H ∞ / H ∞ optimization technique to distributed fault detection (FD) for multi-agent systems (MASs). To simultaneously consider the effects of unknown disturbances and faults in the MASs and event-triggered transmission errors (ETTEs) on the residual signal, a residual generator related to the event-triggered threshold coefficient is designed. The generated residuals achieve the best compromise between robustness to unknown disturbances and sensitivity to faults. The obtained results have a better detection performance by using H ∞ / H ∞ optimization technique compared with other optimized schemes. In addition, the applicability of the proposed H ∞ / H ∞ optimized distributed FD scheme is verified by the simulation of a vehicle lateral dynamic system. [ABSTRACT FROM AUTHOR]

In this paper, a distributed fixed time gradient algorithm for neurodynamic systems is proposed for solving optimization problems with local inequality constraints. The algorithm is designed using fixed time theory and sliding model control techniques, where each agent has a local objective function known only to itself, and the optimal solution of each local objective function sum can be obtained in a fixed time by the information interaction between neighbors under the condition of local inequality constraints. In addition, the upper bound of the fixed time can be obtained and it is proved theoretically that the upper bound of the fixed time is independent of the initial value. Finally, the stability and effectiveness of the algorithm are verified by numerical examples. [ABSTRACT FROM AUTHOR]

Multi-objective immune algorithm (MOIA) is a heuristic algorithm based on artificial immune system model. Due to its characteristics of antibody clonal selection, automatic antigen recognition and immune memory in the immune system, artificial immune algorithm has become a research hotspot in the field of multi-objective optimization after the evolutionary algorithms. In this paper, most MOIAs can be classified into three main categories according to the type of problem solving, i.e., they are mostly designed to solve multi-objective optimization problems (MOPs), dynamic MOPs, and constrained MOPs. In this paper, a comprehensive survey is presented to summarize most existing MOIAs, in which their corresponding characteristics, principles and theoretical analyses are discussed in details. Moreover, the performance of MOIAs on solving various kinds of MOPs and many-objective optimization problems is also studied in our experimental comparisons. Finally, a brief conclusion is given to summarize the current drawbacks, challenges, and some future directions for MOIAs. [ABSTRACT FROM AUTHOR]

Class imbalance distribution widely exists in real-world engineering. However, the mainstream optimization algorithms that seek to minimize error will trap the deep learning model in sub-optimums when facing extreme class imbalance. It seriously harms the classification precision, especially in the minor classes. The essential reason is that the gradients of the classifier weights are imbalanced among the components from different classes. In this paper, we propose Attraction–Repulsion-Balanced Loss (ARB-Loss) to balance the different components of the gradients. We perform experiments on large-scale classification and segmentation datasets, and our ARB-Loss can achieve state-of-the-art performance via only one-stage training instead of 2-stage learning like nowadays SOTA works. [ABSTRACT FROM AUTHOR]

ℓ 1 regularization has been used for logistic regression to circumvent the overfitting and use the estimated sparse coefficient for feature selection. However, the challenge of such regularization is that the ℓ 1 regularization is not differentiable, making the standard convex optimization algorithm not applicable to this problem. This paper presents a simple projection neural network for ℓ 1 -regularized logistics regression. In contrast to many available solvers in the literature, the proposed neural network does not require any extra auxiliary variable nor smooth approximation, and its complexity is almost identical to that of the gradient descent for logistic regression without ℓ 1 regularization, thanks to the projection operator. We also investigate the convergence of the proposed neural network by using the Lyapunov theory and show that it converges to a solution of the problem with any arbitrary initial value. The proposed neural solution significantly outperforms state-of-the-art methods concerning the execution time and is competitive in terms of accuracy and AUROC. [ABSTRACT FROM AUTHOR]

In this paper, we propose a control framework for intelligent surgical robots under the Remote Center of Motion (RCM). The goal of a surgical robot is to assist surgeons in performing complex surgeries. RCM constraint implies that the surgical tip attached to the end-effector of the surgical robot does not slide away from the point of the incision while performing surgery. Implementation of a control algorithm to comply with RCM constraints is a complicated task because of the nonlinear model of the surgical robots and stringent conditions of accuracy imposed by the patient's safety. This paper proposes an optimization-driven approach to perform the surgical maneuver under RCM constraints. We then applied a bio-inspired optimization algorithm to solve the problem efficiently. For testing the performance of ZNNBAS, we used MATLAB to simulate a surgical procedure. A 7-DOF surgical robot (KUKA LBR IIWA 7) was used as a test bench for running the simulations. The simulation results show that the ZNNBAS is comparable with BAS, PSO, and GA and efficiently and robustly performed the task commanded maneuvers while enforcing the RCM constraints. [ABSTRACT FROM AUTHOR]

Many real world problems are multi-objective. Thus, the need for multi-objective learning and optimization algorithms is inevitable. Although the multi-objective optimization algorithms are well-studied, the multi-objective learning algorithms have attracted less attention. In this paper, a fuzzy multi-objective reinforcement learning algorithm is proposed, and we refer to it as the multi-objective fuzzy Q-learning (MOFQL) algorithm. The algorithm is implemented to solve a bi-objective reach-avoid game. The majority of the multi-objective reinforcement algorithms proposed address solving problems in the discrete state-action domain. However, the MOFQL algorithm can also handle problems in a continuous state-action domain. A fuzzy inference system (FIS) is implemented to estimate the value function for the bi-objective problem. We used a temporal difference (TD) approach to update the fuzzy rules. The proposed method is a multi-policy multi-objective algorithm and can find the non-convex regions of the Pareto front. [ABSTRACT FROM AUTHOR]

Unsupervised feature selection, one of the important dimensionality reduction methods for high-dimensional data, has been paid more and more attention by researchers due to its effectiveness of processing the data without labels. However, most unsupervised feature selection methods do not take into account the fact that real data tends to be mixed with noise or outliers, or only consider the sensitivity of the model to them. In addition, the local manifold structure of data is usually preserved by a simple graph based on fixed k nearest neighbors. On the one hand, the different sample distributions of different data are considered without distinction. On the other hand, simple graph can not well capture the inherent relations between samples when facing multi-modal data. In view of the above problems, a robust unsupervised feature selection algorithm, sparse and minimum-redundant subspace learning with dual regularization (SMRSDR), is proposed in this paper. Specifically, a self-paced learning regularization based on soft weight allocation is introduced into SMRSDR to strictly control the samples' entry into model learning. Normal samples are allowed to enter model learning in order of their importance, while noise or outliers are strictly blocked out of model learning. Besides, considering the difference in sample distribution and the multimodal nature of data, the traditional simple graph Laplacian regularization is replaced by the hypergraph Laplacian regularization based on adaptive nearest neighbors selection. Furthermore, the sparsity and minimum-redundancy constraints are imposed on the subspace learning framework so that the most representative and minimum-redundant features can be selected. Finally, the objective function of SMRSDR is solved by an alternating iterative optimization algorithm. And a series of experiments are performed to comprehensively prove the effectiveness and superiority of SMRSDR. [ABSTRACT FROM AUTHOR]

Linear Discriminant Analysis (LDA) is one of the most representative supervised robust dimensionality reduction methods for handling high-dimensional data. High-dimensional datasets tend to contain more outliers and other sorts of noise, whereas most of the existing LDA models incorrectly consider the arithmetic mean of samples as the optimal mean, leading to the deviation of the data mean and thus reduce the robustness of LDA. In this paper, we propose a novel robust trace ratio objective in which the calculation of the difference between sample and class mean is converted to the calculation of the difference between each pair of samples. Besides, the within-class scatter and the total scatter are measured by capped ℓ p -norm. As a result, this novel reformulation can automatically avoid mean calculation and meanwhile mitigate the negative effect of outliers on the objective function. Furthermore, an iterative optimization algorithm is derived to obtain the solution of the model. Extensive experimental results on several benchmark datasets show the superior performance of the proposed method. [ABSTRACT FROM AUTHOR]

Evolutionary multi-objective optimization problems have attracted increasingly attention in the evolutionary computing community. Now a lot of efforts have been devoted to this direction. For example, the proposed Pareto-dominated, decomposition-based and indicator-based methods have improved the scalability of multi-objective evolutionary algorithms (MOEAs). However, with the increase of the number of objectives, the portion of non-dominated individuals in the population is too large, and the distinguishability of individuals in the objective space decreases, which will affect the selection of elite solutions, thereby losing the balance of convergence and diversity. In this paper, a self-adaptive stochastic ranking method (SSR) is proposed to adaptively balance the convergence and diversity in high-dimensional space according to the state of the population. It has been embedded into the MOEA/D framework to form a novel multi-objective optimization algorithm, named MOEA/D-SSR. In addition, an improved shift-based density estimation strategy (ISDE) is adopted to enhance the convergence and diversity. Compared with the existing MOEAs on benchmark suite DTLZ and WFG with up to 10 objectives, the performance of the our algorithm has been verified. Experimental results show that the proposed algorithm is competitive compared with the most advanced MOEAs. [ABSTRACT FROM AUTHOR]

To describe objects more comprehensively and accurately, multi-view learning has attracted considerable attention. Recently, graph embedding based multi-view feature selection methods have been proposed and shown efficient in many real applications. The existing methods generally construct one common graph matrix to exploit the local structure of multi-view data via the linear weight fusion or learning one common graph matrix across all views. However, since all views share the identical graph structure, this emphasizes the consistency too much, resulting in restricting the diversity among different views. In this paper, a tensor low-rank constrained graph embedding method is proposed for multi-view unsupervised feature selection. To embody the view-specific information of each view, our model constructs the graph structure for each corresponding view, respectively. To capture the consistency across views, a tensor low-rank regularization constraint is imposed on the tensor data formed by these graph matrices. An efficient optimization algorithm with theoretical convergence guarantee is designed to solve the proposed method. Extensive experimental results validate that the proposed method outperforms some state-of-the-art methods. The code of our model can be found at https://www.researchgate.net/publication/353902948_demoTLR. [ABSTRACT FROM AUTHOR]

Edge computing and artificial intelligence (AI), especially deep learning algorithms, are gradually intersecting to build the novel system, namely edge intelligence. However, the development of edge intelligence systems encounters several challenges, and one of these challenges is the computational gap between computation-intensive deep learning algorithms and less-capable edge systems. Due to the computational gap, many edge intelligence systems cannot meet the expected performance requirements. To bridge the gap, a plethora of new techniques and optimization methods were proposed in the past years: lightweight deep learning models, network compression, and efficient neural architecture search. Although some reviews or surveys have partially covered this large body of literature, we lack a systematic and comprehensive review to discuss all aspects of these deep learning techniques which are critical for edge intelligence implementation. As various and diverse methods, applicable to edge systems, are proposed, a holistic review would enable edge computing engineers and the community to understand the state-of-the-art deep learning techniques that are instrumental for edge intelligence and to facilitate the development of edge intelligence systems. This paper surveys the representative and latest deep learning techniques that are useful for edge intelligence systems, including hand-crafted models, model compression, hardware-aware neural architecture search, and adaptive deep learning models. Finally, based on observations and simple experiments we conducted, we discuss some future directions. [ABSTRACT FROM AUTHOR]

The prediction of short term power load owns a great influence on performance of the whole electric system. Raising the result of power load forecasting is always a research spot. This paper proposes a method combined Elman neural network (ENN) and the particle swarm optimization (PSO) for the short-term power load forecasting. Firstly, this paper introduces the principle of ENN and PSO algorithm respectively and analyzes performance of network influenced by parameters of ENN. Then the particle swarm optimization algorithm is applied to searching the optimal learning rate of ENN. To study the capability of the method proposed in this paper, the method is utilized for the short-term power load forecasting, which is suitable to be solved by ENN. Besides, a comparison experiment on this method (PSO–ENN) with general regression neural network (GRNN), the original ENN and the traditional back-propagation neural network (BPNN) is given out to illustrate effectiveness of PSO–ENN. [ABSTRACT FROM AUTHOR]

Machine learning algorithms have been used widely in various applications and areas. To fit a machine learning model into different problems, its hyper-parameters must be tuned. Selecting the best hyper-parameter configuration for machine learning models has a direct impact on the model's performance. It often requires deep knowledge of machine learning algorithms and appropriate hyper-parameter optimization techniques. Although several automatic optimization techniques exist, they have different strengths and drawbacks when applied to different types of problems. In this paper, optimizing the hyper-parameters of common machine learning models is studied. We introduce several state-of-the-art optimization techniques and discuss how to apply them to machine learning algorithms. Many available libraries and frameworks developed for hyper-parameter optimization problems are provided, and some open challenges of hyper-parameter optimization research are also discussed in this paper. Moreover, experiments are conducted on benchmark datasets to compare the performance of different optimization methods and provide practical examples of hyper-parameter optimization. This survey paper will help industrial users, data analysts, and researchers to better develop machine learning models by identifying the proper hyper-parameter configurations effectively. [ABSTRACT FROM AUTHOR]

In this paper, we study multi-agent systems with asymmetric information structure. Due to limited channel capacity in communication network, the information in routing path suffers a transmission delay. Instead of the game theoretic setting, we formulate the problem as an online quadratic optimization problem subject to stochastic systems involving input delay. Since the probability statistics of system noise is unknown, the decision-maker can not utilize the traditional optimal control strategies. Motivated by online convex optimization theory, we introduce the notion of regret, which measures the cumulative performance difference between the optimal statistics known (offline) index value and the statistics unknown (online) index value. The contributions of this paper are twofold. First, utilizing the linear minimum mean square biased estimate, we derive a learning based control policy and then characterize its behavior. Second, under some basic assumptions, we further prove that the regret grows at a sub-linear rate and it is explicitly bounded by O (ln T). [ABSTRACT FROM AUTHOR]

With intrinsic non-uniform illumination variations in laparoscopic images, such images often suffer from insufficient lighting and low visibility. This in turn may cause incorrect targeting, surgical risk, and extended operating time during laparoscopic surgery. Although various methods for nature image enhancement have been proposed in past decades, surgical laparoscopic image enhancement still needs improving due to issues including naturalness, blurred texture, color cast, and extensive computational time. To address these problems, this paper proposes a laparoscopic image enhancement method, based on distributed retinex optimization with refined information fusion. The proposed method minimizes a distributed optimization model with l 2 -norm regularization terms, resulting in a low computational complexity. By integrating the refined image information into the optimization process, our method significantly enhances dark regions while preserving naturalness and texture structures. Experimental results show that our optimization algorithm is more effective than conventional enhancement algorithms in terms of vision augmentation, performance index, and computation time. [ABSTRACT FROM AUTHOR]

In this paper, the finite-iteration tracking issue of multi-agent systems is investigated with the distributed optimization method. Since the infinite-iteration strategy is unrealistic, the finite-iteration approach is adopted to study the problem. Moreover, the tracking performance is undesired for the traditional iterative learning schemes. Then optimization technique is used to cover the drawback. The tracking error is regarded as the optimization objective, and the consensus tracking issue is considered as the distributed optimization problem. An appropriate learning control strategy is designed on the basis of the gradient of the tracking error. Further, the optimization issue can be solved by means of the finite-iteration approach, and the learning algorithm can improve the tracking performance. Theoretical results are proposed by using the norm estimation and some inequalities skills. At last, a numerical simulation is given to illustrate the effectiveness of the main results. [ABSTRACT FROM AUTHOR]

[Display omitted] • A distributed algorithm is proposed for the generalized Nash equilibrium seeking. • The proposed algorithm is available for a more general Nash game. • The proposed algorithm can achieve an exponential convergence rate. In this paper, a distributed optimization algorithm is proposed for aggregative game with coupled constraints. Based on the singular perturbation system, the generalized Nash equilibrium is sought by a group of agents. By employing the average consensus method in the fast manifold, the aggregates in the object function can be estimated via simple information exchanges, as well as the aggregate of dual variables, which provides necessary information for the fully distributed algorithm design. Moreover, the exponential convergence of the proposed algorithm is explored based on the Lyapunov method, the properties of the variational inequality and the characteristic of the singular perturbation system. Application to the resource competition problem in smart grid verifies the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

[Display omitted] • Proposed a novel two-step label embedding semantic-guided hashing method. • Proposed a fast alternative optimization strategy to address the non-convex problem. • Evaluated the effectiveness of the proposed method via three famous datasets. Hashing technologies have been widely used for information retrieval tasks due to their efficient retrieval and storage capabilities. Generally, most of the current supervised learning only utilizes labels to construct a binary similarity matrix of instance pairs and ignores the rich semantic information contained in the labels. Indeed, the reason why supervised hashing is better than unsupervised hashing is that the labels itself has strong discriminative information. Therefore, how to effectively explore the label information is one of the ways to improve the performance of retrieval tasks. In addition, existing hashing methods have the problems of high time consumption and weak scalability when facing large-scale data. To remedy these problems, in this paper, we present a flexible two-step label embedding hashing method named L abel E mbedding S emantic- G uided H ashing (LESGH). In the first step, LESGH leverages an asymmetric discrete learning framework to learn discriminative compact hash codes only from label information, and adds the constraints of bit-balance and bit-decorrelation to boost the quality of the hash code generation. In the second step, LESGH learns the hash projection function through the generated hash codes in the first step. Moreover, an effective and fast iterative discrete optimization algorithm is presented to solve the discrete problem instead of using the relaxation-based scheme. In doing so, we can not only simplify the optimization process, but also easily scale to large-scale data. We conduct several experiments on three public datasets, i.e., WIKI, MIRFlickr and NUS-WIDE, demonstrate that LESGH can improve the retrieval performance over the compared state-of-the-art baselines. [ABSTRACT FROM AUTHOR]

In recent years, a constant and fast information growing has characterized digital applications in the majority of real-life scenarios. Thus, a new information asset, namely Big Data, has been defined and lead to different challenges, mainly related to data storage, management and analysis. Focusing on the last challenge, several Big Data analytics techniques have been developed, based on Machine Learning and Deep Learning paradigms. When dealing with Big Data, traditional approaches often take a lot of time to produce even a single predictive model, due to the extremely high demand of computational resources. The design of approaches specifically oriented to Big Data is required to overcome these computational issues. Most solutions rely on the deployment of Big Data analytics infrastructures on a cluster of machines and/or on parallelization techniques. When deployment and parallelization apply to Machine Learning and Deep Learning, we can refer to the terms Distributed Machine Learning and Distributed Deep Learning, respectively. We here discuss the main principles and features of Distributed Machine Learning and Distributed Deep Learning frameworks. The main contribution of this work is a survey of solutions proposed in the literature, through the investigation of selected features and capabilities. In particular, the survey provides a comparative analysis according to the following classification criteria: implemented parallelization technique, supporting device, supported architecture, implemented communication mode, working mode, and class of algorithms. The paper also gives an overview of the most commonly used criteria and metrics for the performance evaluation of analyzed frameworks; finally, some emerging but promising optimization techniques are reviewed apart from our classification. • Big Data analytics leads to challenges related to data management and analysis. • An analysis of the related solutions proposed in the literature is required. • The analysis is focused on Distributed Machine Learning and Deep Learning. • Overview of framework performance evaluation criteria with optimization techniques. [ABSTRACT FROM AUTHOR]

This paper presents a new approach to reservoir computing (RC) optimization. Reservoir computing (RC) is a framework for building recurrent neural networks (RNNs) that alleviates the well-known learning difficulties in such neural networks by training only the output layer. While the weights of the input and the nonlinear hidden layers are randomly generated, the adjustment of critical hyperparameters, such as the input and the feedback scaling factor, is essential for optimal performance in RC. While recent hardware implementations of RC are crucial for high-speed processing, standard gradient-based hyperparameter optimization is often irrelevant due to potentially uncertain or time-varying internal functions and parameters. In this work, we propose and analyze a stochastic optimization approach using gradient approximations based solely on noisy measurements of the loss function to circumvent this problem. Our numerical and experimental results confirm that the proposed method can provide near-optimal RC hyperparameters with substantial complexity reduction compared to competing methods, validating its potential for RC optimization. • Fixed step-size stochastic optimization is useful in Reservoir Computing • Reservoir Computing optimization becomes agnostic to uncertainties and temporal drift • Fast hyperparameter optimization becomes applicable to hardware Reservoir Computing [ABSTRACT FROM AUTHOR]

This paper empirically analyses the impact of changes to the set of training examples on the neural network error surface. Specific quantitative characteristics of the error surface related to properties such as ruggedness, modality and structure are measured and visualized as the set of training examples changes. Both the case of random subsampling and active learning from a fixed dataset are examined, producing ten different training scenarios. For each scenario eleven error surface characteristics are calculated for five common benchmark problems. The results demonstrate that the error surface characteristics calculated using only a subsample of the available data commonly do not generalize to that of the full dataset. The observed error surface characteristics are significantly impacted by the particular set of examples used to calculate error. Some error surface characteristics are significantly altered by small changes in the set of examples used to calculate error. The main finding from this study is that when the set of training examples may change during training the training of a neural network is in essence a dynamic optimization problem, suggesting that optimization algorithms developed specifically to solve dynamic optimization problems may be more efficient at training neural networks under such conditions. [ABSTRACT FROM AUTHOR]

Wind power forecasting has high application value in power systems. However, due to the intermittence and fluctuation of wind power, it is difficult to predict wind power effectively using a single forecasting model. Therefore, to improve the accuracy and stability of wind power forecasting, an ensemble learning model based on stacking framework is proposed in this paper. First, several decomposition techniques are used to pre-process the original wind power data and an optimal decomposition method is selected through experiments. Then, a quadratic interpolation based on state transition algorithm is proposed to optimize the parameters of the Bernstein polynomial model and the weights of the Hermite neural network (HNN) to obtain two base learners. Finally, the Spearman correlation coefficient is used to analyze the correlation of several base learners. The base learners with low correlation and strong prediction ability are selected as the first-layer forecasting model of the stacking model, and the HNN is used as the second-layer prediction model to obtain the stacking ensemble model. To verify the effectiveness of the proposed model, a large number of comprehensive experiments are carried out with wind power data from a wind farm in Xinjiang, China. Experimental results show that the proposed model has higher prediction accuracy and stability than other single forecasting models. [ABSTRACT FROM AUTHOR]

Dimension reduction technology is playing an increasingly important role in machine learning. One of the most important reduction technologies is linear discriminant analysis (LDA), but the main disadvantage of LDA is that it is unable to find the local manifold structure, which means that it may fail to dispose of multimodal data. However, high-dimensional multimodal data are ubiquitous in many rations. In this paper, we propose a new dimensionality reduction method called auto-weighted local discriminant analysis (ALDA). Our method learns the similarity matrix and updates in the subspace simultaneously so that the neighborships can be evaluated in the optimal subspaces instead of in the original space. Furthermore, the new model is built based on the ℓ 2 , 1 -norm and automatically assigns a small weight to the pairwise points with large distance, and vice versa; thus, the local structure information can be captured by the ALDA. Additionally, an iterative re-weighted optimization algorithm is provided to efficiently solve the proposed model. Finally, extensive experiments conducted on several benchmark datasets and some synthetic datasets demonstrate the effectiveness of ALDA when comparing with some state-of-the-art dimensionality reduction methods. [ABSTRACT FROM AUTHOR]

The future network is expected to support extremely large bandwidth, ultra-low latency or deterministic delay, extremely high reliability, and massive connectivity for novel forward-looking scenarios. As one of the most fundamental parts of the network, routing plays a vital role in a well-performed network. Recently, some new techniques using machine intelligence to optimize the network routing have been proposed. Although they have demonstrated great potential to improve the network performance, it is still a great challenge to apply machine intelligence-based routing in real network environment due to the limitations of current network architectures and protocols. Fortunately, the future network research program is on-going for designing new network paradigms, which provides an opportunity to address those limitations. In this paper, we investigate state-of-the-art techniques in machine intelligence-enabled network routing and discuss the development trends of machine intelligence-enabled routing optimization techniques for the future network. [ABSTRACT FROM AUTHOR]

The fitness landscape is a theory applied to the evolutionary dynamics of biological evolution to explain the behavior of evolutionary algorithms in the solution of optimization problems. With the continuous advancement of evolutionary algorithm optimization, a fitness landscape can present more abundant feature information, such as the local fitness, fitness distance correlation, and landscape roughness. These landscape features reflect the optimal solution distribution, quantity, and local unimodal topology of the optimization problem from various angles. This paper expresses the adaptability landscape features of typical optimization problems, engages in a quantitative analysis of the fitness distance correlation information, evaluates the difficulty of solving the problem within the search space, and obtains the correlation degree classification result. The search strategy adapts the mixed mutation and the fitness distance correlation for differential evolution. Empirical studies show that, the fitness distance correlation search strategy for the differential evolution algorithm can avoid falling into the local optimum, improve accuracy and convergence, and solve the single-objective optimization problem in a more comprehensive manner. [ABSTRACT FROM AUTHOR]

To address the ill-posed nature of image compressive sensing (CS) recovery, it is of great importance to properly exploit image priors. In this paper, we propose a novel group residual based nonlocal low-rank regularization for image CS recovery, namely GRNLR-CS, which performs nonlocal low-rank modeling based on the residual of patch group rather than on the patch group itself as conventional methods. In our GRNLR-CS, the weighted Schatten p-norm is adopted as a non-convex surrogate function to estimate the rank of the group residual. To make our GRNLR-CS method tractable and robust, the split Bregman iteration technique is employed to develop an efficient algorithm to solve the optimization problem of GRNLR-CS. Extensive experimental results demonstrate that our GRNLR-CS method outperforms some state-of-the-art optimization based or deep learning based methods for image CS recovery. [ABSTRACT FROM AUTHOR]

Smart grids can use energy stored in electric vehicles (EVs) through vehicle-to-grid (V2G) technology. When the mobility of electric vehicles is combined with V2G technology, electric vehicles can form an energy transmission network called the EV energy network. The EV energy network relies heavily on renewable energy, but due to the unstable nature of renewable energy, sometimes the energy supply is insufficient. If no adjustment is made for energy shortage, the EV energy network will soon collapse. This paper studies how to make the EV energy network operate normally as long as possible when the energy shortage comes. In order to ensure the survivability of the system in the event of energy shortage, a global optimization algorithm is proposed. New terms and criteria are defined for this scenario to evaluate algorithms' performance. The global optimization algorithm is simulated on real bus line data from Manhattan and the Pioneer Valley Transportation Authority (PVTA). Simulation results show that the algorithm is effective. [ABSTRACT FROM AUTHOR]

In the environment of smart cities, human facial age estimation has become an important research topic due to its wide applications. Although a variety of methods are proposed to depict facial biologic attributes, the underlying relationships between facial appearance and its biological aging process have not been fully explored. Although relationship learning methods have been constructed in terms of modeling either the facial representations or the facial attributes, none of them model the relationships simultaneously. In this paper, we propose a unified model to explore these relationships simultaneously, coined as JREAE. In JREAE, two covariance matrices are symmetrically constructed to capture the underlying correlations from both aspects of input facial features and output age labels. In this way, the potential relationships of both feature and label are not only modeled definitely, but also explored adaptively from the training data, which is significantly different from those methods that define the relationships manually. Then, we extend the JREAE model with deep convolutional architecture (deep-JREAE) for more powerful discrimination. In addition, we present optimization algorithms to solve the proposed models with theoretical convergence and complexity proof. Finally, through extensive experiments, we not only validate the effectiveness and superior of the proposed methods in performance, but also visualize and analyze the resulting relationships. [ABSTRACT FROM AUTHOR]

Abstract In this paper, we study the problem of matrix recovery, which aims to restore a target matrix of authentic samples from grossly corrupted observations. Most of the existing methods, such as the well-known Robust Principal Component Analysis (RPCA), assume that the target matrix we wish to recover is low-rank. However, the underlying data structure is often non-linear in practice, therefore the low-rankness assumption could be violated. To tackle this issue, we propose a novel method for matrix recovery in this paper, which could well handle the case where the target matrix is low-rank in an implicit feature space but high-rank or even full-rank in its original form. Namely, our method pursues the low-rank structure of the target matrix in an implicit feature space. By making use of the specifics of an accelerated proximal gradient based optimization algorithm, the proposed method could recover the target matrix with non-linear structures from its corrupted version. Comprehensive experiments on both synthetic and real datasets demonstrate the superiority of our method. [ABSTRACT FROM AUTHOR]

Abstract In recent years, convolutional neural networks (CNNs) have shown great performance in various fields such as image classification, pattern recognition, and multi-media compression. Two of the feature properties, local connectivity and weight sharing, can reduce the number of parameters and increase processing speed during training and inference. However, as the dimension of data becomes higher and the CNN architecture becomes more complicated, the end-to-end approach or the combined manner of CNN is computationally intensive, which becomes limitation to CNN's further implementation. Therefore, it is necessary and urgent to implement CNN in a faster way. In this paper, we first summarize the acceleration methods that contribute to but not limited to CNN by reviewing a broad variety of research papers. We propose a taxonomy in terms of three levels, i.e. structure level, algorithm level, and implementation level, for acceleration methods. We also analyze the acceleration methods in terms of CNN architecture compression, algorithm optimization, and hardware-based improvement. At last, we give a discussion on different perspectives of these acceleration and optimization methods within each level. The discussion shows that the methods in each level still have large exploration space. By incorporating such a wide range of disciplines, we expect to provide a comprehensive reference for researchers who are interested in CNN acceleration. [ABSTRACT FROM AUTHOR]

Highlights • Tensor based discriminative tracking framework is presented in this paper. • A multi-linear classifier with structured output is proposed for tensor input. • Parameter tensor reconstruction in online updating provides robustness against noise. • Tensor block coordinate descent optimization is introduced in online learning. • The proposed tracker shows superior performance on benchmark videos. Abstract Discriminative tracking algorithms have witnessed continued progress for distinguishing the target from background in unconstrained environments. The learning and detection task in existing visual tracking methods often convert a multidimensional data array into a vector-based observation. By altering the 2-D spatial structure of the image, transformation variants and global noises influence the discriminative ability of target representation, often result in degradation of performance. Different from vector representations, this paper presents a tensor-based large margin discriminative framework for visual tracking that utilizes the supervised tensor learning. In our method, an online structured support tensor classifier is designed which produces the multi-linear decision function, incorporating the nonlinearity of tensor-based feature over the target. In order to provide better spatial cues of target representation against noises and facilitate online tracking, we further introduce truncated tucker decomposition in structured multi-linear learning. The proposed algorithm poses an effective parameter tensor reconstruction in the classifier updating procedure and has a robust discriminative ability against several video background variants. Furthermore, a tensor block coordinate descent optimization is presented to achieve a closed form solution specific to the proposed truncated structured Tucker machine (TSTM). Experiment results on a recent comprehensive tracking benchmark demonstrate a promising performance of the proposed method subjectively and objectively compared with several state-of-the-art algorithms. [ABSTRACT FROM AUTHOR]

This work addresses the coordination problem of multiple robots with the goal of finding specific hazardous targets in an unknown area and dealing with them cooperatively. The desired behavior for the robotic system entails multiple requirements, which may also be conflicting. The paper presents the problem as a constrained bi-objective optimization problem in which mobile robots must perform two specific tasks of exploration and at same time cooperation and coordination for disarming the hazardous targets. These objectives are opposed goals, in which one may be favored, but only at the expense of the other. Therefore, a good trade-off must be found. For this purpose, a nature-inspired approach and an analytical mathematical model to solve this problem considering a single equivalent weighted objective function are presented. The results of proposed coordination model, simulated in a two dimensional terrain, are showed in order to assess the behavior of the proposed solution to tackle this problem. We have analyzed the performance of the approach and the influence of the weights of the objective function under different conditions: static and dynamic. In this latter situation, the robots may fail under the stringent limited budget of energy or for hazardous events. The paper concludes with a critical discussion of the experimental results. [ABSTRACT FROM AUTHOR]

Activation function is crucial to the recent successes of deep neural networks. In this paper, we first propose a new activation function, Multiple Parametric Exponential Linear Units (MPELU), aiming to generalize and unify the rectified and exponential linear units. As the generalized form, MPELU shares the advantages of Parametric Rectified Linear Unit (PReLU) and Exponential Linear Unit (ELU), leading to better classification performance and convergence property. In addition, weight initialization is very important to train very deep networks. The existing methods laid a solid foundation for networks using rectified linear units but not for exponential linear units. This paper complements the current theory and extends it to the wider range. Specifically, we put forward a way of initialization, enabling training of very deep networks using exponential linear units. Experiments demonstrate that the proposed initialization not only helps the training process but leads to better generalization performance. Finally, utilizing the proposed activation function and initialization, we present a deep MPELU residual architecture that achieves state-of-the-art performance on the CIFAR-10/100 datasets. The code is available at https://github.com/Coldmooon/Code-for-MPELU . [ABSTRACT FROM AUTHOR]

In this paper, a competitive mechanism integrated whale optimization algorithm (CMWOA) is proposed to deal with multi-objective optimization problems. By introducing the novel competitive mechanism, a better leader can be generated for guiding the update of whale population, which benefits the convergence of the algorithm. It should also be highlighted that in the competitive mechanism, an improved calculation of crowding distance is adopted which substitutes traditional addition operation with multiplication operation, providing a more accurate depiction of population density. In addition, differential evolution (DE) is concatenated to diversify the population, and the key parameters of DE have been assigned different adjusting strategies to further enhance the overall performance. Proposed CMWOA is evaluated comprehensively on a series of benchmark functions with different shapes of true Pareto front. Results demonstrate that proposed CMWOA outperforms other three methods in most cases regarding several performance indicators. Particularly, influences of model parameters have also been discussed in detail. At last, proposed CMWOA is successfully applied to three real world problems, which further verifies the practicality of proposed algorithm. [ABSTRACT FROM AUTHOR]

With the rapid development of financial market, a growing number of stocks become available on the financial market. How to efficiently select these stocks to achieve higher return and lower risk has become a hot research topic in financial management. This is usually called the portfolio optimization problem (POP). When the cardinality constrained (CC) is added to limit the number of selected stocks to a certain value, the resulting CCPOP is more challenging with the following two difficulties: i) Due to the complexity of CC in finical market, how to efficiently deal with CC in POP to obtain feasible solution is difficult and time-consuming. ii) The objectives of portfolio return and risk always conflict with each other and their relation is difficult to balance. To better deal with above difficulties, this paper focuses on the multi-objective CCPOP (MoCCPOP) and proposes a multiple populations co-evolutionary particle swarm optimization (MPCoPSO) algorithm, which is based on multiple populations for multiple objectives (MPMO) framework and has the following four advantages. Firstly, a hybrid binary and real (HBR) encoding strategy is introduced to better represent the stock selection and the asset weight of the solutions in MoCCPOP. Secondly, a return risk ratio heuristic (R3H) strategy based on the historical return and risk of each stock is proposed as a fast CC handling method to obtain feasible solutions. Thirdly, a new particle update method based on bi-directional local search (BLS) strategy is designed to increase the chance to improve the solution accuracy and to approach the global Pareto front (PF). Last but not least, a hybrid elite competition (HEC) strategy is proposed to assist the archive update, which provides more promising solutions and brings diversity to avoid local PF. The first two strategies help to efficiently deal with the CC challenge, while the last two strategies are efficient in solving the multi-objective challenge. By comparing with some recent well-performing and state-of-the-art multi-objective optimization algorithms, MPCoPSO shows the superior performance in solving the MoCCPOP. [ABSTRACT FROM AUTHOR]

In this paper, a modified collaborative filtering (MCF) algorithm with improved performance is developed for recommendation systems with application in predicting baseline data of Friedreich's Ataxia (FRDA) patients. The proposed MCF algorithm combines the individual merits of both the user-based collaborative filtering (UBCF) method and the item-based collaborative filtering (IBCF) method, where both the positively and negatively correlated neighbors are taken into account. The weighting parameters are introduced to quantify the degrees of utilizations of the UBCF and IBCF methods in the rating prediction, and the particle swarm optimization algorithm is applied to optimize the weighting parameters in order to achieve an adequate tradeoff between the positively and negatively correlated neighbors in terms of predicting the rating values. To demonstrate the prediction performance of the proposed MCF algorithm, the developed MCF algorithm is employed to assist with the baseline data collection for the FRDA patients. The effectiveness of the proposed MCF algorithm is confirmed by extensive experiments and, furthermore, it is shown that our algorithm outperforms some conventional approaches. [ABSTRACT FROM AUTHOR]

Short-term traffic forecasting is an important part of contemporary intelligent transportation systems. In this paper, based on echo state network optimized by improved fruit fly optimization algorithm (ESN-IFOA), a model is proposed to provide a five-minute forecast of traffic volume. In IFOA, parallel searching strategy and uniform crossover operator are applied to enhance searching range and communication between swarms. Simultaneously, the source allocation strategy is proposed to calculate the number of fruit flies generated by each swarm in the next iteration. The five main parameters of ESN are optimized by the proposed IFOA. Massive of prediction results and algorithm comparisons demonstrates that ESN-IFOA has very good forecasting ability for the five-minutes forecast of traffic volume. [ABSTRACT FROM AUTHOR]

This paper investigates a online learning optimization problem in a distributed manner, where a set of agents aim to cooperatively minimize the sum of local time-varying cost functions while communications among agents depend on a sequence of time-varying directed graphs. For such a problem, we first propose a online distributed stochastic (sub)gradient push-sum algorithm by utilizing distributed optimization methods and push-sum protocols. Then we analyze the regret bounds for the proposed algorithm on the cases when the cost functions are convex and strongly convex, respectively. The bound on the expected regret for convex functions grows sub-linearly with order of O (T) , where T is the time horizon. When the cost functions are strongly convex with Lipschitz gradients, the regret bound has an improved rate with order of O (ln T). Numerical simulations on the localization in wireless sensor networks are used to show the effectiveness of the proposed algorithm. [ABSTRACT FROM AUTHOR]

This paper discusses the classifier ensemble problem with sparsity and diversity learning, which is a central issue in machine learning. The current approach for reducing the size and increasing the accuracy of a classifier ensemble is to formulate it as a convex quadratic programming problem, which is a relaxation problem, and then solve it by using the existing methods for convex quadratic programming or by computing closed-form solutions. This paper presents a novel computational approach for solving the classifier ensemble problem with sparsity and diversity learning without any recourse to relaxation problems and their associated methods. We first show that the classifier ensemble problem can be expressed as a minimization problem for the sum of certain convex functions over the intersection of fixed point sets of quasi-nonexpansive mappings. Next, we propose fixed point optimization algorithms for solving the minimization problem and show that the algorithms converge to the solution of the minimization problem. It is shown that the proposed algorithms can directly solve the classifier ensemble problem with sparsity and diversity learning. Finally, we compare the performance of the proposed sparsity and diversity learning methods against an existing method in classification experiments using data sets from the UCI machine learning repository and the LIBSVM. The experimental results show that the proposed methods have higher classification accuracies than the existing method. [ABSTRACT FROM AUTHOR]

The problem of choosing error penalty parameter C for optimization extreme learning machine (OELM) is that it can take any positive value for different applications and it is therefore hard to choose correctly. In this paper, we reformulated OELM to take a new regularization parameter ν (ν-OELM) which is inspired by Schölkopf et al. The regularization in terms of ν is bounded between 0 and 1, and is easier to interpret as compared to C . This paper shows that: (1) ν-OELM and ν-SVM have similar dual optimization formulation, but ν-OELM has less optimization constraints due to its special capability of class separation and (2) experiment results on both artificial and real binary classification problems show that ν-OELM tends to achieve better generalization performance than ν-SVM, OELM and other popular machine learning approaches, and it is computationally efficient on high dimension data sets. Additionally, the optimal parameter ν in ν-OELM can be easily selected from few candidates. [ABSTRACT FROM AUTHOR]

An instance is often represented from different aspects (views or modalities), which leads to high-dimensional features and even multiple labels. In this paper, we focus on the feature selection problem in multi-label classification, for which a trivial solution is handling the labels dividedly. Obviously, such a scheme may not work well by leaving the label relationship out of consideration. Recently, several research works conduct feature selection directly under a multi-label framework by implicitly or explicitly modeling label relationship. However, these works assume that all labels share the same feature subset or subspace, which is not reasonable enough for some scenarios since different labels tend to convey different semantics. To address this problem, we develop a novel approach in this paper to select label-dependent features for multi-label classification. Specifically, we (1) formulate a convex model based on a more general and practical assumption that different labels convey different semantics with specific features; (2) design an alternating optimization algorithm based on Nesterov's method and L 1 -ball projection for efficiently finding the optimal solution, which can realize multi-label classification, feature selection, and label relationship estimation simultaneously. Finally, experiments on publicly available datasets show that the proposed algorithm achieves better performance than several related methods. [ABSTRACT FROM AUTHOR]

This paper studies a resource optimization problem for a multi-agent network where all agents have local objective functions and local constraint sets. Meanwhile, the decision variables of all the agents need to satisfy a set of globally coupled inequality constraints. Then, a distributed continuous-time algorithm is designed with the help of the nonsmooth analysis theory and projection operator method. The proposed algorithm does not require each agent to send the gradient information of the cost function to its neighbors, which prevents the gradient information of agents leaking out. Moreover, the convergence analysis shows that the proposed algorithm can converge to the optimal solution starting from any initial allocation. Finally, a case study is presented for a resource allocation problem of a hybrid water-power network. [ABSTRACT FROM AUTHOR]

This paper addresses the issue of group consensus for a novel network model, which possesses a two-layer structure and consists of multiple interactive agents. Agents in the network are divided into several different groups according to hybrid interactions, then two neighboring agents can belong to the same group, different groups but in the same layer or different layers. Correspondingly, three kinds of relationships exist between neighboring agents in the network. Furthermore, consider that only agents in the first layer are pinned by virtual leaders, then based on the graph theory, Lyapunov function method, and optimization technique, some criteria for realizing group consensus of the network under fixed topology and switching topologies are established, which mainly depend on the topology and time intervals of the switching signal. It is found out that when the group consensus is achieved, the agents in the first layer follow the leaders asymptotically, while the agents in the second layer converge to opposite states of the leaders. Finally, a detailed numerical example is given for illustration. [ABSTRACT FROM AUTHOR]

As one of standard approaches to train deep neural networks, dropout has been applied to regularize large models to avoid overfitting, and the improvement in performance by dropout has been explained as avoiding co-adaptation between nodes. However, when correlations between nodes are compared after training the networks with or without dropout, one question arises if co-adaptation avoidance explains the dropout effect completely. In this paper, we propose an additional explanation of why dropout works and propose a new technique to design better activation functions. First, we show that dropout can be explained as an optimization technique to push the input towards the saturation area of nonlinear activation function by accelerating gradient information flowing even in the saturation area in backpropagation. Based on this explanation, we propose a new technique for activation functions, gradient acceleration in activation function (GAAF) , that accelerates gradients to flow even in the saturation area. Then, input to the activation function can climb onto the saturation area which makes the network more robust because the model converges on a flat region. Experiment results support our explanation of dropout and confirm that the proposed GAAF technique improves image classification performance with expected properties. [ABSTRACT FROM AUTHOR]

Compressed sensing has proved itself as a useful technique for accelerating time-consuming fMRI acquisition. However, its intrinsic iterative algorithm of solving optimization problems limits its practical usage. In addition, it may still suffer from residual errors and artifacts when more aggressive acceleration factors are adopted. Deep neural networks have recently shown great potential in computer vision and image processing. Nevertheless, few attempts have been made concerning fMRI reconstruction. In this paper, we propose a deep residual network for faster and better image reconstruction with 20x acceleration. The network is made up of various residual blocks, and is trained to identify the mapping relationship between an aliased image and a fully-recovered image. Mean square error criterion refined with data-consistency loss is employed to evaluate the 'distance' between the network output and ground truth. Results showed that the proposed method can achieve superior image quality and better preserves dynamic features than other state-of-art methods. In addition, the reconstruction can be extremely fast with a one-way deployment on a feed-forward network. [ABSTRACT FROM AUTHOR]

Recently, the Transformer architecture has achieved state-of-the-art performance in many natural language processing tasks. One key component in the Transformer architecture is the attention layer, which captures the relation between tokens. In this paper, we show that the weight of the attention layer has scale-invariant property, i.e. the output is invariant to a rescaling of weights. However, optimization algorithms in the vector space of weight such as SGD are not scaling invariant. This mismatch will potentially hurt the optimization process. To solve the mismatch, we seek a new parameter space for attention layer that is both scale-invariant and can sufficiently represent the output of attention, so that we can employ optimization algorithms in the scale-invariant parameter space. To achieve this goal, we first show that the output of the attention layer can be represented using scale-invariant variables, which is called paths. Then, we define basis paths which are an independent subset of all paths and are sufficient to represent all other paths. We prove that the Scale-Invariant (SI) space for the attention layer is composed of the basis path. Finally, we design an Attention Basis Path Identification(ABPI) Method to identify the basis paths and propose optimizing the attention layer directly in its SI space. Several experiments on benchmark datasets show that we can obtain more effective neural networks with the attention layer by optimizing the attention layer directly in its SI space. [ABSTRACT FROM AUTHOR]

• This study proposes a method to create the aim-object layer. With the assistance of the aim-object layer, a novel type of NFS, termed the AANFS, was constructed which does not require the same number of neurons in the premise and consequent layers. • A new kind of fuzzy set, SCFS, is used to improve conventional fuzzy sets in a fuzzy inference system and enables models the flexibility in changing the number of outputs depending on the application, without significantly increasing parameters. • A WOA-RLSE hybrid learning algorithm is proposed to optimize the parameters of the AANFS model based on the strategy of divide-and-conquer. This paper proposes an aim-object-based asymmetric neuro-fuzzy system that is different from conventional models in two ways. First, this system has an asymmetric structure with different numbers of neurons in the premise and consequent layers. Secondly, with the assistance of the sphere complex fuzzy set, depending on the application, our model can alter the number of outputs. In addition, a hybrid learning algorithm combining the whale optimization algorithm and the recursive least-square estimator is proposed to optimize the proposed model. The results of the experiment show that the proposed model can simultaneously predict multiple targets with fewer parameters and maintain a performance level similar to that of the conventional neuro-fuzzy system. [ABSTRACT FROM AUTHOR]

In this paper, we present a deep learning-based approach, namely a dual CNN–RNN for multiple people tracking. We follow tracking-by-detection paradigm by first training a CNN to measure the similarity of two detection boxes. To solve the data association (DA) problem, we build a graph with nodes as detections and edge costs that are the outputs of a CNN. The general minimum cost lifted multi-cut problem (LMP) and corresponding optimization algorithms are utilized to solve the DA problem. To tackle occlusion and ID-switch problems, an RNN network is proposed to predict the nonlinear motion of people. Moreover, we utilize target motion information to stitch tracklets and build long trajectories. The results of our experiments conducted on Multiple Object Tracking Benchmark 2016 (MOT2016) confirm the efficiency of the proposed algorithm. [ABSTRACT FROM AUTHOR]