Search

In this paper, we propose a novel iterative optimization model for bottom-up saliency detection. By exploring bottom-up saliency principles and semisupervised learning approaches, we design a high-performance saliency analysis method for wide ranging scenes. The proposed algorithm consists of two stages: 1) we develop a boundary homogeneity model to characterize the general position and the contour of the salient objects and 2) we propose a novel iterative optimization model, termed gradual saliency optimization, for further performance improvement. Our main contribution falls on the second stage, where we propose an iterative framework with self-repairing mechanisms for refining saliency maps. In this framework, we further develop a more comprehensive optimization function applying a novel semisupervised learning scheme to enhance the traditional saliency measure. More elaborately, the iterative method can gradually improve the output in each iteration and finally converge to high-quality saliency maps. Based on our experiments on four different public data sets, it can be demonstrated that our approach significantly outperforms the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In learning applications, exploring the cluster structures of the high dimensional data is an important task. It requires projecting or visualizing the cluster structures into a low dimensional space. The challenges are: 1) how to perform the projection or visualization with less information loss and 2) how to preserve the interpretability of the original data. Recent methods address these challenges simultaneously by unsupervised feature selection. They learn the cluster indicators based on the $k$ nearest neighbor similarity graph, then select the features highly correlated with these indicators. Under this direction, many techniques, such as local discriminative analysis, nonnegative spectral analysis, nonnegative matrix factorization, etc., have been successfully introduced to make the selection more accurate. In this paper, we focus on enhancing the unsupervised feature selection in another perspective, namely, making the selection exploit the combination effect of the features. Given the expected feature amount, previous works operate on the whole features then select those of high coefficients one by one as the output. Our proposed method, instead, operates on a group of features initially then update the selection when a better group appears. Compared to the previous methods, the proposed method exploits the combination effect of the features by $\ell {}_{2,0}$ norm. It improves the selection accuracy where the cluster structures are strongly related to a group of features. We conduct the experiments on six open access data sets from different domains. The experimental results show that our proposed method is more accurate than the recent methods which do not specially consider the combination effect of the features. [ABSTRACT FROM AUTHOR]

Imbalanced data with a skewed class distribution are common in many real-world applications. Deep Belief Network (DBN) is a machine learning technique that is effective in classification tasks. However, conventional DBN does not work well for imbalanced data classification because it assumes equal costs for each class. To deal with this problem, cost-sensitive approaches assign different misclassification costs for different classes without disrupting the true data sample distributions. However, due to lack of prior knowledge, the misclassification costs are usually unknown and hard to choose in practice. Moreover, it has not been well studied as to how cost-sensitive learning could improve DBN performance on imbalanced data problems. This paper proposes an evolutionary cost-sensitive deep belief network (ECS-DBN) for imbalanced classification. ECS-DBN uses adaptive differential evolution to optimize the misclassification costs based on the training data that presents an effective approach to incorporating the evaluation measure (i.e., G-mean) into the objective function. We first optimize the misclassification costs, and then apply them to DBN. Adaptive differential evolution optimization is implemented as the optimization algorithm that automatically updates its corresponding parameters without the need of prior domain knowledge. The experiments have shown that the proposed approach consistently outperforms the state of the art on both benchmark data sets and real-world data set for fault diagnosis in tool condition monitoring. [ABSTRACT FROM AUTHOR]

In this paper, we address a class of distributed optimization problems in the presence of interagent communication delays based on passivity. We first focus on unconstrained distributed optimization and provide a passivity-based perspective for distributed optimization algorithms. With the help of the scattering transformation, this perspective allows us to handle arbitrary and unknown constant communication delays in the distributed optimization problem. Then, we extend the results to constrained distributed optimization. Finally, the algorithm is applied to a visual human localization problem. [ABSTRACT FROM AUTHOR]

A major challenge faced in the design of large-scale cyber-physical systems, such as power systems, the Internet of Things or intelligent transportation systems, is that traditional distributed optimal control methods do not scale gracefully, neither in controller synthesis nor in controller implementation, to systems composed of a large number (e.g., on the order of billions) of interacting subsystems. This paper shows that this challenge can now be addressed by leveraging the recently introduced system-level approach (SLA) to controller synthesis. In particular, in the context of the SLA, we define suitable notions of separability for control objective functions and system constraints such that the global optimization problem (or iterate update problems of a distributed optimization algorithm) can be decomposed into parallel subproblems. We then further show that if additional locality (i.e., sparsity) constraints are imposed, then these subproblems can be solved using local models and local decision variables. The SLA is essential to maintain the convexity of the aforementioned problems under locality constraints. As a consequence, the resulting synthesis methods have $O(1)$ complexity relative to the size of the global system. We further show that many optimal control problems of interest, such as (localized) LQR and LQG, $\mathcal {H}_2$ optimal control with joint actuator and sensor regularization, and (localized) mixed $\mathcal {H}_2/\mathcal {L}_1$ optimal control problems, satisfy these notions of separability, and use these problems to explore tradeoffs in performance, actuator, and sensing density, and average versus worst-case performance for a large-scale power inspired system. [ABSTRACT FROM AUTHOR]

In this paper, we establish necessary and sufficient conditions for global linear convergence rate bounds in operator splitting methods for a general class of convex optimization problems, where the associated fixed-point operator is strongly quasi-nonexpansive. We also provide a tight bound on the achievable convergence rate. Most existing results establishing global linear convergence in such methods require restrictive assumptions regarding strong convexity and smoothness of the constituent functions in the optimization problem. However, there are several examples in the literature showing that linear convergence is possible even when these properties do not hold. We provide a unifying analysis method for establishing global linear convergence based on linear regularity and show that many existing results are special cases of our approach. Moreover, we propose a novel linearly convergent splitting method for linear programming. [ABSTRACT FROM AUTHOR]

This paper studies motion planning of a mobile robot under uncertainty. The control objective is to synthesize a finite-memory control policy, such that a high-level task specified as a linear temporal logic formula is satisfied with a desired high probability. Uncertainty is considered in the workspace properties, robot actions, and task outcomes, giving rise to a Markov decision process that models the proposed system. Different from most existing methods, we consider cost optimization both in the prefix and suffix of the system trajectory. We also analyze the potential tradeoff between reducing the mean total cost and maximizing the probability that the task is satisfied. The proposed solution is based on formulating two coupled linear programs, for the prefix and suffix, respectively, and combining them into a multiobjective optimization problem, which provides provable guarantees on the probabilistic satisfiability and the total cost optimality. We show that our method outperforms relevant approaches that employ Round-Robin policies in the trajectory suffix. Furthermore, we propose a new control synthesis algorithm to minimize the frequency of reaching a bad state when the probability of satisfying the tasks is zero, in which case, most existing methods return no solution. We validate the above-mentioned schemes via both numerical simulations and experimental studies. [ABSTRACT FROM AUTHOR]

In this paper, a learning-based predictive control (LPC) scheme is proposed for adaptive optimal control of discrete-time nonlinear systems under stochastic disturbances. The proposed LPC scheme is different from conventional model predictive control (MPC), which uses open-loop optimization or simplified closed-loop optimal control techniques in each horizon. In LPC, the control task in each horizon is formulated as a closed-loop nonlinear optimal control problem and a finite-horizon iterative reinforcement learning (RL) algorithm is developed to obtain the closed-loop optimal/suboptimal solutions. Therefore, in LPC, RL and adaptive dynamic programming (ADP) are used as a new class of closed-loop learning-based optimization techniques for nonlinear predictive control with stochastic disturbances. Moreover, LPC also decomposes the infinite-horizon optimal control problem in previous RL and ADP methods into a series of finite horizon problems, so that the computational costs are reduced and the learning efficiency can be improved. Convergence of the finite-horizon iterative RL algorithm in each prediction horizon and the Lyapunov stability of the closed-loop control system are proved. Moreover, by using successive policy updates between adjoint time horizons, LPC also has lower computational costs than conventional MPC which has independent optimization procedures between two different prediction horizons. Simulation results illustrate that compared with conventional nonlinear MPC as well as ADP, the proposed LPC scheme can obtain a better performance both in terms of policy optimality and computational efficiency. [ABSTRACT FROM AUTHOR]

As a variant of nonnegative matrix factorization (NMF), symmetric NMF (SNMF) has shown to be effective for capturing the cluster structure embedded in the graph representation. In contrast to the existing SNMF-based clustering methods that empirically construct the similarity matrix and rigidly introduce the supervisory information to the assignment matrix, in this paper, we propose a novel SNMF-based semisupervised clustering method, namely, pairwise constraint propagation-induced SNMF (PCPSNMF). By formulating a single-constrained optimization problem, PCPSNMF is capable of learning the similarity and assignment matricesadaptively and simultaneously, in which a small amount of supervisory information in the form of pairwise constraints is introduced in a flexible way to guide the construction of the similarity matrix, and the two matrices communicate with each other to achieve mutual refinement until convergence. In addition, we propose an efficient alternating iterative algorithm to solve the optimization problem, whose convergence is theoretically proven. Experimental results over several benchmark image data sets demonstrate that PCPSNMF is less sensitive to initialization and produces higher clustering performance, compared with the state-of-the-art methods. [ABSTRACT FROM AUTHOR]

In this paper, a novel learning method calledpostboosting using extended G-mean(PBG) is proposed for online sequential multiclass imbalance learning (OS-MIL) in neural networks. PBG is effective due to three reasons. 1) Through postadjusting a classification boundary under extended G-mean, the challenging issue of imbalanced class distribution for sequentially arriving multiclass data can be effectively resolved. 2) A newly derived update rule for online sequential learning is proposed, which produces a high G-mean for current model and simultaneously possesses almost the same information of its previous models. 3) Adynamic adjustment mechanismprovided by extended G-mean is valid to deal with the unresolved challenging dense-majority problem and two dynamic changing issues, namely, dynamic changing data scarcity (DCDS) and dynamic changing data diversity (DCDD). Compared to other OS-MIL methods, PBG is highly effective on resolving DCDS, while PBG is the only method to resolve dense-majority and DCDD. Furthermore, PBG can directly and effectively handle unscaled data stream. Experiments have been conducted for PBG and two popular OS-MIL methods for neural networks under massive binary and multiclass data sets. Through the analyses of experimental results, PBG is shown to outperform the other compared methods on all data sets in various aspects including the issues of data scarcity, dense-majority, DCDS, DCDD, and unscaled data. [ABSTRACT FROM AUTHOR]

Top- $k$ performance has recently received increasing attention in large data categories. Advances, like a top- $k$ multiclass support vector machine (SVM), have consistently improved the top- $k$ accuracy. However, the key ingredient in the state-of-the-art optimization scheme based upon stochastic dual coordinate ascent relies on the sorting method, which yields $O(d\log d)$ complexity. In this paper, we leverage the semismoothness of the problem and propose an optimized top- $k$ multiclass SVM algorithm, which employs semismooth Newton algorithm for the key building block to improve the training speed. Our method enjoys a local superlinear convergence rate in theory. In practice, experimental results confirm the validity. Our algorithm is four times faster than the existing method in large synthetic problems; Moreover, on real-world data sets it also shows significant improvement in training time. [ABSTRACT FROM AUTHOR]

Mean square error (MSE) is the most prominent criterion in training neural networks and has been employed in numerous learning problems. In this paper, we suggest a group of novel robust information theoretic backpropagation (BP) methods, as correntropy-based conjugate gradient BP (CCG-BP). CCG-BP algorithms converge faster than the common correntropy-based BP algorithms and have better performance than the common CG-BP algorithms based on MSE, especially in nonGaussian environments and in cases with impulsive noise or heavy-tailed distributions noise. In addition, a convergence analysis of this new type of method is particularly considered. Numerical results for several samples of function approximation, synthetic function estimation, and chaotic time series prediction illustrate that our new BP method is more robust than the MSE-based method in the sense of impulsive noise, especially when SNR is low. [ABSTRACT FROM AUTHOR]

Many practical applications such as gene expression analysis, multitask learning, image recognition, signal processing, and medical data analysis pursue a sparse solution for the feature selection purpose and particularly favor the nonzeros evenly distributed in different groups. The exclusive sparsity norm has been widely used to serve to this purpose. However, it still lacks systematical studies for exclusive sparsity norm optimization. This paper offers two main contributions from the optimization perspective: 1) we provide several efficient algorithms to solve exclusive sparsity norm minimization with either smooth loss or hinge loss (nonsmooth loss). All algorithms achieve the optimal convergence rate $O(1/k^{2})$. ($k$ is the iteration number.) To the best of our knowledge, this is the first time to guarantee such convergence rate for the general exclusive sparsity norm minimization and 2) when the group information is unavailable to define the exclusive sparsity norm, we propose to use the random grouping scheme to construct groups and prove that if the number of groups is appropriately chosen, the nonzeros (true features) would be grouped in the ideal way with high probability. Empirical studies validate the efficiency of the proposed algorithms, and the effectiveness of random grouping scheme on the proposed exclusive support vector machine formulation. [ABSTRACT FROM AUTHOR]

Designing powerful models able to handle weakly labeled data are a crucial problem in machine learning. In this paper, we propose a new multiple instance learning (MIL) framework. Examples are represented as bags of instances, but we depart from standard MIL assumptions by introducing a symmetric strategy (SyMIL) that seeks discriminative instances in positive and negative bags. The idea is to use the instance the most distant from the hyper-plan to classify the bag. We provide a theoretical analysis featuring the generalization properties of our model. We derive a large margin formulation of our problem, which is cast as a difference of convex functions, and optimized using concave-convex procedure. We provide a primal version optimizing with stochastic subgradient descent and a dual version optimizing with one-slack cutting-plane. Successful experimental results are reported on standard MIL and weakly supervised object detection data sets: SyMIL significantly outperforms competitive methods (mi/MI/Latent-SVM), and gives very competitive performance compared to state-of-the-art works. We also analyze the selected instances of symmetric and asymmetric approaches on weakly supervised object detection and text classification tasks. Finally, we show complementarity of SyMIL with recent works on learning with label proportions on standard MIL data sets. [ABSTRACT FROM AUTHOR]

In this paper, a neural-dynamic optimization-based nonlinear model predictive control (NMPC) is developed for the multiple nonholonomic mobile robots formation. First, a model-based monocular vision method is developed to obtain the location information of the leader. Then, a separation-bearing-orientation scheme (SBOS) control strategy is proposed. During the formation motion, the leader robot is controlled to track the desired trajectory and the desired leader–follower relationship can be maintained through the SBOS method. Finally, the model predictive control (MPC) is utilized to maintain the desired leader–follower relationship. To solve the MPC generated constrained quadratic programming problem, the neural-dynamic optimization approach is used to search for the global optimal solution. Compared to other existing formation control approaches, the proposed solution is that the NMPC scheme exploit prime-dual neural network for online optimization. Finally, by using several actual mobile robots, the effectiveness of the proposed approach has been verified through the experimental studies. [ABSTRACT FROM AUTHOR]

In this paper, we present an approach for minimizing the computational complexity of the trained convolutional neural networks (ConvNets). The idea is to approximate all elements of a given ConvNet and replace the original convolutional filters and parameters (pooling and bias coefficients; and activation function) with an efficient approximations capable of extreme reductions in computational complexity. Low-complexity convolution filters are obtained through a binary (zero and one) linear programming scheme based on the Frobenius norm over sets of dyadic rationals. The resulting matrices allow for multiplication-free computations requiring only addition and bit-shifting operations. Such low-complexity structures pave the way for low power, efficient hardware designs. We applied our approach on three use cases of different complexities: 1) a “light” but efficient ConvNet for face detection (with around 1000 parameters); 2) another one for hand-written digit classification (with more than 180 000 parameters); and 3) a significantly larger ConvNet: AlexNet with $\approx 1.2$ million matrices. We evaluated the overall performance on the respective tasks for different levels of approximations. In all considered applications, very low-complexity approximations have been derived maintaining an almost equal classification performance. [ABSTRACT FROM AUTHOR]

Given a set of target detections over several time periods, this paper addresses the multitarget tracking (MTT) problem of optimally assigning detections to targets and estimating the trajectory of the targets over time. MTT has been studied in the literature via predominantly probabilistic methods. In contrast, we propose the use of mixed integer optimization (MIO) along with relaxations and local-search-based heuristic algorithms that are: scalable, as they provide near optimal solutions for six targets and ten time periods in milliseconds to seconds; general, as they make no probabilistic assumptions on the detection process; robust, as they can accommodate missed and false detections of the targets; and easily implementable, as they use at most two tuning parameters. We evaluate the performance of the new methods using a novel metric for the complexity of an instance, and find that they provide high quality solutions both reliably and quickly for a large range of scenarios, resulting in a promising approach to the area of MTT. [ABSTRACT FROM AUTHOR]

The problem of low-rank approximation with convex constraints, which appears in data analysis, system identification, model order reduction, low-order controller design, and low-complexity modeling is considered. Given a matrix, the objective is to find a low-rank approximation that meets rank and convex constraints while minimizing the distance to the matrix in the squared Frobenius norm. In many situations, this nonconvex problem is convexified by nuclear-norm regularization. However, we will see that the approximations obtained by this method may be far from optimal. In this paper, we propose an alternative convex relaxation that uses the convex envelope of the squared Frobenius norm and the rank constraint. With this approach, easily verifiable conditions are obtained under which the solutions to the convex relaxation and the original nonconvex problem coincide. A semidefinite programming representation of the convex envelope is derived, which allows us to apply this approach to several known problems. Our example on optimal low-rank Hankel approximation/model reduction illustrates that the proposed convex relaxation performs consistently better than nuclear-norm regularization and may outperform balanced truncation. [ABSTRACT FROM AUTHOR]

This paper proposes a distributed model predictive control (MPC) approach for a family of discrete-time linear systems with local (uncoupled) and global (coupled) constraints. The proposed approach is based on the dual problem of an overall MPC optimization problem involving all systems, which is then solved distributively using a modified distributed Nesterov-accelerated-gradient algorithm. To further reduce the computational requirement, this approach allows for early termination of the distributed gradient algorithm. This is made possible via a consensus algorithm that determines the satisfaction of the termination condition and by appropriate tightening of the coupled constraints. Under reasonable assumptions, the approach is able to produce a suboptimal solution as long as the network of the systems is connected while ensuring recursive feasibility and exponential stability of the closed-loop system. The performance of the proposed approach is demonstrated by a numerical example. [ABSTRACT FROM AUTHOR]

This paper develops a new sampling-based method for stability verification of piecewise continuous nonlinear systems via Lyapunov functions. Depending on the nonlinear system dynamics, the candidate Lyapunov function and the set of states of interest, verifying stability requires solving complex, possibly non-convex or infeasible optimization problems. To avoid such problems, the proposed approach firstly distributes the verification of Lyapunov's inequality on a finite sampling of a bounded set of states of interest. Secondly, it extends the validity of Lyapunov's inequality to an infinite, bounded set of states by automatically exploiting local continuity properties. A sampling-based method for estimating the domain of attraction (DOA) via level sets of a validated Lyapunov function is also presented. Efficient state-space exploration is achieved using multi-resolution sampling and hyper-rectangles as basic sampling blocks. The operations that need to be performed for each sampling point in the state-space can be carried out in parallel, which improves scalability. The proposed methodology is illustrated for various benchmark examples from the literature. [ABSTRACT FROM AUTHOR]

There are two ultimate goals in multiobjective optimization. The primary goal is to obtain a set of Pareto-optimal solutions while the secondary goal is to obtain evenly distributed solutions to characterize the efficient frontier. In this paper, a collaborative neurodynamic approach to multiobjective optimization is presented to attain both goals of Pareto optimality and solution diversity. The multiple objectives are first scalarized using a weighted Chebyshev function. Multiple projection neural networks are employed to search for Pareto-optimal solutions with the help of a particle swarm optimization (PSO) algorithm in reintialization. To diversify the Pareto-optimal solutions, a holistic approach is proposed by maximizing the hypervolume (HV) using again a PSO algorithm. The experimental results show that the proposed approach outperforms three other state-of-the-art multiobjective algorithms (i.e., HMOEA/D, MOEA/DD, and NSGAIII) most of times on 37 benchmark datasets in terms of HV and inverted generational distance. [ABSTRACT FROM AUTHOR]

Building highly nonlinear and nonparametric models is central to several state-of-the-art machine learning systems. Kernel methods form an important class of techniques that induce a reproducing kernel Hilbert space (RKHS) for inferring non-linear models through the construction of similarity functions from data. These methods are particularly preferred in cases where the training data sizes are limited and when prior knowledge of the data similarities is available. Despite their usefulness, they are limited by the computational complexity and their inability to support end-to-end learning with a task-specific objective. On the other hand, deep neural networks have become the de facto solution for end-to-end inference in several learning paradigms. In this paper, we explore the idea of using deep architectures to perform kernel machine optimization, for both computational efficiency and end-to-end inferencing. To this end, we develop the deep kernel machine optimization framework, that creates an ensemble of dense embeddings using Nyström kernel approximations and utilizes deep learning to generate task-specific representations through the fusion of the embeddings. Intuitively, the filters of the network are trained to fuse information from an ensemble of linear subspaces in the RKHS. Furthermore, we introduce the kernel dropout regularization to enable improved training convergence. Finally, we extend this framework to the multiple kernel case, by coupling a global fusion layer with pretrained deep kernel machines for each of the constituent kernels. Using case studies with limited training data, and lack of explicit feature sources, we demonstrate the effectiveness of our framework over conventional model inferencing techniques. [ABSTRACT FROM AUTHOR]

The independent component analysis (ICA) is a widely used method for solving blind separation problems. The ICA assumes that the sources are independent of each other and extracts them by maximizing their non-Gaussianity as the objective function. There are the two types of non-Gaussianity of the sources (the super-Gaussian type with the positive kurtosis and the sub-Gaussian one with the negative kurtosis). In this paper, we propose a new objective function unifying the two types of non-Gaussianity naturally, which is derived by applying the Gaussian approximation to the distribution of sources in the second-order polynomial feature space. The proposed objective function [called the adaptive ICA function (AIF)] is a simple form given as a summation of weighted fourth-order statistics, where the weights are adaptively estimated by the current kurtoses. The first practical advantage of the AIF is that it can extract the sources one by one in the descending order of the criterion of non-Gaussianity. It can solve the permutation ambiguity problem. The second and more important advantage is that it can estimate the number of non-Gaussian sources by the Akaike information criterion irrespective of the specific form of their distributions. In order to utilize the above-mentioned advantages of the AIF, we construct a new algorithm named the ordering ICA by extending the fast ICA. Experimental results verify that the ordering ICA can estimate the number of non-Gaussian sources correctly in both artificial and real data sets. [ABSTRACT FROM AUTHOR]

Main issues in visual servoing of manipulators mainly include rapid convergence of feature errors to zero and the safety of joints regarding joint physical limits. To address the two issues, in this paper, an image-based visual servoing scheme is proposed for manipulators with an eye-in-hand configuration. Compared with existing schemes, the proposed one does not require performing pseudoinversion for the image Jacobian matrix or inversion for the Jacobian matrix associated with the forward kinematics of the manipulators. Theoretical analysis shows that the proposed scheme not only guarantees the asymptotic convergence of feature errors to zero but also the compliance with joint angle and velocity limits of the manipulators. Besides, simulation results based on a PUMA560 manipulator with a camera mounted on the end effector verify the theoretical conclusions and the efficacy of the proposed scheme. [ABSTRACT FROM AUTHOR]

Unsupervised hashing can desirably support scalable content-based image retrieval for its appealing advantages of semantic label independence, memory, and search efficiency. However, the learned hash codes are embedded with limited discriminative semantics due to the intrinsic limitation of image representation. To address the problem, in this paper, we propose a novel hashing approach, dubbed as discrete semantic transfer hashing (DSTH). The key idea is to directly augment the semantics of discrete image hash codes by exploring auxiliary contextual modalities. To this end, a unified hashing framework is formulated to simultaneously preserve visual similarities of images and perform semantic transfer from contextual modalities. Furthermore, to guarantee direct semantic transfer and avoid information loss, we explicitly impose the discrete constraint, bit-uncorrelation constraint, and bit-balance constraint on hash codes. A novel and effective discrete optimization method based on augmented Lagrangian multiplier is developed to iteratively solve the optimization problem. The whole learning process has linear computation complexity and desirable scalability. Experiments on three benchmark data sets demonstrate the superiority of DSTH compared with several state-of-the-art approaches. [ABSTRACT FROM AUTHOR]

This paper considers the problem of synthesizing output feedback controllers subject to sparsity constraints. This problem is known to be generically NP-hard, unless the plant satisfies the quadratic invariance property. Our main results show that, even if this property does not hold, tractable convex relaxations with optimality certificates can be obtained by recasting the problem into a polynomial optimization through the use of polyhedral Lyapunov functions. Combining these ideas with rank minimization tools leads to a computationally attractive algorithm. As an alternative, we present a second relaxation, with lower computational complexity, based on finding the best sparse estimate of a desired control action. These results are illustrated with several examples. [ABSTRACT FROM AUTHOR]

Pricing control is an important problem in service systems and it aims to control customer behaviors through an economic way, instead of administrative commands. In this paper, we study a dynamic pricing and service rate control problem in an open Jackson network with limited capacity. The goal is to determine the optimal admission prices and the optimal service rates at every state such that the long-run average social welfare is maximized. The original problem is decomposed into a rate-setting problem plus a price-setting problem. To solve the rate-setting problem, we derive a difference formula based on the sensitivity-based optimization theory. When the cost rate function is convex in service rates and the value rate function is concave in arrival rates, we decompose the rate-setting problem into a series of convex optimization subproblems. When the rate functions have linear structure, these subproblems are even simpler and a bang–bang control is optimal. For the price-setting problem, we determine the state-dependent prices so as to induce the optimal arrival rates obtained by the rate-setting problem. We propose a recursive algorithm to numerically compute the conditional expected delays at every state. Finally, we conduct numerical experiments to explore the optimality properties and some useful insights for this dynamic pricing control problem. [ABSTRACT FROM AUTHOR]

A proper abstraction of a large-scale linear consensus network with a dense coupling graph is one whose number of coupling links is proportional to its number of subsystems and its performance is comparable to the original network. Optimal design problems for an abstracted network are more amenable to efficient optimization algorithms. From the implementation point of view, maintaining such networks are usually more favorable and cost-effective due to their reduced communication requirements across a network. Therefore, approximating a given dense linear consensus network by a suitable abstract network is an important analysis and synthesis problem. In this paper, we develop a framework to compute an abstraction of a given large-scale linear consensus network with guaranteed performance bounds using a nearly linear time algorithm. First, the existence of abstractions of a given network is proven. Then, we present an efficient and fast algorithm for computing a proper abstraction of a given network. Finally, we illustrate the effectiveness of our theoretical findings via several numerical simulations. [ABSTRACT FROM AUTHOR]

This paper proposes a new family of algorithms for training neural networks (NNs). These are based on recent developments in the field of nonconvex optimization, going under the general name of successive convex approximation techniques. The basic idea is to iteratively replace the original (nonconvex, highly dimensional) learning problem with a sequence of (strongly convex) approximations, which are both accurate and simple to optimize. Different from similar ideas (e.g., quasi-Newton algorithms), the approximations can be constructed using only first-order information of the NN function, in a stochastic fashion, while exploiting the overall structure of the learning problem for a faster convergence. We discuss several use cases, based on different choices for the loss function (e.g., squared loss and cross-entropy loss), and for the regularization of the NN’s weights. We experiment on several medium-sized benchmark problems and on a large-scale data set involving simulated physical data. The results show how the algorithm outperforms the state-of-the-art techniques, providing faster convergence to a better minimum. Additionally, we show how the algorithm can be easily parallelized over multiple computational units without hindering its performance. In particular, each computational unit can optimize a tailored surrogate function defined on a randomly assigned subset of the input variables, whose dimension can be selected depending entirely on the available computational power. [ABSTRACT FROM AUTHOR]

This paper examines a matrix-regularized multiple kernel learning (MKL) technique based on a notion of $(r,~p)$ norms. For the problem of learning a linear combination in the support vector machine-based framework, model complexity is typically controlled using various regularization strategies on the combined kernel weights. Recent research has developed a generalized $\ell _{p}$ -norm MKL framework with tunable variable $p$ ($p\ge 1$) to support controlled intrinsic sparsity. Unfortunately, this “1-D” vector $\ell _{p}$ -norm hardly exploits potentially useful information on how the base kernels “interact.” To allow for higher order kernel-pair relationships, we extend the “1-D” vector $\ell _{p}$ -MKL to the “2-D” matrix $(r,~p)$ norms ($1\le r,~p<\infty $). We develop a new formulation and an efficient optimization strategy for $(r,~p)$ -MKL with guaranteed convergence. A theoretical analysis and experiments on seven UCI data sets shed light on the superiority of $(r,~p)$ -MKL over $\ell _{p}$ -MKL in various scenarios. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel adaptive piecewise linear model for dynamic system identification. It has four unique features. First, the model designs a new kind of basis function for function approximation. It maintains the uniform shape for each basis function, so as to achieve a satisfactory tradeoff between generalization ability and model complexity. Second, the model takes the structure of basis functions as decision variables to optimize the formulated identification problems instead of taking expansion coefficients as decision variables as proposed by many existing approaches. Third, we establish an incremental design strategy to solve the system identification problems. In each step of the identification, the selection of optimal basis function is a Lipschitz continuous optimization problem that is likely to be easily handled with some mature toolboxes. This incremental design strategy greatly reduces the estimation cost. Fourth, we introduce a smoothing mechanism to avoid overfitting, when the output of dynamic systems is disturbed by noise. Tests on several benchmark dynamic systems demonstrate the potential of the proposed model. [ABSTRACT FROM AUTHOR]

Inverse reinforcement learning (IRL) attempts to use demonstrations of “expert” decision making in a Markov decision process to infer a corresponding policy that shares the “structured, purposeful” qualities of the expert's actions. In this paper, we extend the maximum causal entropy framework, a notable paradigm in IRL, to the infinite time horizon setting. We consider two formulations (maximum discounted causal entropy and maximum average causal entropy) appropriate for the infinite horizon case and show that both result in optimization programs that can be reformulated as convex optimization problems; thus, admitting efficient computation. We then develop a gradient-based algorithm for the maximum discounted causal entropy formulation that enjoys the desired feature of being model agnostic, a property that is absent in many previous IRL algorithms. We propose the stationary soft Bellman policy, a key building block in the gradient-based algorithm, and study its properties in depth, which not only lead to theoretical insight into its analytical properties, but also help motivate a large toolkit of methods for implementing the gradient-based algorithm. Finally, we select three algorithms of this type and apply them to two problem instances involving demonstration data from a simple controlled queuing network model inspired by problems in air traffic management. [ABSTRACT FROM AUTHOR]

Periodic optimal control problems (POCPs) based on dynamic models holding invariants are often problematic to treat using standard numerical methods. The difficulty stems from a failure of standard constraint qualifications and typically hinders the convergence of the numerical solver, or even defeats it. Optimization problems having weak constraint qualifications can be treated using dedicated solvers, at the price of a more involved algorithmic. In this paper, we analyze the constraint qualification of POCPs holding invariants, and propose three simple and computationally inexpensive modifications of the formulation that allow for a recovery of linear independence constraint qualification, while not affecting the second-order sufficient conditions for optimality. Hence, the resulting POCP can be tackled via standard solvers, without special treatment. The application of these approaches is detailed for the case of POCPs holding index-reduced differential-algebraic equations and representations of the $\mathrm{SO}\left(3\right)$ Lie group. [ABSTRACT FROM AUTHOR]

The process of manually labeling instances, essential to a supervised classifier, can be expensive and time-consuming. In such a scenario the semisupervised approach, which makes the use of unlabeled patterns when building the decision function, is a more appealing choice. Indeed, large amounts of unlabeled samples often can be easily obtained. Many optimization techniques have been developed in the last decade to include the unlabeled patterns in the support vector machines formulation. Two broad strategies are followed: continuous and combinatorial. The approach presented in this paper belongs to the latter family and is especially suitable when a fair estimation of the proportion of positive and negative samples is available. Our method is very simple and requires a very light parameter selection. Several medium- and large-scale experiments on both artificial and real-world data sets have been carried out proving the effectiveness and the efficiency of the proposed algorithm. [ABSTRACT FROM AUTHOR]

In this paper, the problem of the smart grid energy management under stochastic dynamics is investigated. In the considered model, at the demand side, it is assumed that customers can act as prosumers who own renewable energy sources and can both produce and consume energy. Due to the coupling between the prosumers’ decisions and the stochastic nature of renewable energy, the interaction among prosumers is formulated as a stochastic game, in which each prosumer seeks to maximize its payoff, in terms of revenues, by controlling its energy consumption and demand. In particular, the subjective behavior of prosumers is explicitly reflected into their payoff functions using the prospect theory, a powerful framework that allows modeling real-life human choices, rather than objective, user-agnostic decisions, as normative models do. For this prospect-based stochastic game, it is shown that there always exists a stationary Nash equilibrium where the prosumers’ trading policies in the equilibrium are independent of the time and their histories of the play. Moreover, to obtain one of such equilibrium policies, a novel distributed algorithm with no information sharing among prosumers is proposed and shown to converge to an $\epsilon$ -Nash equilibrium in which each prosumer is able to achieve its optimal payoff in an equilibrium up to a small additive error $\epsilon$. On the other hand, at the supply side, the interaction between the utility company and the prosumers is formulated as an online optimization problem in which the utility company's goal is to learn its optimal energy allocation rules. For this case, it is shown that such an optimization problem admits a no-regret algorithm meaning that regardless of the actual outcome of the game among the prosumers, the utility company can follow a strategy that mitigates its allocation costs as if it knew the entire demand market a priori. Simulation results justify the convergence of the proposed algorithms and present new insights toward more efficient energy management in the smart grids. [ABSTRACT FROM AUTHOR]

This paper studies the projected saddle-point dynamics associated to a convex–concave function, which we term saddle function. The dynamics consists of gradient descent of the saddle function in variables corresponding to convexity and (projected) gradient ascent in variables corresponding to concavity. We examine the role that the local and/or global nature of the convexity–concavity properties of the saddle function plays in guaranteeing convergence and robustness of the dynamics. Under the assumption that the saddle function is twice continuously differentiable, we provide a novel characterization of the omega-limit set of the trajectories of this dynamics in terms of the diagonal blocks of the Hessian. Using this characterization, we establish global asymptotic convergence of the dynamics under local strong convexity–concavity of the saddle function. When strong convexity–concavity holds globally, we establish three results. First, we identify a Lyapunov function (that decreases strictly along the trajectory) for the projected saddle-point dynamics when the saddle function corresponds to the Lagrangian of a general constrained convex optimization problem. Second, for the particular case when the saddle function is the Lagrangian of an equality-constrained optimization problem, we show input-to-state stability (ISS) of the saddle-point dynamics by providing an ISS Lyapunov function. Third, we use the latter result to design an opportunistic state-triggered implementation of the dynamics. Various examples illustrate our results. [ABSTRACT FROM AUTHOR]

This paper deals with the performance optimization of resource allocation systems with the aim of maximizing the system's throughput under a given budget for acquiring resources. Resources are assumed to be renewable, i.e., they are not consumed by the operations and become available again after they have been released. The systems under consideration are modeled by a subclass of timed Petri nets called deterministic timed weighted marked graphs. In addition, we take into account infinite server semantics, i.e., the degree of self-concurrency of each transition is infinite. We propose an approach that provides an optimal solution, but has a high computational cost. For this reason, we also present two different approaches that can find suboptimal solutions with a reduced computational cost. The performances of the proposed approaches are compared by means of numerical simulations. [ABSTRACT FROM AUTHOR]

In this paper, a control technique named optimized backstepping is first proposed by implementing tracking control for a class of strict-feedback systems, which considers optimization as a design philosophy of the high-order system control. The basic idea is that designing the actual and virtual controls of backstepping is the optimized solutions of the corresponding subsystems so that overall control of the high-order system is optimized. In general, optimization control is designed based on the solution of Hamilton–Jacobi–Bellman equation, but solving the equation is very difficult due to the inherent nonlinearity and intractability. In order to overcome the difficulty, the neural network (NN)-based reinforcement learning strategy of actor–critic architecture is used. In every backstepping step, the actor and critic NNs are constructed for executing control behavior and evaluating control performance, respectively. According to the Lyapunov stability theorem, it is proven that the desired control performance can be obtained. Finally, a simulation example is carried out to further demonstrate the effectiveness of the proposed control approach. [ABSTRACT FROM AUTHOR]

The amount of matrix data has increased rapidly nowadays. How to classify matrix data efficiently is an important issue. In this paper, by discovering the shortages of 2-D linear discriminant analysis and 2-D logistic regression, a novel 2-D framework named rank- $k$ 2-D multinomial logistic regression (2DMLR-RK) is proposed. The 2DMLR-RK is designed for a multiclass matrix classification problem. In the proposed framework, each category is modeled by a left projection matrix and a right projection matrix with rank $k$. The left projection matrices capture the row information of matrix data, and the right projection matrices acquire the column information. We identify the parameter $k$ plays the role of balancing the capacity of learning and generalization of the 2DMLR-RK. In addition, we develop an effective framework for solving the proposed nonconvex optimization problem. The convergence, initialization, and computational complexity are discussed. Extensive experiments on various types of data sets are conducted. Comparing with 1-D methods, 2DMLR-RK not only achieves a better classification accuracy, but also costs less computation time. Comparing with other state-of-the-art 2-D methods, the 2DMLR-RK achieves a better performance for matrix data classification. [ABSTRACT FROM AUTHOR]

This paper proposes an equalization approach to mean-square (MS) feedback stabilization for multi-input/multi-output (MIMO) discrete-time systems over fading channels. An orthogonal encode/decode matrix pair is placed at the two ends of the fading channel, thereby equalizing the complementary sensitivity in the MS sense. By adopting the notion of the MS stability in an input-output setting, and using the signal-to-noise ratio (SNR) and encode/decode matrix as additional design parameters, we are able to convert the networked stabilization over fading channels into an equivalent constrained \mathcal H2 optimal control, and derive the MS stabilizability conditions over linear time-invariant controllers. Our results recover the existing MS stabilizability condition under the state feedback. For output feedback control, we provide a characterization of the minimum network resource for the SNR required to stabilize the networked control system over the fading channel, and develop the synthesis algorithms for design of the orthogonal coding matrix and output feedback controller in different scenarios. Our results show that the MS stabilization problem as formulated in this paper admits the closed-form solution for MIMO plants with unstable block zeros, and is mathematically tractable for more general MIMO plants. Two numerical examples are worked out to illustrate our results. [ABSTRACT FROM PUBLISHER]

Time-interleaved analog-to-digital converter (TIADC) mismatch compensation filters are traditionally finite impulse response (FIR) filters. The advantages of using FIR filters are that FIR filters are inherently stable and that the design techniques are readily available. This paper demonstrates that infinite impulse response (IIR) filters can be designed to compensate for the mismatch error in the TIADC. Two methods for achieving stability are introduced; one is by increasing the group delay and the other is by constraining the magnitudes of the denominator coefficients. Simulation examples show that, for a given spurious frequency power, the proposed IIR TIADC compensation filter yields about 30% saving in the total number of multiplier coefficients when compared to the conventional FIR TIADC compensation filter. Important aspects of the IIR compensation filter such as noise propagation, performance in the presence of identification error, filter redesign to cope with mismatch drifting are also investigated. [ABSTRACT FROM PUBLISHER]

Consider a connected network of agents endowed with local cost functions representing private objectives. Agents seek to find an agreement on some minimizer of the aggregate cost, by means of repeated communications between neighbors. Consensus on the average over the network, usually addressed by gossip algorithms, is a special instance of this problem, corresponding to quadratic private objectives. Consensus on the median, or more generally, consensus on a given quantile, is also a special instance of this problem. In this paper we show that optimizing the aggregate cost function regularized by a total variation (TV) term has appealing properties. First, it can be done very naturally in a distributed way, yielding algorithms that are efficient on numerical simulations. Secondly, the optimum for the regularized cost is shown to be also the optimum for the initial aggregate cost function under assumptions that are simple to state. Finally, these algorithms are robust to unreliable agents that keep injecting some false value in the network. This is remarkable enough, and is not the case, for instance, of gossip algorithms that are entirely ruled by unreliable agents as detailed in the paper. [ABSTRACT FROM PUBLISHER]

This paper presents the stability and criticality analysis of integer linear programs with respect to perturbations in stochastic data given as Markov chains. These perturbations affect the initial distribution, the transition matrix, or the stationary distribution of Markov chains. Stability analysis is concerned with obtaining the set of all perturbations for which a solution remains optimal. This paper gives expressions for stability regions for perturbations in the initial distribution, the transition matrix, the stationary distribution, and the product of elements of the transition matrix and the stationary distribution. Furthermore, criticality measures that describe the sensitivity of the objective function with respect to an element of the problem data are derived. Stability regions that preserve the stochasticity of the problem data are given. Finally, stability regions for perturbations of elements of the transition matrix, given that the problem is not linear in the initial distribution or the transition matrix, are obtained using a small perturbation analysis. The results are applied to sensor placement problems and numerical examples are given. [ABSTRACT FROM PUBLISHER]

This paper presents comprehensive analyses of power-performance tradeoffs in current-mode logic (CML)-based transmitters via equation-based optimization and provides practical design guidelines that can help achieving optimum power efficiency. An accurate equation-based optimization framework was developed for this purpose, and using this framework, key circuit-level design parameters were found with specific design specifications and process technologies. The proposed optimization framework can determine optimum clock edge-rate (i.e., ratio of rise/fall time to clock period) and inter-stage voltage swings that enable significant power savings without affecting performance. Various transmitter design tradeoffs were analyzed for 45-nm, 65-nm, and 90-nm technologies considering data rate, clock edge-rate, and inter-stage swing optimization. Based on the analysis results, we determined the most energy-efficient data rate that can be achieved with acceptable power overhead for each technology node. Also, we identified the optimal clock edge-rate and inter-stage voltage swings in CML gates that achieved minimum transmitter power dissipation. Our analysis results indicate that, when properly optimized, overall transmitter power consumption can be reduced by up to 43% with inter-stage swing optimization with all other design constraints held constant. [ABSTRACT FROM AUTHOR]

Extreme learning machine (ELM) is an emerging learning algorithm for the generalized single hidden layer feedforward neural networks, of which the hidden node parameters are randomly generated and the output weights are analytically computed. However, due to its shallow architecture, feature learning using ELM may not be effective for natural signals (e.g., images/videos), even with a large number of hidden nodes. To address this issue, in this paper, a new ELM-based hierarchical learning framework is proposed for multilayer perceptron. The proposed architecture is divided into two main components: 1) self-taught feature extraction followed by supervised feature classification and 2) they are bridged by random initialized hidden weights. The novelties of this paper are as follows: 1) unsupervised multilayer encoding is conducted for feature extraction, and an ELM-based sparse autoencoder is developed via \ell 1 constraint. By doing so, it achieves more compact and meaningful feature representations than the original ELM; 2) by exploiting the advantages of ELM random feature mapping, the hierarchically encoded outputs are randomly projected before final decision making, which leads to a better generalization with faster learning speed; and 3) unlike the greedy layerwise training of deep learning (DL), the hidden layers of the proposed framework are trained in a forward manner. Once the previous layer is established, the weights of the current layer are fixed without fine-tuning. Therefore, it has much better learning efficiency than the DL. Extensive experiments on various widely used classification data sets show that the proposed algorithm achieves better and faster convergence than the existing state-of-the-art hierarchical learning methods. Furthermore, multiple applications in computer vision further confirm the generality and capability of the proposed learning scheme. [ABSTRACT FROM AUTHOR]

This paper presents a novel design for a fused floating-point four-term dot product unit. The dot product is one of the most frequently used operations for a wide variety of graphics, digital signal processing (DSP) and statistical applications. The proposed design computes the four-term dot product in a single unit to achieve better performance and accuracy compared to a network of traditional floating-point multipliers and adders, which is referred to as a discrete design. In order to further improve the performance of the fused floating-point four-term dot product unit, a new exponent compare and significand alignment scheme, dual-reduction, early normalization, four-input Leading Zero Anticipation (LZA), and compound addition and rounding are applied. The proposed design is implemented for both single and double precision and synthesized with a 45 nm CMOS standard-cell library. In order to evaluate the improvement of the proposed design, the area, latency, and power consumption are investigated. The proposed design reduces the area, power consumption and latency by about 40% compared to the discrete design. [ABSTRACT FROM PUBLISHER]

This paper studies the problem of minimizing a sum of (possible nonsmooth) convex functions that are corresponding to multiple interacting nodes, subject to a convex state constraint set. Time-varying directed network is considered here. Two types of computational constraints are investigated in this paper: one where the information of gradients is not available and the other where the projection steps can only be calculated approximately. We devise a distributed zeroth-order method, the implementation of which needs only functional evaluations and approximate projection. In particular, we show that the proposed method generates expected function value sequences that converge to the optimal value, provided that the projection errors decrease at appropriate rates. [ABSTRACT FROM AUTHOR]

Self-organizing map (SOM)-based motif mining, despite being a promising approach for problem solving, mostly fails to offer a consistent interpretation of clusters with respect to the mixed composition of signal and noise in the nodes. The main reason behind this shortcoming comes from the similarity metrics used in data assignment, specially designed with the biological interpretation for this domain, which are not meant to consider the inevitable noise mixture in the clusters. This limits the explicability of the majority of clusters that are supposedly noise dominated, degrading the overall system clarity in motif discovery. This paper aims to improve the explicability aspect of learning process by introducing a composite similarity function (CSF) that is specially designed for the k -mer-to-cluster similarity measure with respect to the degree of motif properties and embedded noise in the cluster. Our proposed motif finding algorithm in this paper is built on our previous work robust elicitation algorithms for discovering (READ) [1] and termed READ Deoxyribonucleic acid motifs using CSFs (READ _{\rm csf} ), which performs slightly better than READ and shows some remarkable improvements over SOM-based SOMBRERO and SOMEA tools in terms of F$ -measure on the testing data sets. A real data set containing multiple motifs is used to explore the potential of the READ _{\rm csf} for more challenging biological data mining tasks. Visual comparisons with the verified logos extracted from JASPAR database demonstrate that our algorithm is promising to discover multiple motifs simultaneously. [ABSTRACT FROM PUBLISHER]

In a hierarchical Stackelberg v/s Stackelberg game, a collection of players called leaders play a Nash game constrained by the equilibrium conditions of a distinct Nash game played amongst another set of players, called followers. Generically, follower equilibria are non-unique as a function of leader strategies and the decision problems of leaders are highly nonconvex and lacking in standard regularity conditions. Consequently, the provision of sufficient conditions for the existence of global or even local equilibria remains a largely open question. In this paper, we present what is possibly the first general existence result for equilibria for this class of games. Importantly, we impose no single-valuedness assumption on the equilibrium of the follower-level game. Specifically, under the assumption that the objectives of the leaders admit a quasi-potential function, a notion introduced in this paper, the global and local minimizers of a suitably defined optimization problem are shown to be the global and local equilibria of the game. In effect, existence of equilibria can be guaranteed by the solvability of an optimization problem which holds under mild conditions. We motivate quasi-potential games through an application in communication networks. [ABSTRACT FROM PUBLISHER]

Learning optimal spatio-temporal filters is a key to feature extraction for single-trial electroencephalogram (EEG) classification. The challenges are controlling the complexity of the learning algorithm so as to alleviate the curse of dimensionality and attaining computational efficiency to facilitate online applications, e.g., brain–computer interfaces (BCIs). To tackle these barriers, this paper presents a novel algorithm, termed regularized spatio-temporal filtering and classification (RSTFC), for single-trial EEG classification. RSTFC consists of two modules. In the feature extraction module, an l2 -regularized algorithm is developed for supervised spatio-temporal filtering of the EEG signals. Unlike the existing supervised spatio-temporal filter optimization algorithms, the developed algorithm can simultaneously optimize spatial and high-order temporal filters in an eigenvalue decomposition framework and thus be implemented highly efficiently. In the classification module, a convex optimization algorithm for sparse Fisher linear discriminant analysis is proposed for simultaneous feature selection and classification of the typically high-dimensional spatio-temporally filtered signals. The effectiveness of RSTFC is demonstrated by comparing it with several state-of-the-arts methods on three brain-computer interface (BCI) competition data sets collected from 17 subjects. Results indicate that RSTFC yields significantly higher classification accuracies than the competing methods. This paper also discusses the advantage of optimizing channel-specific temporal filters over optimizing a temporal filter common to all channels. [ABSTRACT FROM PUBLISHER]