Search

Recently, there have been many studies attempting to take advantage of advancements in Artificial Intelligence (AI) in Analog and Mixed-Signal (AMS) circuit design. Automated circuit sizing optimization and improving the accuracy of performance models are the two predominant uses of AI in AMS circuit design. This paper first introduces and explains the basic concepts in AI especially the ones that are more suitable to this application. Next, it surveys some recent studies of various AI techniques for AMS circuit design. Then, it discusses the main approaches as well as the pros and cons of each method. Finally, it gives meaningful insights about the current challenges and open issues, as well as recommends approaches for specific applications. [ABSTRACT FROM AUTHOR]

Training of Deep Neural Network (DNN) at the edge faces the challenge of high energy consumption due to the requirements of a large number of memory accesses for gradient calculations. Therefore, it is necessary to minimize data fetches to perform training of a DNN model on the edge. In this paper, a novel technique has been proposed to reduce the memory access for the training of fully connected layers in transfer learning. By analyzing the memory access patterns in the backpropagation phase in fully connected layers, the memory access can be optimized. We introduce a new method to update the weights by introducing the delta term for every node of output and fully connected layer. Delta term aims to reduce memory access for the parameters which are required to access repeatedly during the training process of fully connected layers. The proposed technique shows 0.13x-13.93x energy savings for the training of fully connected layers for famous DNN architectures on multiple processor architectures. The proposed technique can be used to perform transfer learning on-chip to reduce energy consumption as well as memory access. [ABSTRACT FROM AUTHOR]

Quantization is a promising technique to compress the size of Convolutional Neural Network (CNN) models. Recently, various precision-scalable designs have been presented to reduce the computational complexity in CNNs. However, most of them adopt straightforward calculation scheme to implement the CNN, which causes high bandwidth requirement and low hardware utilization efficiency. This paper proposes a new precision-scalable architecture which can fully reduce the computational complexity in CNN inference and meanwhile has a finely simplified calculation scheme. Based on the proposed scheme, a well-optimized multiplier called Compositional Processing Element (C-PE) is devised. Compared with the previous multipliers, the new C-PE requires less area and power. Furthermore, two levels of optimization are introduced to the design to relieve the bandwidth problem and increase the hardware utilization efficiency. Implemented under the TSMC 90nm CMOS technology, the whole design achieves 6-68.1 fps in various precisions on VGG16 benchmark and a 49.8TOPS/W energy efficiency at 500MHz when scaled to 28nm, which is much better than previous precision-scalable ones. [ABSTRACT FROM AUTHOR]

The computation-intensive circuit simulation makes the analog circuit sizing challenging for large-scale/complicated analog/RF circuits. A Bayesian optimization approach has been proposed recently for the optimization problems involving the evaluations of black-box functions with high computational cost in either objective functions or constraints. In this paper, we propose a weighted expected improvement-based Bayesian optimization approach for automated analog circuit sizing. Gaussian processes (GP) are used as the online surrogate models for circuit performances. Expected improvement is selected as the acquisition function to balance the exploration and exploitation during the optimization procedure. The expected improvement is weighted by the probability of satisfying the constraints. In this paper, we propose a complete Bayesian optimization framework for the optimization of analog circuits with constraints for the first time. The existing GP model-based optimization methods for analog circuits take the GP models as either offline models or as assistance for the evolutionary algorithms. We also extend the Bayesian optimization algorithm to handle multi-objective optimization problems. Compared with the state-of-the-art approaches listed in this paper, the proposed Bayesian optimization method achieves better optimization results with significantly less number of simulations. [ABSTRACT FROM PUBLISHER]

Convolutional neural networks (CNNs) are widely employed in modern computer vision algorithms, where the input image is convolved iteratively by many filters to extract the knowledge behind it. However, while the depth of convolutional layers gets deeper and deeper in recent years, the enormous computational complexity makes it difficult to be deployed on embedded systems with limited hardware resources. In this paper, inspired by rate-distortion optimization in image and video coding, we propose a computation-performance optimization (CPO) method to remove the redundant convolution filters in a CNN with performance constraints. To prove the effectiveness of the proposed method, CPO is applied to the networks for image super-resolution and image classification. Under almost the same PSNR drop and accuracy drop for performance evaluation in these two tasks, we can achieve the best parameter and computation reduction when compared with previous works. [ABSTRACT FROM AUTHOR]

In this paper, we present a framework for analytically estimating the output quality of common digital signal processing (DSP) blocks that utilize approximate adders. The framework is based on considering the error of approximate adders as an additive noise (approximation noise) that disturbs the output of the DSP block in question. A signal processing theoretical modeling approach for describing the power of the approximation noise which is the integral of error spectral density over the bandwidth, is developed. The output qualities of DSP blocks, such as finite impulse response filter, discrete cosine transform, and fast Fourier transform, which utilize approximate adders, are thus estimated. The accuracy of the proposed framework is evaluated by comparing mathematical model predictions to simulation results by using the signal-to-noise ratio (SNR) metric. The inaccuracy of the SNRs predicted by the framework was, on average, less than 2.5dB compared with that obtained from simulations. Therefore, a mathematical optimization approach based on Lagrange Multipliers for optimizing design parameters is also presented. The optimization is realized by choosing a proper configuration of the target block, such as determining the data width of the inexact computation part for each approximate adder in the design. [ABSTRACT FROM AUTHOR]

This paper considers the problem of selecting the top-m designs using simulation with input uncertainty. The performance of each design is measured by its worst case performance. The objective of this paper is to maximize the probability of correctly selecting the top-m designs given a fixed simulation budget. Due to the complexity of probability of correct selection (PCS), we develop a lower bound for the PCS and derive an asymptotically optimal budget allocation rule. Useful insights on characterizing the efficient budget allocation rule with input uncertainty are provided. Meanwhile, a sequential simulation procedure is suggested to implement the allocation rule. A series of numerical experiments indicate that the proposed simulation budget allocation rule can outperform all existing selection rules. [ABSTRACT FROM AUTHOR]

Zero-sum asymmetric information games model decision-making scenarios involving two competing players who have different information about the game being played. A particular case is that of nested information, where one (informed) player has superior information over the other (uninformed) player. This paper considers the case of nested information in repeated zero-sum games and studies the computation of strategies for both the informed and uninformed players for finite-horizon and discounted infinite-horizon nested information games. For finite-horizon settings, we exploit that for both players, the security strategy, and also the opponent's corresponding best response, depend only on the informed player's history of actions. Using this property, we formulate an linear program (LP) computation of player strategies that is linear in the size of the uninformed player's action set. For the infinite-horizon discounted game, we construct LP formulations to compute the approximated security strategies for both players, and show that the worst-case performance difference between the approximated security strategies and the security strategies converges to zero exponentially. Finally, we illustrate the results on a network interdiction game between an informed system administrator and an uniformed intruder. [ABSTRACT FROM AUTHOR]

Recent reports show that projection neural networks with a low-dimensional state space can enhance computation speed obviously. This paper proposes two projection neural networks with reduced model dimension and complexity (RDPNNs) for solving nonlinear programming (NP) problems. Compared with existing projection neural networks for solving NP, the proposed two RDPNNs have a low-dimensional state space and low model complexity. Under the condition that the Hessian matrix of the associated Lagrangian function is positive semi-definite and positive definite at each Karush–Kuhn–Tucker point, the proposed two RDPNNs are proven to be globally stable in the sense of Lyapunov and converge globally to a point satisfying the reduced optimality condition of NP. Therefore, the proposed two RDPNNs are theoretically guaranteed to solve convex NP problems and a class of nonconvex NP problems. Computed results show that the proposed two RDPNNs have a faster computation speed than the existing projection neural networks for solving NP problems. [ABSTRACT FROM AUTHOR]

This paper presents a novel nonlinear dictionary learning (DL) model to address the energy disaggregation (ED) problem, i.e., decomposing the electricity signal of a home to its operating devices. First, ED is modeled as a new temporal DL problem where a set of dictionary atoms is learned to capture the most representative temporal features of electricity signals. The sparse codes corresponding to these atoms show the contribution of each device in the total electricity consumption. To learn powerful atoms, a novel deep temporal DL (DTDL) model is proposed that computes complex nonlinear dictionaries in the latent space of a long short-term memory autoencoder (LSTM-AE). While the LSTM-AE captures the deep temporal manifold of electricity signals, the DTDL model finds the most representative atoms inside this manifold. To simultaneously optimize the dictionary and the deep temporal manifold, a new optimization algorithm is proposed that alternates between finding the optimal LSTM-AE and the optimal dictionary. To the best of authors’ knowledge, DTDL is the only DL model that understands the deep temporal structures of the data. Experiments on the Reference ED Data Set show an outstanding performance compared with the recent state-of-the-art algorithms in terms of precision, recall, accuracy, and F-score. [ABSTRACT FROM AUTHOR]

In this paper, we consider distributed optimization problems over a multiagent network, where each agent can only partially evaluate the objective function, and it is allowed to exchange messages with its immediate neighbors. Differently from all existing works on distributed optimization, our focus is given to optimizing a class of nonconvex problems and under the challenging setting, where each agent can only access the zeroth-order information (i.e., the functional values) of its local functions. For different types of network topologies, such as undirected connected networks or star networks, we develop efficient distributed algorithms and rigorously analyze their convergence and rate of convergence (to the set of stationary solutions). Numerical results are provided to demonstrate the efficiency of the proposed algorithms. [ABSTRACT FROM AUTHOR]

In this paper, we propose a framework of maximizing quadratic submodular energy with a knapsack constraint approximately, to solve certain computer vision problems. The proposed submodular maximization problem can be viewed as a generalization of the classic 0/1 knapsack problem. Importantly, maximization of our knapsack constrained submodular energy function can be solved via dynamic programing. We further introduce a range-reduction step prior to dynamic programing as a two-stage procedure for more efficient maximization. In order to demonstrate the effectiveness of the proposed energy function and its maximization algorithm, we apply it to two representative computer vision tasks: image segmentation and motion trajectory clustering. Experimental results of image segmentation demonstrate that our method outperforms the classic segmentation algorithms of graph cuts and random walks. Moreover, our framework achieves better performance than state-of-the-art methods on the motion trajectory clustering task. [ABSTRACT FROM AUTHOR]

Simulation is often used to estimate the performance of alternative system designs for selecting the best. For a complex system, high-fidelity simulation is usually time-consuming and expensive. In this paper, we provide a new framework that integrates information from the multifidelity models to increase efficiency for selecting the best. A Gaussian mixture model is introduced to capture performance clustering information in the multifidelity models. Posterior information obtained by a clustering analysis incorporates both cluster-wise information and idiosyncratic information for each design. We propose a new budget allocation method to efficiently allocate high-fidelity simulation replications, utilizing posterior information. Numerical experiments show that the proposed multifidelity framework achieves a significant boost in efficiency. [ABSTRACT FROM AUTHOR]

In this paper, we present a framework for risk-sensitive model predictive control (MPC) of linear systems affected by stochastic multiplicative uncertainty. Our key innovation is to consider a time-consistent, dynamic risk evaluation of the cumulative cost as the objective function to be minimized. This framework is axiomatically justified in terms of time-consistency of risk assessments, is amenable to dynamic optimization, and is unifying in the sense that it captures a full range of risk preferences from risk neutral (i.e., expectation) to worst case. Within this framework, we propose and analyze an online risk-sensitive MPC algorithm that is provably stabilizing. Furthermore, by exploiting the dual representation of time-consistent, dynamic risk measures, we cast the computation of the MPC control law as a convex optimization problem amenable to real-time implementation. Simulation results are presented and discussed. [ABSTRACT FROM AUTHOR]

Convolutional neural networks (CNNs) are powerful machine learning models that have become the state of the art in several problems in the areas of computer vision and image processing. Nevertheless, the knowledge of why and how these models present an impressive performance is still limited. There are visualization techniques that can help us to understand the inner working of neural networks. However, they have mostly been applied to classification models. In this paper, we evaluate the application of visualization methods to networks where the input and output are images of proportional dimensions. The results show that visualization brings visual cues associated with how these systems work, helping in their understanding and improvement. We use the knowledge obtained from the visualization of an image restoration CNN to improve the architecture’s efficiency with no significant degradation of its performance. [ABSTRACT FROM AUTHOR]

Conjugate gradient (CG) methods are a class of important methods for solving linear equations and nonlinear optimization problems. In this paper, we propose a new stochastic CG algorithm with variance reduction and we prove its linear convergence with the Fletcher and Reeves method for strongly convex and smooth functions. We experimentally demonstrate that the CG with variance reduction algorithm converges faster than its counterparts for four learning models, which may be convex, nonconvex or nonsmooth. In addition, its area under the curve performance on six large-scale data sets is comparable to that of the LIBLINEAR solver for the $L2$ -regularized $L2$ -loss but with a significant improvement in computational efficiency. 1 CGVR algorithm is available on github: https://github.com/xbjin/cgvr [ABSTRACT FROM AUTHOR]

In this paper, a novel learning method calledpostboosting using extended G-mean(PBG) is proposed for online sequential multiclass imbalance learning (OS-MIL) in neural networks. PBG is effective due to three reasons. 1) Through postadjusting a classification boundary under extended G-mean, the challenging issue of imbalanced class distribution for sequentially arriving multiclass data can be effectively resolved. 2) A newly derived update rule for online sequential learning is proposed, which produces a high G-mean for current model and simultaneously possesses almost the same information of its previous models. 3) Adynamic adjustment mechanismprovided by extended G-mean is valid to deal with the unresolved challenging dense-majority problem and two dynamic changing issues, namely, dynamic changing data scarcity (DCDS) and dynamic changing data diversity (DCDD). Compared to other OS-MIL methods, PBG is highly effective on resolving DCDS, while PBG is the only method to resolve dense-majority and DCDD. Furthermore, PBG can directly and effectively handle unscaled data stream. Experiments have been conducted for PBG and two popular OS-MIL methods for neural networks under massive binary and multiclass data sets. Through the analyses of experimental results, PBG is shown to outperform the other compared methods on all data sets in various aspects including the issues of data scarcity, dense-majority, DCDS, DCDD, and unscaled data. [ABSTRACT FROM AUTHOR]

In this paper, we present an approach for minimizing the computational complexity of the trained convolutional neural networks (ConvNets). The idea is to approximate all elements of a given ConvNet and replace the original convolutional filters and parameters (pooling and bias coefficients; and activation function) with an efficient approximations capable of extreme reductions in computational complexity. Low-complexity convolution filters are obtained through a binary (zero and one) linear programming scheme based on the Frobenius norm over sets of dyadic rationals. The resulting matrices allow for multiplication-free computations requiring only addition and bit-shifting operations. Such low-complexity structures pave the way for low power, efficient hardware designs. We applied our approach on three use cases of different complexities: 1) a “light” but efficient ConvNet for face detection (with around 1000 parameters); 2) another one for hand-written digit classification (with more than 180 000 parameters); and 3) a significantly larger ConvNet: AlexNet with $\approx 1.2$ million matrices. We evaluated the overall performance on the respective tasks for different levels of approximations. In all considered applications, very low-complexity approximations have been derived maintaining an almost equal classification performance. [ABSTRACT FROM AUTHOR]

Building highly nonlinear and nonparametric models is central to several state-of-the-art machine learning systems. Kernel methods form an important class of techniques that induce a reproducing kernel Hilbert space (RKHS) for inferring non-linear models through the construction of similarity functions from data. These methods are particularly preferred in cases where the training data sizes are limited and when prior knowledge of the data similarities is available. Despite their usefulness, they are limited by the computational complexity and their inability to support end-to-end learning with a task-specific objective. On the other hand, deep neural networks have become the de facto solution for end-to-end inference in several learning paradigms. In this paper, we explore the idea of using deep architectures to perform kernel machine optimization, for both computational efficiency and end-to-end inferencing. To this end, we develop the deep kernel machine optimization framework, that creates an ensemble of dense embeddings using Nyström kernel approximations and utilizes deep learning to generate task-specific representations through the fusion of the embeddings. Intuitively, the filters of the network are trained to fuse information from an ensemble of linear subspaces in the RKHS. Furthermore, we introduce the kernel dropout regularization to enable improved training convergence. Finally, we extend this framework to the multiple kernel case, by coupling a global fusion layer with pretrained deep kernel machines for each of the constituent kernels. Using case studies with limited training data, and lack of explicit feature sources, we demonstrate the effectiveness of our framework over conventional model inferencing techniques. [ABSTRACT FROM AUTHOR]

In this paper, we propose a novel adaptive piecewise linear model for dynamic system identification. It has four unique features. First, the model designs a new kind of basis function for function approximation. It maintains the uniform shape for each basis function, so as to achieve a satisfactory tradeoff between generalization ability and model complexity. Second, the model takes the structure of basis functions as decision variables to optimize the formulated identification problems instead of taking expansion coefficients as decision variables as proposed by many existing approaches. Third, we establish an incremental design strategy to solve the system identification problems. In each step of the identification, the selection of optimal basis function is a Lipschitz continuous optimization problem that is likely to be easily handled with some mature toolboxes. This incremental design strategy greatly reduces the estimation cost. Fourth, we introduce a smoothing mechanism to avoid overfitting, when the output of dynamic systems is disturbed by noise. Tests on several benchmark dynamic systems demonstrate the potential of the proposed model. [ABSTRACT FROM AUTHOR]

This paper deals with the performance optimization of resource allocation systems with the aim of maximizing the system's throughput under a given budget for acquiring resources. Resources are assumed to be renewable, i.e., they are not consumed by the operations and become available again after they have been released. The systems under consideration are modeled by a subclass of timed Petri nets called deterministic timed weighted marked graphs. In addition, we take into account infinite server semantics, i.e., the degree of self-concurrency of each transition is infinite. We propose an approach that provides an optimal solution, but has a high computational cost. For this reason, we also present two different approaches that can find suboptimal solutions with a reduced computational cost. The performances of the proposed approaches are compared by means of numerical simulations. [ABSTRACT FROM AUTHOR]

This paper develops and validates a comprehensive and universally applicable computational concept for solving nonlinear differential equations (NDEs) through a neurocomputing concept based on cellular neural networks (CNNs). High-precision, stability, convergence, and lowest-possible memory requirements are ensured by the CNN processor architecture. A significant challenge solved in this paper is that all these cited computing features are ensured in all system-states (regular or chaotic ones) and in all bifurcation conditions that may be experienced by NDEs.One particular quintessence of this paper is to develop and demonstrate a solver concept that shows and ensures that CNN processors (realized either in hardware or in software) are universal solvers of NDE models. The solving logic or algorithm of given NDEs (possible examples are: Duffing, Mathieu, Van der Pol, Jerk, Chua, Rössler, Lorenz, Burgers, and the transport equations) through a CNN processor system is provided by a set of templates that are computed by our comprehensive templates calculation technique that we call nonlinear adaptive optimization. This paper is therefore a significant contribution and represents a cutting-edge real-time computational engineering approach, especially while considering the various scientific and engineering applications of this ultrafast, energy-and-memory-efficient, and high-precise NDE solver concept. For illustration purposes, three NDE models are demonstratively solved, and related CNN templates are derived and used: the periodically excited Duffing equation, the Mathieu equation, and the transport equation. [ABSTRACT FROM PUBLISHER]

In this paper, we present a unified framework for solving a general class of problems arising in the context of set-membership estimation/identification theory. More precisely, the paper aims at providing an original approach for the computation of optimal conditional and robust projection estimates in a nonlinear estimation setting, where the operator relating the data and the parameter to be estimated is assumed to be a generic multivariate polynomial function, and the uncertainties affecting the data are assumed to belong to semialgebraic sets. By noticing that the computation of both the conditional and the robust projection optimal estimators requires the solution to min-max optimization problems that share the same structure, we propose a unified two-stage approach based on semidefinite-relaxation techniques for solving such estimation problems. The key idea of the proposed procedure is to recognize that the optimal functional of the inner optimization problems can be approximated to any desired precision by a multivariate polynomial function by suitably exploiting recently proposed results in the field of parametric optimization. Two simulation examples are reported to show the effectiveness of the proposed approach. [ABSTRACT FROM AUTHOR]

Recently, an in-memory analog circuit based on crosspoint memristor arrays was reported, which enables solving linear regression problems in one step and can be used to train many other machine learning algorithms. To explore its potential for computing accelerator applications, it is of fundamental importance to improve the computing speed of the circuit, i.e., the circuit response towards correct outputs. In this work, we comprehensively studied the transfer function of this circuit, resulting in a quadratic eigenvalue problem that describes the distribution of poles. The minimal real part of non-zero eigenvalues defines the dominant pole, which in turn dominates the response time. Simulations for multiple linear regression solutions with different datasets evidence that, the computing time does not necessarily increase with problem size. The dominant pole is related to parameters in the circuit, including feedback conductance, and gain bandwidth products of operational amplifiers. By optimizing these parameters synergistically, the dominant pole shifts to higher frequencies and the computing speed is consequently optimized. Our results provide a guideline for design and optimization of in-memory machine learning accelerators with analog memristor arrays. Also, issues including power consumption, impact of noise and variation of sources and memristors are investigated to offer a comprehensive evaluation of the circuit performance. [ABSTRACT FROM AUTHOR]

We illustrate novel optimization techniques via simulations for Hopfield networks constructed from manufacturable three-terminal Silicon-Oxide-Nitride-Oxide-Silicon (SONOS) synaptic circuit elements. We first present a computationally-light, memristor-based, highly accurate static compact model for the SONOS synapses used in our simulations. We then show how to exploit analog errors in programming resistances and current leakage, and the continuous tunability of the SONOS synapses to enable transient chaotic group dynamics, to accelerate the convergence of a Hopfield network. We project improvements in energy consumption and time to solution relative to existing CPUs and GPUs by at least 4 orders of magnitude, and also exceed the projected performance of two-terminal memristor-based crossbars in addition to a 100-fold increase in error-resilient array size (i.e. problem size). [ABSTRACT FROM AUTHOR]

In recent years, information relaxation and duality in dynamic programs have been studied extensively, and the resulted primal-dual approach has become a powerful procedure in solving dynamic programs by providing lower-upper bounds on the optimal value function. Theoretically, with the so-called value-based optimal dual penalty, the optimal value function could be recovered exactly via strong duality. However, in practice, obtaining tight dual bounds usually requires good approximations of the optimal dual penalty, which could be time consuming if analytical computation is not possible and nested simulation has to be used to estimate the conditional expectations inside the dual penalty. In this paper, we will develop a framework of a regression approach to approximating the optimal dual penalty in a nonnested manner, by exploring the structure of the function space consisting of all feasible dual penalties. The resulted approximations maintain to be feasible dual penalties, and thus yielding valid dual bounds on the optimal value function. We show that the proposed framework is computationally efficient, and the resulted dual penalties lead to numerically tractable dual problems. Finally, we apply the framework to a high-dimensional dynamic trading problem to demonstrate its effectiveness in solving the dual problems of complex dynamic programs. [ABSTRACT FROM AUTHOR]

In this paper, a neural network model for solving a class of multiextremal smooth nonconvex constrained optimization problems is proposed. Neural network is designed in such a way that its equilibrium points coincide with the local and global optimal solutions of the corresponding optimization problem. Based on the suitable underestimators for the Lagrangian of the problem, one give geometric criteria for an equilibrium point to be a global minimizer of multiextremal constrained optimization problem with or without bounds on the variables. Both necessary and sufficient global optimality conditions for a class of multiextremal constrained optimization problems are presented to determine a global optimal solution. By study of the resulting dynamic system, it is shown that under given assumptions, steady states of the dynamic system are stable and trajectories of the proposed model converge to the local and global optimal solutions of the problem. Numerical results are given and related graphs are depicted to illustrate the global convergence and performance of the solver for multiextremal constrained optimization problems. [ABSTRACT FROM AUTHOR]

Approximate linear programming (ALP) and its variants have been widely applied to Markov decision processes (MDPs) with a large number of states. A serious limitation of ALP is that it has an intractable number of constraints, as a result of which constraint approximations are of interest. In this paper, we define a linearly relaxed approximation linear program (LRALP) that has a tractable number of constraints, obtained as positive linear combinations of the original constraints of the ALP. The main contribution is a novel performance bound for LRALP. [ABSTRACT FROM PUBLISHER]

The semisupervised least squares support vector machine (LS-S3VM) is an important enhancement of least squares support vector machines in semisupervised learning. Given that most data collected from the real world are without labels, semisupervised approaches are more applicable than standard supervised approaches. Although a few training methods for LS-S3VM exist, the problem of deriving the optimal decision hyperplane efficiently and effectually has not been solved. In this paper, a fully weighted model of LS-S3VM is proposed, and a simple integer programming (IP) model is introduced through an equivalent transformation to solve the model. Based on the distances between the unlabeled data and the decision hyperplane, a new indicator is designed to represent the possibility that the label of an unlabeled datum should be reversed in each iteration during training. Using the indicator, we construct an extended candidate set consisting of the indices of unlabeled data with high possibilities, which integrates more information from unlabeled data. Our algorithm is degenerated into a special scenario of the previous algorithm when the extended candidate set is reduced into a set with only one element. Two strategies are utilized to determine the descent directions based on the extended candidate set. Furthermore, we developed a novel method for locating a good starting point based on the properties of the equivalent IP model. Combined with the extended candidate set and the carefully computed starting point, a fast algorithm to solve LS-S3VM quasi-optimally is proposed. The choice of quasi-optimal solutions results in low computational cost and avoidance of overfitting. Experiments show that our algorithm equipped with the two designed strategies is more effective than other algorithms in at least one of the following three aspects: 1) computational complexity; 2) generalization ability; and 3) flexibility. However, our algorithm and other algorithms have similar levels of performance in the remaining aspects. [ABSTRACT FROM AUTHOR]

Feature selection is one of the most important dimension reduction techniques for its efficiency and interpretation. Since practical data in large scale are usually collected without labels, and labeling these data are dramatically expensive and time-consuming, unsupervised feature selection has become a ubiquitous and challenging problem. Without label information, the fundamental problem of unsupervised feature selection lies in how to characterize the geometry structure of original feature space and produce a faithful feature subset, which preserves the intrinsic structure accurately. In this paper, we characterize the intrinsic local structure by an adaptive reconstruction graph and simultaneously consider its multiconnected-components (multicluster) structure by imposing a rank constraint on the corresponding Laplacian matrix. To achieve a desirable feature subset, we learn the optimal reconstruction graph and selective matrix simultaneously, instead of using a predetermined graph. We exploit an efficient alternative optimization algorithm to solve the proposed challenging problem, together with the theoretical analyses on its convergence and computational complexity. Finally, extensive experiments on clustering task are conducted over several benchmark data sets to verify the effectiveness and superiority of the proposed unsupervised feature selection algorithm. [ABSTRACT FROM AUTHOR]

Identifying parsimonious models is generically a “hard” nonconvex problem. Available approaches typically rely on relaxations such as Group Lasso or nuclear norm minimization. Moreover, incorporating stability and model order constraints into the formalism in such methods entails a substantial increase in computational complexity. Motivated by these challenges, in this paper we present algorithms for parsimonious linear time invariant system identification aimed at identifying low-complexity models which i) incorporate a priori knowledge on the system (e.g., stability), ii) allow for data with missing/nonuniform measurements, and iii) are able to use data obtained from several runs of the system with different unknown initial conditions. The randomized algorithms proposed are based on the concept of atomic norm and provide a numerically efficient way to identify sparse models from large amounts of noisy data. [ABSTRACT FROM PUBLISHER]

This paper presents a novel neural network for solving generally constrained variational inequality problems by constructing a system of double projection equations. By defining proper convex energy functions, the proposed neural network is proved to be stable in the sense of Lyapunov and converges to an exact solution of the original problem for any starting point under the weaker cocoercivity condition or the monotonicity condition of the gradient mapping on the linear equation set. Furthermore, two sufficient conditions are provided to ensure the stability of the proposed neural network for a special case. The proposed model overcomes some shortcomings of existing continuous-time neural networks for constrained variational inequality, and its stability only requires some monotonicity conditions of the underlying mapping and the concavity of nonlinear inequality constraints on the equation set. The validity and transient behavior of the proposed neural network are demonstrated by some simulation results. [ABSTRACT FROM PUBLISHER]

Linear discriminant analysis (LDA) as a well-known supervised dimensionality reduction method has been widely applied in many fields. However, the lack of sparsity in the LDA solution makes interpretation of the results challenging. In this paper, we propose a new model for sparse uncorrelated LDA (ULDA). Our model is based on the characterization of all solutions of the generalized ULDA. We incorporate sparsity into the ULDA transformation by seeking the solution with minimum \ell 1 -norm from all minimum dimension solutions of the generalized ULDA. The problem is then formulated as an \ell 1 -minimization problem with orthogonality constraint. To solve this problem, we devise two algorithms: 1) by applying the linearized alternating direction method of multipliers and 2) by applying the accelerated linearized Bregman method. Simulation studies and high-dimensional real data examples demonstrate that our algorithms not only compute extremely sparse solutions but also perform well in classification. [ABSTRACT FROM AUTHOR]

Saliency detection is used to identify the most important and informative area in a scene, and it is widely used in various vision tasks, including image quality assessment, image matching, and object recognition. Manifold ranking (MR) has been used to great effect for the saliency detection, since it not only incorporates the local spatial information but also utilizes the labeling information from background queries. However, MR completely ignores the feature information extracted from each superpixel. In this paper, we propose an MR-based matrix factorization (MRMF) method to overcome this limitation. MRMF models the ranking problem in the matrix factorization framework and embeds query sample labels in the coefficients. By incorporating spatial information and embedding labels, MRMF enforces similar saliency values on neighboring superpixels and ranks superpixels according to the learned coefficients. We prove that the MRMF has good generalizability, and develops an efficient optimization algorithm based on the Nesterov method. Experiments using popular benchmark data sets illustrate the promise of MRMF compared with the other state-of-the-art saliency detection methods. [ABSTRACT FROM PUBLISHER]

Nonnegative matrix factorization (NMF)-based models possess fine representativeness of a target matrix, which is critically important in collaborative filtering (CF)-based recommender systems. However, current NMF-based CF recommenders suffer from the problem of high computational and storage complexity, as well as slow convergence rate, which prevents them from industrial usage in context of big data. To address these issues, this paper proposes an alternating direction method (ADM)-based nonnegative latent factor (ANLF) model. The main idea is to implement the ADM-based optimization with regard to each single feature, to obtain high convergence rate as well as low complexity. Both computational and storage costs of ANLF are linear with the size of given data in the target matrix, which ensures high efficiency when dealing with extremely sparse matrices usually seen in CF problems. As demonstrated by the experiments on large, real data sets, ANLF also ensures fast convergence and high prediction accuracy, as well as the maintenance of nonnegativity constraints. Moreover, it is simple and easy to implement for real applications of learning systems. [ABSTRACT FROM PUBLISHER]

Selecting the optimal subset is highly beneficial to numerous developments in simulation optimization. This paper studies the problem of maximizing the probability of correctly selecting the top- $m$ designs out of $k$ designs under a computing budget constraint. We develop a new procedure which is more efficient and robust than currently existing procedures in the literature. We also provide an analysis on its asymptotic convergence rate. Based on this analysis, we show that our new procedure achieves a higher convergence rate than other procedures under certain conditions. Numerical testing supports our analytical analysis and shows that the new procedure is significantly more efficient and robust. [ABSTRACT FROM AUTHOR]

The one-class classification problem is a well-known research endeavor in pattern recognition. The problem is also known under different names, such as outlier and novelty/anomaly detection. The core of the problem consists in modeling and recognizing patterns belonging only to a so-called target class. All other patterns are termed nontarget, and therefore, they should be recognized as such. In this paper, we propose a novel one-class classification system that is based on an interplay of different techniques. Primarily, we follow a dissimilarity representation-based approach; we embed the input data into the dissimilarity space (DS) by means of an appropriate parametric dissimilarity measure. This step allows us to process virtually any type of data. The dissimilarity vectors are then represented by weighted Euclidean graphs, which we use to determine the entropy of the data distribution in the DS and at the same time to derive effective decision regions that are modeled as clusters of vertices. Since the dissimilarity measure for the input data is parametric, we optimize its parameters by means of a global optimization scheme, which considers both mesoscopic and structural characteristics of the data represented through the graphs. The proposed one-class classifier is designed to provide both hard (Boolean) and soft decisions about the recognition of test patterns, allowing an accurate description of the classification process. We evaluate the performance of the system on different benchmarking data sets, containing either feature-based or structured patterns. Experimental results demonstrate the effectiveness of the proposed technique. [ABSTRACT FROM PUBLISHER]

In this paper, an improved method to construct and estimate Volterra models using Generalized Orthonormal Basis Functions (GOBF) is presented. The proposed method extends results obtained in previous works, where an exact technique for optimizing the GOBF parameters (poles) for symmetric Volterra models of any order was presented. The proposed extensions take place in two different ways: (i) the new formulation is derived in such a way that each multidimensional kernel of the model is decomposed into a set of independent orthonormal bases (rather than a single, common basis), each of which is parameterized by an individual set of poles responsible for representing the dominant dynamic of the kernel along a particular dimension; and (ii) the new formulation is based on a ladder-structured GOBF architecture that is characterized by having only real-valued parameters to be estimated, regardless of whether the GOBF poles encoded by these parameters are real- or complex-valued. The exact gradients of an error functional with respect to the parameters to be optimized are computed analytically and provide exact search directions for an optimization process that uses only input-output data measured from the dynamic system to be modeled. Computational experiments are presented to illustrate the benefits of the proposed approach when modeling nonlinear systems. [ABSTRACT FROM AUTHOR]

In this paper, we propose new methods for the choice and adaptation of the smoothing parameter of the probabilistic neural network (PNN). These methods are based on three reinforcement learning algorithms: $Q(0)$ -learning, $Q(\lambda )$ -learning, and stateless $Q$ -learning. We regard three types of PNN classifiers: the model that uses single smoothing parameter for the whole network, the model that utilizes single smoothing parameter for each data attribute, and the model that possesses the matrix of smoothing parameters different for each data variable and data class. Reinforcement learning is applied as the method of finding such a value of the smoothing parameter, which ensures the maximization of the prediction ability. PNN models with smoothing parameters computed according to the proposed algorithms are tested on eight databases by calculating the test error with the use of the cross validation procedure. The results are compared with state-of-the-art methods for PNN training published in the literature up to date and, additionally, with PNN whose sigma is determined by means of the conjugate gradient approach. The results demonstrate that the proposed approaches can be used as alternative PNN training procedures. [ABSTRACT FROM AUTHOR]

Stochastic Trust-Region Response-Surface method (STRONG) is a new response-surface-based framework for simulation optimization. The appeal of STRONG lies in that it preserves the advantages, yet eliminates the disadvantages, of traditional response surface methodology (RSM) that has been used for more than 50 years. Specifically, STRONG does not require human involvement in the search process and can guarantee to converge to the true optimum with probability one (w.p.1). In this paper, we propose an improved framework, called STRONG-X, that enhances the efficiency and efficacy of STRONG to widen its applicability to more practical problems. For efficiency improvement, STRONG-X includes a newly-developed experimental scheme that consists of construction of optimal simulation designs and an assignment strategy for random number streams to obtain computational gains. For efficacy improvement, a new variant, called STRONG-XG, is developed to achieve convergence under generally-distributed responses, as opposed to STRONG and STRONG-X where convergence is guaranteed only when the response is normal. An extensive numerical study is conducted to evaluate the efficiency and efficacy of STRONG-X and STRONG-XG. Moreover, two illustrative examples are provided to show the viability of STRONG-X and STRONG-XG in practical settings. [ABSTRACT FROM AUTHOR]

This paper is concerned with a multi-period portfolio management problem over a finite horizon. The objective is to seek the optimal investment policy series which maximizes the weighted sum of a linear combination of the expected return and the variance of portfolio over all the investment periods. This formulation enables the investor to adjust weights for any period and have full freedom and control over their best tradeoff between return and risk over each period. We show that such a problem is a convex quadratic programming problem in terms of the decision variables, regardless of price dynamic nature (either linear or nonlinear cases). By solving the stationary equation directly, an optimal solution is developed for its original problem without using embedding method. The solution is simplified for a general linear price model with high-order and coupled asset dynamics and shown to be implementable with historical price data. Simulation is carried out on USA and China stock markets with real data, which demonstrates feasibility and better performance of the proposed solution than the special case considered in the literature. In particular, the proposed solution with suitable nonzero weights on intermediate time periods offers higher return at the same risk level, compared with one involving the terminal wealth only in the objective function. [ABSTRACT FROM AUTHOR]

Several convex relaxations of the optimal power flow (OPF) problem have recently been developed using both bus injection models and branch flow models. In this paper, we prove relations among three convex relaxations: a semidefinite relaxation that computes a full matrix, a chordal relaxation based on a chordal extension of the network graph, and a second-order cone relaxation that computes the smallest partial matrix. We prove a bijection between the feasible sets of the OPF in the bus injection model and the branch flow model, establishing the equivalence of these two models and their second-order cone relaxations. Our results imply that, for radial networks, all these relaxations are equivalent and one should always solve the second-order cone relaxation. For mesh networks, the semidefinite relaxation and the chordal relaxation are equally tight and both are strictly tighter than the second-order cone relaxation. Therefore, for mesh networks, one should either solve the chordal relaxation or the SOCP relaxation, trading off tightness and the required computational effort. Simulations are used to illustrate these results. [ABSTRACT FROM PUBLISHER]

Regression plays a major role in many scientific and engineering problems. The goal of regression is to learn the unknown underlying function from a set of sample vectors with known outcomes. In recent years, kernel methods in regression have facilitated the estimation of nonlinear functions. However, two major (interconnected) problems remain open. The first problem is given by the bias-versus-variance tradeoff. If the model used to estimate the underlying function is too flexible (i.e., high model complexity), the variance will be very large. If the model is fixed (i.e., low complexity), the bias will be large. The second problem is to define an approach for selecting the appropriate parameters of the kernel function. To address these two problems, this paper derives a new smoothing kernel criterion, which measures the roughness of the estimated function as a measure of model complexity. Then, we use multiobjective optimization to derive a criterion for selecting the parameters of that kernel. The goal of this criterion is to find a tradeoff between the bias and the variance of the learned function. That is, the goal is to increase the model fit while keeping the model complexity in check. We provide extensive experimental evaluations using a variety of problems in machine learning, pattern recognition, and computer vision. The results demonstrate that the proposed approach yields smaller estimation errors as compared with methods in the state of the art. [ABSTRACT FROM AUTHOR]

In the design of sigma-delta modulators ($\Sigma \Delta $ Ms), different variables need to be optimized together in order to maximize the performance. This design task has the added difficulty of dealing with the non-linear behavior of the quantizer. Although a linearized model of the quantizer can be used, this may result in significant discrepancies between the predicted and actual behavior of the $\Sigma \Delta \text{M}$. To better predict the behavior of a given design, we propose a design methodology for $\Sigma \Delta $ Ms based on a genetic algorithm (GA) that uses both linear equations and simulations. In order to reduce the computation time, the design solution is initially evaluated using equations and only if the performance is deemed good enough, it is subjected to a more refined simulation. This more precise simulation takes into account thermal noise, finite output swing, and gain (among other non-idealities) of the building blocks of the modulator. Moreover, Monte Carlo (MC) analyses are performed during the optimization in order to assess the sensitivity to component variations of the solutions. In order to demonstrate the validity and robustness of the proposed optimization methodology, several $\Sigma \Delta $ Ms designs are presented, together with the corresponding measured results. [ABSTRACT FROM AUTHOR]