Search

Given several different new product development projects and limited resources, this paper is concerned with the optimal allocation of resources among the projects. This is clearly a multi-objective optimization problem (MOOP), because each new product development project has both a profit expectation and a loss expectation, and such expectations vary according to allocated resources. In such a case, the goal of multi-objective new product development (MONPD) is to maximize the profit expectation while minimizing the loss expectation. As is well known, Pareto optimality and the Pareto front are extremely important to resolve MOOPs. Unlike many other MOOP methods which provide only a single Pareto optimal solution or an approximation of the Pareto front, this paper reports a novel method to calculate the complete Pareto front for the MONPD. Some theoretical conditions and a ripple-spreading algorithm together play a crucial role in finding the complete Pareto front for the MONPD. Simulation results illustrate that the reported method, by calculating the complete Pareto front, can provide the best support to decision makers in the MONPD. [ABSTRACT FROM AUTHOR]

Abstract: Extreme learning machine (ELM) is one of the most popular and important learning algorithms. It comes from single-hidden-layer feedforward neural networks. It has been proved that ELM can achieve better performance than support vector machine (SVM) in regression and classification. In this paper, mathematically, with regression problem, the step 3 of ELM is studied. First of all, the equation are reformulated as an optimal model. With the optimality, the necessary conditions of optimal solution are presented. The equation is replaced by . We can prove that the latter must have one solution at least. Second, optimal approximation solution is discussed in cases of is column full rank, row full rank, neither column nor row full rank. In the last case, the rank-1 and rank-2 methods are used to get optimal approximation solution. In theory, this paper present a better algorithm for ELM. [Copyright &y& Elsevier]

Recently, the application of rank minimization to image denoising has shown remarkable denoising results. However, due to iterative nature of low rank optimization, the estimation of noise for next iteration is an essential requirement after each iteration. In the existing approaches, the noise is estimated at the level of patches using l 2 norm based comparison of the observed image and its denoised version. Because of l 2 norm comparison, these patch-wise noise estimates may be affected in the presence of moderate and severe levels of noise. Also, the patch-wise noise estimates are inferred without considering the geometric details of a given image. Moreover, the number of iterations is pre-specified, heuristically, depending on the known initial noise in the image which limits the applicability of the existing algorithms for practical situation where the noise level is not a priori. In this paper, we attempt to address these limitations by introducing the notion of residual noise for iterative rank minimization. The estimation of residual noise also involves image geometry and statistical characteristics along with patch-wise comparison. In addition to more reliable noise estimation, the residual noise can also be employed as an effective stopping criterion for iterative denoising process. Furthermore, the residual noise estimation is a key factor for computation of thresholding weights, which in turn, yield a more reliable truncated singular value decomposition (SVD) for iterative rank minimization algorithms. Based on more precise truncated SVD, we propose to preserve geometric features like edges and texture in the presence of severe noise. Experimental results show that the proposed method provides better or equivalent results as compared to various state-of-the-art denoising algorithms. [ABSTRACT FROM AUTHOR]

For the purpose of enhancing the discriminability of convolutional neural networks (CNNs) and facilitating the optimization, we investigate the activation function for a neural network and the corresponding initialization method in this paper. Firstly, a trainable activation function with a multi-layer structure (named “Multi-layer Maxout Network”, MMN) is proposed. MMN is a multi-layer structured maxout, inheriting advantages of both a non-saturated activation function and a trainable activation function approximator. Secondly, we derive a robust initialization method specifically for the MMN activation with a theoretical proof, which works for the maxout activation as well. Our novel initialization method could reduce internal covariate shift when signals are propagated through layers and solve the so called “exploding/vanishing gradient” problem, which leads a more efficient training procedure of deep neural networks. Experimental results show that our proposed model yields better performance on three image classification benchmark datasets (CIFAR-10, CIFAR-100 and ImageNet) than quite a few state-of-the-art methods and our novel initialization method improves performance further. Furthermore, the influence of MMN in different hidden layers is analyzed, and a trade-off scheme between the accuracy and computing resources is given. [ABSTRACT FROM AUTHOR]

In this paper, a hybrid recursive least squares (HRLS) algorithm for online identification using sequential chunk-by-chunk observations is proposed. By employing the optimization-based least squares (O-LS), the HRLS can be initialized with any chunk of data samples and works successively in two recursive procedures for updating the inverse matrix with minimal dimension and least rank-deficiency, and thereby contributing to fast and stable online identification. Since norms of the output weight and training errors are minimized simultaneously, the HRLS achieves high accuracy in terms of both generalization and approximation. Simulation studies and comprehensive comparisons demonstrate that the HRLS is numerically more stable and superior to other algorithms in terms of accuracy and speed. [ABSTRACT FROM AUTHOR]

Approximate nearest neighbors (ANN) search for large scale data has attracted considerable attention due to the fact that large amounts of data are easily available. Recently, hashing has been widely adopted for similarity search because of its good potential for low storage cost and fast query speed. Among of them, when semantic similarity information is available, supervised hashing methods show better performance than unsupervised ones. However, supervised hashing methods need explicit similarity information which is not available in some scenarios. In addition, they have the problems of difficult optimization and time consuming for training, which make them unpracticable to large scale data. In this paper, we propose an unsupervised hashing method – Unsupervised Euclidean Hashing (USEH), which learns and generates hashing codes to preserve the Euclidean distance relationship between data. Specifically, USEH first utilizes Locality-Sensitive Hashing (LSH) to generate pseudo labels; then, it adopts a sequential learning strategy to learn the hash functions, one bit at a time, which can generate very discriminative codes. Moreover, USEH avoids explicitly computing the similarity matrix by decomposing it into the product of a label matrix and its transposition, which makes the training complexity of USEH linear to the size of training samples when the number of training samples is much greater than the dimension of feature. Thus, it can efficiently work on large scale data. We test USEH on two large scale datasets – SIFT1M and GIST1M. Experimental results show that USEH is comparable to state-of-the-art unsupervised hashing methods. [ABSTRACT FROM AUTHOR]

Max-min distance analysis (MMDA) for dimensionality reduction has been presented to guarantee class separation. However, class centers may be nonuniformly distributed and thus optimal classification accuracy may not be obtained. In this paper, we first give a novel method based on MMDA, called fractional-step max-min distance analysis (FMMDA), which relaxes max-min pairwise distances in fractional steps. The method can make a relatively uniform distribution of class centers and approximately maintain class separation. Then we present a more efficient method, called regularized max-min distance analysis (RMMDA), which achieves the same effect as FMMDA by integrating the Fisher criterion into MMDA. Moreover, we present the speedup and kernel versions of the methods to accelerate an optimization procedure and deal with the data distribution problem, respectively. Finally, we analyze the computational complexities of our methods. Empirical studies demonstrate that our methods can outperform or be comparable to some state-of-the-art discriminant analysis methods in terms of classification accuracy. [ABSTRACT FROM AUTHOR]

Abstract: This paper presents a Functional Weights Wavelet Neural Network-based state-dependent AR (FWWNN-AR) model with the main objective to address the modeling and prediction problem of nonlinear time series. The FWWNN-AR model is a state-dependent autoregressive (SD-AR) model, which has its coefficients approximated by a set of Functional Weights Wavelet Neural Network (FWWNN). The FWWNN is an enhanced type of wavelet neural network comprising of five layers: input, wavelet, product, output and functional weight layer that computes the weights as function of inputs thus making the weights to vary with the inputs and to share the dynamics with the wavelet compartment. The FWWNN-AR model possesses both the advantages of the state-dependent AR model in the description of nonlinear dynamics using few nodes and of the FWWNN in functional approximation considering mutually the time and frequency spaces. It learns the nonlinear dynamics from three distinct levels: AR level, Wavelet compartment level and functional weights level. A Structured Nonlinear Parameter Optimization Method (SNPOM) is applied to estimate the FWWNN-AR model parameters. This learning approach divides the parameter search space into linear and nonlinear subspaces and centers the search in the nonlinear subspace, but at each iteration in the optimization process, a search in the nonlinear (or linear) subspace is executed on the basis of the estimated values just obtained in linear (or nonlinear) subspace. The search in the nonlinear subspace uses a method similar to the Levemberg–Marquardt Method (LMM), and the search in the linear subspace uses the Least Square Method (LSM). The proposed model is validated by comparing its performances and effectiveness with those achieved by some well known models on both generated and real nonlinear time series. [Copyright &y& Elsevier]

Abstract: Recently, graph-based spectral clustering algorithms have been developing rapidly, which are proposed as discrete combinatorial optimization problems and approximately solved by relaxing them into tractable eigenvalue decomposition problems. In this paper, we first review the current existing spectral clustering algorithms in a unified-framework way and give a straightforward explanation about spectral clustering. We also present a novel model for generalizing the unsupervised spectral clustering to semi-supervised spectral clustering. Under this model, prior information given by some instance-level constraints can be generalized to space-level constraints. We find that (undirected) graph built on the enlarged prior information is more meaningful, hence the boundaries of the clusters are more correct. Experimental results based on toy data, real-world data and image segmentation demonstrate the advantages of the proposed model. [Copyright &y& Elsevier]

Abstract: Joint approximate diagonalization (JAD) is one of well-known methods for solving blind source separation. JAD diagonalizes many cumulant matrices of given observed signals as accurately as possible, where the optimization for each pair of signals is repeated until the convergence. In each pair optimization, JAD should decide whether the pair is actually optimized by a convergence decision condition, where a fixed threshold has been employed in many cases. Though a sufficiently small threshold is desirable for the accuracy of results, the speed of convergence is quite slow if the threshold is too small. In this paper, we propose a new decision condition with an adaptive threshold for JAD under a probabilistic framework. First, it is assumed that the errors in JAD (non-diagonal elements in cumulant matrices) are given by the exponential distribution. Next, it is shown that the maximum likelihood estimation of the probabilistic model is equivalent to JAD. Then, an adaptive threshold is theoretically derived by utilizing the model selection theory. Numerical experiments verify the efficiency of the proposed method for blind source separation of artificial sources and natural images. It is also shown that the proposed method is especially effective when the number of samples is limited. [Copyright &y& Elsevier]

Abstract: Twin support vector regression (TSVR) obtains faster learning speed by solving a pair of smaller sized support vector machine (SVM)-typed problems than classical support vector regression (SVR). In this paper, a primal version for TSVR, termed primal TSVR (PTSVR), is first presented. By introducing a quadratic function to approximate its loss function, PTSVR directly optimizes the pair of quadratic programming problems (QPPs) of TSVR in the primal space based on a series of sets of linear equations. PTSVR can obviously improve the learning speed of TSVR without loss of the generalization. To improve the prediction speed, a greedy-based sparse TSVR (STSVR) in the primal space is further suggested. STSVR uses a simple back-fitting strategy to iteratively select its basis functions and update the augmented vectors. Computational results on several synthetic as well as benchmark datasets confirm the merits of PTSVR and STSVR. [ABSTRACT FROM AUTHOR]

Abstract: Geometric methods provide an intuitive and theoretically solid viewpoint for the solution of many optimization problems in the fields of pattern recognition and machine learning. The support vector machine (SVM) classification is a typical optimization task that has achieved excellent generalization performance in a wide variety of applications. In this paper, the notion of “scaled convex hull” (SCH) is presented, through which the nonseparable SVM classifications can be approximately transformed to separable ones: by a suitable selection of the reduction factor, the initially overlapping SCHs (each is generated by the training patterns of each class) can be reduced to become separable, then the maximal margin classifier between them can be trained, which is an approximation of the standard nonseparable SVM. As a practical application of the SCH framework, the popular Gilbert''s algorithm has been generalized to approximately solve general (linear and nonlinear, separable and nonseparable) SVM classification problems both accurately and efficiently. The experiments show that the proposed method may achieve better performance than the state-of-the-art methods, an improved sequential minimal optimization and Gilbert''s algorithm based on the reduced convex hull (RCH), in terms of the number of kernel evaluations and the execution time. [Copyright &y& Elsevier]

Abstract: The design of radial basis function neural networks (RBFNNs) still remains as a difficult task when they are applied to classification or to regression problems. The difficulty arises when the parameters that define an RBFNN have to be set, these are: the number of RBFs, the position of their centers and the length of their radii. Another issue that has to be faced when applying these models to real world applications is to select the variables that the RBFNN will use as inputs. The literature presents several methodologies to perform these two tasks separately, however, due to the intrinsic parallelism of the genetic algorithms, a parallel implementation will allow the algorithm proposed in this paper to evolve solutions for both problems at the same time. The parallelization of the algorithm not only consists in the evolution of the two problems but in the specialization of the crossover and mutation operators in order to evolve the different elements to be optimized when designing RBFNNs. The subjacent genetic algorithm is the non-sorting dominated genetic algorithm II (NSGA-II) that helps to keep a balance between the size of the network and its approximation accuracy in order to avoid overfitted networks. Another of the novelties of the proposed algorithm is the incorporation of local search algorithms in three stages of the algorithm: initialization of the population, evolution of the individuals and final optimization of the Pareto front. The initialization of the individuals is performed hybridizing clustering techniques with the mutual information (MI) theory to select the input variables. As the experiments will show, the synergy of the different paradigms and techniques combined by the presented algorithm allow to obtain very accurate models using the most significant input variables. [Copyright &y& Elsevier]