Search

PAC learning theory is the foundation of computational learning theory. VC-dimension, Rademacher complexity, and empirical risk-minimization principle are three concepts for deriving a generalization error bound for a trained machine. The fundamental theorem of learning theory relates PAC learnability, VC-dimension, and empirical risk-minimization principle. Another basic theorem in computational learning theory is no-free-lunch theorem. These topics are addressed in this chapter. more...

Sparse coding is a matrix factorization technique. It models a target signal as a sparse linear combination of atoms (elementary signals) drawn from a dictionary (a fixed collection). Sparse coding has become a popular paradigm in signal processing, statistics, and machine learning. This chapter introduces compressed sensing, sparse representation/sparse coding, tensor compressed sensing, and sparse PCA. more...

SVM [12, 201] is one of the most popular nonparametric classification algorithms. It is optimal and is based on computational learning theory [200, 202]. The goal of SVM is to minimize the VC dimension by finding the optimal hyperplane between classes, with the maximal margin, where the margin is defined as the distance of the closest point in each class to the separating hyperplane. It has a general-purpose linear learning algorithm and a problem-specific kernel that computes the inner product of input data points in a feature space. The key idea of SVM is to project the training set in a high-dimensional space into a lower-dimensional feature space by means of a set of nonlinear kernel functions, where the projections of the training examples are always linearly separable in the feature space. The hippocampus, a brain region critical for learning and memory processes, has been reported to possess pattern separation function similar to SVM [6]. more...

Hardware and parallel implementations can substantially speed up machine learning algorithms to extend their widespread applications. In this chapter, we first introduce various circuit realizations for popular neural network learning methods. We then introduce their parallel implementations on graphic processing units (GPUs), systolic arrays of processors, and parallel computers. more...

We propose a vehicle logo recognition method that uses SIFT representation and SVM classification. At the training phase, for each training example, a region of interest (ROI) containing the vehicle logo is extracted based on the vehicle plate location, SIFT features are extracted from the ROI, and keywords as well as their counts are obtained by clustering the SIFT features. For all the training examples, their keywords as well as the corresponding counts are used as input and their categories are used as output for training an SVM classifier. At the recognition stage, by a similar procedure of the training stage, for each test example, SIFT features of the ROI are extracted, and keywords as well as their counts are generated by clustering. These keywords as well as their counts are used as input to the SVM classifier and the category of the vehicle logo is obtained. The method is dependent on processing of a ROI rather than on accurate location of the vehicle logo. It uses little prior knowledge, and is easy to use. The method provides a satisfactory recognition rate, and thus is a feasible method for fusion of multiple classifiers. more...

Many information processing problems can be transformed into some form of eigenvalue or singular value problems. Eigenvalue decomposition (EVD) and singular value decomposition (SVD) are usually used for solving these problems. In this paper, we give an introduction to various neural network implementations and algorithms for principal component analysis (PCA) and its various extensions. PCA is a statistical method that is directly related to EVD and SVD. Minor component analysis (MCA) is a variant of PCA, which is useful for solving total least squares (TLSs) problems. The algorithms are typical unsupervised learning methods. Some other neural network models for feature extraction, such as localized methods, complex-domain methods, generalized EVD, and SVD, are also described. Topics associated with PCA, such as independent component analysis (ICA) and linear discriminant analysis (LDA), are mentioned in passing in the conclusion. These methods are useful in adaptive signal processing, blind signal separation (BSS), pattern recognition, and information compression. more...

The radial basis function (RBF) network has its foundation in the conventional approximation theory. It has the capability of universal approximation. The RBF network is a popular alternative to the well-known multilayer perceptron (MLP), since it has a simpler structure and a much faster training process. In this paper, we give a comprehensive survey on the RBF network and its learning. Many aspects associated with the RBF network, such as network structure, universal approimation capability, radial basis functions, RBF network learning, structure optimization, normalized RBF networks, application to dynamic system modeling, and nonlinear complex-valued signal processing, are described. We also compare the features and capability of the two models. more...

Based on the minimum mean squared error (MMSE) between the data stream and the linear combiner output, a new multiuser detection (MUD) algorithm that combines space–time (ST) processing and antenna array on direct-sequence CDMA signals is proposed. The proposed ST-MUD algorithm is proved to be equivalent to two existing MMSE-based ST-MUD algorithms, and the theoretical BER performances for all the three algorithms are the same. The most attractive feature of the new ST-MUD algorithm is based on the fact that the new method does not require explicit estimation of channel and signaling information. This avoids any channel estimation error, and the method is thus more robust and more accurate than the other two ST-MUD algorithms in practical implementation. Adaptation of the proposed ST-MUD algorithm is implemented by using training sequences. Performance of this new multiuser detector is compared with that of two existing MMSE multiuser detectors and the conventional single-user space–time rake receiver through simulations. The proposed ST-MUD algorithm provides a performance better than existing algorithms and is especially suitable for practical CDMA systems. more...

One of the main benefits of the cyclostationary beamforming algorithms is their ability to extract signals from co-channel interference with only a knowledge of the cycle frequency. In this paper, we study the popular cyclostationary beamformers, and propose five new algorithms, namely, the adaptive cyclic adaptive beamforming (ACAB), adaptive cross-SCORE (ACS), constrained least-squares (CLS), adaptive phase-SCORE (APS), and maximal constrained autocorrelation (MCA) algorithms. All these algorithms are adaptive and have a computational complexity of O(n 2) complex multiplications, where n is the number of array elements. A comparative study of these algorithms is made based on numerical simulations. Each of these algorithms has specific application scenarios. The ACS and the APS algorithms are particularly suited for very adverse signal environments. The ACAB, MCA and cyclic adaptive beamforming (CAB, from the work of Wu and Wong) algorithms can provide good performance in the case of medium or weak interference, while the CLS algorithm is especially suitable for weak interference. The CAB algorithm is shown to be a special case of the least-square self-coherent restoral (LS-SCORE) algorithm. Some insights as to how one can assign carrier frequency and symbol rate during digital modulation are also suggested. The proposed adaptive algorithms are easy to implement, and thus are very promising for applications in wireless and mobile communications. more...

Tabu search is a single-solution-based stochastic metaheuristic global optimization method. It is a hill-climbing method that imitates human memory structure to improve decision-making. Scatter search is a population-based metaheuristic algorithm. Scatter search and its generalized form called path relinking are intimately related to tabu search, and they derive additional advantages by using adaptive memory mechanism. more...

Human being is the most intelligent creature on this planet. This chapter introduces various search metaheuristics that are inspired by various behaviors of human creative problem-solving process.

This chapter continues to introduce topics on EAs. Convergence of EAs is first analyzed by using scheme theorem, building-block hypothesis, and then by using finite and infinite population models. Various parallel implementations of EAs are then described in detail. Some other associated topics including coevolution and fitness approximation are finally introduced. more...

Estimation of distribution algorithm (EDA) is a most successful paradigm of EAs. EDAs are derived by inspirations from evolutionary computation and machine learning. This chapter describes EDAs as well as several classical EDA implementations. more...

Ants are capable of finding the shortest path between the food and the colony using a pheromone-laying mechanism. ACO is a metaheuristic optimization approach inspired by this foraging behavior of ants. This chapter is dedicated to ACO. more...

This chapter treats several hard problems associated with metaheuristic optimization, namely, dynamic, multimodal, and constrained optimization problems.

The neural method is a powerful nonlinear adaptive approach in various signal-processing scenarios. It is especially suitable for real-time application and hardware implementation. In this paper, we review its application in antenna array signal processing. This paper also serves as a tutorial to the neural method for antenna array signal processing. more...

We propose an urban traffic state detection method based on support vector machine (SVM) and multilayer perception (MLP). Fusing the SVM and MLP classifiers into a cascade two-tier classifier improves the accuracy of the traffic state classification. A cascade two-tier classifier MLP-SVM, a single SVM classifier and a single MLP classifier are then fused to further improve the final detection accuracy. We also implement a Dempster-Shafer (D-S) theory of evidence based classifier. Finally, fusion strategies at the training and implementation phases are proposed to improve the detection accuracy. more...

Biometrics are the personal or physical characteristics of a person. These biometric identities are usually used for identification or verification. Biometric recognition systems are increasingly being deployed as a more natural, more secure, and more efficient means than the conventional password-based method for the recognition of people. Many biometric verification systems have been developed for global security. more...

The brain is a strongly recurrent structure. This massive recurrence suggests a major role of self-feeding dynamics in the processes of perceiving, acting and learning, and in maintaining the organism alive more...

Imagine that you are attending a cocktail party, the surrounding is full of chatting and noise, and somebody is talking about you. In this case, your ears are particularly sensitive to this speaker. This is the cocktail-party problem, which can be solved by blind source separation (BSS). more...

The human brain stores the information in synapses or in reverberating loops of electrical activity. Most of existing associative memory models store information in synapses.

The kernel method was originally invented in Aizerman et al. (Autom. Remote Control, 25, 821–837, 1964). The key idea is to project the training set in a lower-dimensional space into a high-dimensional kernel (feature) space by means of a set of nonlinear kernel functions. more...

Matrix factorization or factor analysis is an important task that is helpful in the analysis of high-dimensional real-world data. SVD is a classical method for matrix factorization, which gives the optimal low-rank approximation to a real-valued matrix in terms of the squared error. Many application areas, including information retrieval, pattern recognition, and data mining, require processing of binary rather than real data. more...

Learning is a fundamental capability of neural networks. Learning rules are algorithms for finding suitable weights W and/or other network parameters.

Clustering is a fundamental tool for data analysis. It finds wide applications in many engineering and scientific fields including pattern recognition, feature extraction, vector quantization, image segmentation, bioinformatics, and data mining. Clustering is a classical method for the prototype selection of kernel-based neural networks such as the RBF network, and is most useful for neurofuzzy systems. more...

The Bayesian network model was introduced by Pearl in 1985 [147]. It is the best known family of graphical models in artificial intelligence (AI). Bayesian networks are a powerful tool of common knowledge representation and reasoning for partial beliefs under uncertainty. They are probabilistic models that combine probability theory and graph theory. more...

MLPs are feedforward networks with one or more layers of units between the input and output layers. The output units represent a hyperplane in the space of the input patterns. The architecture of MLP is illustrated in Fig. 4.1. more...

Conventional competitive learning-based clustering algorithms like \(C\)-means and LVQ are plagued by a severe initialization problem [57, 106]. If the initial values of the prototypes are not in the convex hull formed by the input data, clustering may not produce meaningful results. more...

Different learning algorithms have different accuracies. The no free lunch theorem asserts that no single learning algorithm always achieves the best performance in any domain. They can be combined to attain higher accuracy. Data fusion is the process of fusing multiple records representing the same real-world object into a single, consistent, and clean representation. Fusion of data for improving prediction accuracy and reliability is an important problem in machine learning. more...

The Hopfield model [27, 28] is the most popular dynamic model. It is biologically plausible since it functions like the human retina [36].

Learning is an approximation problem, which is closely related to the conventional approximation techniques, such as generalized splines and regularization techniques. The RBF network has its origin in performing exact interpolation of a set of data points in a multidimensional space [81]. The RBF network is a universal approximator, and it is a popular alternative to the MLP, since it has a simpler structure and a much faster training process. Both models are widely used for classification and function approximation. more...

Pedestrian detection and tracking are widely applied to intelligent video surveillance, intelligent transportation, automotive autonomous driving or driving-assistance systems. We select OpenCV as the development tool for implementation of pedestrian detection, tracking, counting and risk warning in a video segment. We introduce a low-dimensional soft-output SVM pedestrian classifier to implement precise pedestrian detection. Experiments indicate that the system has high recognition accuracy, and can operate in real time. more...

We propose an improved randomized circle detection method. The improved method reduces the computational complexity of the randomized circle detection method by a factor of two. We then apply the proposed method to detection of circular traffic signs. For traffic images taken in complex scenarios, the colors of interest are first segmented, obtaining potential regions of traffic signs. By applying edge detection and improved randomized circle detection method, traffic signs can be exactly located. Experimental results show that the proposed method has a small computational requirement for natural scenes under different lighting conditions and it can fast and accurately locate circular traffic signs. It can also position circular traffic signs with occlusions and variations in shape, size, and color. more...

Traffic violation detection systems are effective tools to help traffic administration to monitor the traffic condition. It can detect traffic violations, such as running red lights, speeding, and vehicle retrogress in real time. In this paper, we propose an improved background-updating algorithm by using wavelet transform on dynamic background, and then track moving vehicles by feature-based tracking method. A complete traffic violation detection system is realized in C++ with OpenCV. Keywords—traffic violation detection, vehicle tracking, red running. more...

Pedestrian detection in a single image takes more time than that in a video due to the requirement of scanning the whole image. In this paper, an improved pedestrian detection system based on region of interest (ROI) is proposed for single images. In the improved pedestrian detection system, principal component analysis (PCA) is introduced to improve the detection rate and accuracy of histogram of oriented gradients (HOG) based support vector machine (SVM) classifier for pedestrian detection. PCA eliminates the redundant HOG feature dimensions that have no contribution for the pedestrian classification. A novel ROI extraction method based on fuzzy C-means (FCM) clustering algorithm is used to select the regions that possibly contain pedestrians in a single image. ROI extraction reduces the number of detection windows, resulting in a significant reduction in detection time of a single image. Computer experiments show that the proposed pedestrian detection system can correctly detect the positions of pedestrians in single images in real time. more...

In order to locate license plates in a complex environment, we develop a novel license plate location algorithm by combing the texture features of plate and features in the RGB color space. A picture is first filtered with a custom convolutional kernel in order to enhance the texture of characters, followed by morphological processing to get region set A. The RGB features of the license plate are then extracted from the picture, and region set B is obtained by performing morphological processing. Region set C is obtained by performing intersection operation of A and B. Finally, through elimination of pseudo-regions, the real license plate region is located. Evaluation of this algorithm on pictures sampled from different illuminations and road conditions demonstrates an accuracy of 98%. more...

Detecting lines correctly from a digital image is an important step in many real-world applications. It has been widely used in the fields of contour extraction, character recognition and medical image analysis, as well as in many other computer vision based applications. In this paper, we present a randomized Hough transform based line detection algorithm that utilizes the edge gradient direction. This method exploits edge gradient direction to determine the main direction of a line by applying a constraint on the randomized Hough transform. It substantially reduces the count of invalid samples in the random sampling process. The proposed sampling strategy is superior to some existing methods in terms of memory requirement and computation time. more...

The Hop field model is a well-known dynamic associative-memory model. In this paper, we investigate various aspects of the Hop field model for associative memory. We conduct a systematic simulation investigation of several storage algorithms for Hop field networks, and conclude that the perceptron learning based storage algorithms can achieve much better storage capacity than the Hebbian learning based algorithms. more...

Due to feedback connections, recurrent neural networks (RNNs) are dynamic models. RNNs can provide more compact structure for approximating dynamic systems compared to feedforward neural networks (FNNs). For some RNN models such as the Hopfield model and the Boltzmann machine, the fixed-point property of the dynamic systems can be used for optimization and associative memory. The Hopfield model is the most important RNN model, and the Boltzmann machine as well as some other stochastic dynamic models are proposed as its generalization. These models are especially useful for dealing with combinatorial optimization problems (COPs), which are notorious NPcomplete problems. In this paper, we provide a state-of-the-art introduction to these RNN models, their learning algorithms as well as their analog implementations. Associative memory, COPs, simulated annealing (SA), chaotic neural networks and multilevel Hopfield models are also important topics treated in this paper. more...

Introduction Wireless channels suffer from time-varying impairments such as multipath fading, interference, and noise. Diversity, such as time, frequency, space, polarization, or angle diversity, is typically used to mitigate these impairments. Diversity gain is achieved by receiving independent-fading replicas of the signal. The multiple antenna system employs multiple antennas at either the transmitter or the receiver, and it can be either multiple-input single-output (MISO) for beamforming or transmit diversity at the transmitter, single-input multiple-output (SIMO) for diversity combining at the receiver, or MIMO, depending on the numbers of transmit and receive antennas. The MISO, SIMO, and MIMO channel models can be generated by using the angle-delay scattering function. Multiple antenna systems are generally grouped as smart antenna systems and MIMO systems. A smart antenna system is a subsystem that contains multiple antennas; based on the spatial diversity and signal processing, it significantly increases the performance of wireless communication systems. Direction-finding and beamforming are the two most fundamental topics of smart antennas. Direction-finding is used to estimate the number of emitting sources and their DoAs, while beamforming is used to estimate the signal-of-interest (SOI) in the presence of interference. A MIMO system consists of multiple antennas at both the transmitter and the receiver. They are typically used for transmit diversity and spatial multiplexing. Spatial multiplexing can maximize the system capacity by transmitting at each transmit antenna a different bitstream. more...

Introduction A digital image is a rectangular array of picture elements (pixels), arranged in m rows and n columns. The resolution of the image is m × n . Images can be categorized into bi-level, grayscale, and color images. A natural scene, such as a picture taken by a digital camera or obtained by using a scanner, is typically a continuous-tone image, where the colors vary continuously to the eye and there is a lot of noise in the picture. An artificial image, such as a graphical image, does not have the noise or blurring of a natural image. A cartoon-like image consists of uniform color in each area, but adjacent areas have different colors. The features in each type of image can be exploited to achieve a better compression. For example, for the bi-level image, each pixel is represented by one bit. A pixel has a high probability of being the same as its neighboring pixels, and thus RLE is suitable for compressing such image. The image can be scanned column by column or in zigzag. For the grayscale image, each pixel is represented by n bits, and a pixel tends to be similar to its immediate neighbors but may be not identical, thus RLE is not suitable. By representing the image using a Gray code that differs in only one bit for two consecutive integers, a grayscale image can be separated into n bi-level images, and each can be compressed by using RLE. more...