Search

The idea of using second-order methods to train multi-layer perceptrons is far from a new one. The earliest publications on this subject to reach a wide audience were probably those of Parker (1987) and Watrous (1987); now, a decade later, the catalogue of published MLP research concerned with second-order training methods is substantial1. The aim of this chapter is to explain the major ways in which neural net researchers have sought to adapt the various second-order classical techniques of Chapters 2 and 3 to the particular requirements of MLP training.

This chapter is dedicated to presenting a detailed analysis of the performance of first- and second-order training algorithms when applied to a range of benchmark training tasks. The chapter is in three main sections: Section 5.1 gives details of the chosen benchmark tasks; Section 5.2 provides a detailed description of the architectures, training algorithms and parameter settings used in the tests; and Section 5.3 presents an analysis of the test results in terms of training speed and ‘global reliability’. The training data are reproduced in tabulated form in the Appendix.

With the essential background information of Chapter 2 behind us, we are ready to turn our attention to specific second-order optimisation methods. The survey of multivariate second-order methods presented in this chapter is necessarily selective, with the focus on tried and tested methods that have a reputation for both speed and reliability. Two types of multivariate second-order method are considered — general methods (Section 3.3), and nonlinear least-squares methods (Section 3.4). The former are suitable for finding a minimum of any smooth nonlinear function F(x k ); the latter are suitable only when F(x k )is of the special form given by (1.5).

This chapter serves as an introduction to the main subject of this book — multi-layer perceptron (MLP) training1. The multi-layer perceptron is the most widely-used class of neural network. Much of the popularity of MLPs is attributable to the fact that they have been applied successfully to a wide range of information processing tasks, including pattern classification, function learning and time series prediction. Practical applications for MLPs have been found in such diverse fields as speech recognition, image compression, medical diagnosis, autonomous vehicle control and financial prediction; new applications are being discovered all the time. (For a useful survey of practical MLP applications, see Lisboa (1992).)

Conventional classical methods of supervised learning are inevitably faced with the problem of local minima; evidence is presented that fast training algorithms based on conjugate gradient and quasi-Newton techniques are particularly susceptible to being trapped in sub-optimal solutions. A classical technique is presented which by the use of a homotopy on the range of the target outputs allows supervised learning methods to find a global minimum of the error function in almost every case. The method is straightforward to apply and avoids the high computational overheads associated with stochastic techniques such as simulated annealing and genetic algorithms.