1. Design of deep echo state networks.
- Author
-
Gallicchio, Claudio, Micheli, Alessio, and Pedrelli, Luca
- Subjects
- *
RECURRENT neural networks , *ARTIFICIAL neural networks , *BIOLOGICAL neural networks , *RESERVOIR ecology , *SIGNAL frequency estimation - Abstract
Abstract In this paper, we provide a novel approach to the architectural design of deep Recurrent Neural Networks using signal frequency analysis. In particular, focusing on the Reservoir Computing framework and inspired by the principles related to the inherent effect of layering, we address a fundamental open issue in deep learning, namely the question of how to establish the number of layers in recurrent architectures in the form of deep echo state networks (DeepESNs). The proposed method is first analyzed and refined on a controlled scenario and then it is experimentally assessed on challenging real-world tasks. The achieved results also show the ability of properly designed DeepESNs to outperform RC approaches on a speech recognition task, and to compete with the state-of-the-art in time-series prediction on polyphonic music tasks. Highlights • We address the issue of architectural design for Deep Recurrent Neural Networks. • We focus on the frequency analysis for layering design. • We propose an efficient approach to choose the number of recurrent layers in DeepESN. • We show the empirical advantage of very Deep (> 30 recurrent layers) RNNs. [ABSTRACT FROM AUTHOR]
- Published
- 2018
- Full Text
- View/download PDF