Back to Search
Start Over
Automatic, dynamic, and nearly optimal learning rate specification via local quadratic approximation
- Source :
- Neural networks : the official journal of the International Neural Network Society. 141
- Publication Year :
- 2020
-
Abstract
- In deep learning tasks, the update step size determined by the learning rate at each iteration plays a critical role in gradient-based optimization. However, determining the appropriate learning rate in practice typically relies on subjective judgment. In this work, we propose a novel optimization method based on local quadratic approximation (LQA). In each update step, we locally approximate the loss function along the gradient direction by using a standard quadratic function of the learning rate. Subsequently, we propose an approximation step to obtain a nearly optimal learning rate in a computationally efficient manner. The proposed LQA method has three important features. First, the learning rate is automatically determined in each update step. Second, it is dynamically adjusted according to the current loss function value and parameter estimates. Third, with the gradient direction fixed, the proposed method attains a nearly maximum reduction in the loss function. Extensive experiments were conducted to prove the effectiveness of the proposed LQA method.
- Subjects :
- 0209 industrial biotechnology
Current (mathematics)
Artificial neural network
Computer science
business.industry
Cognitive Neuroscience
Deep learning
02 engineering and technology
Quadratic function
Function (mathematics)
Reduction (complexity)
020901 industrial engineering & automation
Quadratic equation
Deep Learning
Artificial Intelligence
0202 electrical engineering, electronic engineering, information engineering
020201 artificial intelligence & image processing
Artificial intelligence
Gradient descent
business
Algorithm
Algorithms
Subjects
Details
- ISSN :
- 18792782
- Volume :
- 141
- Database :
- OpenAIRE
- Journal :
- Neural networks : the official journal of the International Neural Network Society
- Accession number :
- edsair.doi.dedup.....481a1ad582467799581373bbcd6579ec