1. Perturbed Iterate SGD for Lipschitz Continuous Loss Functions.
- Author
-
Metel, Michael R. and Takeda, Akiko
- Subjects
- *
CONTINUOUS functions , *LIPSCHITZ continuity , *MACHINE learning - Abstract
This paper presents an extension of stochastic gradient descent for the minimization of Lipschitz continuous loss functions. Our motivation is for use in non-smooth non-convex stochastic optimization problems, which are frequently encountered in applications such as machine learning. Using the Clarke ϵ -subdifferential, we prove the non-asymptotic convergence to an approximate stationary point in expectation for the proposed method. From this result, a method with non-asymptotic convergence with high probability, as well as a method with asymptotic convergence to a Clarke stationary point almost surely are developed. Our results hold under the assumption that the stochastic loss function is a Carathéodory function which is almost everywhere Lipschitz continuous in the decision variables. To the best of our knowledge, this is the first non-asymptotic convergence analysis under these minimal assumptions. [ABSTRACT FROM AUTHOR]
- Published
- 2022
- Full Text
- View/download PDF