1. Hamiltonian Monte Carlo on ReLU Neural Networks is Inefficient
- Author
-
Dinh, Vu C., Ho, Lam Si Tung, and Nguyen, Cuong V.
- Subjects
Statistics - Machine Learning ,Computer Science - Machine Learning - Abstract
We analyze the error rates of the Hamiltonian Monte Carlo algorithm with leapfrog integrator for Bayesian neural network inference. We show that due to the non-differentiability of activation functions in the ReLU family, leapfrog HMC for networks with these activation functions has a large local error rate of $\Omega(\epsilon)$ rather than the classical error rate of $O(\epsilon^3)$. This leads to a higher rejection rate of the proposals, making the method inefficient. We then verify our theoretical findings through empirical simulations as well as experiments on a real-world dataset that highlight the inefficiency of HMC inference on ReLU-based neural networks compared to analytical networks., Comment: Paper published at NeurIPS 2024
- Published
- 2024