Abstract
In this paper, we propose a new adaptive stochastic gradient Langevindynamics (ASGLD) algorithmic framework and its two specialized versions, namelyadaptive stochastic gradient (ASG) and adaptive gradient Langevindynamics(AGLD), for non-convex optimization problems. All proposed algorithmscan escape from saddle points with at most O(logd)O(\log d) iterations, which isnearly dimension-free. Further, we show that ASGLD and ASG converge to a localminimum with at most O(logd/ϵ4)O(\log d/\epsilon^4) iterations. Also, ASGLD with fullgradients or ASGLD with a slowly linearly increasing batch size converge to alocal minimum with iterations bounded by O(logd/ϵ2)O(\log d/\epsilon^2), whichoutperforms existing first-order methods.