Optimization Algorithms from a Dynamics Perspective (V): Why the Learning Rate Should Not Be Too Small? · English (unofficial) translations of posts at kexue.fm

The theme of this article is “why we need a finite learning rate.” By “finite,” we mean a rate that is neither too large nor too small, but just right. It is easy to understand that a learning rate that is too large can cause the algorithm to diverge. But why is a learning rate that is too small also undesirable? An intuitive answer is that an excessively small learning rate requires too many iterations, which is an unnecessary waste. Therefore, from the perspectives of “energy saving” and “acceleration,” we avoid using very small learning rates. However, if we disregard computational power and time, would an extremely small learning rate be preferable? A recent paper published by Google on arXiv, “Implicit Gradient Regularization”, attempts to answer this question. It points out that a finite learning rate implicitly introduces a gradient penalty term to the optimization process, and this gradient penalty is helpful for improving generalization performance. Therefore, even without considering factors like computational power and time, one should not use a learning rate that is too small.

Regarding gradient penalty, this blog has discussed it many times. In the articles “A Brief Talk on Adversarial Training: Significance, Methods, and Reflections (with Keras Implementation)” and “Random Thoughts on Generalization: From Random Noise and Gradient Penalty to Virtual Adversarial Training”, we analyzed how adversarial training is, to some extent, equivalent to a gradient penalty on the input. Meanwhile, the “Flooding” technique introduced in “Do We Really Need to Reduce the Training Set Loss to Zero?” is equivalent to a gradient penalty on the parameters. In general, whether it is a gradient penalty on the input or the parameters, it helps improve generalization ability.

The Direction of Steepest Descent

Like other articles in this series, this paper views the optimization process as solving differential equations. Recalling a previous blog post “Optimization Algorithms from a Dynamics Perspective (III): A More Holistic View”, let the loss function be L(\boldsymbol{\theta}). We view \boldsymbol{\theta} as a trajectory \boldsymbol{\theta}(t) changing along some time parameter t. Now we consider its rate of change: \frac{d}{dt}L(\boldsymbol{\theta}(t))=\left\langle\nabla_{\boldsymbol{\theta}}L(\boldsymbol{\theta}(t)),\, \dot{\boldsymbol{\theta}}(t)\right\rangle We hope that L(\boldsymbol{\theta}(t)) decreases over time (the smaller the loss, the better), so we want the above expression to be less than 0. When the magnitude \Vert\dot{\boldsymbol{\theta}}(t)\Vert is fixed, the minimum value of the right-hand side is achieved in the direction opposite to the gradient, -\nabla_{\boldsymbol{\theta}}L(\boldsymbol{\theta}(t)). Therefore, we say the negative gradient direction is the direction of steepest descent. For simplicity, we can directly set: \dot{\boldsymbol{\theta}}(t) = -\nabla_{\boldsymbol{\theta}}L(\boldsymbol{\theta}(t))\triangleq - \boldsymbol{g}(\boldsymbol{\theta}(t)) \label{eq:odes} Then, solving for the parameters \boldsymbol{\theta} is transformed into solving the above system of ordinary differential equations. This is the basic starting point of the “Optimization Algorithms from a Dynamics Perspective” series.

Regularization Hidden in the Learning Rate

However, the practical problem is that we cannot truly solve the system of differential equations [eq:odes]. We can only use numerical iteration, such as the simplest Euler method, to obtain: \boldsymbol{\theta}_{t+\gamma} = \boldsymbol{\theta}_{t} - \gamma \boldsymbol{g}(\boldsymbol{\theta}_t) \label{eq:gd} This is essentially the most primitive gradient descent method, where \gamma is what we commonly call the learning rate. The above equation is essentially a difference equation.

One can imagine that starting from t=0, the points obtained \boldsymbol{\theta}_{\gamma}, \boldsymbol{\theta}_{2\gamma}, \boldsymbol{\theta}_{3\gamma}, \dots will deviate somewhat from the exact solutions \boldsymbol{\theta}(\gamma), \boldsymbol{\theta}(2\gamma), \boldsymbol{\theta}(3\gamma), \dots of the system [eq:odes]. How can we measure the extent of this deviation? We can imagine that \boldsymbol{\theta}_{\gamma}, \boldsymbol{\theta}_{2\gamma}, \boldsymbol{\theta}_{3\gamma}, \dots are actually exact solutions to some differential equation system similar to [eq:odes], but with \boldsymbol{g}(\boldsymbol{\theta}(t)) replaced by some new \tilde{\boldsymbol{g}}(\boldsymbol{\theta}_t). We then just need to compare the difference between \tilde{\boldsymbol{g}}(\boldsymbol{\theta}_t) and \boldsymbol{g}(\boldsymbol{\theta}(t)).

Through derivation, if we only keep terms up to the first order of \gamma, we have: \tilde{\boldsymbol{g}}(\boldsymbol{\theta}_t) = \boldsymbol{g}(\boldsymbol{\theta}_t) + \frac{\gamma}{4}\nabla_{\boldsymbol{\theta}}\Vert \boldsymbol{g}(\boldsymbol{\theta}_t)\Vert^2 = \nabla_{\boldsymbol{\theta}}\left(L(\boldsymbol{\theta}_t) + \frac{1}{4}\gamma\Vert \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta}_t)\Vert^2\right) The derivation process is provided in the next section. As can be seen, this is equivalent to adding a gradient penalty regularization term \frac{1}{4}\gamma\Vert \nabla_{\boldsymbol{\theta}} L(\boldsymbol{\theta})\Vert^2 to the loss function. Gradient penalty terms help the model reach flatter regions, which is beneficial for improving generalization performance. This means that the discretized iteration process implicitly introduces a gradient penalty term, which actually helps the model’s generalization. If \gamma \to 0, this implicit penalty weakens or even disappears.

Therefore, the conclusion is that the learning rate should not be too small; a larger learning rate not only has the benefit of accelerating convergence but also the benefit of improving the model’s generalization ability. Of course, some readers might think: if I directly add the gradient penalty to the loss, can I then use a sufficiently small learning rate? Theoretically, yes. The original paper refers to the practice of adding gradient penalty to the loss as “explicit gradient penalty.”

From Difference Equations to Differential Equations

For the conversion from difference equations to differential equations, we can use the standard “perturbation method,” which has been briefly introduced in this blog (see the tag “Perturbation”). However, a more elegant solution is to directly use the series operation of operators, referring to the previous article “The Art of Operators: Difference, Differentiation, and Bernoulli Numbers”.

We expand \boldsymbol{\theta}_{t+\gamma} using a Taylor series: \boldsymbol{\theta}_{t+\gamma}=\boldsymbol{\theta}_{t}+\gamma \dot{\boldsymbol{\theta}}_{t} + \frac{1}{2}\gamma^2\ddot{\boldsymbol{\theta}}_{t} + \frac{1}{6}\gamma^3\dddot{\boldsymbol{\theta}}_{t} + \cdots If we denote the derivative operator with respect to t as D, then the above equation is actually: \boldsymbol{\theta}_{t+\gamma} = \left(1+\gamma D + \frac{1}{2}\gamma^2 D^2 + \frac{1}{6}\gamma^3 D^3 + \cdots\right)\boldsymbol{\theta}_{t} = e^{\gamma D}\boldsymbol{\theta}_{t} So the difference equation [eq:gd] can be written as: \left(e^{\gamma D} - 1\right)\boldsymbol{\theta}_{t} = - \gamma \boldsymbol{g}(\boldsymbol{\theta}_t) Just like conventional algebraic operations, we have: \begin{aligned} D\boldsymbol{\theta}_{t} =& - \gamma \left(\frac{D}{e^{\gamma D} - 1}\right)\boldsymbol{g}(\boldsymbol{\theta}_t)\\ =& - \left(1 - \frac{1}{2}\gamma D + \frac{1}{12}\gamma^2 D^2 - \frac{1}{720}\gamma^4 D^4 + \cdots\right)\boldsymbol{g}(\boldsymbol{\theta}_t) \end{aligned} The left side is \dot{\boldsymbol{\theta}}_{t}, so the right side is the expression for -\tilde{\boldsymbol{g}}(\boldsymbol{\theta}_t). Retaining terms up to the first order: - \left(1 - \frac{1}{2}\gamma D\right)\boldsymbol{g}(\boldsymbol{\theta}_t) = - \boldsymbol{g}(\boldsymbol{\theta}_t) + \frac{1}{2}\gamma \frac{d}{dt}\boldsymbol{g}(\boldsymbol{\theta}_t) = - \boldsymbol{g}(\boldsymbol{\theta}_t) + \frac{1}{2}\gamma \nabla_{\boldsymbol{\theta}}\boldsymbol{g}(\boldsymbol{\theta}_t)\dot{\boldsymbol{\theta}}_t Which is: \begin{aligned} \dot{\boldsymbol{\theta}}_{t} =& - \boldsymbol{g}(\boldsymbol{\theta}_t) + \frac{1}{2}\gamma \nabla_{\boldsymbol{\theta}}\boldsymbol{g}(\boldsymbol{\theta}_t)\dot{\boldsymbol{\theta}}_t\\ =&- \boldsymbol{g}(\boldsymbol{\theta}_t) + \frac{1}{2}\gamma \nabla_{\boldsymbol{\theta}}\boldsymbol{g}(\boldsymbol{\theta}_t)\left[- \boldsymbol{g}(\boldsymbol{\theta}_t) + \frac{1}{2}\gamma \nabla_{\boldsymbol{\theta}}\boldsymbol{g}(\boldsymbol{\theta}_t)\dot{\boldsymbol{\theta}}_t\right]\\ =&- \boldsymbol{g}(\boldsymbol{\theta}_t) - \frac{1}{2}\gamma \nabla_{\boldsymbol{\theta}}\boldsymbol{g}(\boldsymbol{\theta}_t)\boldsymbol{g}(\boldsymbol{\theta}_t)\quad\text{(neglecting second-order terms)}\\ =&- \boldsymbol{g}(\boldsymbol{\theta}_t) - \frac{1}{4}\gamma \nabla_{\boldsymbol{\theta}}\Vert\boldsymbol{g}(\boldsymbol{\theta}_t)\Vert^2 \end{aligned} Thus, the first-order \tilde{\boldsymbol{g}}(\boldsymbol{\theta}_t)=\boldsymbol{g}(\boldsymbol{\theta}_t) + \frac{1}{4}\gamma \nabla_{\boldsymbol{\theta}}\Vert\boldsymbol{g}(\boldsymbol{\theta}_t)\Vert^2. The derivation is complete.

A Routine Summary

The development and popularization of deep learning are inseparable from the successful application of optimizers based on gradient descent. However, why gradient descent is so successful still lacks a profound explanation. Many researchers, in the process of “alchemy,” more or less summarize some “tricks” that work without knowing why, such as how large the batch size should be or how to adjust the learning rate; everyone likely has their own experience.

Regarding the phenomenon that “the learning rate should not be too small,” everyone should have some intuition. In many cases, it might have been defaultly used as “common sense,” and one might be too lazy to think about the underlying principles. This paper from Google provides a possible explanation for understanding this phenomenon: an appropriate, rather than excessively small, learning rate can bring an implicit gradient penalty term to the optimization process, helping to converge to a flatter region. I believe its analysis process is well worth referencing and learning.