Is L2 Regularization Not as Good as Imagined? It Might Be Due to ``Weight Scale Shifting''

Understanding L2 Regularization

Why add L2 regularization? There are multiple answers to this question. Some answer it from the perspective of Ridge regression, others from Bayesian inference. Here, we provide an understanding from the perspective of perturbation sensitivity.

For two (column) vectors \boldsymbol{w}, \boldsymbol{x}, we have the Cauchy-Schwarz inequality |\boldsymbol{w}^{\top}\boldsymbol{x}| \leq \Vert\boldsymbol{w}\Vert_2 \Vert\boldsymbol{x}\Vert_2. Based on this result, we can prove: \Vert\boldsymbol{W}(\boldsymbol{x}_2 - \boldsymbol{x}_1)\Vert_2 \leq \Vert\boldsymbol{W}\Vert_2 \Vert\boldsymbol{x}_2 - \boldsymbol{x}_1\Vert_2 Here, \Vert\boldsymbol{W}\Vert_2^2 is equal to the sum of the squares of all elements of the matrix \boldsymbol{W}. The proof is not difficult, and interested readers can complete it themselves. This result tells us: the change in \boldsymbol{W}\boldsymbol{x} can be controlled by \Vert\boldsymbol{W}\Vert_2 and \Vert\boldsymbol{x}_2 - \boldsymbol{x}_1\Vert_2. Therefore, if we want the change in \boldsymbol{W}\boldsymbol{x} to be as small as possible when \Vert\boldsymbol{x}_2 - \boldsymbol{x}_1\Vert_2 is very small, we can reduce \Vert\boldsymbol{W}\Vert_2. At this point, we can add a regularization term \mathcal{L}_{reg}=\Vert\boldsymbol{W}\Vert_2^2 to the task objective \mathcal{L}_{task}. It is not difficult to see that this is actually L2 regularization. For related discussions from this perspective, one can also refer to “Lipschitz Constraints in Deep Learning: Generalization and Generative Models” (though note that the notation in the two articles differs slightly).

AdamW Optimizer

When using SGD for optimization, assuming the original iteration is \boldsymbol{\theta}_{t}=\boldsymbol{\theta}_{t-1} - \varepsilon\boldsymbol{g}_{t}, it is easy to prove that after adding the L2 regularization \Vert\boldsymbol{\theta}\Vert_2^2, it becomes: \boldsymbol{\theta}_{t}=(1-\varepsilon\lambda)\boldsymbol{\theta}_{t-1} - \varepsilon\boldsymbol{g}_{t} Since 0 < 1-\varepsilon\lambda < 1, this causes the parameters \boldsymbol{\theta} to have a tendency to “shrink” toward 0 during the optimization process. Such a modification is called “Weight Decay.”

However, the equivalence between L2 regularization and weight decay only holds under the SGD optimizer. If an adaptive learning rate optimizer such as Adagrad or Adam is used, the two are not equivalent. In adaptive learning rate optimizers, the effect of L2 regularization is approximately equivalent to adding -\varepsilon\lambda\text{sign}(\boldsymbol{\theta}_{t-1}) to the optimization process instead of -\varepsilon\lambda\boldsymbol{\theta}_{t-1}. This means that the penalty for each element is uniform, rather than elements with larger absolute values receiving a larger penalty, which partially offsets the effect of L2 regularization. The paper “Decoupled Weight Decay Regularization” first emphasized this issue and proposed the improved AdamW optimizer.

Weight Scale Shifting

We know that the basic structure of a deep learning model is “linear transformation + non-linear activation function,” and one of the most commonly used activation functions today is \text{relu}(x)=\max(x,0). Interestingly, both satisfy “positive homogeneity,” which means that for \varepsilon \geq 0, we have \varepsilon\phi(x)=\phi(\varepsilon x) identically. For other activation functions such as SoftPlus, GELU, Swish, etc., they are actually smooth approximations of \text{relu}, so they can be considered to approximately satisfy “positive homogeneity.”

“Positive homogeneity” makes deep learning models possess a certain invariance to weight scale shifting. Specifically, consider an L-layer model: \begin{aligned} \boldsymbol{h}_L =& \phi(\boldsymbol{W}_L \boldsymbol{h}_{L-1} + \boldsymbol{b}_L) \\ =& \phi(\boldsymbol{W}_L \phi(\boldsymbol{W}_{L-1} \boldsymbol{h}_{L-2} + \boldsymbol{b}_{L-1}) + \boldsymbol{b}_L) \\ =& \cdots\\ =& \phi(\boldsymbol{W}_L \phi(\boldsymbol{W}_{L-1} \phi(\cdots\phi(\boldsymbol{W}_1\boldsymbol{x} + \boldsymbol{b}_1)\cdots) + \boldsymbol{b}_{L-1}) + \boldsymbol{b}_L) \end{aligned} Assuming each parameter introduces a shift \boldsymbol{W}_l = \gamma_l\tilde{\boldsymbol{W}}_l, \boldsymbol{b}_l = \gamma_l\tilde{\boldsymbol{b}}_l, then according to positive homogeneity, we have: \begin{aligned} \boldsymbol{h}_L =& \left(\prod_{l=1}^L \gamma_l\right)\phi(\tilde{\boldsymbol{W}}_L \boldsymbol{h}_{L-1} + \tilde{\boldsymbol{b}}_L) \\ =& \cdots\\ =& \left(\prod_{l=1}^L \gamma_l\right) \phi(\tilde{\boldsymbol{W}}_L \phi(\tilde{\boldsymbol{W}}_{L-1} \phi(\cdots\phi(\tilde{\boldsymbol{W}}_1\boldsymbol{x} + \tilde{\boldsymbol{b}}_1)\cdots) + \tilde{\boldsymbol{b}}_{L-1}) + \tilde{\boldsymbol{b}}_L) \end{aligned} If \prod\limits_{l=1}^L \gamma_l = 1, then the model with parameters \{\boldsymbol{W}_l, \boldsymbol{b}_l\} is completely equivalent to the model with parameters \{\tilde{\boldsymbol{W}}_l, \tilde{\boldsymbol{b}}_l\}. In other words, the model is invariant to weight scale shifting where \prod\limits_{l=1}^L \gamma_l = 1 (WEIght-Scale-Shift-Invariance, WEISSI).

Inconsistency with L2 Regularization

We just said that as long as the scale shift satisfies \prod\limits_{l=1}^L \gamma_l = 1, the models corresponding to the two sets of parameters are equivalent. But the problem is that their corresponding L2 regularizations are not equivalent: \sum_{l=1}^L \Vert\boldsymbol{W}_l\Vert_2^2=\sum_{l=1}^L \gamma_l^2\Vert\tilde{\boldsymbol{W}}_l\Vert_2^2 \neq \sum_{l=1}^L \Vert\tilde{\boldsymbol{W}}_l\Vert_2^2 And it can be proven that if \Vert\boldsymbol{W}_1\Vert_2, \Vert\boldsymbol{W}_2\Vert_2, \dots, \Vert\boldsymbol{W}_L\Vert_2 are fixed, and the constraint \prod\limits_{l=1}^L \gamma_l = 1 is maintained, then the minimum value of \sum_{l=1}^L \Vert\tilde{\boldsymbol{W}}_l\Vert_2^2 occurs at: \Vert\tilde{\boldsymbol{W}}_1\Vert_2=\Vert\tilde{\boldsymbol{W}}_2\Vert_2=\dots=\Vert\tilde{\boldsymbol{W}}_L\Vert_2=\left(\prod_{l=1}^L \Vert\boldsymbol{W}_l\Vert_2\right)^{1/L} In fact, this reflects the inefficiency of L2 regularization. Imagine that we have already trained a set of parameters \{\boldsymbol{W}_l, \boldsymbol{b}_l\}, and the generalization performance of this set of parameters might not be very good. So we hope L2 regularization can help the optimizer find a better set of parameters (sacrificing a bit of \mathcal{L}_{task} to reduce \mathcal{L}_{reg}). However, the above results tell us that due to the existence of weight scale shifting invariance, the model can find a new set of parameters \{\tilde{\boldsymbol{W}}_l, \tilde{\boldsymbol{b}}_l\} that is completely equivalent to the original model (without improving generalization performance), but with a smaller L2 regularization (L2 regularization has taken effect). Simply put, L2 regularization indeed works, but it does not improve the model’s generalization performance, failing to achieve the original intention of using L2 regularization.

WEISSI Regularization

The root of the above problem is that the model is invariant to weight scale shifting, but L2 regularization is not. If we can find a new regularization term that has a similar effect but is also invariant to weight scale shifting, then we can solve this problem. Personally, I feel the original paper’s explanation of this part is not clear enough; the following derivation is based primarily on my personal understanding.

We consider a regularization term of the following general form: \mathcal{L}_{reg}=\sum_{l=1}^L \varphi(\Vert\boldsymbol{W}_l\Vert_2) For L2 regularization, \varphi(x)=x^2. As long as \varphi(x) is a monotonically increasing function of x on [0, +\infty), it can be guaranteed that the optimization goal is to reduce \Vert\boldsymbol{W}_l\Vert. Note that we want the regularization term to have scale shifting invariance; we do not need \varphi(\gamma x) = \varphi(x), but only: \frac{d}{dx}\varphi(\gamma x)=\frac{d}{dx}\varphi(x) \label{eq:varphi} Because the optimization process only needs to use its gradient. Some readers might directly see a solution to this: it is actually the logarithmic function \varphi(x) = \log x. So the newly proposed regularization term is: \mathcal{L}_{reg}=\sum_{l=1}^L \log\Vert\boldsymbol{W}_l\Vert_2=\log \left(\prod_{l=1}^L \Vert\boldsymbol{W}_l\Vert_2\right) In addition, the original paper might have been concerned that the penalty of the above regularization term was not strong enough, so it also added an L1 penalty on the parameter directions. The total form is: \mathcal{L}_{reg}=\lambda_1\sum_{l=1}^L \log\Vert\boldsymbol{W}_l\Vert_2 + \lambda_2\sum_{l=1}^L \big\Vert\boldsymbol{W}_l\big/\Vert\boldsymbol{W}_l\Vert_2\big\Vert_1

Brief Description of Experimental Results

As per tradition, let’s show one of the experimental results from the original paper. Since the authors have compiled it into a paper, it obviously indicates positive results:

For us, it’s simply about knowing there is such a new option, providing one more thing to try when training models. After all, for something like a regularization term, there is no theory that guarantees it will definitely work; you only know the result after using it. No matter how beautifully others describe it, it might not necessarily be useful.