In this blog, we have discussed gradient penalty-related content multiple times. Formally, gradient penalty terms are divided into two types. One is the gradient penalty with respect to the input, \Vert\nabla_{\boldsymbol{x}} f(\boldsymbol{x};\boldsymbol{\theta})\Vert^2, which we discussed in articles such as "A Brief Talk on Adversarial Training: Significance, Methods, and Reflections (with Keras Implementation)" and "Random Thoughts on Generalization: From Random Noise and Gradient Penalty to Virtual Adversarial Training". The other is the gradient penalty with respect to the parameters, \Vert\nabla_{\boldsymbol{\theta}} f(\boldsymbol{x};\boldsymbol{\theta})\Vert^2, which we discussed in articles like "Optimization Algorithms from a Dynamical Perspective (V): Why the Learning Rate Should Not Be Too Small?" and "Do We Really Need to Reduce Training Set Loss to Zero?".
In these related articles, both types of gradient penalties are claimed to have the ability to improve the generalization performance of the model. So, is there any connection between the two? I learned about an inequality between them from a recent paper by Google, "The Geometric Occam’s Razor Implicit in Deep Learning". This partially answers the question, and I feel it might be useful in the future, so I am making a note of it here.
Final Result
Assume an l-layer MLP model, denoted as: \boldsymbol{h}^{(t+1)} = g^{(t)}(\boldsymbol{W}^{(t)}\boldsymbol{h}^{(t)}+\boldsymbol{b}^{(t)}) where g^{(t)} is the activation function of the current layer, t\in\{1,2,\cdots,l\}, and \boldsymbol{h}^{(1)} is denoted as \boldsymbol{x}, which is the original input of the model. For the convenience of subsequent derivation, we denote \boldsymbol{z}^{(t+1)}=\boldsymbol{W}^{(t)}\boldsymbol{h}^{(t)}+\boldsymbol{b}^{(t)}. The set of all parameters is \boldsymbol{\theta}=\{\boldsymbol{W}^{(1)},\boldsymbol{b}^{(1)},\boldsymbol{W}^{(2)},\boldsymbol{b}^{(2)},\cdots,\boldsymbol{W}^{(l)},\boldsymbol{b}^{(l)}\}. Let f be any scalar function of \boldsymbol{h}^{(l+1)}. Then the following inequality holds: \Vert\nabla_{\boldsymbol{x}} f\Vert^2\left(\frac{1 + \Vert \boldsymbol{h}^{(1)}\Vert^2}{\Vert\boldsymbol{W}^{(1)}\Vert^2 \Vert\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(1)}\Vert^2}+\cdots+\frac{1 + \Vert \boldsymbol{h}^{(l)}\Vert^2}{\Vert\boldsymbol{W}^{(l)}\Vert^2 \Vert\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(l)}\Vert^2}\right)\leq \Vert\nabla_{\boldsymbol{\theta}} f\Vert^2 \label{eq:f} In the above formula, \Vert\nabla_{\boldsymbol{x}} f\Vert, \Vert\nabla_{\boldsymbol{\theta}} f\Vert^2, and \Vert \boldsymbol{h}^{(i)}\Vert use the standard l_2 norm (the square root of the sum of squares of each element), while \Vert\boldsymbol{W}^{(1)}\Vert and \Vert\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(1)}\Vert use the "spectral norm" of the matrix (refer to "Lipschitz Constraints in Deep Learning: Generalization and Generative Models"). This inequality shows that the parameter gradient penalty, to some extent, contains the input gradient penalty.
Derivation Process
Clearly, to prove inequality [eq:f], we only need to prove for each parameter: \begin{aligned} \Vert\nabla_{\boldsymbol{x}} f\Vert^2\left(\frac{\Vert \boldsymbol{h}^{(t)}\Vert^2}{\Vert\boldsymbol{W}^{(t)}\Vert^2 \Vert\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(t)}\Vert^2}\right)\leq&\, \Vert\nabla_{\boldsymbol{W}^{(t)}} f\Vert^2 \label{eq:w}\\ \Vert\nabla_{\boldsymbol{x}} f\Vert^2\left(\frac{1}{\Vert\boldsymbol{W}^{(t)}\Vert^2 \Vert\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(t)}\Vert^2}\right)\leq&\, \Vert\nabla_{\boldsymbol{b}^{(t)}} f\Vert^2 \label{eq:b} \end{aligned} Then, by iterating through all t and adding the left and right sides of each equation, we obtain the result. The proof of these two inequalities is essentially a matrix differentiation problem. However, like the author, many readers may not be familiar with matrix differentiation. In this case, the best way is to write out the component forms, which turns it into a scalar differentiation problem.
Specifically, \boldsymbol{z}^{(t+1)}=\boldsymbol{W}^{(t)}\boldsymbol{h}^{(t)}+\boldsymbol{b}^{(t)} is written in component form as: z^{(t+1)}_i = \sum_j w^{(t)}_{i,j} h_j^{(t)} + b^{(t)}_i Then, by the chain rule: \frac{\partial f}{\partial x_i} = \sum_{j,k} \frac{\partial f}{\partial z^{(t+1)}_j} \frac{\partial z^{(t+1)}_j}{\partial h^{(t)}_k} \frac{\partial h^{(t)}_k}{\partial x_i} = \sum_{j,k} \frac{\partial f}{\partial z^{(t+1)}_j} w^{(t)}_{j,k} \frac{\partial h^{(t)}_k}{\partial x_i} \label{eq:l} And: \frac{\partial z^{(t+1)}_j}{\partial w^{(t)}_{m,n}} = \delta_{j,m}h^{(t)}_{n} where \delta_{j,m} is the Kronecker delta. Now we can write: w^{(t)}_{j,k} = \sum_m \delta_{j,m}w^{(t)}_{m,k} = \sum_m \frac{\partial z^{(t+1)}_j}{\partial w^{(t)}_{m,n}} (h^{(t)}_n)^{-1} w^{(t)}_{m,k} Substituting this into [eq:l], we get: \frac{\partial f}{\partial x_i} = \sum_{j,k,m} \frac{\partial f}{\partial z^{(t+1)}_j} \frac{\partial z^{(t+1)}_j}{\partial w^{(t)}_{m,n}} (h^{(t)}_n)^{-1} w^{(t)}_{m,k} \frac{\partial h^{(t)}_k}{\partial x_i}=\sum_{k,m} \frac{\partial f}{\partial w^{(t)}_{m,n}} (h^{(t)}_n)^{-1} w^{(t)}_{m,k} \frac{\partial h^{(t)}_k}{\partial x_i} Multiplying both sides by h^{(t)}_n gives: h^{(t)}_n\frac{\partial f}{\partial x_i} = \sum_{k,m} \frac{\partial f}{\partial w^{(t)}_{m,n}} w^{(t)}_{m,k} \frac{\partial h^{(t)}_k}{\partial x_i} Assuming the original vectors are column vectors and the shape of the matrix is transposed after calculating the gradient, the above can be written in matrix form: \boldsymbol{h}^{(t)}(\nabla_{\boldsymbol{x}} f)^{\top} = (\nabla_{\boldsymbol{W}^{(t)}} f )^{\top} \boldsymbol{W}^{(t)}(\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(t)}) Multiplying both sides on the left by (\boldsymbol{h}^{(t)})^{\top} gives: \Vert\boldsymbol{h}^{(t)}\Vert^2(\nabla_{\boldsymbol{x}} f)^{\top} = (\boldsymbol{h}^{(t)})^{\top}(\nabla_{\boldsymbol{W}^{(t)}} f )^{\top} \boldsymbol{W}^{(t)}(\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(t)}) Taking the norm of both sides: \Vert\boldsymbol{h}^{(t)}\Vert^2 \Vert\nabla_{\boldsymbol{x}} f\Vert = \Vert (\boldsymbol{h}^{(t)})^{\top}(\nabla_{\boldsymbol{W}^{(t)}} f )^{\top} \boldsymbol{W}^{(t)}(\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(t)})\Vert \leq \Vert\boldsymbol{h}^{(t)}\Vert \Vert\nabla_{\boldsymbol{W}^{(t)}} f \Vert \Vert \boldsymbol{W}^{(t)}\Vert \Vert\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(t)}\Vert For the second inequality sign, the matrix norm holds whether using the l_2 norm or the spectral norm. Thus, after choosing the required norm and rearranging, we obtain equation [eq:w]. The proof for equation [eq:b] is similar and will not be repeated here.
Brief Analysis
Some readers might ask how to specifically interpret equation [eq:f]? In fact, I mainly find equation [eq:f] itself interesting and think it might be useful in some scenarios later, so this article is primarily a "note" on it, without a definitive interpretation.
As for the logical sequence of the original paper, it is as follows: in "Optimization Algorithms from a Dynamical Perspective (V): Why the Learning Rate Should Not Be Too Small?", we introduced "Implicit Gradient Regularization" (by the same author as this paper), which pointed out that SGD implicitly contains a gradient penalty term for parameters. Equation [eq:f] then shows that the parameter gradient penalty implicitly contains the input gradient penalty. Furthermore, the input gradient penalty is related to Dirichlet energy, which can serve as a representation of model complexity. Therefore, following this chain of reasoning, the conclusion is: SGD itself tends to choose models with lower complexity.
However, the original paper made a small mistake when interpreting equation [eq:f]. It stated that \Vert \boldsymbol{W}^{(t)}\Vert would be very close to 0 in the initial stage, so the term in the brackets of equation [eq:f] would be very large. Therefore, to reduce the parameter gradient penalty on the right side of equation [eq:f], the input gradient penalty on the left side must be made sufficiently small. However, from "Understanding Model Parameter Initialization Strategies from a Geometric Perspective", we know that commonly used initialization methods are actually close to orthogonal initialization, and the spectral norm of an orthogonal matrix is 1. If activation functions are considered, the spectral norm at initialization is actually greater than 1. Thus, the assumption that \Vert \boldsymbol{W}^{(t)}\Vert is very close to 0 in the initialization stage does not hold.
In fact, for a network that has not collapsed during training, the model parameters and the input/output of each layer basically maintain a stable state. Therefore, throughout the training process, \Vert \boldsymbol{h}^{(t)}\Vert, \Vert\boldsymbol{W}^{(t)}\Vert, and \Vert\nabla_{\boldsymbol{x}}\boldsymbol{h}^{(t)}\Vert do not fluctuate much. Thus, the parameter gradient penalty on the right side is approximately equivalent to the input gradient penalty on the left side. This is my understanding, which does not require the assumption that "\Vert \boldsymbol{W}^{(t)}\Vert is very close to 0."
Summary
This article introduced an inequality between two types of gradient penalty terms and provided a derivation as well as a brief analysis.
Original Address: https://kexue.fm/archives/8796
For more details on reprinting, please refer to: "Scientific Space FAQ"