Yesterday while browsing arXiv, I
discovered a paper from the stars South Korea with a very
straightforward title: “A
Simple yet Effective Way for Improving the Performance of GANs”.
Upon reading it, I found the content to be equally concise, proposing a
method to strengthen the GAN discriminator, which leads to a certain
improvement in generative metrics.
The authors call this method “Cascading Rejection.” I wasn’t sure how to translate it, but Baidu Translate suggested “Cascading Suppression/Rejection,” which seems to capture the essence, so I will use that for now. I am introducing this method not necessarily because of its overwhelming power, but because its geometric meaning is quite interesting and seems to offer some inspiration.
Orthogonal Decomposition
A GAN discriminator generally produces a fixed-length vector \boldsymbol{v} after multiple layers of convolution followed by flattening or pooling. This vector is then used in an inner product with a weight vector \boldsymbol{w} to obtain a scalar score (ignoring bias terms and activation functions for simplicity): D(\boldsymbol{x})=\langle \boldsymbol{v},\boldsymbol{w}\rangle In other words, \boldsymbol{v} is used as the representation of the input image, and the degree of “realness” of the image is judged by the magnitude of the inner product between \boldsymbol{v} and \boldsymbol{w}.
However, \langle \boldsymbol{v},\boldsymbol{w}\rangle only depends on the projection component of \boldsymbol{v} onto \boldsymbol{w}. In other words, when \langle \boldsymbol{v},\boldsymbol{w}\rangle and \boldsymbol{w} are fixed, \boldsymbol{v} can still vary significantly, as shown in the left figure below.
If we assume that an image is real when \langle \boldsymbol{v},\boldsymbol{w}\rangle equals a certain value, the problem is that since \boldsymbol{v} varies so much, does every such \boldsymbol{v} represent a real image? Obviously, not necessarily. This reflects the problem with scoring via inner products: it only considers the projection component on \boldsymbol{w} and ignores the orthogonal component (as shown in the right figure above): \boldsymbol{v}-\Vert \boldsymbol{v}\Vert \cos(\boldsymbol{v},\boldsymbol{w}) \frac{\boldsymbol{w}}{\Vert \boldsymbol{w}\Vert}=\boldsymbol{v}- \frac{\langle\boldsymbol{v},\boldsymbol{w}\rangle}{\Vert \boldsymbol{w}\Vert^2}\boldsymbol{w}
Given this, a natural idea arises: can we use another parameter vector to perform classification on this orthogonal component? Clearly, this is possible, and classifying this orthogonal component will result in a new orthogonal component. Therefore, this process can be iterated: \left\{\begin{aligned} &\boldsymbol{v}_1=\boldsymbol{v}\\ &D_1(\boldsymbol{x})=\langle \boldsymbol{v}_1,\boldsymbol{w}_1\rangle\\ &\boldsymbol{v}_2 = \boldsymbol{v}_1- \frac{\langle\boldsymbol{v}_1,\boldsymbol{w}_1\rangle}{\Vert \boldsymbol{w}_1\Vert^2}\boldsymbol{w}_1\\ &D_2(\boldsymbol{x})=\langle \boldsymbol{v}_2,\boldsymbol{w}_2\rangle\\ &\boldsymbol{v}_3 = \boldsymbol{v}_2- \frac{\langle\boldsymbol{v}_2,\boldsymbol{w}_2\rangle}{\Vert \boldsymbol{w}_2\Vert^2}\boldsymbol{w}_2\\ &D_3(\boldsymbol{x})=\langle \boldsymbol{v}_3,\boldsymbol{w}_3\rangle\\ &\boldsymbol{v}_4 = \boldsymbol{v}_3- \frac{\langle\boldsymbol{v}_3,\boldsymbol{w}_3\rangle}{\Vert \boldsymbol{w}_3\Vert^2}\boldsymbol{w}_3\\ &\qquad\vdots\\ &D_N(\boldsymbol{x})=\langle \boldsymbol{v}_N,\boldsymbol{w}_N\rangle\\ \end{aligned}\right.
Analysis and Reflection
At this point, the core idea of the original paper has basically been covered; the rest involves detailed operations. First, having obtained N scores D_1(\boldsymbol{x}), D_2(\boldsymbol{x}), \dots, D_N(\boldsymbol{x}), a discriminator loss can be applied to each score (using hinge loss directly or cross-entropy after a sigmoid activation). Finally, a weighted average of these N losses is used as the final discriminator loss. This alone can bring performance improvements to GANs. The authors further extended this to CGANs and achieved good results as well.
Compared to the experimental results, I believe the deeper significance of this technique is more noteworthy. In principle, this idea could be applied to general classification problems, not just GANs. By iteratively incorporating orthogonal components into the prediction, we can consider the parameters \boldsymbol{w}_1, \boldsymbol{w}_2, \dots, \boldsymbol{w}_N as representing N different perspectives, where each classification corresponds to a judgment from a different viewpoint.
Thinking of this, I am reminded of Hinton’s Capsules. Although the form is different, the underlying intention seems similar. Capsules aim to represent an entity with a vector rather than a scalar. This “Cascading Rejection” also provides multiple classification results through continuous orthogonal decomposition. That is, to determine if a vector belongs to a class, multiple scores must be given rather than just one, which also has the flavor of “using a vector instead of a scalar.”
Regrettably, I conducted a simple experiment following this logic (on CIFAR-10) and found that the classification accuracy on the validation set dropped slightly (note that this does not contradict the GAN results; GAN performance improves because the discrimination task is made harder, but supervised classification models do not benefit from increased difficulty). However, the degree of overfitting was reduced (i.e., the gap between training and validation accuracy decreased). Of course, my experiment was too simple to draw rigorous conclusions. Nevertheless, I still feel that due to its clear geometric meaning, this technique is worth further consideration.
Summary
This article introduced a technique for improving GAN performance with clear geometric significance and further discussed its potential value.
Original Address: https://kexue.fm/archives/7105
For more details on reposting, please refer to: “Scientific Space FAQ”