[Learning List] Recent Important Advances in GAN Papers

Foreword

Imprecisely speaking, in the current state of GAN research, results are everything. No matter how perfect your theory is, if your experiments cannot generate high-definition images, it is difficult to gain acceptance. Conversely, no matter how “ugly” your theoretical derivation, as long as your experimental results are good enough to generate high-definition images, everyone will flock to your work.

A landmark event for GAN models was NVIDIA’s “Progressive Growing GANs” released last year, which achieved 1024 \times 1024 high-definition face generation for the first time. Keep in mind that typical GANs struggle even with 128 \times 128 faces, so 1024-resolution generation was a major breakthrough. The papers listed below have all achieved 1024-resolution face generation in their experiments. This experimental result alone makes these papers worth our attention.

Of course, generating 1024-resolution images requires not only model improvements but also significant computing power, making it difficult for the average person or lab to replicate. We follow these papers not necessarily to reproduce such large images ourselves, but because the fact that these models can generate them means they must have something worth learning. We can even understand the bottlenecks of GANs through them, helping us avoid detours in our own research.

Paper List

Progressive Growing of GANs for Improved Quality, Stability, and Variation
Paper URL: https://papers.cool/arxiv/1710.10196
Reference Implementation: https://github.com/tkarras/progressive_growing_of_gans
Brief Introduction: This is the Progressive Growing GAN (PGGAN) from NVIDIA mentioned earlier, the first to achieve 1024-resolution face generation. As the name suggests, PGGAN uses a progressive structure to transition from low to high resolution, allowing for the smooth training of high-definition models. The paper also proposes insights and techniques regarding regularization and normalization that are worth considering. However, because it is progressive, it requires training many models in sequence, making PGGAN quite slow.

Which Training Methods for GANs do actually Converge?
Paper URL: https://papers.cool/arxiv/1801.04406
Reference Implementation: https://github.com/LMescheder/GAN_stability
Brief Introduction: This paper contains many mathematical derivations regarding the training stability of GANs, ultimately resulting in a gradient penalty term simpler than that of WGAN-GP. Students concerned with GAN training stability should refer to this. Besides 1024-resolution faces, this paper also conducts experiments on many other datasets with excellent results, all using direct end-to-end training without a progressive structure. My only confusion is: isn’t this penalty term just a special case of WGAN-div? Why did the paper not mention this?

IntroVAE: Introspective Variational Autoencoders for Photographic Image Synthesis
Paper URL: https://papers.cool/arxiv/1807.06358
Reference Implementation: (No high-quality open-source/reproduction seen yet)
Brief Introduction: This is a VAE that “introspects.” It improves VAEs through adversarial training, enabling the generation of high-definition images while simultaneously providing both an encoder and a generator. Beyond generating 1024-resolution images, the conceptual design of this paper is exquisite. While models providing both an encoder and generator are not unique (e.g., BiGAN), IntroVAE is unique because it directly utilizes the encoder as the discriminator, requiring no additional discriminator and thus saving 1/3 of the parameters. The deeper reasons behind this are worth careful analysis.

Large Scale GAN Training for High Fidelity Natural Image Synthesis
Paper URL: https://papers.cool/arxiv/1809.11096
Reference Implementation: https://github.com/AaronLeong/BigGAN-pytorch
Brief Introduction: This is the famous BigGAN. Although this paper does not provide 1024-resolution face results, it provides generation results for natural scene images at 128, 256, and 512 resolutions. Generating natural scenes is many times harder than generating CelebA faces; since it can generate 512-resolution natural scenes, we do not doubt it can easily generate 1024-resolution faces. There are already many popular science introductions to BigGAN online. The paper also proposes its own regularization techniques and shares extensive hyperparameter tuning experience (which parameters lead to good or bad changes), which is highly valuable.

Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flow
Paper URL: https://papers.cool/arxiv/1810.00821
Reference Implementation: https://github.com/akanimax/Variational_Discriminator_Bottleneck
Brief Introduction: This paper uses the Information Bottleneck to control the fitting capability of the discriminator, acting as a regularizer to stabilize GAN training. For an introduction to the Information Bottleneck, you can refer to my previous article. Generally, any method used to prevent overfitting in standard supervised training can theoretically be used in the discriminator, and the Information Bottleneck is one such method. As the title suggests, the paper is not satisfied with only GANs; besides 1024-resolution face experiments, it also includes experiments in imitation learning and reinforcement learning.

A Style-Based Generator Architecture for Generative Adversarial Networks
Paper URL: https://papers.cool/arxiv/1812.04948
Reference Implementation: https://github.com/NVlabs/stylegan
Brief Introduction: This is the new GAN generator architecture released just a few days ago, dubbed “GAN 2.0” by many. It is again from NVIDIA, the same authors as PGGAN, and follows the PGGAN pattern but with a changed generator architecture. They were already generating 1024-resolution images a year ago, and this time is no exception. This new architecture is said to draw inspiration from style transfer models, hence the name “Style-Based Generator.” Upon reading, it is essentially a Conditional GAN (CGAN) architecture, but with the condition and noise swapped. Simply put, noise is treated as the condition, and the condition is treated as noise. Looking at the results, this conceptual shift is very effective. I tried implementing it myself; it works, but there is some mode collapse. It’s best to wait for the official open source. Incidentally, the PGGAN authors brought us the CelebA-HQ dataset a year ago, and now they bring us the new FFHQ dataset. Both the dataset and code are expected to be open-sourced in January next year.

Foreword

Unlike supervised learning tasks—where designing a model and having enough data and compute usually results in a good model—GANs are never just about model design. They are an integration of theory, architecture, and optimization. From a framework perspective, the theoretical framework of GANs became largely complete after WGAN, with subsequent work being mostly minor refinements (including my GAN-QP). From an architectural perspective, DCGAN laid the foundation, and the later ResNet + Upsampling became a standard; the new Style-Based Generator is also maturing. So what remains?

It is the optimization—the training process. I believe that to truly master GANs, one must carefully study the optimization process, perhaps analyzing the training trajectory from a dynamical systems perspective. This might involve the existence, uniqueness, and stability of solutions to differential equations, as well as knowledge of stochastic optimization. In short, only by incorporating the optimization process into the analysis can GANs truly become complete.

The following papers analyze GAN training issues from different perspectives and provide their own solutions.

Paper List

Stabilizing Training of Generative Adversarial Networks through Regularization
Paper URL: https://papers.cool/arxiv/1705.09367
Brief Introduction: This paper derives a GAN regularization term by adding noise, a derivation that theoretically applies to all f-GANs. The experimental results look quite good.

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium
Paper URL: https://papers.cool/arxiv/1706.08500
Brief Introduction: This paper proposes the TTUR training strategy. The general idea is: whereas we previously used the same learning rate to train the discriminator and generator for different numbers of iterations, we can now consider using different learning rates and training each once per iteration. This is clearly more time-efficient. However, although the paper has much theory, its foundation is another existing paper, “Stochastic approximation with two time scales.” It feels like the paper is repeatedly applying this existing theory, which is slightly monotonous.

Which Training Methods for GANs do actually Converge?
Paper URL: https://papers.cool/arxiv/1801.04406
Brief Introduction: I have already introduced this paper, but I am listing it again because it is so classic. It is a must-read for studying GAN training stability, understanding GAN training from the perspective of differential equations. In its stability analysis, this paper primarily cites two others: its “prequel” (by the same author) titled “The Numerics of GANs” and “Gradient descent GAN optimization is locally stable.” Both are classics.

Spectral Normalization for Generative Adversarial Networks
Paper URL: https://papers.cool/arxiv/1802.05957
Brief Introduction: This paper implements the Lipschitz constraint (L-constraint) on the discriminator through spectral normalization. It is arguably the most elegant method for implementing the L-constraint currently available and is widely used. For a related introduction, you can refer to my previous article.

Improving the Improved Training of Wasserstein GANs: A Consistency Term and Its Dual Effect
Paper URL: https://papers.cool/arxiv/1803.01541
Brief Introduction: This adds a new regularization term to WGAN-GP. The idea is simple: directly using the L-constraint (in difference form) as a regularization term, similar to the extra quadratic term in the GAN-QP discriminator. According to the paper’s curves, training is more stable than pure WGAN-GP.