Designing GANs: Another GAN Production Workshop

Total Variation

Let’s take a probability divergence called Total Variation as an example to get a preliminary sense of the key points of deriving probability divergences in the dual space.

First, we have: |p-q|=\max_{t\in[-1, 1]} (p - q)t=\max_{t\in[-1, 1]} pt - qt\label{eq:tv-base} Therefore, for probability distributions p(x) and q(x), we also have: |p(x)-q(x)|=\max_{t(x)\in[-1, 1]} p(x)t(x) - q(x)t(x) Integrating both sides (setting aside the issue of swapping integration and \max for now): \begin{aligned}\int|p(x)-q(x)|dx=&\, \max_{t(x)\in[-1, 1]} \int \big[p(x)t(x) - q(x)t(x)\big]dx\\ =&\, \max_{t(x)\in[-1, 1]} \mathbb{E}_{x\sim p(x)}[t(x)] - \mathbb{E}_{x\sim q(x)}[t(x)] \end{aligned} Here, \int|p(x)-q(x)|dx is known as the Total Variation between two probability distributions. Thus, we have derived the dual form of Total Variation through this process. If we fix p(x) and let q(x) approach p(x), we can minimize the Total Variation: \min_{q(x)}\int|p(x)-q(x)|dx = \min_{q(x)}\max_{t(x)\in[-1, 1]} \mathbb{E}_{x\sim p(x)}[t(x)] - \mathbb{E}_{x\sim q(x)}[t(x)]\label{eq:tv-gan} This yields a form of GAN.

Reviewing the entire process, and putting aside prior knowledge of Total Variation, we can see that the core of the process is actually Equation [eq:tv-base]. Once we have Equation [eq:tv-base], everything else follows naturally. So, what are the characteristics of Equation [eq:tv-base]? In fact, it can be generalized as:

Goal 1 Find functions \phi(t), \psi(t) and a certain range \Omega such that d(p, q) = \max_{t\in\Omega} p\phi(t)+q\psi(t) and d(p,q)\geq 0 as well as d(p,q)=0 \Leftrightarrow p=q.

With such \phi(t), \psi(t), we can derive a GAN similar to [eq:tv-gan]: \min_{q(x)}\int d(p(x), q(x)) dx = \min_{q(x)}\max_{t(x)\in \Omega} \mathbb{E}_{x\sim p(x)}[\phi(t(x))] + \mathbb{E}_{x\sim q(x)}[\psi(t(x))]\label{eq:gan}

Finding the Maximum via Differentiation

Note that in "Goal 1", p and q are just non-negative real numbers, and \phi(t), \psi(t) are just scalar functions. Therefore, the entire goal is purely an extremum problem of a single-variable function, which is quite simplified. Furthermore, by letting r=q/p \in [0, +\infty), it can be transformed into a more concise "Goal 2":

Goal 2 Find functions \phi(t), \psi(t) and a certain range \Omega such that d(r) = \max_{t\in\Omega} \phi(t)+r\psi(t) and the minimum of d(r) is attained at r=1.

Now let’s examine Goal 2. For simplicity, we assume \phi(t) and \psi(t) are smooth functions. In this way, the maximum of \phi(t)+r\psi(t) can be solved by differentiation. In fact, this design is sufficiently representative; when some points are not smooth, we can approximate them with smooth functions and take the limit, such as \text{sign}(x)=\lim_{k\to+\infty}\tanh(kx).

Based on this assumption, to find the maximum of \phi(t)+r\psi(t), we first differentiate and set it to 0: \phi'(t)+r\psi'(t)=0 \quad \Rightarrow \quad r = -\frac{\phi'(t)}{\psi'(t)} \triangleq \omega^{-1}(t)\label{eq:max0} Here we assume the above equation has a unique solution, denoted as t=\omega(r). Thus, -\frac{\phi'(t)}{\psi'(t)} is equivalent to \omega^{-1}(t), which is the inverse function of \omega(r) (again, this is the inverse function, not the reciprocal). Meanwhile, since r\in[0,+\infty), then t\in \Omega=\omega([0,\infty)), meaning the range \Omega of t is the image of \omega(r). Furthermore, since it is invertible, \omega(r) must be either strictly monotonically increasing or strictly monotonically decreasing. Without loss of generality, we assume \omega(r) is strictly monotonically increasing, so \omega^{-1}(t) is also strictly monotonically increasing.

A derivative of 0 only indicates that t is an extremum point, but it doesn’t guarantee it is a maximum point. Let’s determine the conditions for it to be a maximum. We have: \phi'(t)+r\psi'(t)=\big(r-\omega^{-1}(t)\big)\psi'(t) Note that r-\omega^{-1}(t) is strictly monotonically decreasing (with respect to t), so it can have only one zero, and it follows a positive-then-negative pattern. To ensure the entire derivative function also has this property, we set \psi'(t) \triangleq \rho(t) > 0, meaning \rho(t) is always positive. Consequently, the derivative of \phi(t)+r\psi(t) is continuous and follows a positive-then-negative pattern, so \phi(t)+r\psi(t) has only one extremum point, and that point is the maximum.

Finding the Minimum via Differentiation

Let’s summarize the results so far: we assume \omega(r) is strictly monotonically increasing, and \rho(t) is always positive for t\in \Omega, satisfying the following relationship: \left\{\begin{aligned}\phi'(t)=&\,-\omega^{-1}(t)\rho(t)\\ \psi'(t)=&\,\rho(t)\end{aligned} \right. In this case, \phi(t)+r\psi(t) has a unique maximum point t=\omega(r), which is also the global maximum. This completes half of Goal 2. Now let’s examine whether d(r)=\phi(\omega(r))+r\psi(\omega(r)) satisfies the remaining property (i.e., the minimum of d(r) is attained at r=1) when t=\omega(r).

Continuing with differentiation: d'(r)=\big[\phi'(\omega(r))+r\psi'(\omega(r))\big]\omega'(r)+\psi(\omega(r))=\psi(\omega(r))\label{eq:d-r} where the second equality holds because the term in the square brackets is 0 according to Equation [eq:max0]. Now only the \psi(\omega(r)) term remains. Since we assumed \psi'(t)=\rho(t) > 0, \psi(t) is strictly monotonically increasing. Since we also assumed \omega(r) is strictly monotonically increasing, the composite function \psi(\omega(r)) is also strictly monotonically increasing.

Now we add one more assumption: \psi(\omega(1))=0. This means d'(1)=0, so r=1 is an extremum point of d(r). Since \psi(\omega(r)) is continuous and strictly monotonically increasing, d'(r) is negative-then-positive. Therefore, r=1 is the local minimum point and also the global minimum point.

The GAN Model is Delivered

At this point, our derivation is complete. The conditions we obtained are:

Conclusion 1 If \omega(r) is strictly monotonically increasing, \Omega=\omega([0,+\infty)), \rho(t) is always positive for t\in \Omega, and the following relationships are satisfied: \left\{\begin{aligned}\phi'(t)=&\,-\omega^{-1}(t)\rho(t)\\ \psi'(t)=&\,\rho(t)\end{aligned} \right. along with the condition \psi(\omega(1))=0, then \phi(t) and \psi(t) satisfying these conditions can be used to construct a GAN model as in [eq:gan].

For example, let’s verify the original version of GAN: \begin{aligned}&\,\min_{q(x)}\max_{t(x)} \mathbb{E}_{x\sim p(x)}[\log (1-\sigma(t(x)))] + \mathbb{E}_{x\sim q(x)}[\log \sigma(t(x))]\\ =&\, \min_{q(x)}\max_{t(x)} \mathbb{E}_{x\sim p(x)}\left[-\log \left(1+e^{t(x)}\right)\right] + \mathbb{E}_{x\sim q(x)}\left[-\log \left(1+e^{-t(x)}\right)\right] \end{aligned}\label{eq:ori-gan} where \sigma(\cdot) is the sigmoid activation function. The above is the binary cross-entropy where the label for real samples is 0 and the label for fake samples is 1. The right side of the equality is the simplified result. From this, we see \phi(t)=-\log \left(1+e^t\right) and \psi(t)=-\log \left(1+e^{-t}\right). Let’s make a small adjustment and let \psi(t)=\log 2-\log \left(1+e^{-t}\right); obviously, this does not change the original optimization problem. Now we have \rho(t)=\psi'(t)=\frac{1}{1+e^t}, which is clearly always greater than 0, and \omega^{-1}(t)=-\phi'(t)/\psi'(t)=e^t, i.e., t=\omega(r)=\log r, which is also clearly strictly monotonically increasing. Finally, verifying \psi(\omega(1)), we find it indeed equals 0.

From this example, we can derive two inferences:

1. The condition \psi(\omega(1))=0 is not strictly necessary. Even if \psi(\omega(1))=0 is not satisfied initially, we can add a constant to \psi(t) to make it satisfy \psi(\omega(1))=0, and adding a constant does not change the original optimization problem.
2. If we swap the labels, letting the label for real samples be 1 and the label for fake samples be 0, the resulting properties are exactly the opposite: the calculated \rho(t) is always negative, and \omega(r) is strictly monotonically decreasing. This shows that Conclusion 1 provided in this article is a sufficient condition for forming a GAN, not a necessary one.

Different forms of \rho(t), \omega(r) can correspond to the same GAN model. For instance, if we choose t=\omega(r)=\frac{r}{r+1}, then r=\omega^{-1}(t)=\frac{t}{1-t}. If we choose \rho(t)=\frac{1}{t}, then: \left\{\begin{aligned}\phi'(t)=&\,-\frac{t}{1-t}\times\frac{1}{t}\\ \psi'(t)=&\,\frac{1}{t}\end{aligned} \right. Integrating gives \phi(t)=\log(1-t) and \psi(t)=\log t. Furthermore, note that \Omega=\omega([0,+\infty))=[0,1), and \rho(t)=\frac{1}{t} excludes t=0, so the feasible domain is (0,1) (in fact, for experiments, boundary points can be ignored). Thus, the derived GAN is: \min_{q(x)}\max_{t(x)\in (0,1)} \mathbb{E}_{x\sim p(x)}[\log (1-t(x))] + \mathbb{E}_{x\sim q(x)}[\log t(x)] This is equivalent to the original GAN, just without explicitly writing out the activation function that ensures t(x)\in (0,1).

Let’s calculate another example. Choose t=\omega(r)=\frac{1}{2}\log r, i.e., r=\omega^{-1}(t)=e^{2t}, and choose \rho(t)=e^{-t}. We can calculate \phi(t)=-e^t, \psi(t)=-e^{-t}. Thus, we get a GAN variant: \min_{q(x)}\max_{t(x)} \mathbb{E}_{x\sim p(x)}\left[-e^{t(x)}\right] + \mathbb{E}_{x\sim q(x)}\left[-e^{-t(x)}\right]

The original paper also used the above Conclusion to calculate many strange GANs. Interested readers can read the original paper themselves; I will not repeat the derivations here.

Connection with f-GAN

In the derivation above, the result d(r) of the \max step has its minimum at r=1. Recalling Goal 1 and substituting d(r) back into Equation [eq:gan], we find that the optimization objective is actually: \int p(x) d\left(\frac{q(x)}{p(x)}\right)dx\label{eq:f-gan} Does it look familiar? Yes, it looks like the definition of f-divergence. And recalling the derivation at [eq:d-r], we know that the derivative of d(r) is strictly monotonically increasing, which means d(r) is a convex function. So the above formula is indeed an f-divergence! In other words, although the authors of this paper took a path that seems completely different, their results can actually be derived from the results of f-GAN. It doesn’t bring anything fundamentally new.

Is this paper completely equivalent to f-GAN? Unfortunately, not yet. The results in the original paper are only a subset of f-GAN. In other words, any GAN model variant it can derive can also be derived by f-GAN, but the reverse is not necessarily true.

This is because, looking back at the entire derivation process, its core idea is to directly generalize the measurement formula of "points" to "functions". For example, generalizing |p-q|=0 \Leftrightarrow p=q at the beginning to \int |p(x)-q(x)|dx=0 \Leftrightarrow p(x)=q(x). Because of this idea, all derivation processes can be carried out using only single-variable calculus. But the problem is that not all conclusions of \int d(p(x),q(x))dx=0 \Leftrightarrow p(x)=q(x) imply d(p,q)=0 \Leftrightarrow p=q. For example, the KL divergence is \int p(x)\log \frac{p(x)}{q(x)}dx, and its being zero implies p(x)=q(x), but p\log\frac{p}{q}=0 does not necessarily imply p=q. Therefore, the original paper at least cannot derive the GAN corresponding to KL divergence.

In that case, does the original paper have no value? If we look only at the "products," it indeed has no value because f-GAN can produce everything it produces. However, we should not only care about the "products"; sometimes we should also care about the "production process." In fact, I believe the academic value of the original paper lies in providing a reference method for analyzing and discovering GANs directly in the dual space, adding another perspective to our understanding of GAN models.

It Can Actually Be Generalized

Whether it’s f-GAN or the original paper, the losses for the generator and discriminator in the derived GAN models are of the same form, just in different directions. But in fact, in the GAN variants we currently use most frequently, the losses for the generator and discriminator are not the same. For example, a more common form of the original GAN is: \begin{aligned}&\max_{t(x)} \mathbb{E}_{x\sim p(x)}[\log (1-\sigma(t(x)))] + \mathbb{E}_{x\sim q(x)}[\log \sigma(t(x))]\\ &\min_{q(x)} \mathbb{E}_{x\sim q(x)}[-\log (1-\sigma(t(x)))] \end{aligned}\label{eq:ori-gan-2} Other examples include LSGAN, Hinge GAN, and so on. Therefore, if we only consider variants where the generator and discriminator losses are of the same form, it is actually not sufficient.

In fact, this paper could have gone one step further to obtain even more results than f-GAN. Unfortunately, the authors seem to have trapped their thinking in a dead end and did not notice this. In fact, achieving this is very simple: in the process above, through the step \max_{t\in\Omega} \phi(t)+r\psi(t), we solved for t=\omega(r) and then substituted it back into the original \phi(t)+r\psi(t) to get d(r). But in fact, we don’t have to substitute it back into the original formula! We can substitute it back into any formula of the form \alpha(t)+r\beta(t). According to Equation [eq:f-gan] and combined with the requirements for f-divergence listed at the beginning, as long as d(r)=\alpha(\omega(r))+r\beta(\omega(r)) is a convex function (where d(1)=0 can be achieved through translation), or according to the previous reasoning, d(r) is any function with its minimum at r=1. Summarizing:

Conclusion 2 If \omega(r) is strictly monotonically increasing, \Omega=\omega([0,+\infty)), \rho(t) is always positive for t\in \Omega, and the following relationships are satisfied: \left\{\begin{aligned}\phi'(t)=&\,-\omega^{-1}(t)\rho(t)\\ \psi'(t)=&\,\rho(t)\end{aligned} \right. and functions \alpha(t), \beta(t) are such that d(r)=\alpha(\omega(r))+r\beta(\omega(r)) is a convex function, or such that d(r) is a function with its minimum at r=1, then \phi(t), \psi(t), \alpha(t), \beta(t) satisfying these conditions can be used to construct the following GAN model (where the \min_{q(x)} part has omitted the \alpha(t) part which is independent of q(x)): \begin{aligned}\max_{t(x)\in \Omega} &\, \mathbb{E}_{x\sim p(x)}[\phi(t(x))] + \mathbb{E}_{x\sim q(x)}[\psi(t(x))]\\ \min_{q(x)}&\, \mathbb{E}_{x\sim q(x)}[\beta(t(x))] \end{aligned}

Some Examples After Generalization

Using Conclusion 2 to construct GANs is quite flexible. It can construct many examples that f-GAN cannot find because it allows the generator and discriminator losses to be inconsistent.

For example, when calculating the original GAN, we got t=\omega(r)=\log r, and r\log r happens to be a convex function. So we can let \alpha(t)=0 and \beta(t)=t (Note: \alpha(t) can be 0, but \beta(t) cannot; think about why?), yielding d(r)=r\log r. The GAN at this time is: \begin{aligned}\max_{t(x)} &\, \mathbb{E}_{x\sim p(x)}[\log (1-\sigma(t(x)))] + \mathbb{E}_{x\sim q(x)}[\log \sigma(t(x))]\\ \min_{q(x)}&\, \mathbb{E}_{x\sim q(x)}[t(x)] \end{aligned} This is a very useful GAN variant. Also, since r\log r is a convex function, (1+r)\log(1+r) is also convex. Then we can let \alpha(t)=\beta(t)=\log(1+r)=\log(1+e^t), which corresponds exactly to d(r)=(r+1)\log(1+r). Since \log(1+e^t)=-\log(1-\sigma(t)), the corresponding GAN is [eq:ori-gan-2], which is the more commonly used original version of GAN and is more effective than [eq:ori-gan].

Let’s take another example. Let t=\omega(r)=\frac{a + b r}{1 + r}. Here we assume b > a, so \Omega=(a,b) (ignoring boundary points for now), and r=\omega^{-1}(t)=\frac{t-a}{b-t}. We take \rho(t)=2(b-t), which satisfies the requirement of being always positive. This yields \phi(t)=-(t-a)^2 and \psi(t)=-(t-b)^2. Next, take d(r)=\frac{(r-1)^2}{r+1}, which clearly reaches its minimum at r=1. Then let \alpha(t)=\beta(t) and try to reverse-engineer the form of \beta(t), i.e., d(r)=(1+r)\beta(t), which leads to \beta(t)=\left(\frac{r-1}{r+1}\right)^2. Substituting r=\frac{t-a}{b-t} gives \beta(t)=\left(\frac{2}{b-a}t-\frac{a+b}{b-a}\right)^2. For simplicity, we can let b-a=2, then \beta(t)=\left(t-\frac{a+b}{2}\right)^2. The final GAN form is: \begin{aligned}\max_{t(x)\in (a,b)} &\, \mathbb{E}_{x\sim p(x)}\left[-(t-a)^2\right] + \mathbb{E}_{x\sim q(x)}\left[-(t-b)^2\right]\\ \min_{q(x)}&\, \mathbb{E}_{x\sim q(x)}\left[\left(t-\frac{a+b}{2}\right)^2\right] \end{aligned} This is actually LSGAN. Readers might be confused because LSGAN doesn’t have the restriction t(x)\in (a,b). In fact, this restriction can be removed because even after removing it, the optimal solution still lies within (a,b).

Clearly, these GAN variants obtained based on the generalized Conclusion 2 are very valuable, and these variants cannot be obtained by f-GAN. Therefore, if the original paper could supplement this part of the generalization, it would be quite elegant.