Introduction to f-GAN: A Production Workshop for GAN Models

f-divergence

First, let’s provide a basic introduction to f-divergence. The so-called f-divergence is a generalization of the KL divergence: \mathcal{D}_f(P\Vert Q) = \int q(x) f\left(\frac{p(x)}{q(x)}\right)dx\label{eq:f-div} Note that according to the common convention, the term inside the parentheses is p/q rather than q/p. Do not naturally assume it is q/p based on the form of the KL divergence.

It can be seen that this form covers many measures between probability distributions that we have encountered. Here, I will directly bring in the table from the paper (partially):

Convex Functions

A natural question arises: what are the common characteristics of the various f functions listed above?

The answer is:

1. They are mappings from non-negative real numbers to real numbers (\mathbb{R}^* \to \mathbb{R});

2. f(1)=0;

3. They are all convex functions.

The first point is standard, and the second point f(1)=0 ensures that \mathcal{D}_f(P\Vert P)=0. How should we understand the third point about convex functions? It is actually a basic application of the properties of convex functions, specifically Jensen’s inequality: \mathbb{E}\big[f(x)\big]\geq f\big(\mathbb{E}[x]\big)\label{eq:tuhanshu-xingzhi} In other words, "the average of the function is greater than or equal to the function of the average." Some tutorials use this property as the definition of a convex function. If f(u) is a smooth function, we generally determine if it is convex by checking if the second derivative f''(u) is always greater than or equal to 0.

Using [eq:tuhanshu-xingzhi], we have: \begin{aligned}\int q(x) f\left(\frac{p(x)}{q(x)}\right)dx =& \mathbb{E}_{x\sim q(x)} \left[f\left(\frac{p(x)}{q(x)}\right)\right]\\ \geq& f\left(\mathbb{E}_{x\sim q(x)} \left[\frac{p(x)}{q(x)}\right]\right)\\ =& f\left(\int q(x) \frac{p(x)}{q(x)}dx\right)\\ =& f\left(\int p(x)dx\right)\\ =& f(1) = 0 \end{aligned} That is to say, these three conditions ensure that the f-divergence is non-negative, and when the two distributions are identical, the f-divergence is 0. This allows \mathcal{D}_f to be used to measure the difference between distributions. Of course, the f-divergence does not strictly guarantee that \mathcal{D}_f(P\Vert Q) > 0 when P \neq Q in principle. However, we usually choose strictly convex f (i.e., f''(u) > 0 everywhere), which guarantees \mathcal{D}_f(P\Vert Q) > 0 when P \neq Q. In this case, \mathcal{D}_f(P\Vert Q)=0 \Leftrightarrow P=Q. (Note: Even so, in most cases, \mathcal{D}_f(P\Vert Q) is still not a "distance" satisfying the axiomatic definition, but this is not closely related to the theme of this article and is mentioned only in passing.)

Convex Conjugate

Now let’s discuss convex functions from a more mathematical perspective. Generally, let the domain of the convex function be \mathbb{D} (for this article, \mathbb{D}=\mathbb{R}_+). Choose any point \xi and find the tangent line of y=f(u) at u=\xi: y = f(\xi) + f'(\xi)(u - \xi) Consider the difference function: h(u) = f(u) - f(\xi) - f'(\xi)(u - \xi) Intuitively, a convex function is one whose graph always lies above its (any) tangent line. Therefore, for a convex function, the following inequality always holds: f(u) - f(\xi) - f'(\xi)(u - \xi)\geq 0 Rearranging this gives: f(u) \geq f(\xi) - f'(\xi) \xi + f'(\xi)u Since the inequality holds universally and the equality can be achieved, we can derive: f(u) = \max_{\xi\in\mathbb{D}}\big\{f(\xi) - f'(\xi) \xi + f'(\xi)u\big\} Using new notation, let t=f'(\xi), and solve for \xi (for a convex function, this can always be done; readers can try to prove this themselves), then let: g(t) = - f(\xi) + f'(\xi) \xi Then we have: f(u) = \max_{t\in f'(\mathbb{D})}\big\{t u - g(t)\big\} Here g(t) is called the conjugate function of f(u). Note the expression inside the braces: once f is given, g is also determined, and the entire expression is linear with respect to u. Overall, we have done the following:

We have provided a linear approximation for a convex function, and by maximizing the parameters inside, we can reach the original value.

Note that for a given u, we must maximize t once to get a result as close as possible to f(u); otherwise, substituting an arbitrary t only guarantees a lower bound without ensuring the error size. This is why it is called the "local variational method," because maximization (variation) must be performed at every point (locally). In this way, we can understand t as actually being a function of u, i.e.: f(u) = \max_{T\text{ is a function with range }f'(\mathbb{D})}\big\{T(u) u - g(T(u))\big\}\label{eq:max-conj}

The above discussion actually provides the method for calculating the convex conjugate. Here we directly provide the conjugate functions for the convex functions in the previous table.

(Note: Here W is the Lambert W function.)

f-divergence Estimation

What is the difficulty in calculating f-divergence? According to the definition [eq:f-div], we need to know both probability distributions P and Q simultaneously to calculate their f-divergence. However, in machine learning, this is difficult to achieve. Sometimes we only know the analytical form of one probability distribution at most, while the other distribution only has sampled data; in many cases, we don’t know either distribution and only have corresponding samples (meaning we need to compare the similarity between two sets of samples). Therefore, we cannot directly calculate f-divergence according to [eq:f-div].

Combining [eq:f-div] and [eq:max-conj], we get: \begin{aligned}\mathcal{D}_f(P\Vert Q) =& \max_{T}\int q(x) \left[\frac{p(x)}{q(x)}T\left(\frac{p(x)}{q(x)}\right)-g\left(T\left(\frac{p(x)}{q(x)}\right)\right)\right]dx\\ =& \max_{T}\int\left[p(x)\cdot T\left(\frac{p(x)}{q(x)}\right)-q(x)\cdot g\left(T\left(\frac{p(x)}{q(x)}\right)\right)\right]dx\end{aligned} Denoting T\left(\frac{p(x)}{q(x)}\right) as a single function T(x), we have: \mathcal{D}_f(P\Vert Q) = \max_{T}\Big(\mathbb{E}_{x\sim p(x)}[T(x)]-\mathbb{E}_{x\sim q(x)}[g(T(x))]\Big)\label{eq:f-div-e} Equation [eq:f-div-e] is the basic formula for estimating f-divergence. It means: Sample from the two distributions separately, then calculate the average values of T(x) and g(T(x)) respectively, and optimize T to make their difference as large as possible. The final result is the approximate value of the f-divergence. Obviously, T(x) can be fitted with a sufficiently complex neural network; we only need to optimize the parameters of the neural network.

Note that in the discussion of convex functions, when maximizing the objective, there are restrictions on the range of T. Therefore, in the last layer of T, we must design an appropriate activation function so that T satisfies the required range. Of course, the choice of activation function is not unique; some reference activation functions are listed in the previous table. Although the choice of activation function is theoretically arbitrary, for ease of optimization, several principles should be followed:

1. The corresponding domain should be \mathbb{R}, and the range should be the required range (boundary points can be ignored);

2. It is best to choose a globally smooth function rather than simple truncation. For example, if the required range is \mathbb{R}_+, do not use relu(x) directly; consider e^x or \log(1+e^x);

3. Note that the second term of equation [eq:f-div-e] contains g(T(x)), which is a composite calculation of g and T. Therefore, when choosing an activation function, it is best to make its composition with g relatively simple.

GAN Wholesale

After all this discussion, we have almost reached the end of the article, yet we haven’t formally mentioned GANs. In fact, GANs can be considered a byproduct of the entire article.

GANs aim to train a generator that maps a Gaussian distribution to the distribution of our desired dataset. This requires comparing the difference between two distributions. After the previous process, this is actually very simple: just pick any f-divergence. Then use equation [eq:f-div-e] to estimate the f-divergence. Once estimated, we have a model for the f-divergence. Doesn’t the generator want to reduce the difference between distributions? Just minimize the f-divergence. So, written as an expression, it is: \min_G\max_{T}\Big(\mathbb{E}_{x\sim p(x)}[T(x)]-\mathbb{E}_{x=G(z),z\sim q(z)}[g(T(x))]\Big)\label{eq:f-div-gan} Or vice versa: \min_G\max_{T}\Big(\mathbb{E}_{x=G(z),z\sim q(z)}[T(x)]-\mathbb{E}_{x\sim p(x)}[g(T(x))]\Big)\label{eq:f-div-gan-2} And that’s it!

Need a few examples? Okay, let’s look at the JS divergence first. Substituting everything into the formula step by step, you will find the final result is (omitting the constant term of \log 2): \min_G\max_{D}\Big(\mathbb{E}_{x\sim p(x)}[\log D(x)] + \mathbb{E}_{x=G(z),z\sim q(z)}[\log(1-D(x))]\Big) where D is activated by \sigma(x)=1/(1+e^{-x}). This is the original version of GAN.

Try the Hellinger distance? The result is: \min_G\max_{D}\Big(-\mathbb{E}_{x\sim p(x)}[e^{D(x)}] - \mathbb{E}_{x=G(z),z\sim q(z)}[e^{-D(x)}]\Big) where D(x) has a linear activation. This doesn’t seem to have a name yet? However, experiments have already been conducted on it in the paper.

What about KL divergence? Since KL divergence is asymmetric, there are two results: \min_G\max_{D}\Big(\mathbb{E}_{x\sim p(x)}[D(x)] - \mathbb{E}_{x=G(z),z\sim q(z)}[e^{D(x)-1}]\Big) or \min_G\max_{D}\Big(\mathbb{E}_{x=G(z),z\sim q(z)}[D(x)] - \mathbb{E}_{x\sim p(x)}[e^{D(x)-1}]\Big) where D(x) also has a linear activation.

Well, I won’t give any more examples. In fact, these f-divergences are essentially similar, and it’s hard to see much difference in effect. However, it can be noted that JS divergence and Hellinger distance are both symmetric and bounded, which are very good properties that we will use later.