In the previous articles of this series, we have understood VAEs from multiple perspectives. Generally, VAEs are used to obtain a generative model or to create a better encoding model; these are the conventional uses of VAEs. However, besides these common applications, there are some "niche requirements," such as using them to estimate the probability density of x, which is often used in data compression.
In this article, we will understand and derive the VAE model from the perspective of estimating probability density.
Two Problems
The so-called estimation of probability density involves fitting a batch of samples x_1, x_2, \dots, x_N \sim \tilde{p}(x) with a parameterized family of probability densities q_{\theta}(x). The goal of fitting is generally to minimize the negative log-likelihood: \mathbb{E}_{x\sim \tilde{p}(x)}[-\log q_{\theta}(x)] = -\frac{1}{N}\sum_{i=1}^N \log q_{\theta}(x_i) \label{eq:mle} However, this is purely a theoretical form, and many problems remain unsolved, which can be categorized into two major issues:
What kind of q_{\theta}(x) should be used for fitting?
What method should be used to solve the aforementioned objective?
Mixture Models
Regarding the first question, we naturally hope that the fitting capability of q_{\theta}(x) is as strong as possible, ideally having the ability to fit any probability distribution. Unfortunately, although neural networks theoretically have universal approximation capabilities, that refers to the ability to fit functions, not necessarily probability distributions. A probability distribution must satisfy q_{\theta}(x) \geq 0 and \int q_{\theta}(x) dx = 1, the latter of which is usually difficult to guarantee.
Since a direct approach is difficult, we think from an indirect perspective and construct a mixture model: q_{\theta}(x) = \int q_{\theta}(x|z)q(z)dz = \mathbb{E}_{z\sim q(z)}[q_{\theta}(x|z)] \label{eq:q} Where q(z) is usually chosen as a simple parameter-free distribution, such as a standard normal distribution; while q_{\theta}(x|z) is a parameterized simple distribution conditioned on z, such as a standard normal distribution whose mean and variance are related to z.
From the perspective of generative models, the above model is interpreted as a two-step operation: first sampling z from q(z), and then passing it into q_{\theta}(x|z) to generate x. However, the focus of this article is on estimating probability density. The reason we choose such a q_{\theta}(x|z) is that it has sufficient capacity to fit complex distributions. The final q_{\theta}(x) is represented as the average of multiple simple distributions q_{\theta}(x|z). Readers familiar with Gaussian Mixture Models (GMM) should know that such models possess very strong fitting capabilities, even theoretically being able to fit any distribution, so the fitting capacity of the distribution is guaranteed.
Importance Sampling
However, the integral in Equation [eq:q] cannot be easily calculated. Or rather, only distributions that cannot be expressed simply and explicitly possess sufficiently strong fitting capabilities. Therefore, if we want to estimate it, we must do so by sampling according to \mathbb{E}_{z\sim q(z)}[q_{\theta}(x|z)]. However, in practical scenarios, the dimensions of z and x are relatively high, and high-dimensional spaces suffer from the "curse of dimensionality." This means that in high-dimensional space, even if we sample millions or tens of millions of points, it is difficult to sufficiently cover the space, making it hard to accurately estimate \mathbb{E}_{z\sim q(z)}[q_{\theta}(x|z)].
To address this, we need to find a way to narrow down the sampling space. First, we usually control the variance of q_{\theta}(x|z) to be relatively small. Consequently, for a given x, there will not be many z values that make q_{\theta}(x|z) large; for most z, the calculated q_{\theta}(x|z) will be very close to zero. Thus, we only need to find a way to sample z values that make q_{\theta}(x|z) relatively large to obtain a good estimate of \mathbb{E}_{z\sim q(z)}[q_{\theta}(x|z)].
Specifically, we introduce a new distribution p_{\theta}(z|x), assuming that the z values making q_{\theta}(x|z) large follow this distribution. Thus, we have: q_{\theta}(x) = \int q_{\theta}(x|z)q(z)dz = \int q_{\theta}(x|z)\frac{q(z)}{p_{\theta}(z|x)}p_{\theta}(z|x)dz = \mathbb{E}_{z\sim p_{\theta}(z|x)}\left[q_{\theta}(x|z)\frac{q(z)}{p_{\theta}(z|x)}\right] In this way, we transform "aimless" sampling from q(z) into more targeted sampling from p_{\theta}(z|x). Since the variance of q_{\theta}(x|z) is controlled to be small, the variance of p_{\theta}(z|x) will naturally not be large, and sampling efficiency is improved. Note that in the generative model perspective, p_{\theta}(z|x) is viewed as an approximation of the posterior distribution, but from the perspective of estimating probability density, it is purely an importance weighting function and does not require special interpretation of its meaning.
Training Objective
At this point, we have solved the first problem: what distribution to use and how to better calculate this distribution. The remaining question is how to train it.
In fact, with the concept of importance sampling, we don’t need to consider things like the ELBO (Evidence Lower Bound). We can directly use the objective in Equation [eq:mle]. Substituting the expression for q_{\theta}(x), we get: \mathbb{E}_{x\sim \tilde{p}(x)}\left[-\log \mathbb{E}_{z\sim p_{\theta}(z|x)}\left[q_{\theta}(x|z)\frac{q(z)}{p_{\theta}(z|x)}\right]\right] In fact, if we sample only one z through reparameterization in the \mathbb{E}_{z\sim p_{\theta}(z|x)} step, the training objective becomes: \mathbb{E}_{x\sim \tilde{p}(x)}\left[-\log q_{\theta}(x|z)\frac{q(z)}{p_{\theta}(z|x)}\right],\quad z\sim p_{\theta}(z|x) This is already the conventional VAE training objective. If we sample M > 1 points, then it becomes: \mathbb{E}_{x\sim \tilde{p}(x)}\left[-\log \left(\frac{1}{M}\sum_{i=1}^M q_{\theta}(x|z_i)\frac{q(z_i)}{p_{\theta}(z_i|x)}\right)\right],\quad z_1,z_2,\dots,z_M\sim p_{\theta}(z|x) This is the "Importance Weighted Autoencoder," which comes from "Importance Weighted Autoencoders" and is regarded as an enhancement of the VAE. Overall, through the perspective of importance sampling, we can bypass the tedious derivations of the traditional VAE’s ELBO and avoid the joint distribution perspective introduced in "Variational Autoencoders (Part 2): From a Bayesian Perspective", directly obtaining the VAE model and even its improved versions.
Summary
This article introduced the Variational Autoencoder (VAE) from the starting point of estimating the probability density of samples. Combined with importance sampling, we can obtain a rapid derivation of the VAE, completely avoiding many tedious details such as the ELBO.
Original Address: https://kexue.fm/archives/8791
For more details on reprinting, please refer to: "Scientific Space FAQ"