This article is very short, mainly describing a very useful, not complicated, but surprisingly long-overlooked fact.
In the article Mutual Information in Deep Learning: Unsupervised Feature Extraction, we obtained the final loss for the Deep INFOMAX model through a weighted combination of a prior distribution loss and a maximized mutual information loss. While that story was complete, in a sense, it was just a cobbled-together loss. This article aims to prove that the loss can be naturally derived from Variational Autoencoders (VAE).
Process
To repeat once more, the loss that the Variational Autoencoder (VAE) needs to optimize is: \begin{aligned} &KL(\tilde{p}(x)p(z|x)\Vert q(z)q(x|z))\\ =&\iint \tilde{p}(x)p(z|x)\log \frac{\tilde{p}(x)p(z|x)}{q(x|z)q(z)} dzdx \end{aligned} Related discussions have appeared many times on this blog. VAE includes both an encoder and a decoder. If we only need to encode features, training a decoder seems redundant. Therefore, the focus is on how to remove the decoder.
It’s actually quite simple: split the VAE loss into two parts: \begin{aligned} &KL(\tilde{p}(x)p(z|x)\Vert q(z)q(x|z))\\ =&\iint \tilde{p}(x)p(z|x)\log \frac{p(z|x)}{q(z)} dzdx-\iint \tilde{p}(x)p(z|x)\log \frac{q(x|z)}{\tilde{p}(x)} dzdx \end{aligned} The first term is the KL divergence of the prior distribution, and isn’t the \log \frac{q(x|z)}{\tilde{p}(x)} in the second term actually the pointwise mutual information of x and z? If q(x|z) has infinite fitting capacity, eventually we will inevitably have \tilde{p}(x)p(z|x) = q(x|z)p(z) (Bayes’ rule), so the second term is: KL(q(x|z)p(z)\Vert \tilde{p}(x)p(z))=KL(\tilde{p}(x)p(z|x)\Vert \tilde{p}(x)p(z)) which is the mutual information between the two random variables x and z. The negative sign in front means we want to maximize the mutual information.
The rest of the process is the same as in Mutual Information in Deep Learning: Unsupervised Feature Extraction, and is omitted here.
Conclusion
As stated at the beginning, this article is very short and concise. Its main purpose is to provide a new understanding of the Variational Autoencoder loss (minimizing prior distribution + maximizing mutual information), which then naturally leads to the loss for Deep INFOMAX.
If I hadn’t already written Mutual Information in Deep Learning: Unsupervised Feature Extraction, I would have definitely used this starting point to explain Deep INFOMAX. However, since that article has been out for a few days, I had to write this short post as a supplement.
When reposting, please include the address of this article: https://kexue.fm/archives/6088
For more detailed reposting matters, please refer to: Scientific Space FAQ