If you have been following Scientific Space, you might have noticed that I have always had a preference for extremum principles. For instance, I previously wrote a series of articles titled Natural Extremum to introduce some extremum problems and the calculus of variations. In physics, I favor the form of the Principle of Least Action. In data mining, I am deeply interested in the Maximum Entropy Model based on the Maximum Entropy Principle. Recently, while working through the exercises in Quantum Mechanics and Path Integrals, I also developed a great interest in the variational principle discussed in Chapter 11.
For anything new, my learning method is to understand its principles and ideas with an example as simple as possible, and then gradually increase the complexity, so that I do not get lost. The variational principle is a powerful method for estimating path integrals. Since path integrals are functional integrals—or infinite-dimensional integrals—it is natural to wonder if there is a similar estimation principle for finite-dimensional integrals, such as the simplest one-dimensional integral. In fact, there is, and it is not complicated. Understanding it helps in grasping the core idea of the variational principle. Unfortunately, I have not found existing materials describing this simplified version of the principle, which may be due to the limited resources I have consulted.
Starting from Gaussian-type Integrals
The variational principle is essentially an application of Jensen’s inequality. We start with the following integral: \label{jifen} I(\epsilon)=\int_{-\infty}^{\infty}e^{-x^2-\epsilon x^4}dx We have previously studied this integral from the perspective of perturbation expansion. This integral is often regarded as a one-dimensional version of complex path integrals; generally, one can obtain effective ideas for studying path integrals by examining it. To estimate the above integral, a natural thought is whether it can be represented by a simple Gaussian integral: \label{jinsi} \int_{-\infty}^{\infty}e^{-px^2}dx=\sqrt{\frac{\pi}{p}} The key question is how to choose p such that [jinsi] is as close as possible to [jifen]. We rewrite [jifen] as: \label{bianhuan} I(\epsilon)=\sqrt{\frac{\pi}{p}}\int_{-\infty}^{\infty}e^{(p-1)x^2-\epsilon x^4}\left(\sqrt{\frac{p}{\pi}}e^{-px^2}\right)dx How should we view [bianhuan]? I have deliberately written it in a specific form—from a statistical perspective, it is the average of e^{(p-1)x^2-\epsilon x^4} under the weighting \sqrt{\frac{p}{\pi}}e^{-px^2}. Therefore, we might as well write it as: \label{pingjun} I(\epsilon)=\sqrt{\frac{\pi}{p}}\left\langle e^{(p-1)x^2-\epsilon x^4}\right\rangle To remind you again, this integral is the average under the weight \sqrt{\frac{p}{\pi}}e^{-px^2}. Up to this point, what we have done is merely a formal transformation; there has been no substantial progress, as we still haven’t calculated the integral we need.
Now let us recall Jensen’s inequality. It states that if f(x) is a convex function (definitions of convexity may vary across different books; here, a convex function refers to one where the second derivative is non-negative), then: \label{jensen} f(\langle x \rangle)\leq \langle f(x) \rangle Readers might usually see Jensen’s inequality in a different form, but please consider it carefully: doesn’t Jensen’s inequality express exactly this meaning? Jensen’s inequality has a clear physical interpretation: if several points lie on the graph of a convex function, then their centroid lies above the function’s graph.
Using Jensen’s inequality, we can further simplify [pingjun]. Since e^x is clearly a convex function, we have: \label{pingjun-jensen} I(\epsilon)=\sqrt{\frac{\pi}{p}}\left\langle e^{(p-1)x^2-\epsilon x^4}\right\rangle \geq \sqrt{\frac{\pi}{p}}e^{\left\langle (p-1)x^2-\epsilon x^4\right\rangle} We can calculate \left\langle (p-1)x^2-\epsilon x^4\right\rangle as follows: \left\langle (p-1)x^2-\epsilon x^4\right\rangle=\int_{-\infty}^{\infty}\left[(p-1)x^2-\epsilon x^4\right]\left(\sqrt{\frac{p}{\pi}}e^{-px^2}\right)dx=\frac{2 (p-1) p-3 \epsilon}{4 p^2} Thus: \label{pingjun-guji} I(\epsilon)\geq\sqrt{\frac{\pi}{p}}\exp\left[\frac{2 (p-1) p-3 \epsilon}{4 p^2}\right] Regardless of the value of p, the rightmost side will always be less than or equal to I(\epsilon). Naturally, to get as close as possible to I(\epsilon), we should choose p such that the right side of the inequality is maximized! This is the origin of the extremum principle. By differentiating, we find that the right side reaches its maximum when: p=\frac{1}{2} \left(\sqrt{12 \epsilon+1}+1\right) Substituting this into [pingjun-guji], we obtain: \label{pingjun-guji2} I(\epsilon)\geq \sqrt{\frac{2 \pi }{\sqrt{12 \epsilon+1}+1}}\exp\left[\frac{3 \epsilon}{\left(\sqrt{12 \epsilon+1}+1\right)^2}\right]=\hat{I}(\epsilon) Below are some results:
| \epsilon=0 | \epsilon=1 | \epsilon=10 | \epsilon=100 | |
|---|---|---|---|---|
| I(\epsilon) | 1.77245 | 1.36843 | 0.921961 | 0.554577 |
| \hat{I}(\epsilon) | 1.77245 | 1.34547 | 0.891204 | 0.531509 |
Comparing these with the results in http://kexue.fm/archives/3280/, we can see that \hat{I}(\epsilon) is a quite good estimate, and it is indeed true that \hat{I}(\epsilon)\leq I(\epsilon).
My Summary
From the process above, we can see some of the power of the extremum principle: it provides a good estimate of the integral that is stable; regardless of the magnitude of the parameter \epsilon, a reasonable estimate can be obtained. Secondly, it does not just provide an estimate of the integral, but the result is a lower bound of the integral, which is quite valuable. Generally, we can obtain an effective estimate of an expression through various methods, but it is often difficult to determine the relationship between the estimated value and the exact value (unless accuracy is significantly sacrificed). The extremum principle achieves both—maintaining high accuracy while providing a bound.
The extremum principle involved in this article originates from Jensen’s inequality. The core lies in the fact that e^x is a convex function, and e^{\left\langle h(x)\right\rangle} is often easier to calculate than \left\langle e^{h(x)}\right\rangle. Therefore, the direction for generalization is obvious: replace the weighting with something else, or replace e^x with a general convex function, as long as f(\left\langle h(x)\right\rangle) is easier to calculate than \left\langle f(h(x))\right\rangle.
Of course, the disadvantages of the extremum principle are also very apparent. First, it is not easy to find approximation functions that are both good and integrable. Second, even if the integration can be performed, the subsequent process of finding the extremum is not necessarily simple, as it involves solving systems of non-linear equations. Once approximations are used, it often degenerates into the results of perturbation expansion, losing the unique advantages of the variational method. Finally, the estimation from the extremum principle has a less obvious drawback: it is difficult to further improve the accuracy.
When reposting, please include the address of this article: https://kexue.fm/archives/3630
For more detailed reposting matters, please refer to: Scientific Space FAQ