Whenever I see two seemingly unrelated branches of mathematics ingeniously connected, I am always struck with wonder by the magical beauty of mathematics. Long ago, when I saw how number theory problems were combined with complex variable methods through the generating function method—giving rise to the wonderful field of "Analytic Number Theory"—I marveled at the elegance of generating functions! Unfortunately, I never got around to properly organizing and writing about those topics. Today, while reading Mr. Tsung-Dao Lee’s Mathematical Methods in Physics, I saw how he connected complex variables with random walks in a masterfully crafted way, which once again left me in awe. Finally, unable to suppress my excitement, I am writing here about probability theory and generating functions.
When number theory and complex variables combine, they form "Analytic Number Theory." Following this logic, the combination of probability and complex variables should result in a field called "Analytic Probability." However, when searching online, I did not find this term in common use. Nevertheless, I will use this term in this article. Although the name is not popular, the methods of analytic probability are not new; they can be traced back to the great mathematician Laplace and his work Théorie Analytique des Probabilités. Despite this, it seems this ingenious and beautiful method has not received the full recognition it deserves.
I believe that even as a concise computational tool, the beautiful technique of the generating function method should be known as widely as possible by science enthusiasts, let alone by students of mathematics.
Discrete Probability
Suppose the possible values of a random variable X are discrete: x_1, x_2, x_3, \dots. For simplicity here, we consider X to take positive integer values (discrete non-integers can be considered similarly). The number of values for X can be finite or infinite. Their corresponding probabilities are p_1 = P(X=x_1), p_2 = P(X=x_2), p_3 = P(X=x_3), \dots. Then, we consider the generating function (or "mother function"): f(z) = \sum_{n} p_n z^{x_n} In other words, the coefficient of the z^{\alpha} term in f(z) is the probability that X takes the value \alpha.
For example, if P(X=n) = 2^{-n} for n=1, 2, 3, \dots, then the corresponding generating function is: f(z) = \sum_{n=1}^{\infty} 2^{-n} z^n = \frac{1}{1-z/2} - 1
The generating function of a probability distribution has the following basic properties:
f(1)=1: This represents the sum of the probabilities of all possible values, which is, of course, 1.
The expected value of X: E(X) = \left. z \frac{d}{dz} f(z) \right|_{z=1} = \left. \frac{d}{d(\ln z)} f(z) \right|_{z=1} The derivative above is known as the "logarithmic derivative." This equality can be derived simply by substituting the definition of expectation.
Furthermore: E(X^n) = \left. \left( \frac{d}{d(\ln z)} \right)^n f(z) \right|_{z=1} If f(z) and g(z) represent the generating functions of the probability distributions of two independent trials, then the generating function of the probability distribution after both trials is f(z)g(z). This can also be derived from the definition.
For example, in a single trial where the probability of success is p, success takes the value 1 and failure takes the value 0. The corresponding generating function is p z^1 + (1-p) z^0 = pz + (1-p). After n trials, the generating function is: f(z) = [pz + (1-p)]^n Expanding this expression, we get: f(z) = \sum_{i=0}^n \binom{n}{i} p^i (1-p)^{n-i} z^i This tells us the probability of X taking a value after n trials: P(X=i) = \binom{n}{i} p^i (1-p)^{n-i} And it is easy to calculate: \begin{aligned} E(X) &= \left. \frac{d}{d(\ln z)} [pz + (1-p)]^n \right|_{z=1} = \left. npz[pz + (1-p)]^{n-1} \right|_{z=1} = np \\ E(X^2) &= \left. \frac{d}{d(\ln z)} \{npz[pz + (1-p)]^{n-1}\} \right|_{z=1} = np + n(n-1) p^2 \end{aligned} Therefore: Var(X) = np + n(n-1)p^2 - (np)^2 = np(1-p) And so on. These are our basic results regarding the Binomial Distribution.
Continuous Probability
The above method can potentially be extended to continuous probability. Readers might wonder: the discussion above is based on Taylor series, which are discrete; how can they be extended to the continuous case? However, for mathematics, the most magical part lies in the fact that "after breaking through conventional thinking, one often sees another vast horizon." To this end, let us first define the following probability event. Suppose the possible values of X are any real numbers in (-\infty, +\infty), and P(x \leq X \leq x+dx) = P(x)dx. That is, P(x) is the probability density. Following the form of the generating function above, we consider: f(z) = \lim_{dx \to 0} \sum_{x} P(x) z^x dx = \int_{-\infty}^{+\infty} P(x) z^x dx Using an integral to replace the summation should not be surprising, but we haven’t seen this specific form of function before. However, since it is still a power series, there is no reason we must use the notation z. Let’s change the notation and replace z with e^{-i\omega}. The previous discussion still holds, and we find that f(z) becomes: F(\omega) = \int_{-\infty}^{+\infty} P(x) e^{-i\omega x} dx This is the Fourier transform of P(x)!
In this way, the Fourier transform gains a clear meaning in probability theory: it represents the generating function of a probability distribution. Therefore, it is easy to see that if we know the image after the Fourier transform, we can know the original function, as both contain the same information. The formula for finding the original probability distribution from F(\omega) is precisely the inverse Fourier transform: P(x) = \frac{1}{2\pi} \int_{-\infty}^{+\infty} F(\omega) e^{i\omega x} d\omega
Correspondingly, the generating function (Fourier transform) for continuous probability also has several similar properties:
Total probability F(0)=1;
For the expectation: E(X) = \left. i \frac{d}{d\omega} F(\omega) \right|_{\omega=0}
For the moments of various orders: E(X^n) = \left. i^n \frac{d^n}{d\omega^n} F(\omega) \right|_{\omega=0}
Later, we will attempt to use the method of analytic probability to explore the problem of random walks (Brownian motion). We will see that using this method to investigate random walks is remarkably and delightfully simple.
When reposting, please include the original address of this article: https://kexue.fm/archives/2550
For more detailed information regarding reposting, please refer to: Scientific Space FAQ