Statistics is the process of estimating a true distribution through a large number of samples. A term that often accompanies statistics is “smoothing,” which refers to the process of discounting or adjusting statistical results. The idea behind smoothing is that if the sample space is very large, the statistical results will be sparse. Due to various accidental factors, small statistical counts become unreliable. For instance, a frequency of 1 might just be a coincidental result, and its actual probability is not necessarily close to 1/N; similarly, a frequency of 0 does not necessarily mean the event will never occur. Therefore, we need to smooth the statistical results to make the conclusions more reliable.
There are many methods for smoothing. Here, we introduce a smoothing formula based on the forgetting hypothesis. Suppose our task is to count the frequency of each character from a corpus. We simulate the process of forgetting in the human brain: assume that every time a character appears, the amount of memory in our brain increases by 1. However, if the character does not appear within a certain period (regardless of how large this period is for now), the memory in the brain decreases to a proportion \beta of its previous value. Assuming the character appears periodically, the memory amount A_n satisfies the following recurrence relation: A_{n+1} = \beta A_n + 1 The general solution to this recurrence is: A_n = \frac{1-\beta^n}{1-\beta} Its limit is: A = \frac{1}{1-\beta} We consider this limit to be our true statistical result—that is, the result after smoothing.
If the total number of characters in a corpus is N and the frequency of a certain character is F, assuming the characters are uniformly distributed, the appearance period of that character would be N/F. We can assume that the law of forgetting follows an exponential decay (guessed from the Ebbinghaus curve; see: https://en.wikipedia.org/wiki/Forgetting_curve, or readers may speculate other forms of forgetting laws). Thus, there exists a constant \alpha < 1 such that: \beta = \alpha^{N/F} In this way, the data smoothing formula we obtain is: \hat{F} = \frac{1}{1-\alpha^{N/F}}
Let us analyze this formula more closely. It is not difficult to imagine that \alpha must be very close to 1; otherwise, everything would be forgotten almost instantly. Secondly, we should expect that when F is sufficiently large, the smoothed result \hat{F} should be close to F, as the more sufficient the statistics, the more credible the results. Therefore, we assume that when F=N, \hat{F}=F. In this case, we should multiply by a factor: \hat{F} = \frac{N(1-\alpha)}{1-\alpha^{N/F}} This is the final smoothing formula.
If we adopt the results provided by “Uncle Han” (Han Shu), we might consider using: \alpha = 0.99999962... However, we notice that as F \to 0: \hat{F} = \lim_{F\to 0}\frac{N(1-\alpha)}{1-\alpha^{N/F}}=N(1-\alpha) This implies that this smoothing formula also assigns a frequency greater than 0, specifically N(1-\alpha), to characters with a statistical frequency of 0. Clearly, fixing \alpha here seems somewhat inappropriate. Why not fix N(1-\alpha) instead? If we assign a frequency \gamma to all characters that have never appeared, we can solve for \alpha: \alpha = 1 - \frac{\gamma}{N} Thus: \hat{F} = \frac{\gamma}{1-(1 - \gamma/N)^{N/F}} At this point, starting from the forgetting hypothesis, we have obtained a final smoothing formula that contains only one parameter with a clear meaning. Finally, we note that when N is sufficiently large: (1 - \gamma/N)^N \approx e^{-\gamma} Therefore: \hat{F} \approx \frac{\gamma}{1-e^{-\gamma/F}} This also serves as a usable smoothing formula. Finally, let’s look at its Taylor expansion: \hat{F} \approx \frac{\gamma}{1-e^{-\gamma/F}} = F + \frac{\gamma}{2} + \frac{\gamma^2}{12F^2} + \dots If we only take the first two terms, we can see that the effect of this approach is actually similar to Add-1 smoothing (Laplace smoothing). In this sense, perhaps we can say that the forgetting hypothesis provides a more substantial theoretical basis for Add-1 smoothing?
When reposting, please include the original address of this article: https://kexue.fm/archives/4182
For more details regarding reposting, please refer to: “Scientific Space FAQ”