The Even-Order Taylor Expansion of $ (x)$ at $x=0$ is Always Positive

The Proof Process

This appears to be a very strong result. Is the proof complicated? In fact, the proof is very simple. Let f_n(x) = \sum\limits_{k=0}^n \frac{x^k}{k!} When n is even, we have \lim\limits_{x\to\pm\infty} f_n(x)=+\infty, meaning the function overall opens upwards. Therefore, we only need to prove that its minimum value is greater than 0. Since it is a smooth and continuous polynomial function, the global minimum must be at one of its local minima. Looking at it from another perspective, we only need to prove that the function values at all its critical points (whether local maxima or minima) are greater than 0.

The method to find critical points is, naturally, to take the derivative. A beautiful property of f_n(x) is that its derivative satisfies: f_n'(x) = f_{n-1}(x) Critical points satisfy f_n'(x)=0, which means they satisfy f_{n-1}(x)=0. At such points, we have: f_n(x) = f_{n-1}(x) + \frac{x^n}{n!} = \frac{x^n}{n!} \geq 0 \quad (\text{when } n \text{ is even}) Thus, we have proven that the function values at all critical points of f_n(x) are non-negative. Therefore, f_n(x) \geq 0 holds constantly. Furthermore, one can verify that x=0 is not a critical point, so the \geq can be replaced with >. This completes the proof.

Application Scenarios

In fact, I saw this conclusion in a new Arxiv paper titled “Exploring Alternatives to Softmax Function”. The original paper provides a relatively complex proof based on mathematical induction; the proof above is my own construction, which is relatively simpler and clearer.

So, why did the original paper need this conclusion? As the name suggests, it is to explore alternatives to softmax. We know that in machine learning, the common method to transform output into a probability distribution is to apply softmax: \text{softmax}(\boldsymbol{x})_i = \frac{e^{x_i}}{\sum\limits_{k=1}^n e^{x_k}} Since f_n(x) > 0 when n is even, and f_n(x) is an approximation of e^x within a certain range, replacing e^x with f_n(x) can serve as a reasonable normalization function: \text{taylor-softmax}(\boldsymbol{x}, n)_i = \frac{f_n(x_i)}{\sum\limits_{k=1}^n f_n(x_k)} The original paper conducted several experiments showing that taylor-softmax offers some improvement over the conventional softmax:

A Brief Commentary

However, in my view, these experimental results are hardly convincing, as the baseline performance used is too low (it’s 2020, at least run a ResNet, right?). Furthermore, the original paper does not provide much intuitive understanding of this alternative; it simply performs basic experiments and claims it works, which is somewhat crude.

Nevertheless, despite the many shortcomings of the original paper, I believe the proposed taylor-softmax might actually be effective. The process from softmax to taylor-softmax is essentially changing the activation function from an exponential function to a polynomial function. What is the difference? We know that when |x| is large, e^x increases or decays very rapidly. This directly leads to the phenomenon where softmax often gives excessively high confidence (probability values being either 0 or 1). In contrast, polynomial functions do not grow as aggressively, making them less prone to over-confidence issues and, consequently, less likely to overfit.

Similar modifications appear in the classic dimensionality reduction method t-SNE. The predecessor of t-SNE was SNE, which constructed a probability distribution using an exponential form similar to softmax. It was found to suffer from the “Crowding Problem” (refer to “The Principle of Minimum Entropy (IV): ’Birds of a Feather’ from Libraries to Word Vectors”). Finally, t-SNE replaced the exponential with a quadratic function, which worked much better. I feel that the idea behind taylor-softmax shares some common ground with t-SNE.

Maintaining Monotonicity

In fact, it can also be proven that f_n(x) has only one global minimum point, so its graph is always “U-shaped,” as shown below:

Some readers with a penchant for rigor might worry about the non-monotonicity of f_n(x), feeling that a non-monotonic function might hide certain issues. In reality, there is currently no clear evidence that the transformation into a probability distribution must be monotonic. Of course, if you are still concerned, you can simply truncate it. As mentioned, f_n(x) has only one minimum point x_n^*. The part greater than the minimum point is monotonically increasing; for the part less than the minimum point, we can simply set it equal to the minimum value. That is, define: \tilde{f}_{n}(x)=\left\{\begin{aligned}&f_n(x), & x > x_n^*\\ &f_n(x_n^*), & x\leq x_n^*\end{aligned}\right. Then use \tilde{f}_{n}(x) instead of f_{n}(x) for normalization. For a fixed n, x_n^* and f_n(x_n^*) can be calculated in advance using numerical methods:

Summary

The main purpose of this text is to introduce the interesting conclusion that “the even-order Taylor expansion of e^x is always positive” and to briefly mention its application in searching for softmax alternatives.