Generally speaking, the objects processed by neural networks are continuous floating-point numbers, and the standard outputs are also continuous numerical values. However, in practical problems, we often require discrete results. For example, in classification problems, we hope to output the correct category; while the "category" is discrete, only the "probability of the category" is continuous. Furthermore, the evaluation metrics for many of our tasks are actually discrete, such as accuracy and F1 score in classification, or BLEU in machine translation, and so on.
Taking the classification problem as an example again, the common evaluation metric is accuracy, while the common loss function is cross-entropy. A decrease in cross-entropy is indeed associated with an increase in accuracy to some extent, but they do not have an absolute monotonic correlation. In other words, if the cross-entropy decreases, the accuracy does not necessarily increase. Obviously, if we could use the negative of the accuracy as the loss function, that would be ideal. However, accuracy is non-differentiable (involving operations like \text{argmax}), so it cannot be used directly.
In such cases, there are generally two types of solutions. One is to employ reinforcement learning, setting the accuracy as the reward function; this is an "overkill" approach. The other is to attempt to find a smooth and differentiable approximation formula for the accuracy. This article explores common smooth approximations of non-differentiable functions, sometimes referred to as "smoothing" and sometimes as "softening."
max
The foundation for most of the content discussed later is the smooth approximation of the \max operation. We have: \max(x_1,x_2,\dots,x_n) = \lim_{K\to +\infty}\frac{1}{K}\log\left(\sum_{i=1}^n e^{K x_i}\right) Therefore, by choosing a constant K, we have the approximation: \max(x_1,x_2,\dots,x_n) \approx \frac{1}{K}\log\left(\sum_{i=1}^n e^{K x_i}\right) In models, we can often set K=1, which is equivalent to integrating K into the model itself. Thus, most simply, we have: \begin{aligned} \max(x_1,x_2,\dots,x_n) \approx&\, \log\left(\sum_{i=1}^n e^{x_i}\right) \\ \triangleq&\, \text{logsumexp}(x_1, x_2, \dots, x_n) \end{aligned} \label{eq:max-approx}
Here, \text{logsumexp} appears, which is a very common operator. In this context, it is a smooth approximation of the \max function. Indeed, the smooth approximation of the \max function is actually \text{logsumexp}, not \text{softmax}, which looks similar literally. For related derivations, you can also refer to my previous post "Seeking a Smooth Maximum Function".
softmax
I just mentioned that \text{softmax} is not a smooth approximation of \max. So, what is it a smooth approximation of? It is actually a smooth approximation of \text{onehot}(\text{argmax}(\boldsymbol{x})), which means first finding the position of the maximum value and then generating a vector of the same length, where the position of the maximum value is set to 1 and all other positions are set to 0. For example: \quad \to \quad [0, 0, 0, 1, 0] We can simply provide a derivation process from \text{logsumexp} to \text{softmax}. Consider the vector \boldsymbol{x}=[x_1, x_2, \dots, x_n], and then consider: \boldsymbol{x}'=[x_1, x_2, \dots, x_n] - \max(x_1, x_2, \dots, x_n) That is, subtracting the overall maximum from each element. The position of the maximum in this new vector is the same as in the original vector, i.e., \text{onehot}(\text{argmax}(\boldsymbol{x}))=\text{onehot}(\text{argmax}(\boldsymbol{x}')). Without loss of generality, consider the case where x_1, x_2, \dots, x_n are distinct. Then the maximum value of the new vector is clearly 0, and all other elements are negative. Consequently, we can consider: e^{\boldsymbol{x}'}=[e^{x_1 - \max(x_1, x_2, \dots, x_n)}, e^{x_2 - \max(x_1, x_2, \dots, x_n)}, \dots, e^{x_n - \max(x_1, x_2, \dots, x_n)}] as an approximation of \text{onehot}(\text{argmax}(\boldsymbol{x}')). Since the maximum value is 0, the corresponding position is e^0=1, while the others are negative, making them close to 0 after taking the exponential.
Finally, by substituting the approximation [eq:max-approx] into the above equation and simplifying, we obtain: \begin{aligned} \text{onehot}(\text{argmax}(\boldsymbol{x}))=&\, \text{onehot}(\text{argmax}(\boldsymbol{x}'))\\ \approx&\, \left(\frac{e^{x_1}}{\sum\limits_{i=1}^n e^{x_i}}, \frac{e^{x_2}}{\sum\limits_{i=1}^n e^{x_i}}, \dots, \frac{e^{x_n}}{\sum\limits_{i=1}^n e^{x_i}}\right)\\ \triangleq&\, \text{softmax}(x_1, x_2, \dots, x_n) \end{aligned}
argmax
\text{argmax} refers to directly providing the index (an integer) where the maximum value of the vector is located. For example: \quad \to \quad 4 Here we follow the common usage habit where indices start from 1, so the result is 4; however, in programming languages, they usually start from 0, so the result would typically be 3.
For a smooth approximation of \text{argmax}, we naturally hope to output a floating-point number close to 4. To construct such an approximation, we first observe that \text{argmax} is actually equal to: \text{sum}\Big(\underbrace{[1, 2, 3, 4, 5]}_{\text{Index vector [1, 2, ..., n]}}\,\, \otimes\,\,\underbrace{[0, 0, 0, 1, 0]}_{\text{onehot}(\text{argmax}(\boldsymbol{x}))}\Big) which is the dot product of the array [1, 2, \dots, n] and \text{onehot}(\text{argmax}(\boldsymbol{x})). Constructing the softened version of \text{argmax} is then simple: replace \text{onehot}(\text{argmax}(\boldsymbol{x})) with \text{softmax}(\boldsymbol{x}): \text{argmax} (\boldsymbol{x}) \approx \sum_{i=1}^n i\times \text{softmax}(\boldsymbol{x})_i
Accuracy
The several approximations discussed above are basically derived by finding the correct form based on one-hot vectors and then approximating the one-hot vector with softmax. Using this idea, we can derive smooth approximations for many operators, such as accuracy.
For simplicity, let’s introduce the notation \boldsymbol{1}_k, which represents a one-hot vector with 1 at the k-th position. Suppose in a classification problem, the target category is i and the predicted category is j. We can consider the one-hot vectors \boldsymbol{1}_i and \boldsymbol{1}_j, and then consider their inner product: \langle \boldsymbol{1}_i, \boldsymbol{1}_j\rangle = \left\{\begin{aligned}&1,\,\,(i=j)\\ &0,\,\,(i\neq j)\end{aligned}\right. That is, when the two categories are the same, the inner product is exactly 1, and when they are different, the inner product is exactly 0. Thus, the inner product of the one-hot vectors corresponding to the target and predicted categories exactly defines a "correct prediction" counting function. With a counting function, we can calculate the accuracy: \text{Accuracy} = \frac{1}{|\mathcal{B}|}\sum_{\boldsymbol{x}\in\mathcal{B}}\langle \boldsymbol{1}_i(\boldsymbol{x}), \boldsymbol{1}_j(\boldsymbol{x})\rangle where \mathcal{B} represents the current batch. The above formula is precisely the function to calculate the accuracy within a batch. In neural networks, to ensure differentiability, the final output can only be a probability distribution (the result after softmax). Therefore, the smooth approximation of accuracy is to replace the one-hot vector of the predicted category with the probability distribution: \text{Accuracy} \approx \frac{1}{|\mathcal{B}|}\sum_{\boldsymbol{x}\in\mathcal{B}}\langle \boldsymbol{1}_i(\boldsymbol{x}), p(\boldsymbol{x})\rangle Similarly, smooth approximations for metrics like recall and F1 can be derived. Taking binary classification as an example, let p(\boldsymbol{x}) be the probability of the positive class and t(\boldsymbol{x}) be the label of sample \boldsymbol{x} (0 or 1). Then the smooth approximation of the F1 score for the positive class is: \text{Positive F1} \approx \frac{2 \sum\limits_{\boldsymbol{x}\in\mathcal{B}}t(\boldsymbol{x}) p(\boldsymbol{x})}{\sum\limits_{\boldsymbol{x}\in\mathcal{B}}\big[t(\boldsymbol{x}) + p(\boldsymbol{x})\big]}
The accuracy approximation formula derived this way is differentiable and its negative can be used directly as a loss. However, in the sampling estimation process, it is a biased estimate of F1 (since the denominator also contains a summation over the batch), which sometimes affects the optimization trajectory or even leads to divergence. Therefore, in general, it is best not to use it directly. Instead, first train with ordinary cross-entropy until it is almost finished, and then use the negative of F1 as the loss for fine-tuning.
soft-k-max
\text{softmax} is a smooth approximation of "setting the position of the maximum value to 1 and others to 0." What if we want a smooth approximation of "setting the positions of the top k values to 1 and others to 0"? We can call this \text{soft-}k\text{-max}.
I have not constructed a simple form for \text{soft-}k\text{-max}, but it can be constructed recursively:
Input is \boldsymbol{x}, initialize \boldsymbol{p}^{(0)} as an all-zero vector;
Execute \boldsymbol{x} = \boldsymbol{x} - \min(\boldsymbol{x}) (to ensure all elements are non-negative).For i=1,2,\dots,k, execute:
\boldsymbol{y} = (1 - \boldsymbol{p}^{(i-1)})\otimes\boldsymbol{x};
\boldsymbol{p}^{(i)} = \boldsymbol{p}^{(i-1)} + \text{softmax}(\boldsymbol{y})Return \boldsymbol{p}^{(k)}.
As for the principle, if you replace \text{softmax}(\boldsymbol{y}) with \text{onehot}(\text{argmax}(\boldsymbol{y})) and recurse, it becomes clear. It essentially finds the maximum, then "removes" that maximum position so that the next largest value becomes the maximum, then applies \text{softmax} again, and repeats the process.
Summary
Function smoothing is an interesting mathematical topic that frequently appears in machine learning. On one hand, it is a technique for making certain operations differentiable, allowing models to be solved directly using backpropagation without "resorting" to reinforcement learning. On the other hand, in some cases, it can enhance the interpretability of the model, as the corresponding non-differentiable functions often have better interpretability. After training with the smooth version, it might be possible to revert to the non-differentiable version to explain the model’s output.
Of course, appreciating it as pure mathematical beauty is also quite pleasant.
Reprinting please include the original address of this article: https://kexue.fm/archives/6620
For more detailed reprinting matters, please refer to: "Scientific Space FAQ"