Introduction
In a discussion in a QQ group today, I came across "Focal Loss." After some searching, I found it is a loss function proposed by Kaiming He’s team in their paper "Focal Loss for Dense Object Detection", which they used to improve the performance of image object detection. Although I rarely work on computer vision tasks and don’t follow image-related applications closely, Focal Loss is essentially a loss function designed to solve class imbalance and differences in classification difficulty in classification problems. In short, this work has been widely acclaimed. You can also check the discussion on Zhihu: "How to evaluate Kaiming’s Focal Loss for Dense Object Detection?"
When I first saw this loss, it felt quite magical and seemed very useful. This is because in NLP, there are also many tasks with class imbalance. The most classic example is sequence labeling, where categories are severely imbalanced. For instance, in Named Entity Recognition (NER), it is obvious that entities in a sentence are much fewer than non-entities. This is a case of severe class imbalance. I tried applying it to my QA model based on sequence labeling and saw a slight improvement. Indeed, it is a good loss function.
Upon closer comparison, I realized that this loss shares the same underlying principle as a loss function I conceived last night! This prompted me to write this blog post. I will analyze the problem from my own perspective, eventually arriving at Focal Loss, and also present the similar loss I derived last night.
Hard Truncation
This entire article starts from the binary classification problem, but the same ideas can be applied to multi-class problems. The standard loss for binary classification is cross-entropy: L_{ce} = -y\log \hat{y} - (1-y)\log(1-\hat{y})=\left\{\begin{aligned}&-\log(\hat{y}),\,\text{when } y=1\\ &-\log(1-\hat{y}),\,\text{when } y=0\end{aligned}\right. where y\in\{0,1\} is the true label and \hat{y} is the predicted value. Usually, for binary classification, we use the sigmoid function for activation \hat{y}=\sigma(x), so it is equivalent to: L_{ce} = -y\log \sigma(x) - (1-y)\log\sigma(-x)=\left\{\begin{aligned}&-\log \sigma(x),\,\text{when } y=1\\ &-\log\sigma(-x),\,\text{when } y=0\end{aligned}\right. (Note that 1-\sigma(x)=\sigma(-x).)
In a blog post from the first half of the year, "Text Sentiment Classification (IV): A Better Loss Function", I proposed a "hard truncation" loss based on the idea of "focusing energy on hard-to-classify samples," which took the form: L^{\cdot} = \lambda(y,\hat{y})\cdot L_{ce} where \lambda(y,\hat{y})=\left\{\begin{aligned}&0,\,(y=1 \text{ and } \hat{y} > 0.5) \text{ or } (y=0 \text{ and } \hat{y} < 0.5)\\ &1,\,\text{otherwise}\end{aligned}\right. The logic here is: if the predicted value for a positive sample is greater than 0.5, or the predicted value for a negative sample is less than 0.5, I stop updating it and concentrate attention on those samples that are not predicted accurately. While this threshold can be adjusted and partially achieves the goal, it significantly increases the number of required iterations.
The reason is as follows: taking a positive sample as an example, I only tell the model to stop updating if the prediction is greater than 0.5, but I don’t tell it to "maintain" a value greater than 0.5. Therefore, in the next stage, its predicted value might very likely drop below 0.5 again. Of course, if that happens, it will be updated again in the next round. This repeated iteration can theoretically achieve the goal, but it increases the iteration count significantly. Thus, the key to improvement is: "not just telling the model to stop updating when the positive sample’s prediction is above 0.5, but telling it that once it is above 0.5, it only needs to maintain that state." (It’s like a teacher who stops caring once a student passes; that clearly doesn’t work. If a student has passed, the goal should be to find ways to maintain that state or even improve, rather than ignoring them.)
Softening the Loss
The deficiency of hard truncation lies in the fact that the factor \lambda(y,\hat{y}) is non-differentiable, or rather, we treat its derivative as 0. Consequently, this term provides no help to the gradient, and we cannot get reasonable feedback from it (meaning the model doesn’t know what "maintaining" implies).
One way to solve this is to "soften" the loss. "Softening" means approximating non-differentiable functions with differentiable ones, which in mathematics is called "smoothing." After this treatment, what was originally non-differentiable becomes differentiable. A similar example can be found in the K-means section of "Gradient Descent and EM Algorithm: From the Same Origin". First, let’s rewrite L^{\cdot}: L^{\cdot} =\left\{\begin{aligned}&-\theta(0.5-\hat{y})\log(\hat{y}),\,\text{when } y=1\\ &-\theta(\hat{y}-0.5)\log(1-\hat{y}),\,\text{when } y=0\end{aligned}\right. Here \theta is the unit step function: \theta(x) = \left\{\begin{aligned}&1, x > 0\\ &\frac{1}{2}, x = 0\\ &0, x < 0\end{aligned}\right. This L^{\cdot} is completely equivalent to the original one. It is also equivalent to (since \sigma(0)=0.5): L^{\cdot} =\left\{\begin{aligned}&-\theta(-x)\log \sigma(x),\,\text{when } y=1\\ &-\theta(x)\log\sigma(-x),\,\text{when } y=0\end{aligned}\right. Now the path is clear. To "soften" this loss, we must "soften" \theta(x), and softening it is as easy as using the sigmoid function! We have: \theta(x) = \lim_{K\to +\infty} \sigma(Kx) So, quite obviously, we replace \theta(x) with \sigma(Kx): L^{\cdot \cdot }=\left\{\begin{aligned}&-\sigma(-Kx)\log \sigma(x),\,\text{when } y=1\\ &-\sigma(Kx)\log\sigma(-x),\,\text{when } y=0\end{aligned}\right. This is the loss I conceived last night. Clearly, it is also very easy to implement.
Now, let’s compare it with Focal Loss.
Focal Loss
Kaiming He’s Focal Loss takes the form: L_{fl}=\left\{\begin{aligned}&-(1-\hat{y})^{\gamma}\log \hat{y},&\text{when } y=1\\ &-\hat{y}^{\gamma}\log (1-\hat{y}),&\text{when } y=0\end{aligned}\right. If we apply the prediction \hat{y}=\sigma(x), we get: L_{fl}=\left\{\begin{aligned}&-\sigma^{\gamma}(-x)\log \sigma(x),&\text{when } y=1\\ &-\sigma^{\gamma}(x)\log\sigma(-x),&\text{when } y=0\end{aligned}\right. Notably, if both K and \gamma are set to 1, then L^{\cdot \cdot}=L_{fl}!
In fact, the roles of K and \gamma are the same: they both adjust the steepness of the weight curve, just in different ways. Note that L^{\cdot \cdot} or L_{fl} already inherently contains a solution for imbalanced samples; or rather, class imbalance is essentially a manifestation of differences in classification difficulty. For example, if negative samples are far more numerous than positive samples, the model will naturally lean towards the majority negative class (imagine predicting all samples as negative). In this case, the \hat{y}^{\gamma} or \sigma(Kx) for the negative class will be very small, while the (1-\hat{y})^{\gamma} or \sigma(-Kx) for the positive class will be very large. At this point, the model will start to concentrate its energy on the positive samples.
Furthermore, Kaiming He found that adding a weight adjustment to L_{fl} yields a slight improvement: L_{fl}=\left\{\begin{aligned}&-\alpha(1-\hat{y})^{\gamma}\log \hat{y},&\text{when } y=1\\ &-(1-\alpha)\hat{y}^{\gamma}\log (1-\hat{y}),&\text{when } y=0\end{aligned}\right. Through a series of hyperparameter tuning, he found that \alpha=0.25, \gamma=2 worked best (on his model). Note that in his task, positive samples are the minority, meaning that originally positive samples could not "compete" with negative samples. However, after being "manipulated" by (1-\hat{y})^{\gamma} and \hat{y}^{\gamma}, the situation might have even reversed, requiring a down-weighting of the positive samples. However, I believe such adjustments are empirical results. Theoretically, it is difficult to have a guiding principle to determine the value of \alpha. Without massive computing power for tuning, it might be better to simply set \alpha=0.5 (equal weighting).
Multi-class Classification
The form of Focal Loss for multi-class classification is also easy to obtain: L_{fl}=-(1-\hat{y}_t)^{\gamma}\log \hat{y}_t where \hat{y}_t is the predicted value for the target class, usually the result after softmax. How can my conceived L^{\cdot \cdot} be generalized to multi-class? It is also simple: L^{\cdot \cdot }=-\text{softmax}(-Kx_t)\log \text{softmax}(x_t) Here x_t is also the predicted value for the target class, but it is the result before softmax (the logit).
Conclusion
What? You came up with the same idea as Kaiming He? No, no, no. This article is merely an introduction to Kaiming He’s Focal Loss. More accurately, it is an introduction to some schemes for dealing with class imbalance and classification difficulty variance, while providing my own perspective as much as possible. Of course, writing in this manner might inevitably seem like a poor imitation or an attempt to associate myself with greatness; I hope the readers will understand.
When reposting, please include the original address of this article: https://kexue.fm/archives/4733
For more detailed reposting matters, please refer to: "Scientific Space FAQ"