Revisiting Class Imbalance: Connections Between Weight Adjustment and Modified Loss Functions

From Smooth Accuracy to Cross Entropy

The analysis here primarily focuses on binary classification using the sigmoid function, but most conclusions can be extended in parallel to multi-class classification using softmax. Let x be the input, y \in \{0, 1\} be the target, and p_{\theta}(x) \in [0, 1] be the model. Ideally, we should optimize whatever metric we use for evaluation. For classification problems, the most straightforward metric is accuracy, but accuracy does not provide effective gradients and thus cannot be used directly for training.

To address this, we use a smoothed version of the metric. As discussed in the previous article "Notes on Function Smoothing: Differentiable Approximations of Non-differentiable Functions", a smoothed approximation of accuracy is: \text{ACC}_{\text{smooth}} = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y p_{\theta}(x) + (1 - y)(1 - p_{\theta}(x))\big] where \mathcal{D} is the training dataset. Therefore, in principle, we should minimize -\text{ACC}_{\text{smooth}}. However, in practice, directly optimizing this objective does not yield good results. It is better to optimize the cross entropy: \text{cross\_entropy} = \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[-y \log p_{\theta}(x) - (1 - y)\log(1 - p_{\theta}(x))\big] This is quite intriguing: if \text{ACC}_{\text{smooth}} is closer to our evaluation metric, why is cross entropy more beneficial for the evaluation metric?

This requires an explanation through gradients. For p_{\theta}(x), it is typically activated via a sigmoid function, i.e., p_{\theta}(x) = \sigma(z_{\theta}(x)), where \sigma(t) = \frac{1}{1+e^{-t}}. Its derivative is \sigma'(t) = \sigma(t)(1 - \sigma(t)), and z_{\theta}(x) is what we usually call the logits.

Suppose y=1. Then the corresponding -\text{ACC}_{\text{smooth}} is -p_{\theta}(x) = -\sigma(z_{\theta}(x)), and its gradient is: -\nabla_{\theta} p_{\theta}(x) = - p_{\theta}(x) (1 - p_{\theta}(x))\nabla_{\theta}z_{\theta}(x) As mentioned, since y=1, the training objective is p_{\theta}(x) \to 1. We expect that when p_{\theta}(x) is close to 0 (large error), it should produce a large gradient, and when p_{\theta}(x) is close to 1 (small error), it should produce a small gradient. However, the above -\nabla_{\theta} p_{\theta}(x) clearly does not behave this way. Its adjustment factor p_{\theta}(x)(1 - p_{\theta}(x)) reaches its maximum at 0.5, while it reaches its minimum at both 0 and 1. This means that if the error is too large, the gradient actually becomes small, leading to low optimization efficiency and poor overall performance. Conversely, for cross entropy, we have: -\nabla_{\theta} \log p_{\theta}(x) = - (1 - p_{\theta}(x))\nabla_{\theta}z_{\theta}(x) This exactly removes the p_{\theta}(x) factor that causes the negative effect in the gradient. Consequently, the optimization efficiency is higher, and the final result is better. The above analysis is for y=1; if y=0, the conclusion remains the same.

From Smooth F1 to Weighted Cross Entropy

From this process, we can sense that various modifications to the loss function are essentially just adjusting the gradients. By obtaining more reasonable gradients, we can achieve more effective optimization and obtain better models. Furthermore, reflecting on the conversion process above, the gradient of the original approximation objective was -\nabla_{\theta}p_{\theta}(x), but -\nabla_{\theta}\log p_{\theta}(x) worked better. If we don’t analyze the underlying reasons deeply and simply treat p \to \log p as an "axiom", would it hold? Would it lead to interesting results?

For example, when negative samples far outnumber positive samples, our evaluation metric is usually no longer accuracy (otherwise, simply outputting 0 for everything would yield high accuracy). We typically care about the F1 score of the positive class. Directly optimizing F1 is also difficult, so we need a smooth version. The article "Notes on Function Smoothing" also provides this result: \text{F1}_{\text{smooth}} = \frac{2 \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y p_{\theta}(x)\big]}{\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y + p_{\theta}(x)\big]} So our minimization objective was originally -\text{F1}_{\text{smooth}}. According to the aforementioned "axiom," let’s first directly calculate the gradient of -\text{F1}_{\text{smooth}}: \begin{aligned} &-\nabla_{\theta}\frac{2 \mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y p_{\theta}(x)\big]}{\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y + p_{\theta}(x)\big]}\\ =&-2\frac{\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y \nabla_{\theta}p_{\theta}(x)\big]}{\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y + p_{\theta}(x)\big]} + 2\frac{\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y p_{\theta}(x)\big]\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[\nabla_{\theta}p_{\theta}(x)\big]}{\left(\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y + p_{\theta}(x)\big]\right)^2}\\ =&-\frac{2\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[\big(y-\text{F1}_{\text{smooth}}/2\big)\nabla_{\theta}p_{\theta}(x)\big]}{\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y + p_{\theta}(x)\big]} \end{aligned} Where \frac{2}{\mathbb{E}_{(x,y)\sim\mathcal{D}}[y + p_{\theta}(x)]} is a global scaling factor. We are mainly concerned with the gradient of each sample, so the result is: -\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[\big(y-\text{F1}_{\text{smooth}}/2\big)\nabla_{\theta}p_{\theta}(x)\big] According to the p \to \log p "axiom" (and -p \to \log(1-p) for negative samples), we obtain the final gradient as: -\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y\cdot\big(1-\text{F1}_{\text{smooth}}/2\big)\cdot\nabla_{\theta}\log p_{\theta}(x) + (1 - y)\cdot\text{F1}_{\text{smooth}}/2\cdot\nabla_{\theta}\log (1-p_{\theta}(x))\big] This is equivalent to the gradient of the optimization objective: -\mathbb{E}_{(x,y)\sim\mathcal{D}}\big[y\cdot\big(1-\text{F1}_{\text{smooth}}/2\big)\cdot\log p_{\theta}(x) + (1 - y)\cdot\text{F1}_{\text{smooth}}/2\cdot\log (1-p_{\theta}(x))\big] (where \text{F1}_{\text{smooth}} is treated as a constant for the gradient). Thus, this is essentially using 1-\text{F1}_{\text{smooth}}/2 to adjust the cross entropy of positive samples and \text{F1}_{\text{smooth}}/2 to adjust the cross entropy of negative samples.

From Margin Expansion to Logits Adjustment

In fact, regardless of the evaluation metric, we always hope to predict every sample as correctly as possible. The problem is that categories with fewer samples do not generalize well because they are not learned sufficiently.

Let’s think about this problem from a geometric perspective. Ideally, in the encoding space, each class of samples occupies its own "territory," and the territories of different classes do not intersect. The poor generalization of minority classes is mainly reflected in the fact that their "territory" is relatively small and is often "suppressed" by classes with many samples. Consequently, "survival" becomes an issue, let alone accounting for new samples that did not appear in the training set.

How can we solve this? It’s quite intuitive: if the samples in the minority classes were all "powerhouses"—the kind that can take on ten others—then even with fewer samples, they could hold their own in the "territory war." Let’s consider an n-class classification problem. Suppose the encoding vector of a sample is f_{\theta}(x) and the class vector is u_y. The similarity between the sample and the class vector is generally measured by the inner product \langle f_{\theta}(x), u_y\rangle. Assume each sample can occupy a "territory" with radius r_y. This means that for any z satisfying \Vert z - f_{\theta}(x)\Vert \leq r_y, z is considered an encoding vector for that sample. This implies that for any z satisfying this condition, its similarity to u_y should be greater than its similarity to other classes.

Now we consider: \langle z, u_y\rangle = \langle f_{\theta}(x), u_y\rangle + \langle z - f_{\theta}(x), u_y\rangle Since \Vert z - f_{\theta}(x)\Vert \leq r_y, it is obvious that: \langle f_{\theta}(x), u_y\rangle - r_y\Vert u_y\Vert \leq \langle z, u_y\rangle \leq \langle f_{\theta}(x), u_y\rangle + r_y\Vert u_y\Vert Therefore, to achieve the goal that "the similarity of z to u_y should be greater than its similarity to other classes," we only need "the minimum similarity of z to u_y to be greater than its maximum similarity to other classes." Thus, our optimization objective becomes: -\log\frac{e^{\langle f_{\theta}(x), u_y\rangle - r_y\Vert u_y\Vert}}{e^{\langle f_{\theta}(x), u_y\rangle - r_y\Vert u_y\Vert}+\sum\limits_{i\neq y} e^{\langle f_{\theta}(x), u_i\rangle + r_y\Vert u_i\Vert}} As we can see, this is essentially equivalent to softmax variants with margins, such as AM-Softmax or Circle Loss. The specific form is not important; we just need to set a larger margin for classes with fewer samples (making each sample in the minority class more "powerful"). How should we design the margin for each class? The previous article "Alleviating Class Imbalance through Mutual Information Ideas" provided a scheme: m_y = -\tau \log p(y), where p(y) is the prior distribution. Then we have: -\log\frac{e^{\langle f_{\theta}(x), u_y\rangle + \tau \log p(y)}}{\sum\limits_{i} e^{\langle f_{\theta}(x), u_i\rangle + \tau \log p(i)}} In this way, we connect to the Logit Adjustment loss, or rather, provide a geometric intuition for Logit Adjustment loss. Essentially, Logit Adjustment is also adjusting weights, but while general weight adjustment happens after the \log of the loss function, Logit Adjustment happens before the \log.

Summary

This article has reflected on the phenomenon of class imbalance and its countermeasures, primarily hoping to reveal the ideas behind modified loss functions through relatively intuitive guidance. From this, we can find that these schemes are essentially all about adjusting sample weights or class weights. The analytical approach in this article is relatively informal, basically consisting of the author’s brainstorming. If there are any errors or omissions, readers are kindly requested to point them out.

Original Address: https://kexue.fm/archives/7708

For more details on reposting, please refer to: Scientific Space FAQ