(Note: The relevant content of this article has been organized into the paper “ZLPR: A Novel Loss for Multi-label Classification.” If you need to cite it, you can directly cite the English paper. Thank you.)
Generally, when dealing with conventional multi-class classification problems, we use a fully connected layer at the end of the model to output scores for each class, then apply softmax activation and use cross entropy as the loss function. In this article, we attempt to generalize the “Softmax + Cross Entropy” scheme to multi-label classification scenarios, hoping to obtain a loss function for multi-label classification tasks that does not require special adjustment of class weights or thresholds.
From Single-Label to Multi-Label
Generally, multi-class classification refers to single-label classification, i.e., selecting 1 target category from n candidate categories. Assuming the scores for each class are s_1, s_2, \dots, s_n and the target class is t \in \{1, 2, \dots, n\}, the loss used is: -\log \frac{e^{s_t}}{\sum\limits_{i=1}^n e^{s_i}} = - s_t + \log \sum\limits_{i=1}^n e^{s_i} \label{eq:log-softmax} The optimization direction of this loss is to make the target class score s_t the maximum among s_1, s_2, \dots, s_n. For more content related to softmax, you can also refer to articles like “Seeking a Smooth Maximum Function” and “Discussion on Function Smoothing: Differentiable Approximation of Non-differentiable Functions.”
Now we turn to multi-label classification, i.e., selecting k target categories from n candidate categories. In this case, a naive approach is to use sigmoid activation, turning it into n binary classification problems and using the sum of binary cross-entropies as the loss. Obviously, when n \gg k, this approach faces serious class imbalance issues. At this point, some balancing strategies are needed, such as manually adjusting the weights of positive and negative samples or using Focal Loss. After training is complete, the optimal threshold still needs to be determined based on the validation set.
At this point, a natural confusion arises: Why does “n choose k” require so much more work than “n choose 1”?
I believe this is quite unscientific. After all, intuitively, n choose k should just be a natural extension of n choose 1, so it shouldn’t require so much more effort. Even if n choose k is more complex, the difficulty should transition gradually. However, if it turns into multiple binary classifications, n choose 1 actually becomes the most difficult because the class imbalance is most severe. Formally, single-label classification is easier than multi-label classification because single-label has “Softmax + Cross Entropy” available, which does not suffer from class imbalance, whereas “Sigmoid + Cross Entropy” in multi-label classification does.
Therefore, the ideal solution should be to generalize “Softmax + Cross Entropy” to multi-label classification.
Searching Far and Wide
To consider this generalization, I made several attempts and rejected many results. Finally, I determined a relatively elegant scheme: constructing a combinatorial form of softmax as a generalization of single-label softmax. In this section, we will first assume k is a fixed constant, then discuss the automatic determination of k in general cases, eventually arriving at an effective generalization.
Combinatorial Softmax
First, we consider the scenario where k is a fixed constant, which means that during prediction, we directly output the k categories with the highest scores. What about training? As a natural extension of softmax, we can consider the following as the loss: -\log \frac{e^{s_{t_1}+s_{t_2}+\dots+s_{t_k}}}{\sum\limits_{1\leq i_1 < i_2 < \cdots < i_k\leq n}e^{s_{i_1}+s_{i_2}+\dots+s_{i_k}}} = \log Z_k - (s_{t_1}+s_{t_2}+\dots+s_{t_k}) where t_1, t_2, \dots, t_k are the k target labels, and Z_k = \sum\limits_{1\leq i_1 < i_2 < \cdots < i_k\leq n}e^{s_{i_1}+s_{i_2}+\dots+s_{i_k}} is the partition function. Clearly, the above formula is a softmax constructed with the total score of any k categories s_{i_1}+s_{i_2}+\dots+s_{i_k} as the basic unit, so it is a reasonable generalization of single-label softmax. Alternatively, it can be understood as still being a single-label classification problem, but one of choosing 1 from C_n^k.
In this scheme, the difficult part is the calculation of Z_k, which is the sum of exponentials of C_n^k terms. However, we can use Newton’s identities to help us calculate it recursively. Let S_k = \sum\limits_{i=1}^n e^{k s_i}, then: \begin{aligned} Z_1 =&\, S_1\\ 2Z_2 =&\, Z_1 S_1 - S_2\\ 3Z_3 = &\, Z_2 S_1 - Z_1 S_2 + S_3\\ \vdots\\ k Z_k = &\, Z_{k-1} S_1 - Z_{k-2} S_2 + \dots + (-1)^{k-2} Z_1 S_{k-1} + (-1)^{k-1} S_k \end{aligned} So to calculate Z_k, we only need to calculate k recursive steps, which can be done within a reasonable time. In the prediction stage, we directly output the k classes with the highest scores.
Automatic Threshold Determination
The above discussion concerns multi-label classification with a fixed number of outputs, but the number of target labels in general multi-label classification is uncertain. To this end, we determine a maximum number of target labels K \geq k and add a label 0 as a padding label. At this point, the loss becomes: \log \overline{Z}_K - (s_{t_1}+s_{t_2}+\dots+s_{t_k}+\underbrace{s_0+\dots+s_0}_{K-k \text{ items}}) And: \begin{aligned} \overline{Z}_K =&\, \sum\limits_{1\leq i_1 < i_2 < \cdots < i_K\leq n}e^{s_{i_1}+s_{i_2}+\dots+s_{i_K}} + \sum\limits_{0 = i_1 = \dots = i_j < i_{j+1} < \cdots < i_K\leq n}e^{s_{i_1}+s_{i_2}+\dots+s_{i_K}}\\ =&\, Z_K + e^{s_0} \overline{Z}_{K-1} \end{aligned} It looks complicated, but it’s actually simple. It is still based on the total score of K categories, but it allows and only allows the 0-th class to repeat. During prediction, we still output the K classes with the highest scores, but allow the 0-th class to be output repeatedly. The equivalent effect is using s_0 as a threshold and only outputting classes with scores greater than s_0. The final formula shows that \overline{Z}_K can also be calculated recursively, so there is no difficulty in implementation.
A Simpler Path Found
It seemed that “searching far and wide” finally yielded results: the theory was there, implementation wasn’t difficult, and it seemed time to run experiments and see the effects. If the results were good, maybe even a paper could be written. It looked like a bright future! However...
Fortunately or unfortunately, while verifying its effectiveness, I consulted some experts. Under their suggestion, I looked at the Circle Loss which I hadn’t examined closely before. I saw the unified loss form therein (Equation (1) in the original paper) and realized that this unified form contained a much simpler generalization scheme.
So, the unfortunate part is that such a simpler scheme already existed, so no matter how much I “searched far and wide,” it didn’t mean much anymore. The fortunate part is that I found this better scheme; otherwise, if I had hurriedly published the previous scheme only to find it wasn’t as simple or effective as existing ones, I would have been quite embarrassed.
Unified Loss Form
Let’s look at the single-label classification cross entropy \eqref{eq:log-softmax} in another way: -\log \frac{e^{s_t}}{\sum\limits_{i=1}^n e^{s_i}} = -\log \frac{1}{\sum\limits_{i=1}^n e^{s_i-s_t}} = \log \sum\limits_{i=1}^n e^{s_i-s_t} = \log \left(1 + \sum\limits_{i=1,i\neq t}^n e^{s_i-s_t}\right) Why is this loss effective? From the articles “Seeking a Smooth Maximum Function” and “Discussion on Function Smoothing: Differentiable Approximation of Non-differentiable Functions,” we know that \text{logsumexp} is actually a smooth approximation of \max, so we have: \log \left(1 + \sum\limits_{i=1,i\neq t}^n e^{s_i-s_t}\right) \approx \max \begin{pmatrix} 0 \\ s_1 - s_t \\ \vdots \\ s_{t-1} - s_t \\ s_{t+1} - s_t \\ \vdots \\ s_n - s_t \end{pmatrix} The characteristic of this loss is that all non-target class scores \{s_1, \dots, s_{t-1}, s_{t+1}, \dots, s_n\} are compared pairwise with the target class score \{s_t\}, and the maximum of their differences should be as small as possible (less than zero), thus achieving the effect that “the target class score is greater than every non-target class score.”
Therefore, if we have a multi-label classification scenario with multiple target classes, we also hope that “every target class score is not less than every non-target class score,” so the following form of loss emerges: \log \left(1 + \sum\limits_{i\in\Omega_{neg},j\in\Omega_{pos}} e^{s_i-s_j}\right) = \log \left(1 + \sum\limits_{i\in\Omega_{neg}} e^{s_i}\sum\limits_{j\in\Omega_{pos}} e^{-s_j}\right) \label{eq:unified} where \Omega_{pos}, \Omega_{neg} are the sets of positive and negative categories for the sample, respectively. The form of this loss is easy to understand: if we want s_i < s_j, we add e^{s_i - s_j} into the \log. If we add a scaling factor \gamma and a margin m, we get the unified form in the Circle Loss paper: \log \left(1 + \sum\limits_{i\in\Omega_{neg},j\in\Omega_{pos}} e^{\gamma(s_i-s_j + m)}\right) = \log \left(1 + \sum\limits_{i\in\Omega_{neg}} e^{\gamma (s_i + m)}\sum\limits_{j\in\Omega_{pos}} e^{-\gamma s_j}\right) As a side note, the above is Equation (1) of the Circle Loss paper, but Equation (1) is not called Circle Loss; Equation (4) is. However, I believe Equation (1) is the most interesting part of the entire paper.
Application to Multi-Label Classification
\gamma and m are generally considered in metric learning, so here we only care about Equation \eqref{eq:unified}. If k is fixed in n-choose-k multi-label classification, then Equation \eqref{eq:unified} can be used directly as the loss, and the k categories with the largest scores are output during prediction.
For multi-label classification where k is not fixed, we need a threshold to determine which classes to output. To this end, we also introduce an extra class 0, hoping that the scores of target classes are all greater than s_0, and the scores of non-target classes are all less than s_0. As mentioned before, if we want s_i < s_j, we add e^{s_i - s_j} to the \log, so now Equation \eqref{eq:unified} becomes: \begin{aligned} &\log \left(1 + \sum\limits_{i\in\Omega_{neg},j\in\Omega_{pos}} e^{s_i-s_j}+\sum\limits_{i\in\Omega_{neg}} e^{s_i-s_0}+\sum\limits_{j\in\Omega_{pos}} e^{s_0-s_j}\right)\\ =&\log \left(e^{s_0} + \sum\limits_{i\in\Omega_{neg}} e^{s_i}\right) + \log \left(e^{-s_0} + \sum\limits_{j\in\Omega_{pos}} e^{-s_j}\right)\\ \end{aligned} If we set the threshold to 0, it simplifies to: \log \left(1 + \sum\limits_{i\in\Omega_{neg}} e^{s_i}\right) + \log \left(1 + \sum\limits_{j\in\Omega_{pos}} e^{-s_j}\right) \label{eq:final} This is the final loss form we obtained—a natural and concise generalization of “softmax + cross entropy” for multi-label classification tasks. It does not suffer from class imbalance because it does not turn multi-label classification into multiple binary classification problems, but rather into pairwise comparisons between target and non-target class scores. By virtue of the good properties of \text{logsumexp}, it automatically balances the weight of each term.
Here is a reference implementation in Keras:
def multilabel_categorical_crossentropy(y_true, y_pred):
"""Cross entropy for multi-label classification
Note: y_true and y_pred have the same shape. Elements in y_true are
either 0 or 1, where 1 indicates the target class and 0
indicates a non-target class.
Warning: Please ensure the range of y_pred is all real numbers.
In other words, y_pred should generally not have an
activation function, especially not sigmoid or softmax!
In the prediction stage, output classes where y_pred > 0.
If you have questions, please read and understand this
article carefully.
"""
y_pred = (1 - 2 * y_true) * y_pred
y_pred_neg = y_pred - y_true * 1e12
y_pred_pos = y_pred - (1 - y_true) * 1e12
zeros = K.zeros_like(y_pred[..., :1])
y_pred_neg = K.concatenate([y_pred_neg, zeros], axis=-1)
y_pred_pos = K.concatenate([y_pred_pos, zeros], axis=-1)
neg_loss = K.logsumexp(y_pred_neg, axis=-1)
pos_loss = K.logsumexp(y_pred_pos, axis=-1)
return neg_loss + pos_lossConclusion
The final conclusion is Equation \eqref{eq:final}, which is the unified loss for multi-label classification problems sought in this article. Everyone is welcome to test it and report the results. I have experimented with several multi-label classification tasks, and all could match the performance of binary classification schemes under fine-tuned weights.
It should be noted that besides standard multi-label classification problems, some common task formats can also be considered multi-label classification, such as sequence labeling based on 0/1 tagging. A typical example is my “half-pointer, half-tagging” design. Therefore, from this perspective, there are many tasks that can be regarded as multi-label classification to test Equation \eqref{eq:final}. I indeed tried \eqref{eq:final} in the previous triplet extraction example task_relation_extraction.py, and it finally achieved results consistent with those here.
Of course, finally, it must be stated that although theoretically Equation \eqref{eq:final} as a loss function for multi-label classification can automatically solve many problems, there is ultimately no absolutely perfect solution that guarantees improvement. Therefore, when you use it to replace your original multi-label classification scheme, there is no guarantee that there will be an improvement, especially if you have already handled the class imbalance problem through fine-tuning weights. The gain from Equation \eqref{eq:final} might be very limited. After all, the original intention of Equation \eqref{eq:final} is just to allow us to achieve most of the effects without excessive parameter tuning.
When reposting, please include the original article address: https://kexue.fm/archives/7359
For more detailed reposting matters, please refer to: “Scientific Space FAQ”