Text sentiment classification is essentially a binary classification problem. In fact, for classification models, there is often a common issue: the optimization objective is inconsistent with the evaluation metrics. Generally, for classification (including multi-class), we use cross-entropy as the loss function, which originates from Maximum Likelihood Estimation (refer to "Gradient Descent and EM Algorithm: Same Origin, Same Lineage"). However, our final evaluation goal is not to see how small the cross-entropy is, but to look at the model’s accuracy. Generally speaking, if the cross-entropy is very small, the accuracy will be high, but this relationship is not necessarily guaranteed.
Aiming for the Average vs. Aiming for the Top
A more common example is: a mathematics teacher is working hard to improve the students’ average score, but the final assessment metric is the pass rate (60 points to pass). If the average score is 100 (meaning all students scored 100), then naturally the pass rate is 100%, which is ideal. But reality is not always so beautiful. As long as the average score hasn’t reached 100, a higher average does not necessarily mean a higher pass rate. For example, if two people score 40 and 90 respectively, the average is 65, but the pass rate is only 50%. If both people score 60, the average is 60, but the pass rate is 100%. This means that the average score can be used as a target, but this target is not directly linked to the assessment metric.
So, to improve the final assessment goal, what should this teacher do? Obviously, first look at which students have already passed. Leave the passing students alone for now and focus on providing extra tutoring for the students who failed. In this way, in principle, many failing students can reach 60 points. It is also possible that some students who originally passed might fall below 60, but this process can be iterated, eventually bringing everyone above 60. Of course, the final average score might not be very high, but there’s no choice—after all, the assessment metric is the pass rate.
A Better Update Scheme
For binary classification models, we always hope the model outputs 1 for positive samples and 0 for negative samples. However, due to limitations in model fitting capacity, this is generally unachievable. In practice, during prediction, we consider anything greater than 0.5 as a positive sample and anything less than 0.5 as a negative sample. This implies that we can update the model "selectively." For example, if we set a threshold of 0.6, and the model’s output for a positive sample is already greater than 0.6, I will not update the model based on this sample. If the model’s output for a negative sample is less than 0.4, I will not update based on that sample either. Only samples falling between 0.4 and 0.6 will trigger a model update. In this way, the model will "concentrate its energy" on those "ambiguous" samples, thereby achieving better classification results. This is consistent with the traditional SVM philosophy.
Furthermore, such an approach can theoretically prevent overfitting, as it prevents the model from "desperately" fitting those samples that are already easy to fit (to further reduce the loss function). This is like a teacher who only cares about top students, hoping they improve from 80 to 90, without finding ways to improve the grades of struggling students—this is clearly not a good teacher.
Modified Cross-Entropy Loss
How can we achieve the goal mentioned above? It’s simple: just adjust the loss function. Here, we mainly draw on the ideas of hinge loss and triplet loss. The commonly used cross-entropy loss function is: L_{old} = -\sum_y y_{true} \log y_{pred}
Select a threshold m=0.6 (in principle, any threshold greater than 0.5 works). Introduce the unit step function \theta(x): \theta(x) = \left\{\begin{aligned}&1, x > 0\\ &\frac{1}{2}, x = 0\\ &0, x < 0\end{aligned}\right.
Now, consider the new loss function: L_{new} = -\sum_y \lambda(y_{true}, y_{pred}) y_{true}\log y_{pred} where \lambda(y_{true}, y_{pred}) = 1-\theta(y_{true}-m)\theta(y_{pred}-m)-\theta(1-m-y_{true})\theta(1-m-y_{pred}) L_{new} adds a correction term \lambda(y_{true}, y_{pred}) to the cross-entropy. What does this term mean? When a positive sample enters, y_{true}=1, so: \lambda(1, y_{pred})=1-\theta(y_{pred}-m) At this point, if y_{pred} > m, then \lambda(1, y_{pred})=0, and the cross-entropy automatically becomes 0 (reaching the minimum). Conversely, if y_{pred} < m, then \lambda(1, y_{pred})=1, and the cross-entropy is maintained. That is to say, if the output for a positive sample is already greater than m, it is no longer updated (because the minimum is reached, and the gradient can be considered 0); it only continues to update if it is less than m. Similarly, for negative samples, if the output is already less than 1-m, it stops updating; it only continues if it is greater than 1-m.
In this way, simply by replacing the original cross-entropy loss with the modified cross-entropy L_{new}, we can achieve the goal we designed at the beginning.
Experimental Testing Based on IMDB
The theory sounds beautiful, but is the actual effect as imagined? Let’s experiment immediately.
To make the results more comparable, I chose a standard task in text sentiment classification: the IMDB movie review classification. The tool used is the latest version of Keras (2.0). Most of the code can be found in the Keras examples, including LSTM and CNN.
First, the LSTM version:
from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Embedding, LSTM, Dense
from keras.datasets import imdb
from keras import backend as K
margin = 0.6
theta = lambda t: (K.sign(t)+1.)/2.
max_features = 20000
maxlen = 80
batch_size = 32
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
model = Sequential()
model.add(Embedding(max_features, 128))
model.add(LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(Dense(1, activation='sigmoid'))
def loss(y_true, y_pred):
return - (1 - theta(y_true - margin) * theta(y_pred - margin)
- theta(1 - margin - y_true) * theta(1 - margin - y_pred)
) * (y_true * K.log(y_pred + 1e-8) + (1 - y_true) * K.log(1 - y_pred + 1e-8))
model.compile(loss=loss,
optimizer='adam',
metrics=['accuracy'])
model.fit(x_train, y_train,
batch_size=batch_size,
epochs=15,
validation_data=(x_test, y_test))The code is basically copied from the official example, no
explanation needed. After running, the model achieved a training
accuracy of 99.01% and a test accuracy of 82.26%. If the loss is changed
directly to binary_crossentropy (with nothing else
changed), the training accuracy is 99.56% and the test accuracy is
81.02%. This indicates that the new loss function indeed helps prevent
overfitting and improves accuracy. Of course, there might be random
errors in this test, but the average of multiple runs still shows that
the new loss function brings about a 0.5% to 1% improvement in accuracy
(naturally, you cannot expect a leap in performance just by slightly
changing the loss function).
Now let’s look at the CNN version:
from keras.preprocessing import sequence
from keras.models import Sequential
from keras.layers import Embedding, Dense, Dropout, Activation
from keras.layers import Conv1D, GlobalMaxPooling1D
from keras.datasets import imdb
from keras import backend as K
margin = 0.6
theta = lambda t: (K.sign(t)+1.)/2.
max_features = 5000
maxlen = 400
batch_size = 32
embedding_dims = 50
filters = 250
kernel_size = 3
hidden_dims = 250
epochs = 10
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
model = Sequential()
model.add(Embedding(max_features,
embedding_dims,
input_length=maxlen))
model.add(Dropout(0.2))
model.add(Conv1D(filters,
kernel_size,
padding='valid',
activation='relu',
strides=1))
model.add(GlobalMaxPooling1D())
model.add(Dense(hidden_dims))
model.add(Dropout(0.2))
model.add(Activation('relu'))
model.add(Dense(1))
model.add(Activation('sigmoid'))
def loss(y_true, y_pred):
return - (1 - theta(y_true - margin) * theta(y_pred - margin)
- theta(1 - margin - y_true) * theta(1 - margin - y_pred)
) * (y_true * K.log(y_pred + 1e-8) + (1 - y_true) * K.log(1 - y_pred + 1e-8))
model.compile(loss=loss,
optimizer='adam',
metrics=['accuracy'])
model.fit(x_train, y_train,
batch_size=batch_size,
epochs=epochs,
validation_data=(x_test, y_test))After running, the model obtained a training accuracy of 98.66% and a
test accuracy of 88.24%. The result for pure
binary_crossentropy was 98.90% training accuracy and 88.14%
test accuracy. The two results are basically consistent within the range
of fluctuation. However, during the training process, the test results
using the new loss function remained stable at around 88.2%, while when
using cross-entropy, it jumped to 89%, then to 87%, and then back to
88%. That is to say, although the final accuracies were similar, the
fluctuations were larger with cross-entropy. We have reason to believe
that the model trained with the new loss function has better
generalization capabilities.
In Summary
This article mainly draws on the ideas of hinge loss and triplet loss to adjust the cross-entropy loss used for binary classification, making it more effective at fitting samples that are predicted incorrectly. Experiments also show that, in some sense, the new loss function can indeed bring a small improvement.
Furthermore, this idea can actually be applied to multi-class classification or even regression problems. I won’t give detailed examples here, but I will share more when I encounter specific analyses.
When reposting, please include the original address of this article: https://kexue.fm/archives/4293
For more detailed reposting matters, please refer to: "Scientific Space FAQ"