CoSENT (I): A More Effective Sentence Vector Scheme than Sentence-BERT

The Ineffective Cosine

For simplicity, let’s first consider "Yes/No type" data, i.e., samples of "(Sentence 1, Sentence 2, Is Similar)". Suppose two sentences yield vectors u and v after passing through the encoding model. Since the retrieval stage calculates the cosine similarity \cos(u,v)=\frac{\langle u,v\rangle}{\Vert u\Vert \Vert v\Vert}, a natural idea is to design a loss function based on \cos(u,v), such as: \begin{aligned} t\cdot (1 - \cos(u, v)) + (1 - t) \cdot (1 + \cos(u,v)) \label{eq:cos-1}\\ t\cdot (1 - \cos(u, v))^2 + (1 - t) \cdot \cos^2(u,v) \label{eq:cos-2} \end{aligned} where t\in\{0,1\} indicates whether they are similar. Many similar loss functions can be written; the general idea is to make the similarity of positive pairs as large as possible and the similarity of negative pairs as small as possible. However, experimental results from directly optimizing these targets are often very poor (at least significantly worse than InferSent), and in some cases, even worse than random initialization.

The Difficult Threshold

This is because the negative pairs labeled in text matching corpora are usually "hard negatives"—commonly, they are semantically different but have significant literal overlap. At this point, if we use Equation [eq:cos-1] as the loss function, the target for positive pairs is 1 and the target for negative pairs is -1. If we use Equation [eq:cos-2], the target for positive pairs is 1 and the target for negative pairs is 0. In either case, the target for negative pairs is "too low." For "hard negatives," although the semantics are different, they are still "similar" to some extent; the similarity shouldn’t be as low as 0 or -1. Forcing them toward 0 or -1 usually results in over-learning, leading to a loss of generalization ability, or making optimization so difficult that the model fails to learn at all.

To verify this conclusion, one only needs to replace the negative samples in the training set with randomly sampled pairs (viewed as weaker negative pairs) and train with the above loss; you will find that the performance actually improves. If the negative pairs are not changed, one way to alleviate this problem is to set a higher threshold for negative pairs, such as: t\cdot (1 - \cos(u, v)) + (1 - t) \cdot \max(\cos(u,v),0.7) In this way, as long as the similarity of negative pairs is lower than 0.7, they are no longer optimized, making it less prone to over-learning. However, this is only a mitigation; the performance is hard to optimize to the best level, and choosing this threshold remains a difficult problem.

Further Reflection

Why are InferSent and Sentence-BERT effective? In the "Closed-door Construction" section of "Enhancing RoFormer-Sim with Open Source Manually Labeled Data", I gave an explanation based on fault tolerance. After some thought, I have a new understanding of this problem.

Generally, even if negative pairs are "hard negatives," the literal similarity of positive pairs is overall greater than that of negative pairs. Thus, even for an initial model, the distance \Vert u-v\Vert for positive pairs is generally smaller, while the distance \Vert u-v\Vert for negative pairs is generally larger. We can imagine that u-v for positive pairs is mainly distributed near a sphere with a smaller radius, while u-v for negative pairs is distributed near a sphere with a larger radius. That is to say, u-v itself has a clustering tendency in the initial stage. We then only need to strengthen this clustering tendency based on label information, keeping u-v smaller for positive pairs and larger for negative pairs. A direct way is to follow u-v with a Dense classifier. However, conventional classifiers are based on inner products; they cannot distinguish between two classes distributed on different spheres. So we add the absolute value to get |u-v|, turning the sphere into a local spherical cap (or turning the sphere into a cone), at which point a Dense classification layer can be used. This is what I believe to be the origin of |u-v|.

As for the concatenation of u, v, I believe it is used to eliminate anisotropy. Sentence vector models like "BERT + [CLS]" have severe anisotropy in the initial stage, which has a serious negative impact on sentence vector performance. |u-v| is only the relative distance between vectors and cannot significantly improve this anisotropy. By concatenating u, v followed by a Dense layer, since the class vectors of the Dense layer are randomly initialized, it is equivalent to giving u and v a random optimization direction, forcing them to "spread out" and move away from the current anisotropic state.

Potential Problems

Although InferSent and Sentence-BERT are effective, they also have obvious problems.

For instance, as mentioned earlier, the reason they work is that there is a clustering tendency in the initial stage, and label training only strengthens this clustering information. Therefore, "having a clustering tendency in the initial stage" becomes quite important. This means the effect is quite dependent on the initial model; for example, the final effect of "BERT + Mean Pooling" is better than "BERT + [CLS]" because the former has better discriminative power in the initial stage.

Furthermore, InferSent and Sentence-BERT are ultimately schemes where training and prediction are inconsistent. Therefore, there is a certain probability of "training collapse," specifically manifested as the training loss still decreasing and training accuracy still increasing, but the evaluation metrics based on cosine values (such as the Spearman coefficient) significantly decreasing, even on the training set. This indicates that training is proceeding normally, but it has deviated from the classification basis of "positive pair u-v is smaller, negative pair u-v is larger," causing the cosine values to collapse.

InferSent and Sentence-BERT also suffer from tuning difficulties. This is also due to the inconsistency between training and prediction, making it difficult to determine which adjustments in the training process will bring positive help to the prediction results.

New Loss Function

Let \Omega_{pos} be the set of all positive pairs and \Omega_{neg} be the set of all negative pairs. We hope that for any positive pair (i,j)\in \Omega_{pos} and negative pair (k,l)\in \Omega_{neg}, we have: \cos(u_i,u_j) > \cos(u_k, u_l) where u_i,u_j,u_k,u_l are their respective sentence vectors. Simply put, we only want the similarity of positive pairs to be greater than that of negative pairs; as for how much greater, the model can decide for itself. In fact, the common evaluation metric for semantic similarity, Spearman, is the same: it only depends on the relative order of the prediction results, not the specific values.

In "Generalizing ’Softmax + Cross Entropy’ to Multi-label Classification", we introduced an effective solution for this type of requirement, which is formula (1) in Circle Loss theory: \log \left(1 + \sum\limits_{i\in\Omega_{neg},j\in\Omega_{pos}} e^{s_i-s_j}\right) Simply put, if you want to achieve s_i < s_j, then add the term e^{s_i-s_j} into the \log. Corresponding to our scenario, we can obtain the loss function: \log \left(1 + \sum\limits_{(i,j)\in\Omega_{pos},(k,l)\in\Omega_{neg}} e^{\lambda(\cos(u_k, u_l) - \cos(u_i, u_j))}\right) \label{eq:cosent} where \lambda > 0 is a hyperparameter; in the experiments in this article, it is set to 20. This is the core content of CoSENT, a new loss function for optimizing cosine values.

Universal Ranking

Some readers might question: even if Equation [eq:cosent] is usable, it only applies to binary classification data. What about NLI data, which is 3-class?

In fact, Equation [eq:cosent] is essentially a loss function designed for ranking. It can be written more generally as: \log \left(1 + \sum\limits_{\text{sim}(i,j) > \text{sim}(k,l)} e^{\lambda(\cos(u_k, u_l) - \cos(u_i, u_j))}\right) \label{eq:cosent-2} That is to say, as long as we believe the true similarity of pair (i,j) should be greater than the true similarity of (k,l), we can add e^{\lambda(\cos(u_k, u_l) - \cos(u_i, u_j))} into the \log. In other words, as long as we can design an order for the sample pairs, we can use Equation [eq:cosent-2].

For NLI data, it has three labels: "Entailment", "Neutral", and "Contradiction". We can naturally assume that the similarity of two "Entailment" sentences is greater than that of two "Neutral" sentences, and the similarity of two "Neutral" sentences is greater than that of two "Contradiction" sentences. Thus, based on these three labels, we can rank the NLI sentence pairs. With this ranking, NLI data can also be trained using CoSENT. Similarly, for data like STS-B, which is inherently scoring-based, it is even more suitable for CoSENT because the scoring labels themselves are ranking information.

Of course, if there is no such ordinal relationship between multiple categories, CoSENT cannot be used. However, for multi-class sentence pair data where an ordinal relationship cannot be constructed, I am also skeptical about whether InferSent and Sentence-BERT can produce reasonable sentence vector models. Currently, I haven’t seen such datasets, so it cannot be verified.

Excellent Results

I conducted experiments on CoSENT across multiple Chinese datasets, comparing two schemes: training on the original training set and training on the NLI dataset. Most experimental results show that CoSENT is significantly better than Sentence-BERT. The test datasets are the same as those in "Which Unsupervised Semantic Similarity is Stronger? A Comprehensive Evaluation". Each dataset was divided into train, valid, and test parts. The evaluation metric is the Spearman coefficient between the predicted values and the labels.

Experiment Code: https://github.com/bojone/CoSENT

Below are the results on the test sets after training on their respective train sets:

Below are the results on the test sets after training on the open-source NLI data:

As can be seen, CoSENT shows significant improvements in most tasks, and the decrease in a few tasks is very small (within 1%). The average improvement for native training exceeds 6%, and the average improvement for NLI training is around 1%.

In addition, CoSENT has a faster convergence speed. For example, in the native training of "BERT+CoSENT+ATEC", the valid result of the first epoch is 48.78, while the corresponding "Sentence-BERT+ATEC" is only 41.54; in the native training of "RoBERTa+CoSENT+PAWSX", the valid result of the first epoch is 57.66, while the corresponding "Sentence-RoBERTa+PAWSX" is only 10.84; and so on.

Connections and Differences

Some readers might ask: what is the difference between Equation [eq:cosent] or [eq:cosent-2] and SimCSE or contrastive learning? From the form of the loss function, there are some similarities, but the meanings are completely different.

Standard SimCSE only requires positive pairs (constructed through Dropout or manual labeling), and then it treats all other samples in the batch as negative samples. Supervised SimCSE requires triplet data; it actually adds hard negatives to the standard SimCSE, meaning negative samples include not only all other samples in the batch but also labeled hard negatives. At the same time, positive samples are still indispensable, so triplet data in the format of "(Original Sentence, Similar Sentence, Dissimilar Sentence)" is needed.

As for CoSENT, it only uses labeled positive and negative sentence pairs and does not involve the process of randomly sampling other samples in the batch to construct negative samples. We can also understand it as contrastive learning, but it is contrastive learning of "sentence pairs" rather than "samples" like SimCSE. That is to say, its "unit" is a pair of sentences rather than a single sentence.