Enhancing RoFormer-Sim with Open-Source Human-Annotated Data

Recap

I have found that since the release of SimBERT, the most frequent question from readers is likely:

Why is the similarity between "I like Beijing" and "I don’t like Beijing" so high? Aren’t their meanings opposite?

Especially after the release of RoFormer-Sim, similar questions appear almost every week or two. Furthermore, not only in my own "Scientific Space" exchange group but also in other NLP-related groups, similar questions pop up from time to time, indicating that such confusion is widespread.

So, how should we understand this?

First, the notion of "opposite meaning" is somewhat incorrect. From the perspective of similarity, there are only "similar" and "dissimilar" descriptions; there is no formal concept of "opposite." In principle, no two sentences are absolutely unrelated, so theoretically, no two sentences have a similarity of 0, let alone a clearly defined "opposite." On the contrary, what we usually consider "antonyms" are, objectively speaking, quite similar words. For example, "like" and "hate" share many commonalities: both are verbs, both describe emotional tendencies, and their usage is similar. So how can we say these two words are "not similar at all" or even "opposite"? When we call them antonyms, we mean they are in an opposing relationship within a very small dimension. Note that it is only one dimension, not all of them. This implies that our perception itself is non-objective (if they are similar in so many dimensions and dissimilar in only one, and we call them "antonyms," is that not non-objective?).

Similarly, in my understanding, from an objective perspective, "I like Beijing" and "I don’t like Beijing" are very similar. Therefore, it is reasonable for the model to give a high similarity score, and unreasonable to give a low one. Of course, I am not saying that "I like Beijing" and "I don’t like Beijing" are similar in every scenario; they indeed have opposing dimensions. The problem is that unsupervised and weakly supervised learning produce objective results. If we believe "I like Beijing" and "I don’t like Beijing" are dissimilar, it means we have subjectively selected the dimension we want to compare, rather than considering all objective dimensions. Since this is a subjective human behavior, we should not expect unsupervised or weakly supervised methods to learn it. The best way is to use annotated data for supervised learning.

To put it bluntly:

The model is not wrong; the human is. If the human insists they are right, then please tell the model it is wrong through supervised learning with annotated data.

Classification

Through the above discussion, we should understand the necessity of supervised learning with annotated data. Not all problems can be solved through unsupervised or weakly supervised means. If one insists on an unsupervised or weakly supervised solution, the cost may be far greater than labeling a few pieces of data.

As for Chinese human-annotated data related to similarity, there are currently three types collected:

1. Binary Type: This is the most common type. The main format is "(Sentence 1, Sentence 2, Is Similar)". Datasets like ATEC, BQ, LCQMC, and PAWSX collected here are of this type.

2. NLI Type: NLI stands for Natural Language Inference. The sample format is "(Sentence 1, Sentence 2, Entailment/Neutral/Contradiction)". It can be viewed as a more granular similarity dataset. Current Chinese NLI datasets are translated from English versions, with links located at CNSD.

3. Scoring Type: This is the most granular similarity corpus. The format is "(Sentence 1, Sentence 2, Similarity Degree)". This similarity degree is generally a finer-grained scale than 0/1. The current Chinese dataset available is STS-B, which is also translated from the corresponding English dataset.

Since the first two types have larger volumes, for convenience, we directly set a threshold to convert the third type (STS-B) into the first type. Thus, we utilize two data formats: 1. Binary classification of sentence pairs; 2. Three-class classification of sentence pairs.

Unexpected Findings

As mentioned at the beginning, although it is supervised training, it is not "obvious and easy." This is mainly because the choice of training method was somewhat unexpected. For simplicity, let’s start with the binary classification training samples.

Assume the sentence vectors obtained after the two sentences pass through the encoder are u and v. Since we usually use their cosine value \cos(u,v)=\frac{\langle u,v\rangle}{\Vert u\Vert \Vert v\Vert} for ranking during the retrieval phase, a natural idea is to design a loss function based on \cos(u,v). Some easily conceived ones are:

\begin{aligned} &t\cdot (\cos(u,v) - 1)^2 + (1 - t)\cdot \cos^2(u,v) \\ &t\cdot (\cos(u,v) - 1)^2 + (1 - t)\cdot (\cos(u,v) + 1)^2 \\ &t\cdot \max(0.9 - \cos(u,v), 0) + (1-t)\cdot \max(\cos(u,v) - 0.1, 0) \end{aligned}

where t\in\{0,1\} is the label for the sentence pair. The general idea of these losses is to make \cos(u,v) for positive pairs as large as possible and \cos(u,v) for negative pairs as small as possible.

However, in my experiments, such training schemes (where training and prediction are consistent) actually performed worse than a scheme that appears inconsistent between training and prediction, which originated from InferSent and was adopted by Sentence-BERT. Specifically, Sentence-BERT concatenates u, v, |u-v| (where |u-v| is the vector formed by taking the absolute value of each element of u-v) as features, followed by a fully connected layer for binary classification (or three-class classification for NLI datasets).

Of course, this is only the training scheme. When using it, the sentence vectors are still extracted and retrieved using cosine similarity. Thus, the scheme used by InferSent and Sentence-BERT is actually an inconsistent training and prediction scheme. Training does not directly involve \cos(u,v), yet \cos(u,v) can be used for retrieval during prediction, and it performs quite well, which is indeed unexpected.

Speculation

I was also quite puzzled by this. I noticed that in the Sentence-BERT paper, they compared the final effects of different feature concatenation methods, showing that the concatenation of u, v, |u-v| yielded the best results. If only parts of them were kept, the performance dropped significantly, as shown in the table below.

Inspired by this table, I "concocted" an explanation. First, we know that humans are very "picky," especially for similarity tasks; we usually consider only very strict similarity as "similar," but our training data is often not that precise. On one hand, the annotations themselves may contain noise; on the other hand, for some sample pairs, annotators might mark them as positive because they share the same topic (rather than semantics). That is to say, annotated data is usually not as strict as we require. If we directly use the annotation results to learn our ranking metric, it might bring unexpected biases.

Looking back at the approach of concatenating u, v, |u-v| followed by a fully connected layer, its scoring function is equivalent to: s = \langle u, w_1\rangle + \langle v, w_2\rangle + \langle |u-v|, w_3\rangle where w_1, w_2, w_3 are the corresponding parameter vectors. The first two terms \langle u, w_1\rangle + \langle v, w_2\rangle, if large, do not necessarily mean u and v are close; similarly, if small, they don’t mean u and v are far apart. Their role is more like a "topic classification" model used to identify whether the topics of u and v are consistent. As for the third term, we know |u-v|=0 \Leftrightarrow u=v, so the third term has the ability to judge the degree of proximity between the two vectors, perhaps representing true "semantic similarity."

In summary, we can consider that the approach of concatenating u, v, |u-v| followed by a fully connected layer includes both a score for judging topic consistency and a score for semantic similarity. It separates "topic" and "semantics," enhancing the model’s fault tolerance to data, thereby making the finally learned vectors better reflect pure and precise "semantics."

The Best of Both Worlds

Through the Sentence-BERT scheme and utilizing open-source similarity datasets, we can learn a reasonably good sentence vector model (i.e., a retrieval model). Extracting features and using cosine similarity as a metric yields good results. However, the problem is that SimBERT and RoFormer-Sim were never intended to be simple retrieval models; they aim to "have their cake and eat it too"—possessing both good retrieval effects and the ability to generate similar sentences.

To this end, after training a Sentence-BERT using the above method, we use the scheme introduced in "SimBERTv2 is here! RoFormer-Sim Model Integrating Retrieval and Generation" to distill the retrieval performance of Sentence-BERT into RoFormer-Sim. This improves the retrieval model’s performance while retaining similar sentence generation. Furthermore, distillation between models of the same size often improves performance slightly, so the retrieval effect of our distilled RoFormer-Sim is actually better than the directly trained Sentence-BERT.

Performance Demonstration

We have open-sourced the RoFormer-Sim trained with annotated data as follows (weights with -ft in the filename):

https://github.com/ZhuiyiTechnology/roformer-sim

Below are the test results (test set) for several tasks from "Which Unsupervised Semantic Similarity is Stronger? A Comprehensive Evaluation":

As can be seen, there is a significant improvement in performance, and the small version also performs quite impressively. Of course, after supervised training, improvement is inevitable, so the comparison in this table is not the main point. But for users, as long as there is a ready-to-use model, it doesn’t matter how it was created, right? Readers might be more concerned about whether this new model solves the "pain points" of previous retrieval models, such as whether it can widen the gap between "I like Beijing" and "I don’t like Beijing." Let’s look at some examples (base version; the results for the small version are nearly identical):

>>> similarity('The weather is nice today', 'The weather is very good today')
0.9769838
>>> similarity('The weather is nice today', 'The weather is not good today')
0.62359834
>>> similarity('I like Beijing', 'I really like Beijing')
0.9921096
>>> similarity('I like Beijing', 'I don't like Beijing')
0.5291042
>>> similarity('The movie is good', 'The movie is great')
0.96764225
>>> similarity('The movie is good', 'The movie is not good')
0.6312722
>>> similarity('Red apple', 'Green apple')
0.6974633
>>> similarity('Recommend a red car to me', 'Recommend a black car to me')
0.7191832
>>> similarity('Recommend a red car to me', 'Recommend a red car')
0.9866457
>>> similarity('Recommend a red car to me', 'Please give me a red car')
0.9460306

From the examples, it can be seen that after supervised training, the model indeed reflects similarity scores that are more in line with common human perception. For instance, the similarity decreases significantly after adding the word "not" (we found this effect is mainly brought by the NLI dataset). Also, it becomes more sensitive to colors like "red" and "black." Especially in the last three examples, it demonstrates that its retrieval ranking results are more consistent with our conventional intent recognition scenarios.

Summary

This article introduced our process of using annotated data to enhance RoFormer-Sim and open-sourced the corresponding trained models, providing a better open-source baseline for Chinese similarity models.

When reposting, please include the original address of this article: https://kexue.fm/archives/8541

For more detailed reposting matters, please refer to: "Scientific Space FAQ"