Last year, we released the SimBERT model, which has been one of our more successful open-source models and has received recognition from many readers. Simply put, SimBERT is a model that integrates generation and retrieval. It can be used as a relatively high baseline for sentence vectors and can also be used to automatically generate similar questions, serving as a pioneering tool for auxiliary data augmentation.
Recently, we further integrated and optimized SimBERT-related technologies based on the RoFormer base model, and finally released the upgraded RoFormer-Sim model.
Introduction
RoFormer-Sim is an upgraded version of SimBERT; we can also colloquially call it “SimBERTv2,” while SimBERT refers to the old version by default. From the outside, except for the base architecture being replaced by RoFormer, there is no obvious difference between RoFormer-Sim and SimBERT. In fact, their main differences lie in the training details, which we can compare using two formulas:
\begin{array}{c} \text{SimBERT} = \text{BERT} + \text{UniLM} + \text{Contrastive Learning} \\[5pt] \text{RoFormer-Sim} = \text{RoFormer} + \text{UniLM} + \text{Contrastive Learning} + \text{BART} + \text{Distillation} \end{array}
In addition, RoFormer-Sim uses more training data and has been
extended to general sentence patterns. That is to say, unlike SimBERT,
which was limited to interrogative sentences (questions), RoFormer-Sim
can be used to generate similar sentences for general statements, making
it applicable to more scenarios. Other training details include
RoFormer-Sim using a larger batch_size and
maxlen, which we will introduce further later.
Open Source Address: https://github.com/ZhuiyiTechnology/roformer-sim
Corpus
The key to both SimBERT and RoFormer-Sim lies in the construction of the training corpus. The training corpus for RoFormer-Sim consists of two parts: 1. Question-type similar sentences; 2. General-type similar sentences. For question-type similar sentences, we followed the SimBERT approach by collecting similar questions from Baidu Zhidao and then cleaning them through rules; this part is already very mature for us. For general-type similar sentences, we had no ready-made place to collect them, so we proposed two schemes that can construct (pseudo) similar sentence pairs in an unsupervised manner to some extent.
The first scheme is based on the idea that “answers to the same question are similar.” If we have existing QA corpora where one question has multiple answers, we can segment each answer into sentences and then use an existing similarity function to compare the similarity between answers, selecting sentence pairs with similarity exceeding a certain threshold as similar sentence pairs.
The second scheme is based on the idea that “sentences within the same passage are similar.” This is simpler and more direct: segment each passage into sentences, calculate the similarity between all pairs using an existing similarity function, and select pairs exceeding a certain threshold. Obviously, the rationality of this scheme is weaker, so its threshold is set higher.
This involves an “existing similarity function.” We directly used a variant of Jaccard similarity; in other words, only a rule-based, character-level similarity is needed. Semantic associations are obtained through the internal correlations within the passage and the generalization ability of the pre-trained model itself. Through the first scheme, we constructed about 4.5 million (pseudo) similar sentence pairs from several reading comprehension datasets. Through the second scheme, we constructed about 4.7 million (pseudo) similar sentence pairs from over 30GB of parallel corpora. The crawled questions reached about 30 million similar sentence groups (one group can form multiple pairs). From this perspective, the number of questions far exceeds general sentence patterns, so we sampled them at a 1:1 ratio to balance the samples for each sentence type.
Generation
The training method for RoFormer-Sim is basically the same as SimBERT, as shown in the figure below. A slight difference is that to enhance the model’s generation capability, when constructing the training corpus, we also randomly replaced some tokens of the input sentence with [MASK]. This pre-training method was first proposed by BART. Our difference from BART is: BART is “input a noisy sentence, output the original sentence,” while we are “input a noisy sentence, output a similar sentence to the original one.” Theoretically, our task is even more difficult.
There are no particularly good evaluation metrics for generation effects; we can just look at some examples:
gen_synonyms(u'Which is better, Guangzhou or Shenzhen?')
[
'Shenzhen or Guangzhou, which is better?',
'Guangzhou or Shenzhen, which is better',
'Which is even better, Guangzhou or Shenzhen?',
'Which is even better, Shenzhen or Guangzhou?',
'Shenzhen and Guangzhou, which one is better?',
'Which one is a bit better, Shenzhen or Guangzhou?',
'Is Shenzhen better or Guangzhou better?',
'Which place is a bit better, Guangzhou or Shenzhen?',
'Is Guangzhou better or Shenzhen better?',
'Guangzhou or Shenzhen, which is a bit better',
'Which has better development, Guangzhou or Shenzhen?',
'Is Shenzhen better or Guangzhou better',
'Which city is better, Shenzhen or Guangzhou',
'Is Shenzhen better than Guangzhou?',
'In the end, which is better, Shenzhen or Guangzhou? Why?',
'Is Shenzhen actually better or is Guangzhou better',
'Generally, is Shenzhen better or Guangzhou better',
'Which has slightly better development, Guangzhou or Shenzhen',
'Between the better ones, is Shenzhen or Guangzhou better?',
'Where is Shenzhen better than Guangzhou?'
]
gen_synonyms(u'Science and technology are the primary productive forces.')
[
'Science and technology are the primary productive forces!',
'Science and technology are the primary productive forces',
'1. Science and technology are the primary productive forces.',
'First, science and technology are the primary productive forces.',
'Firstly, science and technology are the primary productive forces.',
'The primary productive force is science and technology.',
'Because science and technology are the primary productive forces.',
'Science and technology are known as the primary productive forces.',
'That is, science and technology are the primary productive forces.',
'Are science and technology the primary productive forces',
'Tech is the primary productive force.',
'Therefore, science and technology are the primary productive forces.',
'Secondly, science and technology are the primary productive forces.',
'Science and technology are truly the primary productive forces.',
'Are science and technology the primary productive forces?',
'Second, science and technology are the primary productive forces.',
'So it is said that science and technology are the primary productive forces.',
'Science and technology indeed are the primary productive forces.',
'Science and technology remain the primary productive forces',
'Is it correct that science and technology are the primary productive forces?'
]Overall, similar sentence augmentation for arbitrary sentence patterns has been preliminarily achieved, but the augmentation effect for questions is better than for general sentence patterns. This is because the quality of questions in the training corpus is significantly higher than that of general sentence patterns. Since BART-style training was performed, in addition to direct similar sentence generation, we can also manually mask certain parts and let the model expand divergently on its own, for example:
# Masking "primary" (indices 6, 7 in the original Chinese string)
gen_synonyms(u'Science and technology are the primary productive forces.', mask_idxs=[6, 7])
[
"Science and technology are the primary productive forces",
"2. Science and technology are the primary productive forces.",
"Science and technology are the primary productive forces, and also the secondary productive forces.",
"Science and technology are the primary productive forces, scientific development is the secondary.",
"9. Science and technology are the primary productive forces.",
"First, science and technology are a kind of productive force.",
"Science and technology are productive forces.",
"Science and technology are the secondary productive forces.",
"The phrase 'Science and technology are the primary productive forces' is proposed now.",
"1. Science and technology are a type of productive force.",
"What does 'Science and technology are the primary productive forces' mean",
"Science and technology are a major productive force.",
"One: Science and technology are the highest productive forces.",
"Refers to science and technology not being the primary productive forces.",
"Science and technology are the secondary productive forces, and the primary is the secondary.",
"Two, science and technology are a productive force.",
"The world's first productive force is science and technology.",
"Science and technology are one of the socialist productive forces.",
"Second, science and technology are also the secondary productive forces.",
"Tech is all productive forces."
]We invite everyone to discover more ways to play with this.
Retrieval
Adding general sentence pattern corpora and introducing BART-style training are changes that relatively improved the generation model’s performance. However, we unexpectedly found that the performance of the retrieval model (i.e., the sentence encoding model) decreased. The estimated reason might be that while more corpora and greater noise increased the difficulty for the generation model, for contrastive learning, these samples of different sentence patterns or with noise served as negative samples that were actually easier to distinguish. For example, if a batch contains both interrogative and declarative sentences, the model can easily identify many negative samples through sentence structure (rather than semantics), thereby reducing its ability to understand semantics.
Of course, the essential positioning of both SimBERT and RoFormer-Sim is as similar sentence augmentation models; the retrieval model is just its “by-product.” However, we still hope this “by-product” can be as good as possible. To this end, after training RoFormer-Sim, we further transferred the retrieval performance of SimBERT to RoFormer-Sim through distillation, making the retrieval performance of RoFormer-Sim basically equal to or even better than SimBERT. The distillation method is simple: for the same batch of sentences, if the sentence vectors from SimBERT are u_1, u_2, \cdots, u_n and the sentence vectors from RoFormer-Sim are v_1, v_2, \cdots, v_n, then we learn using the following loss:
\mathcal{L}_{\text{sim}} = \frac{\lambda}{n^2}\sum_{i=1}^n\sum_{j=1}^n (\cos(u_i,u_j)-\cos(v_i,v_j))^2
where \lambda=100. Of course, to prevent the model from “forgetting” the generation model, the generation loss is also added during distillation, i.e., \mathcal{L}=\mathcal{L}_{\text{sim}}+\mathcal{L}_{\text{gen}}. The distillation for the base version does not require many steps; it can be completed in roughly 5,000 steps.
Following “Which Unsupervised Semantic Similarity Method is Stronger? We Did a Comprehensive Evaluation,” we used the same tasks to compare the retrieval effects of SimBERT and RoFormer (the three data points in each cell represent the effects of “without whitening,” “with whitening,” and “with whitening-256,” same as the previous evaluation):
\small{\begin{array}{l|ccccc} \hline & \text{ATEC} & \text{BQ} & \text{LCQMC} & \text{PAWSX} & \text{STS-B} \\ \hline \text{V1}\text{-P1} & 38.50 / \color{red}{23.64} / \color{red}{30.79} & 48.54 / \color{red}{31.78} / \color{red}{40.01} & 76.23 / \color{red}{75.05} / \color{red}{74.50} & 15.10 / \color{green}{18.49} / \color{green}{15.64} & 74.14 / \color{red}{73.37} / \color{green}{75.29} \\ \text{V1}\text{-P2} & 38.93 / \color{red}{27.06} / \color{red}{30.79} & 49.93 / \color{red}{35.38} / \color{red}{40.14} & 75.56 / \color{red}{73.45} / \color{red}{74.39} & 14.52 / \color{green}{18.51} / \color{green}{15.74} & 73.18 / \color{green}{73.43} / \color{green}{75.12} \\ \text{V1}\text{-P3} & 36.50 / \color{red}{31.32} / \color{red}{31.24} & 45.78 / \color{red}{29.17} / \color{red}{40.98} & 74.42 / \color{red}{73.79} / \color{red}{73.43} & 15.33 / \color{green}{18.39} / \color{green}{15.87} & 67.31 / \color{green}{70.70} / \color{green}{72.00} \\ \text{V1}\text{-P4} & 33.53 / \color{red}{29.04} / \color{red}{28.78} & 45.28 / \color{red}{34.70} / \color{red}{39.00} & 73.20 / \color{red}{71.22} / \color{red}{72.09} & 14.16 / \color{green}{17.32} / \color{green}{14.39} & 66.98 / \color{green}{70.55} / \color{green}{71.43} \\ \hline \text{V2}\text{-P1} & 39.52 / \color{red}{25.31} / \color{red}{31.10} & 50.26 / \color{red}{33.47} / \color{red}{40.16} & 76.02 / \color{red}{74.92} / \color{red}{74.58} & 14.37 / \color{green}{19.31} / \color{green}{14.81} & 74.46 / \color{red}{71.00} / \color{green}{76.29} \\ \text{V2}\text{-P2} & 39.71 / \color{red}{32.60} / \color{red}{30.89} & 50.80 / \color{red}{37.62} / \color{red}{40.12} & 75.83 / \color{red}{73.45} / \color{red}{74.52} & 13.87 / \color{green}{19.50} / \color{green}{14.88} & 73.47 / \color{green}{74.56} / \color{green}{76.40} \\ \text{V2}\text{-P3} & 39.55 / \color{red}{24.61} / \color{red}{31.82} & 50.25 / \color{red}{29.59} / \color{red}{41.43} & 74.90 / \color{red}{73.95} / \color{red}{74.06} & 14.57 / \color{green}{18.85} / \color{green}{15.26} & 68.89 / \color{green}{71.40} / \color{green}{73.36} \\ \text{V2}\text{-P4} & 36.02 / \color{red}{29.71} / \color{red}{29.61} & 48.22 / \color{red}{35.02} / \color{red}{39.52} & 73.76 / \color{red}{71.19} / \color{red}{72.68} & 13.60 / \color{green}{16.67} / \color{green}{13.86} & 68.39 / \color{green}{71.04} / \color{green}{72.43} \\ \hline \end{array}}
As can be seen from the table, regardless of whether whitening is added, RoFormer-Sim outperforms SimBERT on most tasks. This shows that the retrieval effect of RoFormer-Sim after distillation can indeed be improved, and this “by-product” is not too bad.
Using the same method, we also created a small version of RoFormer-Sim. In this case, the base version of RoFormer-Sim was used as the teacher model for distillation, but the number of distillation steps needed to be higher (around 500,000). The final results are as follows:
\small{\begin{array}{l|ccccc} \hline & \text{ATEC} & \text{BQ} & \text{LCQMC} & \text{PAWSX} & \text{STS-B} \\ \hline \text{V1}_{\text{small}}\text{-P1} & 30.68 / \color{red}{27.56} / \color{red}{29.07} & 43.41 / \color{red}{30.89} / \color{red}{39.78} & 74.73 / \color{red}{73.21} / \color{red}{73.50} & 15.89 / \color{green}{17.96} / \color{green}{16.75} & 70.54 / \color{green}{71.39} / \color{green}{72.14} \\ \text{V1}_{\text{small}}\text{-P2} & 31.00 / \color{red}{29.14} / \color{red}{29.11} & 43.76 / \color{red}{36.86} / \color{red}{39.84} & 74.21 / \color{red}{73.14} / \color{red}{73.67} & 16.17 / \color{green}{18.12} / \color{green}{16.81} & 70.10 / \color{green}{71.40} / \color{green}{72.28} \\ \text{V1}_{\text{small}}\text{-P3} & 30.03 / \color{red}{21.24} / \color{red}{29.30} & 43.72 / \color{red}{31.69} / \color{red}{40.81} & 72.12 / \color{red}{70.27} / \color{red}{70.52} & 16.93 / \color{green}{21.68} / \color{green}{18.75} & 66.55 / \color{red}{66.11} / \color{green}{69.19} \\ \text{V1}_{\text{small}}\text{-P4} & 29.52 / \color{red}{28.41} / \color{red}{28.57} & 43.52 / \color{red}{36.56} / \color{red}{40.49} & 70.33 / \color{red}{68.75} / \color{red}{69.01} & 15.39 / \color{green}{21.57} / \color{green}{16.34} & 64.73 / \color{green}{68.12} / \color{green}{68.24} \\ \hline \text{V2}_{\text{small}}\text{-P1} & 37.33 / \color{red}{23.59} / \color{red}{31.31} & 47.90 / \color{red}{29.21} / \color{red}{42.07} & 74.72 / \color{green}{74.94} / \color{red}{74.69} & 13.41 / \color{green}{15.30} / \color{green}{13.61} & 71.48 / \color{red}{69.01} / \color{green}{75.10} \\ \text{V2}_{\text{small}}\text{-P2} & 37.42 / \color{red}{31.25} / \color{red}{31.18} & 49.15 / \color{red}{38.01} / \color{red}{41.98} & 75.21 / \color{red}{73.47} / \color{red}{74.78} & 13.38 / \color{green}{15.87} / \color{green}{13.69} & 72.06 / \color{green}{73.92} / \color{green}{75.69} \\ \text{V2}_{\text{small}}\text{-P3} & 36.71 / \color{red}{30.33} / \color{red}{31.25} & 49.73 / \color{red}{31.03} / \color{red}{42.74} & 74.25 / \color{red}{72.72} / \color{red}{74.19} & 14.58 / \color{green}{18.68} / \color{red}{14.40} & 69.12 / \color{green}{71.07} / \color{green}{72.68} \\ \text{V2}_{\text{small}}\text{-P4} & 32.80 / \color{red}{27.87} / \color{red}{29.65} & 46.80 / \color{red}{36.93} / \color{red}{41.31} & 72.30 / \color{red}{69.94} / \color{green}{72.38} & 13.45 / \color{green}{16.93} / \color{red}{13.38} & 67.21 / \color{green}{70.42} / \color{green}{71.39} \\ \hline \end{array}}
Summary
This article introduced and released our upgraded version of SimBERT—RoFormer-Sim (SimBERTv2). It can be used both to expand similar sentences and as a high baseline for semantic similarity problems. Compared to SimBERT, its biggest feature is the extension of sentence patterns to general types, no longer limited to similar questions. We welcome readers to further explore and share more ways to use it!
When reprinting, please include the original address of this article: https://kexue.fm/archives/8454
For more detailed reprinting matters, please refer to: “Scientific Space FAQ”