Some time ago, we released the weights for a model named SimBERT. Based on Google’s open-source BERT model and designed using the UniLM philosophy from Microsoft, SimBERT is a model further fine-tuned on a task that integrates retrieval and generation. Consequently, it possesses both similar question generation and similar sentence retrieval capabilities. However, at that time, apart from releasing a weight file and an example script, we did not provide a further explanation of the model’s principles and training process. In this article, we will supplement that information.
Open Source Address: https://github.com/ZhuiyiTechnology/simbert
UniLM
UniLM is a Transformer model that integrates NLU (Natural Language Understanding) and NLG (Natural Language Generation) capabilities. It was proposed by Microsoft in May last year, and in February this year, it was upgraded to v2. Our previous article, "From Language Models to Seq2Seq: Transformer is All About the Mask", briefly introduced UniLM, and it has already been integrated into bert4keras.
The core of UniLM is to grant the model Seq2Seq capabilities through a special Attention Mask. Suppose the input is "What do you want to eat?" and the target sentence is "White Cut Chicken." UniLM concatenates these two sentences into one: [CLS] What do you want to eat [SEP] White Cut Chicken [SEP], and then applies the Attention Mask as shown in the figure:
In other words, the tokens in [CLS] What do you want to eat [SEP] have bidirectional Attention between them, while the tokens in White Cut Chicken [SEP] have unidirectional Attention. This allows for the recursive prediction of the tokens in White Cut Chicken [SEP], thereby giving it text generation capabilities.
Seq2Seq only demonstrates that UniLM has NLG capabilities. Why did we say earlier that it possesses both NLU and NLG capabilities? Because of UniLM’s special Attention Mask, the 6 tokens in [CLS] What do you want to eat [SEP] only perform Attention among themselves and have absolutely no relationship with White Cut Chicken [SEP]. This means that even though White Cut Chicken [SEP] is appended afterward, it does not affect the encoding vectors of the first 6 tokens. To put it more clearly, the first 6 encoding vectors are equivalent to the encoding results when only [CLS] What do you want to eat [SEP] is present. If the vector of [CLS] represents the sentence vector, then it is the sentence vector for "What do you want to eat", not the sentence vector after adding "White Cut Chicken".
Due to this characteristic, UniLM also randomly adds some [MASK] tokens during input. This way, the input part can perform the MLM (Masked Language Model) task, and the output part can perform the Seq2Seq task. MLM enhances NLU capabilities, while Seq2Seq enhances NLG capabilities—achieving two goals at once.
SimBERT
Once UniLM is understood, it is not difficult to understand the training method of SimBERT. SimBERT belongs to supervised training. The training corpus consists of self-collected similar sentence pairs. The Seq2Seq part is constructed through a similar sentence generation task where one sentence predicts another. As mentioned earlier, the [CLS] vector effectively represents the input sentence vector, so it can simultaneously be used to train a retrieval task, as shown below:
Suppose SENT_a and SENT_b are a pair of
similar sentences. In the same batch, both [CLS] SENT_a [SEP] SENT_b [SEP] and [CLS] SENT_b [SEP] SENT_a [SEP] are added to
the training for a similar sentence generation task. This is the Seq2Seq
part.
On the other hand, the [CLS] vectors
for the entire batch are extracted to obtain a sentence vector matrix
\boldsymbol{V} \in \mathbb{R}^{b \times
d} (where b is the
batch_size and d is the
hidden_size). Then, l_2
normalization is performed on the d
dimension to obtain \tilde{\boldsymbol{V}}. We then calculate the
pairwise inner product to get a b \times
b similarity matrix \tilde{\boldsymbol{V}}\tilde{\boldsymbol{V}}^{\top}.
This is multiplied by a scale (we used 30), and the diagonal is masked.
Finally, a softmax is performed on each row to train it as a
classification task, where the target label for each sample is its
similar sentence (since the self-similarity has been masked). Simply
put, all non-similar samples within the batch are treated as negative
samples, and softmax is used to increase the similarity of the similar
samples while decreasing the similarity of the others.
Ultimately, the key is that "the [CLS] vector effectively represents the input sentence vector," so it can be used for NLU-related tasks. The final loss is the sum of the Seq2Seq loss and the similar sentence classification loss.
Other Details
Since the source code has been released, you can read the code for more training details. The model is implemented using Keras + bert4keras. The code is quite clear, so most doubts should be resolved by reading it.
Effect demonstration:
>>> gen_synonyms(u'Which is better, WeChat or Alipay?')
[
u'WeChat and Alipay, which is better?',
u'Which is better WeChat or Alipay',
u'Alipay and WeChat which is better',
u'Alipay and WeChat which is better ah',
u'Which one is more useful, WeChat or Alipay?',
u'Which is more useful WeChat or Alipay',
u'Alipay and WeChat which one is better',
u'Alipay and WeChat which is more useful',
u'Which one is better to use, WeChat or Alipay?',
u'Which one to choose, WeChat or Alipay',
u'Is WeChat better or is Alipay more useful',
u'WeChat vs Alipay which one',
u'Alipay and WeChat which is a bit more useful?',
u'Alipay is better or WeChat',
u'Which one is actually better, WeChat or Alipay',
u'Alipay and WeChat which has better practicality',
u'Okay, Alipay and WeChat which is safer?',
u'WeChat Alipay which is more useful? What is the difference',
u'What is the difference between WeChat and Alipay? Who is more useful',
u'Alipay and WeChat which is more fun'
]
>>> most_similar(u'How to get a certificate of first marriage and no children', 20)
[
(u'How to handle getting a certificate of first marriage and no children?', 0.9728098),
(u'How to open a certificate for first marriage and no children status?', 0.9612292),
(u'Where to get a certificate of first marriage and no children?', 0.94987774),
(u'Where is the certificate of first marriage and no children issued?', 0.9476072),
(u'Does the male side also need to get a first marriage certificate?', 0.7712214),
(u'Besides the village, can the workplace issue a first marriage certificate?', 0.63224965),
(u'How to send for having a baby', 0.40672967),
(u'It requires you to go to the local public security bureau to issue a change certificate', 0.39978087),
(u'What to do if Taobao shop certification fails', 0.39477515),
(u'Hello, it requires a change certificate issued by the local public security bureau', 0.39288986),
(u'How to apply for a credit card without a work certificate', 0.37745982),
(u'How to buy high-speed rail tickets for minors who don\'t have ID cards yet', 0.36504325),
(u'They won\'t issue a tobacco license, what should I do?', 0.35596085),
(u'How to have a baby', 0.3493368),
(u'How to open a welfare lottery station', 0.34158638),
(u'How to handle a Shenyang tobacco license? Is it easy?', 0.33718678),
(u'What are the characteristics of male infertility', 0.33530876),
(u'What to do if one copy of the marriage certificate is lost for divorce', 0.33166665),
(u'How to go to the local tax bureau to issue an invoice?', 0.33079252),
(u'What should be noted for male infertility checks?', 0.3274408)
]Readers might be concerned about the training data. To answer collectively: regarding the training data, it is not convenient to make it public or share it privately, so please do not ask about the data. The data source was crawled from recommended similar questions on Baidu Zhidao and then filtered through simple algorithms. If readers already have many question pairs, they can also use common retrieval algorithms to retrieve some similar sentences to use as training data. In short, there are no particularly strict requirements for training data; theoretically, any data with a certain degree of similarity will work.
As for the training hardware, the open-source model was trained on a
single TITAN RTX (22G VRAM, batch_size=128) for about 4
days. There are no hard requirements for VRAM or time; it depends on the
actual situation. If the VRAM is not that large, simply reduce the
batch_size appropriately. If the corpus itself is not very
large, the training time does not need to be that long (roughly enough
to traverse the dataset completely a few times).
That is all I can think of for now. If you have any other questions, feel free to leave a comment for discussion.
Summary
This article introduced the training principles of the SimBERT model we released earlier and open-sourced the training code. By training based on the UniLM philosophy, SimBERT possesses both retrieval and generation capabilities. Everyone is welcome to use and test it!
When reprinting, please include the original address of this article: https://kexue.fm/archives/7427
For more detailed reprinting matters, please refer to: "Scientific Space FAQ"