Recently, I joined the fun and participated in the “Xiao Bu Assistant Dialogue Short Text Semantic Matching” track of the Global AI Technology Innovation Contest. The task is a standard binary classification for short text sentence pairs. In this era where pre-trained Transformers are ubiquitous, such a task usually doesn’t present much difficulty. However, what makes this competition interesting is that the data is desensitized—meaning every character has been mapped to a numeric ID, and we cannot access the original text.
In this situation, can we still use pre-trained models like BERT? Certainly, but it requires some techniques and potentially some additional pre-training. In this post, I will share a baseline that combines classification, pre-training, and semi-supervised learning, which can be applied to desensitized data tasks.
Model Overview
The core idea of the model is actually a variant of PET (Pattern-Exploiting Training), which I introduced in the previous article “Is GPT-3 Necessary? No, BERT’s MLM Model Can Also Do Few-Shot Learning”. We use a single Masked Language Model (MLM) to complete everything, as shown in the diagram below:
As you can see, the entire model is just an MLM model. Specifically, we added two tokens, [YES] and [NO], to the vocabulary to represent the similarity between sentences. We use the output vector corresponding to [CLS] to predict the label of the sentence pair ([YES] or [NO]). The corpus is constructed by concatenating the sentence pairs in the usual way, randomly masking some tokens in both sentences, and then predicting those tokens at their corresponding output positions.
In this way, we simultaneously perform the sentence pair classification task (via the prediction result of [CLS]) and the MLM pre-training task (for other masked tokens). Furthermore, unlabeled samples (such as the test set) can also be thrown into training, as long as we don’t predict the [CLS] token for them. Thus, through the MLM model, we integrate classification, pre-training, and semi-supervised learning.
Reusing BERT
Can BERT still be used with desensitized data? Of course. For BERT, desensitized data simply means the Embedding layer is different; the other layers are still highly valuable. Therefore, reusing BERT primarily involves re-aligning the Embedding layer through pre-training.
In this process, initialization is crucial. First, we extract special tokens like [UNK], [CLS], and [SEP] from BERT’s Embedding layer and keep them unchanged. Next, we calculate the character frequencies for both the desensitized data and plain text data (plain text refers to any open-source general corpus, not necessarily the original text corresponding to the desensitized data). Then, we perform a simple alignment between the plain text vocabulary and the desensitized vocabulary based on frequency. This allows us to initialize the desensitized Embedding layer by taking the corresponding BERT Embeddings from the plain text characters.
Simply put, I use the BERT Embedding of the most frequent plain text character to initialize the most frequent desensitized character, and so on, to achieve a basic vocabulary alignment. My comparative experiments show that this operation significantly accelerates the model’s convergence.
Code Sharing
With that, the model introduction is basically complete. Using this
approach with the bert-base version, I achieved a score of
0.866 on the leaderboard and 0.952 offline (single model, no K-fold
ensemble; it seems everyone is experiencing a large gap between online
and offline scores). Here is my implementation using
bert4keras:
Github Address: https://github.com/bojone/oppo-text-match
Regarding the character frequencies of plain text data, I have already pre-calculated a list and synchronized it to Github; you can use it directly. I recommend training for 100 epochs, which takes about 6 hours on a 3090.
By the way, if you want to use the Large version of BERT, I do not recommend using the RoBERTa-wwm-ext-large released by HIT. The reason was mentioned in “Is GPT-3 Necessary? No, BERT’s MLM Model Can Also Do Few-Shot Learning”: for some reason, that version randomly initialized the weights of the MLM head, but we need to use the pre-trained MLM weights. If you need a Large version, I recommend the BERT Large released by Tencent UER.
Summary
There isn’t much else to say; I’ve just shared a simple baseline for the competition and written a quick blog post. I hope it helps everyone!
When reprinting, please include the original address of this article: https://kexue.fm/archives/8213
For more detailed reprinting matters, please refer to: “Scientific Space FAQ”