At the beginning of this year, inspired by BERT-flow, I conceived a method called "BERT-whitening," which once became the new SOTA for semantic similarity (refer to "You May Not Need BERT-flow: A Linear Transformation Comparable to BERT-flow", and the paper "Whitening Sentence Representations for Better Semantics and Faster Retrieval"). However, "the good times did not last long." Shortly after BERT-whitening was submitted to Arxiv, at least two new papers appeared on Arxiv with results significantly better than BERT-whitening.
The first is "Generating Datasets with Pretrained Language Models". This paper uses templates to unsupervisedly construct data pairs from GPT2_XL to train similarity models. Personally, I think while it is somewhat inspiring and the results are decent, the cost of reproduction and the number of variables are too high. The other is the protagonist of this article, "SimCSE: Simple Contrastive Learning of Sentence Embeddings". The SimCSE proposed in this paper significantly outperforms BERT-flow and BERT-whitening on English data, and the method is particularly simple!
So, is SimCSE equally effective in Chinese? Can it significantly improve the results of Chinese semantic similarity? This article aims to perform some supplementary experiments.
Open Source Address: https://github.com/bojone/SimCSE
SimCSE
First, let’s briefly introduce SimCSE. In fact, SimCSE can be seen as a simplified version of SimBERT (for SimBERT, please read "Having Your Cake and Eating It Too: SimBERT Model Integrating Retrieval and Generation"). Its simplifications are as follows:
SimCSE removes the generation part of SimBERT and only keeps the retrieval model;
Since SimCSE has no labeled data, it treats each sentence itself as a similar sentence to be passed in.
To put it bluntly, it is essentially a contrastive learning model trained with (self, self) as positive examples and (self, others) as negative examples. Of course, it is not actually that simple. If only exactly identical samples are used as positive examples, the generalization ability will be greatly reduced. Generally, we use some data augmentation methods to make the two samples of a positive pair slightly different. However, how to perform data augmentation in NLP is a difficult problem in itself. SimCSE proposes an extremely simple solution: directly use Dropout as data augmentation!
Specifically, N sentences pass through an Encoder with Dropout to obtain vectors \boldsymbol{h}^{(0)}_1, \boldsymbol{h}^{(0)}_2, \dots, \boldsymbol{h}^{(0)}_N. Then, the same batch of sentences is passed through the Encoder again (this time with a different random Dropout) to obtain vectors \boldsymbol{h}^{(1)}_1, \boldsymbol{h}^{(1)}_2, \dots, \boldsymbol{h}^{(1)}_N. We can treat (\boldsymbol{h}^{(0)}_i, \boldsymbol{h}^{(1)}_i) as a pair of (slightly different) positive examples. The training objective is:
-\sum_{i=1}^N\sum_{\alpha=0,1}\log \frac{e^{\cos(\boldsymbol{h}^{(\alpha)}_i, \boldsymbol{h}^{(1-\alpha)}_i)/\tau}}{\sum\limits_{j=1,j\neq i}^N e^{\cos(\boldsymbol{h}^{(\alpha)}_i, \boldsymbol{h}^{(\alpha)}_j)/\tau} + \sum\limits_j^N e^{\cos(\boldsymbol{h}^{(\alpha)}_i, \boldsymbol{h}^{(1-\alpha)}_j)/\tau}}
English Results
The original paper’s (English) experiments are quite extensive; readers can read the original text carefully. However, it should be noted that the evaluation metrics in the main table of the original paper are inconsistent with those of BERT-flow and BERT-whitening. The table with consistent metrics is in the appendix:
No matter how you compare them, SimCSE is clearly superior to BERT-flow and BERT-whitening. Is this advantage of SimCSE universal? Does it have this advantage in Chinese? We will conduct experiments immediately.
Experimental Configuration
Our Chinese experiments are basically aligned with "Which Unsupervised Semantic Similarity is Stronger? We Did a Comprehensive Evaluation", including the 5 tasks previously tested, 4 types of Pooling, and all base, small, and tiny versions of the models. The large version was not run because it OOM’ed (Out Of Memory) under the same configuration.
Open Source Address: https://github.com/bojone/SimCSE
After hyperparameter tuning, I found that the optimal parameters for SimCSE on Chinese tasks are not exactly the same as those in the original paper. The specific differences are as follows:
The original paper used
batch_size=512; here it isbatch_size=64(I really can’t afford such a luxurious batch size).The learning rate in the original paper was 5e-5; here it is 1e-5.
The optimal dropout rate in the original paper was 0.1; here it is 0.3.
The unsupervised SimCSE in the original paper was trained on extra data; here, 10,000 task data points were randomly selected for training.
The original unsupervised training included an MLM task; here, only SimCSE training is performed.
To clarify the last point: the unsupervised SimCSE in the original paper was trained on 1 million sentences selected from Wikipedia. For the Chinese experiments, for convenience and fairness in comparison, the task data was used directly for training (only sentences were used, no labels, so it is still unsupervised). However, except for PAWSX, the other 4 tasks do not require all data for training. After testing, only 10,000 randomly selected training samples trained for one epoch are needed to reach the optimal effect (more or fewer samples lead to worse results).
Chinese Results
All Chinese experimental results for SimCSE are as follows:
\small{\begin{array}{l|ccccc} \hline & \text{ATEC} & \text{BQ} & \text{LCQMC} & \text{PAWSX} & \text{STS-B} \\ \hline \text{BERT}\text{-P1} & 16.59 / 20.61 / \textcolor{green}{33.14} & 29.35 / 25.76 / \textcolor{green}{50.67} & 41.71 / 48.92 / \textcolor{green}{69.99} & 15.15 / 17.03 / \textcolor{red}{12.95} & 34.65 / 61.19 / \textcolor{green}{69.04} \\ \text{BERT}\text{-P2} & 9.46 / 22.16 / \textcolor{green}{25.18} & 16.97 / 18.97 / \textcolor{green}{41.19} & 28.42 / 49.61 / \textcolor{green}{56.45} & 13.93 / 16.08 / \textcolor{red}{12.46} & 21.66 / 60.75 / \textcolor{red}{57.63} \\ \text{BERT}\text{-P3} & 20.79 / 18.27 / \textcolor{green}{32.89} & 33.08 / 22.58 / \textcolor{green}{49.58} & 59.22 / 60.12 / \textcolor{green}{71.83} & 16.68 / 18.37 / \textcolor{red}{14.47} & 57.48 / 63.97 / \textcolor{green}{70.08} \\ \text{BERT}\text{-P4} & 24.51 / 27.00 / \textcolor{green}{31.96} & 38.81 / 32.29 / \textcolor{green}{48.40} & 64.75 / 64.75 / \textcolor{green}{71.49} & 15.12 / 17.80 / \textcolor{red}{16.01} & 61.66 / 69.45 / \textcolor{green}{70.03} \\ \hline \text{RoBERTa}\text{-P1} & 24.61 / 29.59 / \textcolor{green}{32.23} & 40.54 / 28.95 / \textcolor{green}{50.61} & 70.55 / 70.82 / \textcolor{green}{74.22} & 16.23 / 17.99 / \textcolor{red}{12.25} & 66.91 / 69.19 / \textcolor{green}{71.13} \\ \text{RoBERTa}\text{-P2} & 20.61 / 28.91 / \textcolor{red}{20.07} & 31.14 / 27.48 / \textcolor{green}{39.92} & 65.43 / 70.62 / \textcolor{red}{62.65} & 15.71 / 17.30 / \textcolor{red}{12.00} & 59.50 / 70.77 / \textcolor{red}{61.49} \\ \text{RoBERTa}\text{-P3} & 26.94 / 29.94 / \textcolor{green}{32.66} & 40.71 / 30.95 / \textcolor{green}{51.03} & 66.80 / 68.00 / \textcolor{green}{73.15} & 16.08 / 19.01 / \textcolor{red}{16.47} & 61.67 / 66.19 / \textcolor{green}{70.14} \\ \text{RoBERTa}\text{-P4} & 27.94 / 28.33 / \textcolor{green}{32.40} & 43.09 / 33.49 / \textcolor{green}{49.78} & 68.43 / 67.86 / \textcolor{green}{72.74} & 15.02 / 17.91 / \textcolor{red}{16.39} & 64.09 / 69.74 / \textcolor{green}{70.11} \\ \hline \text{NEZHA}\text{-P1} & 17.39 / 18.83 / \textcolor{green}{32.14} & 29.63 / 21.94 / \textcolor{green}{46.08} & 40.60 / 50.52 / \textcolor{green}{60.38} & 14.90 / 18.15 / \textcolor{red}{16.60} & 35.84 / 60.84 / \textcolor{green}{68.50} \\ \text{NEZHA}\text{-P2} & 10.96 / 23.08 / \textcolor{red}{15.70} & 17.38 / 28.81 / \textcolor{green}{32.20} & 22.66 / 49.12 / \textcolor{red}{21.07} & 13.45 / 18.05 / \textcolor{red}{12.68} & 21.16 / 60.11 / \textcolor{red}{43.35} \\ \text{NEZHA}\text{-P3} & 23.70 / 21.93 / \textcolor{green}{31.47} & 35.44 / 22.44 / \textcolor{green}{46.69} & 60.94 / 62.10 / \textcolor{green}{69.65} & 18.35 / 21.72 / \textcolor{red}{18.17} & 60.35 / 68.57 / \textcolor{green}{70.68} \\ \text{NEZHA}\text{-P4} & 27.72 / 25.31 / \textcolor{green}{30.26} & 44.18 / 31.47 / \textcolor{green}{46.57} & 65.16 / 66.68 / \textcolor{green}{67.21} & 13.98 / 16.66 / \textcolor{red}{14.41} & 61.94 / 69.55 / \textcolor{red}{68.18} \\ \hline \text{WoBERT}\text{-P1} & 23.88 / 22.45 / \textcolor{green}{32.66} & 43.08 / 32.52 / \textcolor{green}{49.13} & 68.56 / 67.89 / \textcolor{green}{72.99} & 18.15 / 19.92 / \textcolor{red}{12.36} & 64.12 / 66.53 / \textcolor{green}{70.00} \\ \text{WoBERT}\text{-P2} & \text{-} & \text{-} & \text{-} & \text{-} & \text{-} \\ \text{WoBERT}\text{-P3} & 24.62 / 22.74 / \textcolor{green}{34.03} & 40.64 / 28.12 / \textcolor{green}{49.77} & 64.89 / 65.22 / \textcolor{green}{72.44} & 16.83 / 20.56 / \textcolor{red}{14.55} & 59.43 / 66.57 / \textcolor{green}{70.96} \\ \text{WoBERT}\text{-P4} & 25.97 / 27.24 / \textcolor{green}{33.67} & 42.37 / 32.34 / \textcolor{green}{49.09} & 66.53 / 65.62 / \textcolor{green}{71.74} & 15.54 / 18.85 / \textcolor{red}{14.00} & 61.37 / 68.11 / \textcolor{green}{70.00} \\ \hline \text{RoFormer}\text{-P1} & 24.29 / 26.04 / \textcolor{green}{32.33} & 41.91 / 28.13 / \textcolor{green}{49.13} & 64.87 / 60.92 / \textcolor{green}{71.61} & 20.15 / 23.08 / \textcolor{red}{15.25} & 59.91 / 66.96 / \textcolor{green}{69.45} \\ \text{RoFormer}\text{-P2} & \text{-} & \text{-} & \text{-} & \text{-} & \text{-} \\ \text{RoFormer}\text{-P3} & 24.09 / 28.51 / \textcolor{green}{34.23} & 39.09 / 34.92 / \textcolor{green}{50.01} & 63.55 / 63.85 / \textcolor{green}{72.01} & 16.53 / 18.43 / \textcolor{red}{15.25} & 58.98 / 55.30 / \textcolor{green}{71.44} \\ \text{RoFormer}\text{-P4} & 25.92 / 27.38 / \textcolor{green}{34.10} & 41.75 / 32.36 / \textcolor{green}{49.58} & 66.18 / 65.45 / \textcolor{green}{71.84} & 15.30 / 18.36 / \textcolor{red}{15.17} & 61.40 / 68.02 / \textcolor{green}{71.40} \\ \hline \text{SimBERT}\text{-P1} & 38.50 / 23.64 / \textcolor{green}{36.98} & 48.54 / 31.78 / \textcolor{green}{51.47} & 76.23 / 75.05 / \textcolor{red}{74.87} & 15.10 / 18.49 / \textcolor{red}{12.66} & 74.14 / 73.37 / \textcolor{green}{75.12} \\ \text{SimBERT}\text{-P2} & 38.93 / 27.06 / \textcolor{green}{37.00} & 49.93 / 35.38 / \textcolor{green}{50.33} & 75.56 / 73.45 / \textcolor{red}{72.61} & 14.52 / 18.51 / \textcolor{green}{19.72} & 73.18 / 73.43 / \textcolor{green}{75.13} \\ \text{SimBERT}\text{-P3} & 36.50 / 31.32 / \textcolor{green}{37.81} & 45.78 / 29.17 / \textcolor{green}{51.24} & 74.42 / 73.79 / \textcolor{green}{73.85} & 15.33 / 18.39 / \textcolor{red}{12.48} & 67.31 / 70.70 / \textcolor{green}{73.18} \\ \text{SimBERT}\text{-P4} & 33.53 / 29.04 / \textcolor{green}{36.93} & 45.28 / 34.70 / \textcolor{green}{50.09} & 73.20 / 71.22 / \textcolor{green}{73.42} & 14.16 / 17.32 / \textcolor{red}{16.59} & 66.98 / 70.55 / \textcolor{green}{72.64} \\ \hline \text{SimBERT}_{\text{small}}\text{-P1} & 30.68 / 27.56 / \textcolor{green}{31.16} & 43.41 / 30.89 / \textcolor{green}{44.80} & 74.73 / 73.21 / \textcolor{green}{74.32} & 15.89 / 17.96 / \textcolor{red}{14.69} & 70.54 / 71.39 / \textcolor{red}{69.85} \\ \text{SimBERT}_{\text{small}}\text{-P2} & 31.00 / 29.14 / \textcolor{green}{30.76} & 43.76 / 36.86 / \textcolor{green}{45.50} & 74.21 / 73.14 / \textcolor{green}{74.55} & 16.17 / 18.12 / \textcolor{red}{15.18} & 70.10 / 71.40 / \textcolor{red}{69.18} \\ \text{SimBERT}_{\text{small}}\text{-P3} & 30.03 / 21.24 / \textcolor{green}{30.07} & 43.72 / 31.69 / \textcolor{green}{44.27} & 72.12 / 70.27 / \textcolor{green}{71.21} & 16.93 / 21.68 / \textcolor{red}{12.10} & 66.55 / 66.11 / \textcolor{red}{64.95} \\ \text{SimBERT}_{\text{small}}\text{-P4} & 29.52 / 28.41 / \textcolor{green}{28.56} & 43.52 / 36.56 / \textcolor{green}{43.38} & 70.33 / 68.75 / \textcolor{red}{68.35} & 15.39 / 21.57 / \textcolor{red}{14.47} & 64.73 / 68.12 / \textcolor{red}{63.23} \\ \hline \text{SimBERT}_{\text{tiny}}\text{-P1} & 30.51 / 24.67 / \textcolor{green}{30.04} & 44.25 / 31.75 / \textcolor{green}{43.89} & 74.27 / 72.25 / \textcolor{green}{73.47} & 16.01 / 18.07 / \textcolor{red}{12.51} & 70.11 / 66.39 / \textcolor{green}{70.11} \\ \text{SimBERT}_{\text{tiny}}\text{-P2} & 30.01 / 27.66 / \textcolor{green}{29.37} & 44.47 / 37.33 / \textcolor{green}{44.04} & 73.98 / 72.31 / \textcolor{green}{72.93} & 16.55 / 18.15 / \textcolor{red}{13.73} & 70.35 / 70.88 / \textcolor{red}{69.63} \\ \text{SimBERT}_{\text{tiny}}\text{-P3} & 28.47 / 19.68 / \textcolor{green}{28.08} & 42.04 / 29.49 / \textcolor{green}{41.21} & 69.16 / 66.99 / \textcolor{green}{69.85} & 16.18 / 20.11 / \textcolor{red}{12.21} & 64.41 / 66.72 / \textcolor{red}{64.62} \\ \text{SimBERT}_{\text{tiny}}\text{-P4} & 27.77 / 27.67 / \textcolor{red}{26.25} & 41.76 / 37.02 / \textcolor{green}{41.62} & 67.55 / 65.66 / \textcolor{green}{67.34} & 15.06 / 20.49 / \textcolor{red}{13.87} & 62.92 / 66.77 / \textcolor{red}{60.80} \\ \hline \end{array}}
In the table, the data in each cell is in the form "a / b / c", where a is the original result without any processing, b is the result of BERT-whitening (without dimensionality reduction), and c is the result of SimCSE. If c > b, then c is displayed in green; otherwise, it is in red. In other words, the more green there is, the better SimCSE performs compared to BERT-whitening. For other experimental details, please refer to the original code and "Which Unsupervised Semantic Similarity is Stronger? We Did a Comprehensive Evaluation".
Note that due to Dropout and the random sampling of 10,000 samples, the results have a certain degree of randomness. Running the code again will definitely result in fluctuations in the metrics.
Some Conclusions
From the experimental results, it can be seen that except for the "outlier" PAWSX, SimCSE indeed has an overwhelming advantage over BERT-whitening, with some tasks improving by more than 10 points. On BQ, SimCSE even performs better than the supervised-trained SimBERT. Furthermore, models like SimBERT, which have already undergone supervised training, can still achieve further improvements, all of which demonstrate the power of SimCSE. (As for why PAWSX is an "outlier," a simple analysis was already provided in the article "Which Unsupervised Semantic Similarity is Stronger? We Did a Comprehensive Evaluation").
At the same time, we can see that under SimCSE, the
first-last-avg pooling method, which performed well in
BERT-flow and BERT-whitening, no longer has any advantage. Instead,
directly taking the [CLS] vector performs better. Surprisingly, the
performance of the Pooler (adding a Dense layer on top of [CLS]) is
relatively poor, which is quite confusing.
Since BERT-whitening is just a linear transformation, I also experimented with whether SimCSE alone could replicate the effect of this linear transformation. Specifically, I fixed the weights of the Encoder, added a Dense layer without an activation function, and then trained only the final Dense layer with SimCSE as the objective. The results showed that SimCSE in this case was not as good as BERT-whitening. This means that for SimCSE to be effective, the Encoder must be fine-tuned. It also suggests that BERT-whitening might contain something that SimCSE does not. Perhaps combining the two in some way would yield even better results (currently under consideration...).
Summary
This article shared my Chinese experiments on SimCSE. The results show that SimCSE is indeed quite excellent on many tasks and can significantly outperform BERT-whitening.
Please include the address of this article when reposting: https://kexue.fm/archives/8348
For more detailed reposting matters, please refer to: "Scientific Space FAQ"