Which Unsupervised Semantic Similarity Method is the Best? A Comprehensive Evaluation

Purely Unsupervised

The first table presents the results of extracting sentence vectors directly using pre-trained BERT in a completely unsupervised manner. In the BERT-flow paper, it was confirmed that without any post-processing, the best pooling method for extracting sentence vectors based on BERT is the average of all token vectors from the first and last layers, namely first-last-avg (the BERT-flow paper mistakenly identified this as the average of the last two layers and labeled it as last2avg, but it is actually the first and last layers). Therefore, the following results are based on first-last-avg with either flow or whitening applied.

NLI Supervised

The second table presents the results of sentence vectors extracted from the Sentence-BERT model (SBERT) fine-tuned on the NLI dataset. In this case, first-last-avg is also the best, so the baseline for both flow and whitening is the sentence vector produced by first-last-avg. The NLI dataset is a natural language inference dataset, which is similar but not equivalent to semantic similarity. It can serve as supervised pre-training for semantic similarity tasks, but since it does not directly use semantic similarity data, it remains unsupervised relative to the semantic similarity task itself.

Dimension vs. Performance

In these two tables, the bold values represent the best results. A green arrow \color{green}{\uparrow} means that the BERT-whitening result is superior to the BERT-flow model under the same conditions, while a red arrow \color{red}{\downarrow} means the opposite. In other words, more green arrows mean that BERT-whitening performs better. The numbers 256 and 384 following "whitening" refer to the dimensions retained after reduction. As seen from these two tables, BERT-whitening generally outperforms BERT-flow, achieving SOTA on most tasks, and in most cases, dimensionality reduction further improves performance.

To further confirm the effect of dimensionality reduction, we plotted the relationship between retained dimensions and performance:

The figure above demonstrates the change in evaluation metrics relative to retained dimensions after whitening for various models on different tasks. It can be observed that for every task, the optimal performance is not achieved at full dimensionality. This means dimensionality reduction can bring performance gains. Furthermore, many tasks can even be reduced to 1/8 of the original dimensions or more while maintaining or even increasing performance. This fully demonstrates the engineering value of BERT-whitening, as dimensionality reduction significantly speeds up retrieval.

Evaluation Setup

This evaluation involves 11 models, 5 datasets, and 4 pooling methods. Each combination compares three post-processing methods: "no whitening," "whitening," and "whitening with dimensionality reduction." Thus, there are approximately 11 \times 5 \times 4 \times 3 = 660 experimental results, making it quite comprehensive. I say "approximately" because some pooling methods are not available for certain models. Since the computational cost of BERT-flow is significantly higher than BERT-whitening, we did not reproduce the BERT-flow results. However, as seen from the English tasks, the effects of BERT-whitening and BERT-flow are usually close, and BERT-whitening often outperforms BERT-flow, making whitening results representative.

The evaluation metric is the same as the English tasks: Spearman’s correlation coefficient. This is a ranking metric similar to AUC, which only depends on the order of predicted scores and is independent of thresholds, making it suitable for evaluating performance. The reason for not using the more familiar accuracy metric is twofold: first, accuracy depends on specific thresholds; second, the labels for the STS-B dataset are numbers from 1 to 5, not 0/1 labels, making accuracy impossible to calculate. Thus, Spearman’s correlation is used uniformly. If readers must understand it from an accuracy perspective, one can roughly assume "accuracy \approx 0.5 + \text{Spearman} / 2".

The 11 models are as follows:

• BERT: Google’s open-source Chinese BERT base version.

• RoBERTa: HFL’s open-source roberta_wwm_ext base version.

• NEZHA: Huawei’s open-source BERT base version with relative position encoding (wwm).

• WoBERT: A word-based BERT; the Plus version is used here.

• RoFormer: BERT with a new type of position encoding (Rotary).

• BERT_{large}: Tencent UER’s open-source BERT large version.

• RoBERTa_{large}: HFL’s open-source roberta_wwm_ext large version.

• NEZHA-large: Huawei’s open-source BERT large version with relative position encoding (wwm).

• SimBERT: BERT base version trained on similar sentences.

• SimBERT_{small}: BERT small version trained on similar sentences.

• SimBERT_{tiny}: BERT tiny version trained on similar sentences.

The 5 tasks are as follows:

• ATEC: ATEC Semantic Similarity Learning competition dataset, financial customer service scenario.

• BQ: HIT BQ Corpus dataset, question matching in the banking and finance domain.

• LCQMC: HIT LCQMC dataset, question matching covering multiple domains.

• PAWSX: Google’s dataset containing multilingual paraphrase and non-paraphrase pairs. It features high lexical overlap, making it difficult for unsupervised methods. Only the Chinese portion is used.

• STS-B: Calculates correlation between two sentences. Translated from the English version with manual corrections.

The 4 pooling methods are as follows:

• P1: Extract the [CLS] vector from the last layer of the encoder.

• P2: Extract the vector corresponding to the Pooler (used by BERT for NSP). The difference from P1 is an additional linear transformation.

• P3: Average all vectors from the last layer of the encoder.

• P4: Average all vectors from the first and last layers of the encoder.

Summary of Results

All experimental results are summarized in the following three tables. Each element in the tables is in the form a / b / c, representing the score for "no whitening" (a), "with whitening" (b), and "with whitening and appropriate dimensionality reduction" (c). If b \geq a, b is shown in green; otherwise, it is red. If c \geq a, c is shown in green; otherwise, it is red. "Appropriate dimensionality reduction" means reducing to 256 dimensions for base models, 384 for large models, and 128 for tiny and small models.

The first table compares 6 base models. Note that WoBERT and RoFormer do not have NSP tasks, so they lack P2 weights:

The second table compares the 3 large models:

The third table compares SimBERT models of different sizes:

Experimental Conclusions

Similar to the English task tables, green indicates that the whitening operation improved the quality of sentence vectors, while red indicates a decrease. More green indicates that the whitening method is more effective. From the tables above, we can draw several conclusions:

1. The test results for Chinese tasks are much more complex and irregular than those for English tasks. For example, in English tasks, P4 pooling is generally better than others, and large models generally outperform base models, but these trends are not obvious in Chinese tasks.

2. Except for SimBERT, there is generally more green than red, indicating that whitening has a positive effect on improving sentence vectors. Specifically, in the a/b/c format, c has significantly more green than b, showing that dimensionality reduction further improves performance. This means whitening is an algorithm that truly improves both speed and efficiency.

3. In the BQ task, the whitening method almost always led to a decrease. This is similar to the SICK-R task in English, illustrating that "there is no free lunch." Some tasks will cause the "isotropy" assumption to fail, in which case neither BERT-whitening nor BERT-flow will bring improvements.

4. SimBERT is the SOTA for all tasks except PAWSX. Of course, SimBERT is trained with supervised semantic similarity data (though theoretically, there is no overlap between training and test data), so comparing it with other models is not entirely fair. Regardless, SimBERT is open-source and available for everyone to use as a baseline.

5. Applying whitening to SimBERT either leads to a performance drop or brings negligible improvement. This suggests that for sentence vectors trained via supervised methods, further whitening is unnecessary and generally does not help.

6. PAWSX is indeed very difficult; the road for semantic similarity tasks is still long...