CoSENT (2): How Big is the Gap Between Representation-based and Interaction-based Matching?

Automatic Thresholding

In the article "CoSENT (1): A More Effective Sentence Vector Scheme than Sentence-BERT", the metric we used to evaluate CoSENT was the Spearman correlation coefficient. This is a metric that only depends on the relative order of prediction results and does not rely on a threshold, making it suitable for evaluating retrieval scenarios. However, if the evaluation metrics are classification metrics like accuracy or F1, a threshold must be determined. Prediction results greater than this number are considered positive, and those smaller are considered negative, before the metrics can be calculated. In a binary classification scenario, we can effectively determine this threshold using a binary search.

However, searching for a threshold is not unique to binary classification; general multi-class tasks actually have the same requirement. So, let’s expand on this issue here. For example, for an n-class prediction distribution [p_1, p_2, \dots, p_n], we generally use the category with the maximum probability, i.e., \mathop{\text{argmax}}\,(p_1, p_2, \dots, p_n), as the predicted category. But in scenarios with class imbalance, this might not be optimal. We can search for a vector [t_1, t_2, \dots, t_n] using a validation set and then use: \mathop{\text{argmax}}\,(p_1 t_1, p_2 t_2, \dots, p_n t_n) as the predicted category, where [t_1, t_2, \dots, t_n] acts as the threshold in a multi-class scenario.

So, how do we search for [t_1, t_2, \dots, t_n]? The search objective is naturally to maximize the metric, but metrics like accuracy or F1 are non-differentiable, so gradient descent is out of the question. Since the parameters to be searched form a multi-dimensional vector, binary search is not very useful either. Here, I introduce a solution called the "Powell method." The mathematical details of the Powell method are extensive, so I won’t expand on them here. Simply put, the Powell method is an algorithm for solving low-dimensional unconstrained optimization problems. It does not require gradients and is relatively efficient. Here, "low-dimensional" means the parameters to be optimized usually do not exceed 100 (you can’t expect it to solve a neural network). Most importantly, the Powell method has a ready-made implementation in Scipy; you can call it by specifying method=’Powell’ in scipy.optimize.minimize.

For the above problem, the reference code is as follows:

import numpy as np
from scipy.optimize import minimize

def loss(t):
    """Here y_true.shape=[batch_size],
    y_pred.shape=[batch_size, num_classes]
    """
    t = (np.tanh(t) + 1) / 2
    return -np.mean(y_true == (y_pred * t[None]).argmax(1))

options = {'xtol': 1e-10, 'ftol': 1e-10, 'maxiter': 100000}
result = minimize(
    loss, np.zeros_like(y_pred[:1]), method='Powell', options=options
)
thresholds = (np.tanh(result.x) + 1) / 2

Experimental Results

With the automatic threshold determination method, we can now test classification performance. I conducted experiments on four datasets: ATEC, BQ, LCQMC, and PAWSX, comparing three schemes: CoSENT, Sentence-BERT, and Interaction-based (denoted as Interact). For fairness, each method used the Powell method to determine the optimal threshold on the validation set, and then reported the performance on the test set using that threshold, even for the Interaction-based model.

Experimental Code: https://github.com/bojone/CoSENT/tree/main/accuracy

The experimental results are as follows (metric is accuracy):

The experimental results show that from a performance perspective, the Interaction-based method indeed holds the "king" status, but the gap between representation-based methods (CoSENT and Sentence-BERT/RoBERTa) and the Interaction-based method is not as large as I imagined. Objectively speaking, on the ATEC and BQ tasks, there is no significant difference between Interact and CoSENT. On the LCQMC task, there is no significant difference between Interact and Sentence-BERT/RoBERTa.

The only dataset where a significant gap opened up is PAWSX. In "Which Unsupervised Semantic Similarity Method is Stronger? A Comprehensive Evaluation" and "Is it still SOTA for Chinese Tasks? Supplementary Experiments for SimCSE", we found that almost all unsupervised sentence vector methods fail on PAWSX. Why? Because the negative samples in PAWSX are almost all "adversarial samples"—negative samples with very high literal overlap but different meanings. Therefore, for these "highly difficult" negative samples where unsupervised methods "collapse across the board," even when using labeled data for training, deeper interaction is naturally required to better identify them.

Theoretical Limits

Some readers might be curious: Can we theoretically analyze the limit of representation-based schemes? Perhaps surprisingly, this analysis is not difficult, and the answer is:

Theoretically, almost anything an Interaction-based model can achieve, a Representation-based model can also achieve.

How do we arrive at this result? In fact, articles previously introduced on this blog are sufficient. First, let’s assume the similarity of sample pairs is between 0 and 1, and the sample pairs are unordered, i.e., \text{sim}(x,y) = \text{sim}(y,x). If there are n samples, and we calculate the similarity between every two samples (regardless of how the similarity is actually calculated), we get a similarity matrix S. It is a "positive definite symmetric matrix" (or more strictly, semi-positive definite). According to linear algebra, the SVD decomposition of a positive definite symmetric matrix must be in the form S = U \Lambda U^\top, where U is an orthogonal matrix and \Lambda is a diagonal matrix. Then we have S = U \Lambda U^\top = (U\sqrt{\Lambda})(U\sqrt{\Lambda})^\top. This indicates that a positive definite symmetric matrix can always be decomposed into the form S = BB^\top. This is equivalent to saying that each sample i can be represented as an n-dimensional vector v_i such that S_{i,j} = \langle v_i, v_j \rangle.

Up to this point, all results are theoretically guaranteed and exactly equal. However, the current "n-dimensional vector" is far too large, so we should think from the perspective of dimensionality reduction. At this point, the "JL Lemma" (refer to "The Amazing Johnson-Lindenstrauss Lemma: Theory") can step in. It tells us that regardless of the original dimensionality, n vectors can be reduced to \mathcal{O}(\log n) dimensions while keeping the inner product approximately unchanged. In "The Amazing Johnson-Lindenstrauss Lemma: Application", we also estimated that this magnitude should be around 8 \log n. Therefore, for a 768-dimensional BERT-base vector, it is theoretically no problem to fit the pairwise similarities of millions of samples through inner products. Thus, a "representation-based" scheme based on inner products with hundreds of dimensions can theoretically reach the performance of an interaction-based scheme quite accurately.

Then why is there a significant difference between the two on difficult datasets like PAWSX? Personally, I believe this is caused by the contradiction between the "continuity of neural networks and the cosine metric" and the "inherent adversarial nature of text matching."

A neural network itself is a continuous function, and the encoder is responsible for compressing a sentence into a sentence vector. The continuity of the result must be very good. Continuity here means that a small change in the sentence leads to a small change in the sentence vector. At the same time, the continuity of cosine similarity is also very good; that is, if \Delta v is small, the difference between \cos(u, v) and \cos(u, v + \Delta v) is also small. Therefore, overall, the continuity of "representation-based" schemes is very high. But the problem is that human-designed language is inherently adversarial; that is, a tiny change in literal expression can lead to a huge change in the labeled result. A classic example is adding the word "not," which leads to so-called "semantic reversal." Simply put, the continuity is not good.

Consequently, in such tasks, a "representation-based" scheme with very good continuity will find it very difficult to fit a dataset with obvious adversarial characteristics. Of course, we have already analyzed that it is theoretically possible to fit it. So, in practice, it can indeed be fitted, but it requires training for many epochs to "grind away" the original continuity of the representation-based scheme. However, more epochs also lead to more serious overfitting. Therefore, the training loss of CoSENT can also drop to near 0 (indicating that the fitting ability is not a problem), but the performance on the validation set is not as good as the Interaction-based model. As for the Interaction-based model, it is exposed to both samples from the beginning, and in the subsequent layers, the model can fit and amplify the differences on its own. Thus, the contradiction between continuity and adversarial nature is not as severe in the Interaction-based scheme, leading to better results.

Conclusion

This article explored the performance gap between representation-based matching and interaction-based matching from both theoretical and experimental perspectives. Additionally, it discussed the automatic search for thresholds in multi-class problems.

Original Address: https://kexue.fm/archives/8860