You May Not Need BERT-flow: A Linear Transformation Comparable to BERT-flow

Assumptions of Cosine Similarity

Generally, when we compare or retrieve semantic similarity, we calculate a sentence vector for each sentence and then compare or rank them using the cosine of the angle between them. However, have we ever considered this question: What assumptions does cosine similarity make about the input vectors? Or rather, what conditions must the vectors satisfy for cosine similarity to perform better?

We know that the geometric meaning of the dot product of two vectors \boldsymbol{x} and \boldsymbol{y} is “the product of their respective magnitudes and the cosine of the angle between them.” Therefore, cosine similarity is the dot product of two vectors divided by their respective magnitudes. The corresponding coordinate calculation formula is: \cos(\boldsymbol{x},\boldsymbol{y}) = \frac{\sum\limits_{i=1}^d x_i y_i}{\sqrt{\sum\limits_{i=1}^d x_i^2} \sqrt{\sum\limits_{i=1}^d y_i^2}} \label{eq:cos} However, do not forget one thing: the above equality holds only under an “orthonormal basis.” In other words, the “cosine of the angle” between vectors has a clear geometric meaning, but the right side of the equation is merely an operation on coordinates. Coordinates depend on the chosen basis; if the basis is different, the coordinate formula for the dot product changes, and thus the coordinate formula for the cosine value changes as well.

Therefore, assuming that BERT sentence embeddings already contain sufficient semantics (e.g., they can reconstruct the original sentence), if they perform poorly when using formula [eq:cos] to calculate cosine similarity, the reason might be that the coordinate system of the sentence embeddings at this time is not an orthonormal basis. So, how do we know which specific basis it uses? In principle, there is no way to know, but we can guess. The basis for our guess is that when choosing a basis for a set of vectors, we try to use each basis vector as evenly as possible. From a statistical perspective, this manifests as each component being independent and uniform. If this set of bases is orthonormal, the corresponding vector set should exhibit “isotropy.”

Of course, this is not a rigorous derivation but a heuristic guide. It tells us that if a set of vectors satisfies isotropy, we can consider it to originate from an orthonormal basis, and thus formula [eq:cos] can be used for similarity; conversely, if it does not satisfy isotropy, we can find a way to make it more isotropic before using formula [eq:cos]. BERT-flow precisely thought of using a “flow model” for this purpose.

Reflections on Flow Models

In my view, flow models are truly a type of model that leaves one with mixed feelings; I could write pages of musings about them, but I will try to keep it short here. In mid-2018, OpenAI released the Glow model, which looked very impressive. This attracted me to study flow models further, and I even reproduced Glow. My related work is recorded in “NICE in Flow: Basic Concepts and Implementation” and “RealNVP and Glow in Flow: Inheritance and Sublimation”. If you are not yet familiar with flow models, you are welcome to read those blog posts. Simply put, a flow model is a vector transformation model that can transform the distribution of input data into a standard normal distribution. Since a standard normal distribution is clearly isotropic, BERT-flow chose the flow model.

So, are there any issues with flow models? I actually complained about this before in the article “Invertible ResNet in Flow: The Ultimate Beauty of Violence”. To reiterate:

(Flow models) use clever designs to ensure that the inverse transformation of each layer is simple and that the Jacobian matrix is triangular, making the Jacobian determinant easy to calculate. Such models are theoretically elegant and beautiful, but there is a serious problem: because the inverse must be simple and the Jacobian determinant easy to calculate, the non-linear transformation capability of each layer is very weak. In fact, in models like Glow, only half of the variables are transformed in each layer. Therefore, to ensure sufficient fitting capability, the model must be stacked very deep (for example, for 256 \times 256 face generation, the Glow model stacks about 600 convolutional layers with 200 million parameters), and the computational cost is very high.

Seeing this, readers can understand why I said at the beginning that my first impression of BERT-flow was that it felt “off.” The above critique tells us that flow models are actually quite weak. Then, how large is the flow model used in BERT-flow? It is a Glow model with level=2 and depth=3. These parameters might not mean much to everyone, but basically, it is so small that the entire model adds almost no computational overhead. Therefore, my “off” intuition was:

The flow model itself is very weak, and the flow model used in BERT-flow is even weaker. Thus, it is unlikely that the flow model plays a vital role in BERT-flow. Thinking conversely, perhaps we can find a simpler and more direct method to achieve the same effect as BERT-flow.

Standardizing the Covariance Matrix

After exploration, I did find such a method. As the title of this article suggests, it is just a linear transformation.

The idea is simple: we know that a standard normal distribution has a mean of 0 and a covariance matrix that is the identity matrix. So, why don’t we try transforming the mean of the sentence vectors to 0 and the covariance matrix to the identity matrix? Suppose the set of (row) vectors is \{\boldsymbol{x}_i\}_{i=1}^N. We perform the transformation: \tilde{\boldsymbol{x}}_i = (\boldsymbol{x}_i - \boldsymbol{\mu})\boldsymbol{W} such that the mean of \{\tilde{\boldsymbol{x}}_i\}_{i=1}^N is 0 and the covariance matrix is the identity matrix. Readers familiar with traditional data mining may know that this is actually equivalent to the Whitening operation. Therefore, I call this method BERT-whitening.

Setting the mean to 0 is simple: let \boldsymbol{\mu}=\frac{1}{N}\sum_{i=1}^N \boldsymbol{x}_i. The difficulty lies in solving for the matrix \boldsymbol{W}. Let the covariance matrix of the original data be: \boldsymbol{\Sigma}=\frac{1}{N}\sum_{i=1}^N (\boldsymbol{x}_i - \boldsymbol{\mu})^{\top}(\boldsymbol{x}_i - \boldsymbol{\mu})=\left(\frac{1}{N}\sum_{i=1}^N \boldsymbol{x}_i^{\top}\boldsymbol{x}_i\right) - \boldsymbol{\mu}^{\top}\boldsymbol{\mu} Then it is not difficult to obtain the covariance matrix of the transformed data as \tilde{\boldsymbol{\Sigma}}=\boldsymbol{W}^{\top}\boldsymbol{\Sigma}\boldsymbol{W}. So we actually need to solve the equation: \boldsymbol{W}^{\top}\boldsymbol{\Sigma}\boldsymbol{W}=\boldsymbol{I}\quad\Rightarrow \quad \boldsymbol{\Sigma} = \left(\boldsymbol{W}^{\top}\right)^{-1}\boldsymbol{W}^{-1} = \left(\boldsymbol{W}^{-1}\right)^{\top}\boldsymbol{W}^{-1} We know that the covariance matrix \boldsymbol{\Sigma} is a positive semi-definite symmetric matrix, and it is usually positive definite when there is enough data. It has an SVD decomposition of the following form: \boldsymbol{\Sigma} = \boldsymbol{U}\boldsymbol{\Lambda}\boldsymbol{U}^{\top} where \boldsymbol{U} is an orthogonal matrix and \boldsymbol{\Lambda} is a diagonal matrix with positive diagonal elements. Therefore, letting \boldsymbol{W}^{-1}=\sqrt{\boldsymbol{\Lambda}}\boldsymbol{U}^{\top} completes the solution: \boldsymbol{W} = \boldsymbol{U}\sqrt{\boldsymbol{\Lambda}^{-1}}

The reference code in Numpy is:

def compute_kernel_bias(vecs):
    """Compute kernel and bias
    vecs.shape = [num_samples, embedding_size],
    Final transformation: y = (x + bias).dot(kernel)
    """
    mu = vecs.mean(axis=0, keepdims=True)
    cov = np.cov(vecs.T)
    u, s, vh = np.linalg.svd(cov)
    W = np.dot(u, np.diag(1 / np.sqrt(s)))
    return W, -mu

Some might ask what to do with large corpora. First, the above algorithm only needs to know the mean vector \boldsymbol{\mu}\in\mathbb{R}^{d} and the covariance matrix \boldsymbol{\Sigma}\in\mathbb{R}^{d\times d} of all sentence vectors (d is the dimension of the word vectors). \boldsymbol{\mu} is the mean of all sentence vectors \boldsymbol{x}_i, and the mean can be calculated recursively: \boldsymbol{\mu}_{n+1} = \frac{n}{n+1}\boldsymbol{\mu}_{n} + \frac{1}{n+1}\boldsymbol{x}_{n+1} Similarly, the covariance matrix \boldsymbol{\Sigma} is just the mean of all \boldsymbol{x}_i^{\top}\boldsymbol{x}_i minus \boldsymbol{\mu}^{\top}\boldsymbol{\mu}, which can also be calculated recursively: \boldsymbol{\Sigma}_{n+1} = \frac{n}{n+1}\left(\boldsymbol{\Sigma}_{n}+\boldsymbol{\mu}_{n}^{\top}\boldsymbol{\mu}_{n}\right) + \frac{1}{n+1}\boldsymbol{x}_{n+1}^{\top}\boldsymbol{x}_{n+1}-\boldsymbol{\mu}_{n+1}^{\top}\boldsymbol{\mu}_{n+1} Since it can be calculated recursively, it means we can compute \boldsymbol{\mu} and \boldsymbol{\Sigma} with limited memory. Therefore, BERT-whitening is not a problem for large corpora.

Comparison with BERT-flow

Now, we can test the effect of the aforementioned BERT-whitening. To compare with BERT-flow, I used bert4keras to test on the STS-B task. The reference script is at:

Github Link: https://github.com/bojone/BERT-whitening

The comparison of effects is as follows:

As can be seen, simple BERT-whitening indeed achieves results comparable to BERT-flow. Besides STS-B, my colleagues have performed similar comparisons on Chinese business data, and the results all indicate that the improvement brought by BERT-flow is similar to that of BERT-whitening. This suggests that the introduction of flow models may not be so necessary, as flow model layers are not common layers; they require specialized implementation and a certain amount of work to train. In contrast, the implementation of BERT-whitening is very simple—just a linear transformation—and can be easily applied to any sentence vector model. (Of course, if one insists on arguing, one could also say that whitening is a flow model implemented with a linear transformation...)

Note: I should add here that the “last2avg” mentioned in the BERT-flow paper originally meant the average vector of the outputs of the last two layers, but its code actually uses the average vector of the “first layer + last layer” outputs. See the related discussion in this ISSUE.

Dimensionality Reduction for Better Results

We now know that the transformation matrix \boldsymbol{W} = \boldsymbol{U}\sqrt{\boldsymbol{\Lambda}^{-1}} of BERT-whitening can transform the data’s covariance matrix into an identity matrix. What if we don’t consider \sqrt{\boldsymbol{\Lambda}^{-1}} and just use \boldsymbol{U} for the transformation? It is not hard to see that if we only use \boldsymbol{U}, the data’s covariance matrix becomes \boldsymbol{\Lambda}, which is a diagonal matrix.

As mentioned earlier, \boldsymbol{U} is an orthogonal matrix, which merely rotates the overall data without changing the relative positions between samples; in other words, it is a completely “fidelity-preserving” transformation. Each diagonal element of \boldsymbol{\Lambda} measures the variation of the data in that specific dimension. If its value is very small, it means the variation in that dimension is minimal, close to a constant. This implies that the original sentence vectors might actually reside in a lower-dimensional space. We can then remove these dimensions, which not only reduces dimensionality but also makes the cosine similarity results more reasonable.

In fact, the diagonal matrix \boldsymbol{\Lambda} from SVD is already sorted from largest to smallest. Therefore, we only need to keep the first few dimensions to achieve this dimensionality reduction effect. Readers familiar with linear algebra should recognize that this operation is actually PCA! The code only needs one line changed:

def compute_kernel_bias(vecs, n_components=256):
    """Compute kernel and bias
    vecs.shape = [num_samples, embedding_size],
    Final transformation: y = (x + bias).dot(kernel)
    """
    mu = vecs.mean(axis=0, keepdims=True)
    cov = np.cov(vecs.T)
    u, s, vh = np.linalg.svd(cov)
    W = np.dot(u, np.diag(1 / np.sqrt(s)))
    return W[:, :n_components], -mu

The results are as follows:

From the table above, we can see that by keeping only the first 256 dimensions of the 768 dimensions in the base version, the performance actually improves. Furthermore, due to the dimensionality reduction, vector retrieval speed will definitely be significantly faster. Similarly, keeping only the first 384 dimensions of the 1024 dimensions in the large version improves performance while reducing dimensionality. This result indicates that sentence vectors trained unsupervised are actually “general-purpose.” For applications within specific domains, many features are redundant. Eliminating these redundant features often achieves the effect of increasing both speed and efficiency.

In contrast, flow models are invertible and do not reduce dimensionality. While this is an advantage in some scenarios, it is a disadvantage in many others because it cannot eliminate redundant dimensions, which limits performance. For example, research on GANs shows that a 1024 \times 1024 face image can be randomly generated from a 256-dimensional Gaussian vector. This indicates that these face images actually constitute a fairly low-dimensional manifold. However, if a flow model is used, because invertibility must be maintained, one is forced to use a 1024 \times 1024 \times 3 dimensional Gaussian vector for random generation, which greatly increases computational costs and fails to improve results.

(Note: For subsequent experimental results, please see “Which Unsupervised Semantic Similarity Method is Stronger? We Conducted a Comprehensive Evaluation.”)

Final Conclusion

In summary, the current results are: my experiments show that through a simple linear transformation (BERT-whitening), the performance is basically comparable to the BERT-flow model. This suggests that introducing a flow model into sentence vector models may not be that critical; its calibration of the distribution might only be superficial, whereas directly calibrating the covariance matrix of sentence vectors through a linear transformation can achieve similar effects. At the same time, BERT-whitening also supports dimensionality reduction, achieving the effect of increasing both speed and efficiency.

When reposting, please include the original address of this article: https://kexue.fm/archives/8069

For more detailed reposting matters, please refer to: “Scientific Space FAQ”