This article brings some "heavyweight" news. As the title suggests, it turns out that even the famous deep learning word vector tool, Word2Vec, is nothing more than an SVD!
Of course, to the super-loyal fans of Word2Vec: don’t get too excited. We are only saying that they are equivalent in terms of model structure, not completely identical. Word2Vec still has its unique characteristics. However, after explaining it this way, many questions might suddenly become clear.
Word Vector = One-Hot
Let’s first review an article from last year, "What exactly are Word Vectors and Embeddings?" The main point of that article was: The so-called Embedding layer is simply a one-hot fully connected layer (to emphasize again, this is a complete equivalence, not just "equivalent to"), and word vectors are the parameters of this fully connected layer. As for Word2Vec, it trains the Embedding layer through a greatly simplified language model to obtain word vectors (it has many optimization tricks, but the model structure is just that simple). Word vectors can reduce the risk of overfitting because tools like Word2Vec use large-scale corpora to pre-train this Embedding layer in an unsupervised manner; it has nothing to do with whether you use one-hot, Embedding, or the word vectors themselves.
With this perspective, we can immediately explain why one of our previous methods works. In sentiment classification problems, if we have word vectors, the simplest scheme to obtain a sentence vector is to directly sum or average the word vectors of the words in the sentence; this can achieve about 85% accuracy. In fact, this is also the approach of FastText, the text classification tool released by Facebook (FastText also introduces ngram features to alleviate word order issues, but overall, it still averages feature vectors to get sentence vectors). Why does such a seemingly non-intuitive, "brute-force" scheme achieve such decent accuracy?
Returning to the era of one-hot encoding, this is how we represented a sentence. Suppose we encode the six words "I", "Love", "Science", "Space", "Not", and "Bad" using the following one-hot encodings:
| Word | One-Hot Vector |
|---|---|
| I | [1, 0, 0, 0, 0, 0] |
| Love | [0, 1, 0, 0, 0, 0] |
| Science | [0, 0, 1, 0, 0, 0] |
| Space | [0, 0, 0, 1, 0, 0] |
| Not | [0, 0, 0, 0, 1, 0] |
| Bad | [0, 0, 0, 0, 0, 1] |
Then, without considering word order, the phrase "I love space science" can be represented by the following vector (Bag of Words): \begin{pmatrix}1 & 1 & 1 & 1 & 0 & 0\end{pmatrix} With this vector, to perform classification using a neural network, we can follow it with a fully connected layer with 3 hidden nodes: \begin{aligned} &\begin{pmatrix}1 & 1 & 1 & 1 & 0 & 0\end{pmatrix} \begin{pmatrix} w_{11} & w_{12} & w_{13}\\ w_{21} & w_{22} & w_{23}\\ w_{31} & w_{32} & w_{33}\\ w_{41} & w_{42} & w_{43}\\ w_{51} & w_{52} & w_{53}\\ w_{61} & w_{62} & w_{63} \end{pmatrix} \\ =& \begin{pmatrix}w_{11}+w_{21}+w_{31}+w_{41} & w_{12}+w_{22}+w_{32}+w_{42} & w_{13}+w_{23}+w_{33}+w_{43}\end{pmatrix} \end{aligned}
Wait? Isn’t this just taking the first four vectors and adding them together?
The situation is now clear. If we use a traditional Bag-of-Words model, ignore word order, and follow it with a fully connected layer—and to prevent overfitting, we replace the parameters of this fully connected layer with pre-trained word vectors—then the result is equivalent to directly extracting the corresponding word vectors and summing them! In other words, summing word vectors to get a sentence vector is actually the equivalent of the traditional Bag-of-Words model!
Word2Vec = SVD?
Throughout this discussion, I have included a question mark when talking about the equivalence between Word2Vec and SVD. This is because while their model structures are equivalent, their implementations differ. Whether this counts as "equivalence" depends on the reader’s own definition.
In fact, the concept of word vectors has existed for a long time. Back then, it wasn’t called "Word Embedding" but "distributed representation." The original intent was that the context of a word helps us understand the word itself. Suppose the total vocabulary has N words; a one-hot representation represents each word as an N-dimensional vector. A distributed representation, however, imagines opening a window (a few words before and after the current word) and then counting the distribution of the surrounding words. This distribution is used to represent the current word, and this distribution can also be represented as a corresponding N-dimensional vector. Since a word is represented through its context distribution rather than being an isolated "one-hot" vector, it can represent semantic correlation. But the problem is that it is still N-dimensional; the dimension is too high, and the entire word vector table (co-occurrence matrix) is too sparse.
What can be done? Mathematicians found a solution long ago. For sparse matrices, a scheme that can both reduce dimensionality and improve generalization is to perform SVD decomposition on the matrix. As the first article in this series mentioned, SVD decomposition is equivalent to a three-layer autoencoder. Looking at it today, this scheme says: The original distributed representation of word vectors is N-dimensional and too large; we can use an autoencoder to reduce the dimensionality. Let the number of intermediate nodes in the autoencoder be n. Set n to an appropriate value, and after training the autoencoder, directly use the n-dimensional results of the middle layer as the new word vectors. Therefore, this is an autoencoder scheme with N-dimensional input, n intermediate nodes, and N-dimensional output, which is also equivalent to an SVD decomposition.
So, what about Word2Vec? How should we view the Word2Vec model? One scheme of Word2Vec, CBOW, sums the word vectors of several surrounding words, then connects to an N-dimensional fully connected layer, and uses a softmax to predict the probability of the current word. As discussed in the first half of this article, this summation of word vectors is equivalent to the original Bag-of-Words model followed by a fully connected layer (where the parameters of this layer are the word vector table). From this perspective, Word2Vec is also just a three-layer neural network with N-dimensional input, n intermediate nodes, and N-dimensional output. Thus, in terms of network structure, it is equivalent to an autoencoder, which in turn is equivalent to SVD decomposition.
From an implementation standpoint, the differences are also obvious:
1. This Word2Vec scheme can be seen as using surrounding words to predict the current word, whereas an autoencoder or SVD uses surrounding words to predict surrounding words (reconstruction).
2. Word2Vec uses a softmax at the end to predict probabilities, meaning it implements a non-linear transformation, while an autoencoder or SVD (in their basic forms) do not.
To what extent these two differences affect the quality of word vectors lacks a rigorous mathematical proof, but from practical tests, given the same corpus, the quality of Word2Vec’s word vectors seems to be superior.
A Summary that is Not a Summary
Through a series of brainstorming sessions, this article reflects on the connections between traditional distributed representations and the Word2Vec model, leading to these conclusions. The main purpose of such thinking is to link various aspects together so that our understanding is closer to the essence, which may play an important guiding role in how we apply or even construct models. Overall, the articles in this series tell us that writing many operations or models in matrix form allows us to see the essence of many things and helps guide our thinking on directions for improvement. For example, many models can actually be written as matrix multiplications, and matrix multiplication is equivalent to a layer in a neural network. Since it is a neural network, can we add more layers? Can we add activation functions? Can we use SVD to decompose it? Because according to the second article in this series, SVD has clear clustering significance. Such reflections can help us build or even generalize models.
Original link: https://kexue.fm/archives/4233
For more details on reprinting, please refer to: "Scientific Space FAQ"