Geometric Word Vectors
Although the aforementioned "Matchmaker" analogy was merely a fantasy, the problems faced are real. According to traditional NLP methods, we can count the co-occurrence frequency of any two words as well as the individual frequency of each word, and then calculate their correlation to obtain a "correlation matrix." However, as mentioned before, this co-occurrence matrix is far too massive; it must be compressed and reduced in dimensionality. Simultaneously, data smoothing is required to assign a reasonable estimate to the correlation of word pairs that have not appeared together.
In existing machine learning solutions, we already have experience in reducing the dimensionality of large matrices, such as SVD and pLSA. SVD is for the dimensionality reduction of arbitrary matrices, while pLSA is for the transition probability matrix P(j|i). The underlying ideas are similar: both decompose a large matrix \boldsymbol{A} into the product of two smaller matrices \boldsymbol{A} \approx \boldsymbol{B}\boldsymbol{C}, where the number of rows in \boldsymbol{B} equals the number of rows in \boldsymbol{A}, the number of columns in \boldsymbol{C} equals the number of columns in \boldsymbol{A}, and their own sizes are much smaller than \boldsymbol{A}. If no constraints are placed on \boldsymbol{B} and \boldsymbol{C}, it is SVD; if positive-definite normalization constraints are applied to \boldsymbol{B} and \boldsymbol{C}, it is pLSA.
However, if it is a correlation matrix, the situation is somewhat different. It is positive-definite but not normalized. We need to design a new compression scheme for it. Drawing on the experience of matrix factorization, we can imagine placing all words in an n-dimensional space—that is, representing them with a vector in n-dimensional space—and assuming that their correlation is a certain function of their inner product (Why the inner product? Because matrix multiplication itself is essentially a series of inner products): \frac{P(w_i,w_j)}{P(w_i)P(w_j)}=f\big(\langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle\big) \tag{8} where the bold \boldsymbol{v}_i, \boldsymbol{v}_j represent the word vectors corresponding to words w_i, w_j. From a geometric perspective, we are placing words into an n-dimensional space, using points in space to represent words.
Because geometry feels intuitive to us while semantics feels complex, we ideally hope to reflect semantic relationships through geometric relationships. Below, we will determine the function f based on the geometric properties we desire. In fact, the GloVe word vector paper did something similar, which was very inspiring, but the derivation in GloVe is not particularly elegant. Please note that the perspective here is novel—determining our model from the properties we desire, rather than deriving properties after having a model.
Airport - Airplane + Train = Railway Station
One of the most talked-about characteristics of word vectors is their "word analogy" property, such as the classic "King - Man + Woman = Queen" (whether this property is necessary for word vectors is controversial, but it is at least a plus). However, the contexts of Chinese and English are different, and this specific example is difficult to reproduce in Chinese corpora. Of course, there are many such examples, and there is no need to stick strictly to "foreign examples." For instance, in Chinese corpora, it is easy to find "Airport - Airplane + Train = Railway Station." More accurately: \boldsymbol{v}(\text{Airport}) - \boldsymbol{v}(\text{Airplane}) + \boldsymbol{v}(\text{Train}) = \boldsymbol{v}(\text{Railway Station}) \tag{9}
Why do word vectors possess this property? A recent article, "Skip-Gram – Zipf + Uniform = Vector Additivity," provided a theoretical analysis of this phenomenon. Based on some strong assumptions, the article derived this result. What we are about to do now is something potentially striking: directly use this property as one of the definitions of the word vector model!
Specifically, the additivity of word meaning is directly reflected as the additivity of word vectors; this property is the definition of the word vector model. We intend to start from this property and work backwards to find the function f that was not determined in the previous section. In this way, we not only find a reasonable basis for determining f but also explain the linear arithmetic characteristics of word vectors—because this is fundamentally the definition of the model, not a corollary.
Since it is a linear operation, we can rearrange the terms to get "Airport + Train = Railway Station + Airplane." Now let’s think: from a purely semantic perspective, what does this equation express? As mentioned at the beginning, the assumption of word vector models is basically to derive word meaning from the distribution of context. Since "Airport + Train = Railway Station + Airplane," it clearly means that the common context of "Airport" and "Train" is basically the same as the common context of "Railway Station" and "Airplane." To put it simply, semantic equivalence is like saying, "if two people have very similar criteria for choosing a partner, they must have many things in common." At this point, the form of f is ready to emerge!
Form of the Model
Since the degree of correlation between words is described by correlation, if "Airport + Train = Railway Station + Airplane," then we have: \frac{P(\text{Airport}, \text{Train}; w)}{P(\text{Airport}, \text{Train})P(w)} = \frac{P(\text{Railway Station}, \text{Airplane}; w)}{P(\text{Railway Station}, \text{Airplane})P(w)} \tag{10} Here w is any word in the context. Since we are not particularly concerned with word order but only with the average distribution of the context itself, we can use the naive assumption to simplify the above equation. According to Equation (6) [from previous parts of the series], we get: \frac{P(\text{Airport}, w)}{P(\text{Airport})P(w)} \times \frac{P(\text{Train}, w)}{P(\text{Train})P(w)} = \frac{P(\text{Railway Station}, w)}{P(\text{Railway Station})P(w)} \times \frac{P(\text{Airplane}, w)}{P(\text{Airplane})P(w)} \tag{11} Substituting the previously assumed Equation (8), we get: f\big(\langle \boldsymbol{v}_{\text{Airport}}, \boldsymbol{v}_w\rangle\big) f\big(\langle \boldsymbol{v}_{\text{Train}}, \boldsymbol{v}_w\rangle\big) = f\big(\langle \boldsymbol{v}_{\text{Airplane}}, \boldsymbol{v}_w\rangle\big) f\big(\langle \boldsymbol{v}_{\text{Railway Station}}, \boldsymbol{v}_w\rangle\big) \tag{12} Finally, substituting Equation (9), we obtain: \begin{aligned} & \left. f\big(\langle \boldsymbol{v}_{\text{Airport}}, \boldsymbol{v}_w\rangle\big) f\big(\langle \boldsymbol{v}_{\text{Train}}, \boldsymbol{v}_w\rangle\big) \middle/ f\big(\langle \boldsymbol{v}_{\text{Airplane}}, \boldsymbol{v}_w\rangle\big) \right. \\ = & f\big(\langle \boldsymbol{v}_{\text{Airport}} - \boldsymbol{v}_{\text{Airplane}} + \boldsymbol{v}_{\text{Train}}, \boldsymbol{v}_w\rangle\big) \\ = & f\big(\langle \boldsymbol{v}_{\text{Airport}}, \boldsymbol{v}_w\rangle + \langle \boldsymbol{v}_{\text{Train}}, \boldsymbol{v}_w\rangle - \langle \boldsymbol{v}_{\text{Airplane}}, \boldsymbol{v}_w\rangle\big) \end{aligned} \tag{13} Since \boldsymbol{v}_w is arbitrary, the above equation is equivalent to: f(x+y-z) = f(x)f(y)/f(z) Adding the condition of continuity, the general solution to the above equation is (the solution process can be found in most mathematical analysis textbooks): f(x) = e^{\alpha x} That is, the exponential form. Now we have the following result: in order for the resulting word vectors to have additivity, we need to model the correlation using an exponential model: \frac{P(w_i,w_j)}{P(w_i)P(w_j)} = e^{\langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle} \tag{14} Equivalently, modeling the Pointwise Mutual Information (PMI): \label{eq:model} \text{PMI}(w_i,w_j) = \langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle \tag{15} Thus, we have completed the derivation of the model’s form. From a formal perspective, it resembles an SVD decomposition of the PMI matrix.
Forgetting Normalization
We did not divide by a normalization factor to complete the probability normalization as is done in typical probabilistic models. The consequence of this is: for the model in this article, as well as the GloVe model, we cannot discuss anything related to normalization; otherwise, it would lead to self-contradictory results.
In fact, this is a trade-off of space for time. Because we do not divide by a normalization factor but must keep the results close to normalization, we can only pre-calculate and store the mutual information of all co-occurrence items, which often requires a significant amount of memory. The benefit of this step is that the total number of co-occurrence items is actually quite limited (the number of "word pairs" is always less than the number of sentences). Therefore, when you have a large-scale corpus and sufficient memory, using the GloVe model is often much faster than using the Skip-Gram model of word2vec.
Furthermore, since the model in this article and the Skip-Gram model of word2vec essentially differ by a normalization factor, it is clear that whether some of the derivations in this article can be directly migrated to the Skip-Gram model depends largely on whether its normalization factor is close to 1 after training.
When reposting, please include the original address: https://kexue.fm/archives/4671
For more detailed reposting matters, please refer to: "Scientific Space FAQ"