A More Unique Word Vector Model (V): Interesting Results

The Meaning of Vector Magnitude

It seems that most word vector models rarely concern themselves with the magnitude (norm) of the word vectors. Interestingly, in the word vector model we have developed, the magnitude can represent the importance of a word to a certain extent. We can understand this fact from two perspectives.

In the context of a single window, the probability of the center word reappearing is actually quite small; it is a relatively random event. Therefore, we can roughly assume: P(w,w) \sim P(w) \tag{24} According to our model, we have: e^{\langle\boldsymbol{v}_{w},\boldsymbol{v}_{w}\rangle} = \frac{P(w,w)}{P(w)P(w)} \sim \frac{1}{P(w)} \tag{25} Thus: \Vert\boldsymbol{v}_{w}\Vert^2 \sim -\log P(w) \tag{26} It can be seen that the higher the frequency of a word (the more likely it is to be a stop word or a functional word), the smaller the magnitude of its corresponding word vector. This indicates that the magnitude of these word vectors indeed represents word importance. In fact, the quantity -\log P(w) is similar to IDF and is specifically called ICF (Inverse Corpus Frequency). For more details, please refer to the paper "TF-ICF: A New Term Weighting Scheme for Clustering Dynamic Data Streams".

We can also understand this from another perspective by decomposing each vector into its magnitude and direction: \boldsymbol{v}=\Vert\boldsymbol{v}\Vert\cdot\frac{\boldsymbol{v}}{\Vert\boldsymbol{v}\Vert} \tag{27} Where the magnitude \Vert\boldsymbol{v}\Vert is an independent parameter, and the direction vector \boldsymbol{v}/\Vert\boldsymbol{v}\Vert consists of n-1 independent parameters, where n is the dimension of the word vector. Since the number of parameters differs significantly, the model will naturally choose to adjust the magnitude rather than struggle to adjust the direction if the objective can be met by adjusting the magnitude. According to Equation (15), we have: \log\frac{P(w_i,w_j)}{P(w_i)P(w_j)}=\langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle=\Vert\boldsymbol{v}_i\Vert\cdot \Vert\boldsymbol{v}_j\Vert\cdot \cos\theta_{ij} \tag{28} For words like "of", "the", or "is" which have almost no semantic meaning, in which direction will their word vectors develop? As mentioned earlier, they have high frequencies but rarely form fixed collocations with specific words; they essentially appear randomly. Thus, for any word w_i, we can assume: \log\frac{P(w_i,\text{of})}{P(w_i)P(\text{of})}\approx 0 \tag{29} To achieve this, the most convenient method is naturally \Vert\boldsymbol{v}_{\text{of}}\Vert\approx 0. Since adjusting one parameter can achieve the goal, the model will prefer this. That is to say, for words with high frequency but low overall mutual information (which usually lack specific meaning), the magnitude will automatically approach 0. Therefore, we say that the magnitude of the word vector can represent the importance of the word to some extent.

In a set of word vectors trained using this model on the Baidu Baike corpus, without weight truncation, the top 50 words sorted by magnitude in ascending order are:

It is evident that these words are what we call "stop words" or "functional words," which validates that the magnitude indeed represents the importance of the word itself. This result is somewhat related to whether weights are truncated. If weights are truncated, the resulting order is:

The obvious difference between the two tables is that in the second table, while they are still mostly stop words, some more obvious stop words like "of" and "is" are not at the very front. This is because their word frequencies are extremely high, so the impact of truncation is greater, leading to the possibility of insufficient fitting (simply put, the model focuses more on low-frequency words, while for high-frequency words, it only requires them to be "reasonable"). Why do the period and comma still rank high? Because in a single window of a sentence, the probability of two periods appearing is much lower than the probability of "of" appearing twice. Therefore, the use of the period "." more closely matches the assumptions of our derivation. Correspondingly, since "of" may appear multiple times in a window, the mutual information of "of" with itself should be larger, leading to a larger magnitude.

Word Analogy Experiments

Since we claim that the word analogy property is the definition of this model, does the model actually perform well in word analogies? Let’s look at some examples.

I would also like to point out that in word analogy experiments, some results look beautiful while others look unreliable. In fact, word vectors reflect the statistical laws of the corpus, which are objective. Conversely, some relationships defined by humans are subjective. For a word vector model, "similar" means having a similar context distribution, not a human-defined similarity. Whether the effect is good depends on how much the view "similar context distribution \leftrightarrow similar words" (related to the corpus) differs from the human definition of similarity (unrelated to the corpus, based on subjective thoughts). When you find experimental results are not good, you might want to think about this point.

Related Word Ranking

Note Equation (15), which states that the mutual information of two words is equal to the inner product of their word vectors. A larger mutual information indicates a higher probability of the two words appearing together, while a smaller mutual information indicates they are rarely used together. Therefore, we can use the inner product to rank and find related words for a given word. Of course, the inner product includes the magnitude. As we just mentioned, the magnitude represents word importance. If we ignore importance and purely consider word meaning, we can normalize the vector norms before calculating the inner product. This approach is more stable: \cos\theta_{ij}=\left\langle \frac{\boldsymbol{v}_i}{|\boldsymbol{v}_i|}, \frac{\boldsymbol{v}_j}{|\boldsymbol{v}_j|}\right\rangle \tag{30}

According to probability theory, if the mutual information is 0, the joint probability of two words is exactly the probability of their random combination, indicating they are unrelated. Corresponding to Equation (15), the inner product of the two word vectors is 0. In vector space, an inner product of 0 means the vectors are orthogonal (perpendicular), which we usually interpret as being unrelated. Thus, statistical independence between two words corresponds perfectly to geometric independence. This is one of the formal beauties of the model.

It should be noted that, as mentioned earlier, stop words tend to shrink their magnitude rather than adjust their direction, so their direction may not be meaningful. We can consider the direction of stop words to be random. When we use cosine similarity to find related words, unexpected stop words might appear.

Redefining Similarity

Note that we were talking about related word ranking. Related words and similar words are not the same thing! For example, "single" and "frozen" might be highly related to "dog" (in certain slang contexts), but they are not synonyms. "Science" and "Development Outlook" are also highly related, but they are not synonyms.

So how do we find synonyms? In fact, this question is putting the cart before the horse because the definition of similarity is human-made. For example, "like" and "love" are similar, but what about "like" and "hate"? In a general topic classification task, they might be considered similar, but in a sentiment analysis task, they are opposites. Or consider "run" and "grab"; generally, they are not similar, but in part-of-speech classification, they are similar because they are both verbs.

Returning to the hypothesis of our word vector model: the context distribution reveals the meaning of a word. Therefore, two similar words should have similar context distributions. The "Airport - Airplane + Train = Railway Station" example discussed earlier was based on the same principle, but it required a strict one-to-one correspondence of context words. Here, we only require an approximate correspondence, which is a relaxed condition. To adapt to different levels of similarity needs, we can choose the context ourselves. Specifically, for two given words w_i, w_j and their vectors \boldsymbol{v}_i, \boldsymbol{v}_j, to calculate their similarity, we first write out their mutual information with N pre-specified words: \langle\boldsymbol{v}_i,\boldsymbol{v}_1\rangle,\langle\boldsymbol{v}_i,\boldsymbol{v}_2\rangle,\dots,\langle\boldsymbol{v}_i,\boldsymbol{v}_N\rangle \tag{31} and \langle\boldsymbol{v}_j,\boldsymbol{v}_1\rangle,\langle\boldsymbol{v}_j,\boldsymbol{v}_2\rangle,\dots,\langle\boldsymbol{v}_j,\boldsymbol{v}_N\rangle \tag{32} where N is the total number of words in the vocabulary. If these two words are similar, their context distributions should also be similar, so these two sequences should have a linear correlation. Therefore, we can compare their Pearson product-moment correlation coefficient: \frac{\sum_{k=1}^N \Big(\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle - \overline{\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle}\Big)\Big(\langle\boldsymbol{v}_j,\boldsymbol{v}_k\rangle - \overline{\langle\boldsymbol{v}_j,\boldsymbol{v}_k\rangle}\Big)}{\sqrt{\sum_{k=1}^N \Big(\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle - \overline{\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle}\Big)^2}\sqrt{\sum_{k=1}^N \Big(\langle\boldsymbol{v}_j,\boldsymbol{v}_k\rangle - \overline{\langle\boldsymbol{v}_j,\boldsymbol{v}_k\rangle}\Big)^2}} \tag{33} where \overline{\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle} is the mean of \langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle: \overline{\langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle}=\frac{1}{N}\sum_{k=1}^N \langle\boldsymbol{v}_i,\boldsymbol{v}_k\rangle=\left\langle\boldsymbol{v}_i,\frac{1}{N}\sum_{k=1}^N \boldsymbol{v}_k\right\rangle = \langle\boldsymbol{v}_i,\bar{\boldsymbol{v}}\rangle \tag{34} Thus, the correlation coefficient formula can be simplified to: \frac{\sum_{k=1}^N \langle\boldsymbol{v}_i,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle\langle\boldsymbol{v}_j,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle}{\sqrt{\sum_{k=1}^N \langle\boldsymbol{v}_i,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle^2}\sqrt{\sum_{k=1}^N \langle\boldsymbol{v}_j,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle^2}} \tag{35} Using matrix notation (assuming vectors are row vectors), we have: \begin{aligned} &\sum_{k=1}^N \langle\boldsymbol{v}_i,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle\langle\boldsymbol{v}_j,\boldsymbol{v}_k-\bar{\boldsymbol{v}}\rangle \\ =&\sum_{k=1}^N \boldsymbol{v}_i (\boldsymbol{v}_k-\bar{\boldsymbol{v}})^{\top}(\boldsymbol{v}_k-\bar{\boldsymbol{v}})\boldsymbol{v}_j^{\top} \\ =&\boldsymbol{v}_i \left[\sum_{k=1}^N (\boldsymbol{v}_k-\bar{\boldsymbol{v}})^{\top}(\boldsymbol{v}_k-\bar{\boldsymbol{v}})\right]\boldsymbol{v}_j^{\top} \end{aligned} \tag{36} The term in the square brackets is: \boldsymbol{V}^{\top}\boldsymbol{V},\,\boldsymbol{V}=\begin{pmatrix}\boldsymbol{v}_1-\bar{\boldsymbol{v}}\\ \boldsymbol{v}_2-\bar{\boldsymbol{v}}\\ \vdots \\ \boldsymbol{v}_N-\bar{\boldsymbol{v}}\end{pmatrix} \tag{37} This is the matrix \boldsymbol{V} formed by subtracting the mean from the word vectors. \boldsymbol{V}^{\top}\boldsymbol{V} is an n \times n real symmetric matrix, which can be decomposed (e.g., Cholesky decomposition) as: \boldsymbol{V}^{\top}\boldsymbol{V}=\boldsymbol{U}\boldsymbol{U}^{\top} \tag{38} where \boldsymbol{U} is an n \times n real matrix. Thus, the correlation coefficient formula can be written as: \frac{\boldsymbol{v}_i \boldsymbol{U}\boldsymbol{U}^{\top}\boldsymbol{v}_j^{\top}}{\sqrt{\boldsymbol{v}_i \boldsymbol{U}\boldsymbol{U}^{\top}\boldsymbol{v}_i^{\top}}\sqrt{\boldsymbol{v}_j \boldsymbol{U}\boldsymbol{U}^{\top}\boldsymbol{v}_j^{\top}}}=\frac{\langle\boldsymbol{v}_i \boldsymbol{U},\boldsymbol{v}_j \boldsymbol{U}\rangle}{\Vert\boldsymbol{v}_i \boldsymbol{U}\Vert \times \Vert\boldsymbol{v}_j \boldsymbol{U}\Vert} \tag{39} We find that similarity is still measured by the cosine value of the vectors, but only after they have been transformed by the matrix \boldsymbol{U}.

Finally, how should we choose these N words? We can sort them by frequency in descending order and choose the top N. If N is large (e.g., N=10000), we get semantically related words in general scenarios. If N is small (e.g., N=500), we get grammatically similar words.

Keyword Extraction

As in the article "Incredible Word2Vec (3): Extracting Keywords", a keyword is a word that can summarize the meaning of a sentence. That is, one can guess the overall content of the sentence just by looking at the keywords. Suppose a sentence has k words w_1, w_2, \dots, w_k. The keyword w should maximize: P(w_1,w_2,\dots,w_k|w)\sim \frac{P(w_1,w_2,\dots,w_k;w)}{P(w_1,w_2,\dots,w_k)P(w)} \tag{40} Simply put, we want to maximize the probability of guessing the sentence from the word. Since the sentence is given, P(w_1, \dots, w_k) is a constant. Using the naive assumption and Equation (6): \frac{P(w_1,w_2,\dots,w_k;w)}{P(w_1,w_2,\dots,w_k)P(w)}=\frac{P(w_1,w)}{P(w_1)P(w)}\frac{P(w_2,w)}{P(w_2)P(w)}\dots \frac{P(w_k,w)}{P(w_k)P(w)} \tag{41} Substituting our word vector model, we get: e^{\langle\boldsymbol{v}_1,\boldsymbol{v}_w\rangle}e^{\langle\boldsymbol{v}_2,\boldsymbol{v}_w\rangle}\dots e^{\langle\boldsymbol{v}_k,\boldsymbol{v}_w\rangle}=e^{\left\langle\sum_i \boldsymbol{v}_i, \boldsymbol{v}_w\right\rangle} \tag{42} So it is equivalent to maximizing: \left\langle\sum_i \boldsymbol{v}_i, \boldsymbol{v}_w\right\rangle \tag{43} The problem is now simple: for a given sentence, sum the vectors of all its words to get a sentence vector, then calculate the inner product (or cosine) of the sentence vector with each word vector in the sentence, and sort them in descending order. This simple approach reduces the complexity from \mathcal{O}(k^2) to \mathcal{O}(k). Here are some examples:

Sentence: The second central environmental protection inspection team’s mobilization meeting for inspection work in Zhejiang Province was held in Hangzhou. From August 11 to September 11, the central environmental protection inspection team officially stationed in Zhejiang. This indicates that all enterprises in Zhejiang will face environmental inspections within the next month, meaning they face risks of production restriction, suspension, and closure.
Keyword Ranking: Inspection team, production restriction, inspection team, mobilization meeting, closure, inspection, production suspension, indicates, environmental protection, upcoming.

Sentence: The Yiwu Environmental Protection Bureau stated that due to high cadmium content in alloy raw materials, it ordered some electroplating enterprises to stop production for rectification. It is reported that Yiwu’s zinc-cadmium alloy production has basically stopped. Additionally, enterprises in Ouhai District, Wenzhou, were ordered to stop production until they pass inspection. A new round of environmental protection is unfolding in Zhejiang’s zinc industry.
Keyword Ranking: Zinc alloy, Environmental Protection Bureau, production suspension, Ouhai District, permit, cadmium, electroplating, order, Yiwu City, raw materials.

Sentence: Beef Bourguignon is a classic French dish, called "the most delicious beef humans can cook." This dish has a rich wine aroma and inviting color, and the process is not troublesome. What is the story behind it? How to make a delicious Beef Bourguignon?
Keyword Ranking: Beef stew, wine aroma, classic dish, delicious, inviting, Burgundy, rich, dish, cooking, one.

Even for long sentences, this scheme is quite reliable. Note that while simple, this keyword extraction scheme is not applicable to every word vector model. For example, GloVe vectors do not work this way because their stop words have larger magnitudes; in GloVe, the smaller the inner product, the more likely it is to be a keyword.

Sentence Similarity

Let’s look at another example: sentence similarity, which is a concern for many readers. It is actually similar to keyword extraction.

When are two sentences similar or semantically equivalent? Simply put, if I read the first sentence, I can know what the second sentence is about, and vice versa. In this case, the correlation between the two sentences must be high. Let sentence S_1 have k words w_1, \dots, w_k and sentence S_2 have l words w_{k+1}, \dots, w_{k+l}. Using the naive assumption and Equation (6): \frac{P(S_1,S_2)}{P(S_1)P(S_2)}=\prod_{i=1}^k\prod_{j=k+1}^{k+l} \frac{P(w_i,w_j)}{P(w_i)P(w_j)} \tag{44} Substituting our word vector model: \begin{aligned} \frac{P(S_1,S_2)}{P(S_1)P(S_2)} &= \prod_{i=1}^k\prod_{j=k+1}^{k+l} e^{\langle\boldsymbol{v}_i,\boldsymbol{v}_j\rangle} \\ &= e^{\sum_{i=1}^k\sum_{j=k+1}^{k+l}\langle\boldsymbol{v}_i,\boldsymbol{v}_j\rangle} \\ &= e^{\left\langle\sum_{i=1}^k\boldsymbol{v}_i,\sum_{j=k+1}^{k+l}\boldsymbol{v}_j\right\rangle} \end{aligned} \tag{45} So it is equivalent to ranking by: \left\langle\sum_{i=1}^k\boldsymbol{v}_i,\sum_{j=k+1}^{k+l}\boldsymbol{v}_j\right\rangle \tag{46} The result is simple: sum the word vectors of each sentence to get sentence vectors, then calculate their inner product (or cosine) to get the correlation.

Sentence Vectors

The previous two sections imply that sentence vectors can be obtained by directly summing word vectors. How is the quality of such sentence vectors?

We conducted a simple experiment. Using sentence vectors obtained by summing word vectors (non-truncated version) plus a linear classifier (Logistic Regression), we achieved about 81% accuracy on a sentiment classification task. If a hidden layer is added (input 128, hidden 64 with ReLU, output 1), the accuracy reaches about 88%. In comparison, an LSTM’s accuracy is around 90%. This shows that such sentence vectors are quite remarkable, especially considering they were trained on Baidu Baike (which has no explicit sentiment bias) yet successfully captured sentiment tendencies.

To verify the impact of truncation, we repeated the experiment with truncated word vectors. The Logistic Regression accuracy reached 82%, and the three-layer neural network reached 89%. This suggests that truncation (down-weighting high-frequency words) indeed helps capture semantics better.

import pandas as pd
import jieba
import numpy as np

pos = pd.read_excel('pos.xls', header=None)
neg = pd.read_excel('neg.xls', header=None)
pos[1] = pos[0].apply(lambda s: jieba.lcut(s, HMM=False))
neg[1] = neg[0].apply(lambda s: jieba.lcut(s, HMM=False))
# w2v.sent2vec sums word vectors
pos[2] = pos[1].apply(w2v.sent2vec) 
neg[2] = neg[1].apply(w2v.sent2vec)
pos = np.hstack([np.array(list(pos[2])), np.array([[1] for i in pos[2]])])
neg = np.hstack([np.array(list(neg[2])), np.array([[0] for i in neg[2]])])
data = np.vstack([pos, neg])
np.random.shuffle(data)

from keras.models import Sequential
from keras.layers import Dense
model = Sequential()
model.add(Dense(64, input_shape=(w2v.word_size,), activation='relu'))
model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',
              optimizer='adam',
              metrics=['accuracy'])

batch_size = 128
model.fit(data[:16000,:w2v.word_size], data[:16000,[w2v.word_size]],
          batch_size=batch_size,
          epochs=100,
          validation_data=(data[16000:,:w2v.word_size], data[16000:,[w2v.word_size]]))

Original Address: https://kexue.fm/archives/4677