English (unofficial) translations of posts at kexue.fm
Source

[Incredible Word2Vec] 4. A Different Kind of ``Similarity''

Translated by DeepSeek V4 Pro. Translations can be inaccurate, please refer to the original post for important stuff.

Su Jianlin May 1, 2017 #4368

Definition of Similarity

When we obtain word vectors using Word2Vec, we generally use cosine similarity to compare the degree of similarity between two words, defined as: \cos (\boldsymbol{x}, \boldsymbol{y}) = \frac{\boldsymbol{x}\cdot\boldsymbol{y}}{|\boldsymbol{x}|\times|\boldsymbol{y}|} With this concept of similarity, we can both compare the similarity between any two words and find the words most similar to a given word. In gensim’s Word2Vec implementation, this is achieved via the most_similar function.

Wait! We have quickly provided a formula for calculating similarity, but we have not yet “defined” similarity! Without even defining similarity, how did we arrive at a mathematical formula to evaluate it?

It is important to note that this is not a question that can be casually ignored. Often, we do not know exactly what we are doing before we start doing it. For example, in the previous article regarding keyword extraction, I believe many people never considered what a keyword actually is. Is a keyword simply a word that is “key”? If we consider that a keyword is used to estimate what an article is roughly about, we get a very natural definition of a keyword: keywords = \mathop{\text{argmax}}_{w\in s}p(s|w) From there, we can use various methods to model it.

Returning to the theme of this article, how do we define similarity? The answer is: define the required similarity based on the scenario.

So, what kind of similarity is provided by cosine similarity? In fact, Word2Vec essentially uses the average distribution of the context to describe the current word (since Word2Vec does not consider word order). Since the cosine value is independent of the vector’s magnitude, it describes “relative consistency.” Therefore, a high cosine similarity actually means that these two words are often paired with the same set of words, or more crudely put, within the same sentence, the two words are interchangeable. For example, the words most similar to “Guangzhou” are “Dongguan” and “Shenzhen.” This is because, in many scenarios, replacing “Guangzhou” in a sentence with “Dongguan” or “Shenzhen” still results in a reasonable sentence (the sentence itself is linguistically reasonable, even if it is not factually true; for instance, “Guangzhou is the capital of Guangdong” becomes “Dongguan is the capital of Guangdong”—the sentence is reasonable, but it is not a fact).

>>> s = u'Guangzhou'
>>> pd.Series(model.most_similar(s))
0 (Dongguan, 0.840889930725)
1 (Shenzhen, 0.799216389656)
2 (Foshan, 0.786817014217)
3 (Huizhou, 0.779960155487)
4 (Zhuhai, 0.735232532024)
5 (Xiamen, 0.725090026855)
6 (Wuhan, 0.724122405052)
7 (Shantou, 0.719602525234)
8 (Zengcheng, 0.713532149792)
9 (Shanghai, 0.710560560226)

Correlation: Another Kind of Similarity

As mentioned earlier, the definition of similarity actually depends on the scenario; cosine similarity is just one of them. Sometimes we might feel that “Dongguan” and “Guangzhou” have no connection at all. For an “Old Guangzhou” local, words like “Baiyun Mountain,” “Baiyun Airport,” and “Canton Tower” are the ones most similar to “Guangzhou.” This scenario is also very common. For example, in travel recommendations, when a tourist comes to Guangzhou, they naturally hope that after entering “Guangzhou,” the system automatically outputs words related to Guangzhou like “Baiyun Mountain,” “Baiyun Airport,” and “Canton Tower,” rather than outputting words like “Dongguan” or “Shenzhen.”

This kind of “similarity” is more accurately described as “correlation.” How should we describe it? The answer is Mutual Information, defined as: \log \frac{p(x,y)}{p(x)p(y)}=\log p(y|x) - \log p(y) The larger the mutual information, the more frequently the two words x and y appear together.

Thus, given a word x, we can find the words that frequently appear with word x. This can also be accomplished using the Skip-Gram + Huffman Softmax model in Word2Vec. The code is as follows:

import numpy as np
import gensim
model = gensim.models.word2vec.Word2Vec.load('word2vec_wx')

def predict_proba(oword, iword):
    iword_vec = model[iword]
    oword = model.wv.vocab[oword]
    oword_l = model.syn1[oword.point].T
    dot = np.dot(iword_vec, oword_l)
    lprob = -sum(np.logaddexp(0, -dot) + oword.code*dot) 
    return lprob

from collections import Counter
def relative_words(word):
    r = {i:predict_proba(i, word)-np.log(j.count) for i,j in model.wv.vocab.iteritems()}
    return Counter(r).most_common()

At this point, the related words for “Guangzhou” are:

>>> s = u'Guangzhou'
>>> w = relative_words(s)
>>> pd.Series(w)
0 (Fuzhong Road, -17.390365773)
1 (OHG, -17.4582544641)
2 (Linzhai Town, -17.6119545612)
3 (Pingshan Subdistrict, -17.6462214199)
4 (Dongpu Town, -17.6648893759)
5 (West Wing, -17.6796614955)
6 (Beijing West, -17.6898282385)
7 (left-right-harpoons, -17.6950761384)
8 (K1019, -17.7259853233)
9 (Jingtai Subdistrict, -17.7292421556)
10 (PSW3, -17.7296432222)
11 (Guangzhou Railway Polytechnic, -17.732288911)
12 (13A06, -17.7382891287)
13 (5872, -17.7404719442)
14 (13816217517, -17.7650583156)
15 (Attempted Case, -17.7713452536)
16 (Zengcheng City, -17.7713832873)
17 (Dixiaopu Road, -17.7727940473)
18 (Guangzhou Baiyun Airport, -17.7897457043)
19 (Faust, -17.7956389314)
20 (National Archives, -17.7971039916)
21 (w0766fc, -17.8051687721)
22 (K1020, -17.8106548248)
23 (Chen Baochen, -17.8427718407)
24 (jinriGD, -17.8647825023)
25 (3602114109100031646, -17.8729896156)

It can be seen that the results obtained are basically closely related to Guangzhou. Of course, sometimes we want to emphasize high-frequency words slightly; therefore, we can consider modifying the mutual information formula to: \log \frac{p(x,y)}{p(x)p^{\alpha}(y)}=\log p(y|x) - \alpha\log p(y) where \alpha is a constant slightly less than 1. If we take \alpha=0.9, then we have:

from collections import Counter
def relative_words(word):
    r = {i:predict_proba(i, word)-0.9*np.log(j.count) for i,j in model.wv.vocab.iteritems()}
    return Counter(r).most_common()

The results are rearranged as follows:

>>> s = u'Guangzhou'
>>> w = relative_words(s)
>>> pd.Series(w)
0 (Fuzhong Road, -16.8342976099)
1 (Beijing West, -16.9316053191)
2 (OHG, -16.9532688634)
3 (West Wing, -17.0521852934)
4 (Zengcheng City, -17.0523156839)
5 (Guangzhou Baiyun Airport, -17.0557270208)
6 (Linzhai Town, -17.0867272184)
7 (left-right-harpoons, -17.1061883426)
8 (Pingshan Subdistrict, -17.1485480457)
9 (5872, -17.1627067119)
10 (Dongpu Town, -17.192150594)
11 (PSW3, -17.2013228493)
12 (Faust, -17.2178736991)
13 (Hongfen, -17.2191157626)
14 (National Archives, -17.2218467278)
15 (Attempted Case, -17.2220391092)
16 (Jingtai Subdistrict, -17.2336594498)
17 (Guangxiao Temple, -17.2781121397)
18 (International Freight Forwarding, -17.2810157155)
19 (Dixiaopu Road, -17.2837591345)
20 (Guangzhou Railway Polytechnic, -17.2953441257)
21 (Fangcun, -17.301106775)
22 (Testing Institute, -17.3041253252)
23 (K1019, -17.3085465963)
24 (Chen Baochen, -17.3134413583)
25 (Linhexi, -17.3150577006)

Relatively speaking, the latter result is more readable. Other results are shown below:

>>> s = u'Airplane'
>>> w = relative_words(s)
>>> pd.Series(w)
0 (Macau International Airport, -16.5502216186)
1 (HawkT1, -16.6055740672)
2 (a plane, -16.6105400944)
3 (ground crew, -16.6764712234)
4 (US Army, -16.6781627384)
5 (SU200, -16.6842796275)
6 (takeoff and landing, -16.6910345896)
7 (Shanghai Pudong International Airport, -16.7040362134)
8 (alternate landing, -16.7232609719)
9 (the first one, -16.7304077856)

>>> pd.Series(model.most_similar(s))
0 (takeoff, 0.771412968636)
1 (airliner, 0.758365988731)
2 (helicopter, 0.755871891975)
3 (one (plane), 0.749522089958)
4 (takeoff and landing, 0.726713418961)
5 (landing, 0.723304390907)
6 (a plane, 0.722024559975)
7 (flight, 0.700125515461)
8 (Boeing, 0.697083711624)
9 (jet plane, 0.696866035461)

>>> s = u'Bicycle'
>>> w = relative_words(s)
>>> pd.Series(w)
0 (ride, -16.4410312554)
1 (fly a kite, -16.6607225423)
2 (moped, -16.8390451582)
3 (bicycle, -16.900188791)
4 (tricycle, -17.1053629907)
5 (rental point, -17.1599389605)
6 (electric bike, -17.2038996636)
7 (power-assisted bicycle, -17.2523149342)
8 (multiple vehicles, -17.2629832083)
9 (CRV, -17.2856425014)

>>> pd.Series(model.most_similar(s))
0 (motorcycle, 0.737690329552)
1 (ride, 0.721182465553)
2 (scooter, 0.7102201581)
3 (electric bike, 0.700758457184)
4 (mountain bike, 0.687280654907)
5 (cycling, 0.666575074196)
6 (bike, 0.651858925819)
7 (riding a bike, 0.650207400322)
8 (moped, 0.635745406151)
9 (tricycle, 0.630989730358)

You can try it yourself. It should be noted that, unfortunately, although Huffman Softmax speeds up calculation during the training phase, it is actually slower than native Softmax during the prediction phase when a traversal of the dictionary is required. Therefore, this is not a high-efficiency solution.

What Exactly Was Done

Based on the previous two parts, we can see that “similarity” generally has two scenarios: 1. Often paired with the same set of words; 2. Often appearing together. Both scenarios can be considered as similarity between words, applicable to different needs.

For example, when performing word sense disambiguation for polysemous words, such as determining whether “star” refers to a celestial body or a celebrity, mutual information can be utilized. We can beforehand find corpora where “star” means a celestial body and identify words with high mutual information with “star,” such as “sun,” “planet,” and “earth.” Similarly, we can find corpora where “star” means a celebrity and identify words with high mutual information, such as “entertainment” and “movie.” In a new context, we can then infer the exact meaning based on the surrounding words.

In summary, it is necessary to clarify your own needs and then consider the corresponding method.

Reprinting: Please include the address of this article: https://kexue.fm/archives/4368

For more detailed reprinting matters, please refer to: Scientific Space FAQ