[Incredible Word2Vec] 3. Extracting Keywords

What are Keywords?

Admittedly, TF-IDF and TextRank are two classic algorithms for keyword extraction, and both have their merits. However, the problem is that if a reader has never seen these two algorithms before, they might feel the results are somewhat whimsical, and it would likely be difficult to construct them from scratch. That is to say, while these two algorithms appear simple, they are not necessarily intuitive. For instance, a student who hasn’t studied information theory might find it hard to understand why IDF takes a logarithm. Why not some other function? And how many readers would think, out of the blue, to use the logic of PageRank to judge the importance of a word?

Ultimately, the issue lies here: extracting keywords and text summarization both seem like very natural tasks, but has anyone truly considered what the definition of a keyword is? I am not asking you to look up a dictionary for a textual definition, but rather asking for a mathematical definition. What should be the reasonable mathematical definition of a keyword? Or rather, what is our purpose in obtaining keywords?

Clearly, whether it is keywords or summaries, we hope to grasp the general idea of an article as quickly as possible. If the keyword of an article is "Deep Learning," we know the article is unlikely to be talking at length about "How the Steel Was Tempered." In other words, we can guess the main idea of the text from the keywords. Expressed mathematically, this is the conditional probability: p(s|w_i) where s represents a segment of text and w_i is a certain word in the text. If w_i is a keyword of the text, it should maximize the above probability. That is to say, we only need to calculate the above probability for all words in the sentence, sort them in descending order, and we can extract the keywords. Simply put, keywords are the words that best allow us to guess the original text. How do we estimate this probability? Simply use the Naive Bayes assumption. If s consists of n words w_1, w_2, \dots, w_n, then: p(s|w_i) = p(w_1, w_2, \dots, w_n|w_i) = \prod_{k=1}^n p(w_k|w_i) Thus, we only need to estimate the transition probability p(w_k|w_i) between words to obtain the conditional probability p(s|w_i), thereby completing the keyword extraction.

What is the connection to Word2Vec? To estimate p(w_k|w_i), a large amount of text is needed for statistics. Fortunately, this process is unsupervised, and statistics is a simple matter. However, we have a better tool. Which tool is best at modeling p(w_k|w_i)? Readers might have already guessed: it is clearly Word2Vec! Isn’t the Skip-Gram model of Word2Vec designed exactly to model this probability? Given Word2Vec’s "fast, accurate, and effective" characteristics, there is no reason not to use it! (Of course, as mentioned at the beginning, one does not necessarily have to use the Bayesian assumption, and certainly does not have to use Word2Vec, but the definition of keywords itself should be reasonable.)

Calculating Probabilities with Word2Vec

At this point, readers should understand why I emphasized the combination of Skip-Gram + Huffman Softmax so much in the previous two articles, because this combination models p(w_k|w_i). Of course, due to the nature of Huffman Softmax, calculating p(w_k|w_i) requires some effort. The reference code is as follows:

import numpy as np
import gensim
model = gensim.models.word2vec.Word2Vec.load('word2vec_wx')

def predict_proba(oword, iword):
    iword_vec = model[iword]
    oword = model.wv.vocab[oword]
    oword_l = model.syn1[oword.point].T
    dot = np.dot(iword_vec, oword_l)
    lprob = -sum(np.logaddexp(0, -dot) + oword.code*dot) 
    return lprob

This is basically written by directly referencing the score_sg_pair function in gensim’s Word2Vec. The simple process is: Extract the Huffman encoding (path) of w_k, extract the word vector of w_i, and then, according to the path, calculate the probability of each node on the path and multiply them to get p(w_k|w_i). Since we are calculating the log-probability, multiplication becomes addition. How is the final probability calculated? In fact, according to the Word2Vec formula, the log-probability of each node is: \begin{aligned} &\log \left(\frac{1}{1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}}\right)^{1-d}\left(1-\frac{1}{1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}}\right)^{d}\\ =&-(1-d)\log (1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}) - d \log (1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}) - d \boldsymbol{x}^{\top} \boldsymbol{\theta}\\ =&-\log (1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}) - d \boldsymbol{x}^{\top} \boldsymbol{\theta} \end{aligned} Here \boldsymbol{\theta} is the node vector, \boldsymbol{x} is the input word vector, and d is the encoding of the node (either 0 or 1). However, the official score_sg_pair function is not written this way because: \begin{aligned} &-\log (1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}}) - d \boldsymbol{x}^{\top} \boldsymbol{\theta}\\ =&-\log \bigg[e^{d \boldsymbol{x}^{\top}\boldsymbol{\theta}}(1+e^{-\boldsymbol{x}^{\top} \boldsymbol{\theta}})\bigg]\\ =&-\log \bigg(e^{d \boldsymbol{x}^{\top}\boldsymbol{\theta}}+e^{(d-1)\boldsymbol{x}^{\top} \boldsymbol{\theta}}\bigg)\\ =&-\log \bigg(1+e^{-(-1)^d \boldsymbol{x}^{\top}\boldsymbol{\theta}}\bigg) \end{aligned}

Practice is Paramount

With the above preparation, calculating keywords is now simple:

from collections import Counter
def keywords(s):
    s = [w for w in s if w in model]
    ws = {w:sum([predict_proba(u, w) for u in s]) for w in s}
    return Counter(ws).most_common()

import pandas as pd # Introduced mainly for better display effects
import jieba
s = u'The sun is a star'
pd.Series(keywords(jieba.cut(s)))

The output result is:

0 (Star, -27.9013707845)
1 (Sun, -28.1072913493)
2 (A, -30.482187911)
3 (Is, -36.3372344659)

Other examples:

>>> s=u’According to the Changping District Government website, the Ming Tombs are the most complete and largest imperial burial complex in the world. They were announced as the first batch of national key cultural relics protection units by the State Council in 1961 and were listed in the World Heritage List in 2003.’
>>> pd.Series(keywords(jieba.cut(s)))
0 (Cultural Relics Protection, -261.691625676)
1 (List, -272.297758506)
2 (World Heritage, -273.943120665)
3 (First Batch, -280.781786703)
4 (Listed, -281.663865896)
5 (Ming Tombs, -286.298893108)
6 (Burial Complex, -287.463013816)
...

>>> s=u’The Xiongán New Area emerged suddenly, attracting many out-of-town real estate speculators to buy houses. However, under the background of the local governmentś heavy crackdown on illegal speculation and the freezing of the property market, the investment demand of those carrying money but having nowhere to start in Xiongán New Area has been squeezed out to surrounding areas.’
>>> pd.Series(keywords(jieba.cut(s)))
0 (Speculators, -326.997266407)
1 (Property Market, -336.176584187)
2 (Speculation, -337.190896137)
3 (Buying Houses, -344.613473556)
4 (Purchasing, -346.396359454)
5 (Heavy Crackdown, -350.207272082)
6 (Out-of-town, -355.860419218)

>>> s=u’If designing costumes for a period film, one must consider which dynasty the story takes place in. The Han Dynasty featured wide robes and large sleeves, while the Qing Dynasty featured magua and cheongsam. But on the Peking Opera stage, almost any historical figure, based on their gender, age, status, and personality, can find suitable attire in the existing costumes.’
>>> pd.Series(keywords(jieba.cut(s)))
0 (Dynasty, -485.150966757)
1 (Figure, -493.759615898)
2 (Period Costume, -495.478962392)
3 (Han Dynasty, -503.409908377)
4 (Qing Dynasty, -503.45656029)
5 (Cheongsam, -504.76313228)
6 (Status, -507.624260109)

You can try it yourself. If you want to try it on your own corpus, just train a Word2Vec (Skip-Gram + Huffman Softmax) model on the corpus and then call the code above.

Potential Doubts

According to our initial idea, p(w_k|w_i) should be calculated statistically within the entire sentence, but Word2Vec only uses a window for calculation. Is this reasonable? In fact, although Word2Vec only uses a window, it has successfully established connections between similar words. That is to say, using Word2Vec for the above process essentially aggregates similar words for evaluation. In contrast, the TF-IDF method only aggregates identical words for evaluation. Therefore, we say that Word2Vec keyword extraction can preliminarily incorporate semantic judgment. Furthermore, Word2Vec considers internal associations within the article by accounting for p(w_k|w_i). This has a bit of a TextRank flavor; it is a bigram model, whereas TF-IDF only considers the information content of the word itself, making it a unigram model.

Moreover, Word2Vec is trained based on neural networks and comes with inherent smoothing capabilities. Even if two words have never co-occurred in the text, a relatively reasonable probability can still be obtained.

Of course, the price for this is efficiency: the efficiency of the TF-IDF algorithm is \mathcal{O}(N), while extracting with Word2Vec clearly has an efficiency of \mathcal{O}(N^2), where N is the number of words in the sentence.