From Conditional Probability to Mutual Information
Currently, the principle of most word vector models is that the distribution of a word’s context can reveal its semantics, much like the saying, “Tell me who you associate with, and I will tell you who you are.” Therefore, the core of word vector models is to model the relationship between a word and its context. Except for GloVe, almost all word vector models model the conditional probability P(w|\text{context}). For example, the Skip-gram model in Word2Vec models the conditional probability P(w_2|w_1). However, this quantity has several drawbacks. First, it is asymmetric; that is, P(w_2|w_1) is not necessarily equal to P(w_1|w_2). This means that during modeling, we must distinguish between the context vector and the target vector, and they cannot reside in the same vector space. Second, it is a bounded, normalized quantity, which implies we must use methods like Softmax to compress and normalize it, leading to optimization difficulties.
In fact, in the world of Natural Language Processing (NLP), there is a more symmetric quantity that is more important than simple conditional probability P(w_2|w_1), which is: \frac{P(w_1,w_2)}{P(w_1)P(w_2)}=\frac{P(w_2|w_1)}{P(w_2)} \tag{1} This quantity roughly represents “how many times more likely the two words are to meet in reality compared to meeting by chance.” If it is much greater than 1, it indicates that they tend to co-occur rather than combine randomly. Conversely, if it is much less than 1, it means they deliberately avoid each other. This quantity is pivotal in NLP; for now, let’s call it “relevance.” Of course, its logarithmic value is even more famous, known as Pointwise Mutual Information (PMI): \text{PMI}(w_1,w_2)=\log \frac{P(w_1,w_2)}{P(w_1)P(w_2)} \tag{2}
Based on the above theoretical foundation, we believe that directly modeling relevance is more reasonable than modeling the conditional probability P(w_2|w_1). Therefore, this article revolves around this perspective. Before that, let us further demonstrate the beautiful properties of mutual information itself.
Additivity of Mutual Information
Relevance (and equivalently, mutual information) possesses a very elegant decomposition property under the Naive assumption. The Naive assumption refers to the assumption that features are independent of each other, such that we have P(a,b)=P(a)P(b), which decomposes the joint probability to simplify the model.
For example, consider the mutual information between two variables Q and A. Q and A are not single features but combinations of multiple features: Q=(q_1,\dots,q_k) and A=(a_1,\dots,a_l). Now consider their relevance: \begin{aligned} \frac{P(Q,A)}{P(Q)P(A)} &= \frac{P(q_1,\dots,q_k;a_{1},\dots,a_{l})}{P(q_1,\dots,q_k)P(a_{1},\dots,a_{l})} \\ &= \frac{P(q_1,\dots,q_k|a_{1},\dots,a_{l})}{P(q_1,\dots,q_k)} \end{aligned} \tag{3} Using the Naive assumption, we get: \frac{P(q_1,\dots,q_k|a_{1},\dots,a_{l})}{P(q_1,\dots,q_k)} = \frac{\prod_{i=1}^k P(q_i|a_{1},\dots,a_{l})}{\prod_{i=1}^k P(q_i)} \tag{4} Using Bayes’ theorem, we obtain: \begin{aligned} \frac{\prod_{i=1}^k P(q_i|a_{1},\dots,a_{l})}{\prod_{i=1}^k P(q_i)} &= \frac{\prod_{i=1}^k P(a_{1},\dots,a_{l}|q_i)P(q_i)/P(a_{1},\dots,a_{l})}{\prod_{i=1}^k P(q_i)} \\ &= \prod_{i=1}^k\frac{P(a_{1},\dots,a_{l}|q_i)}{P(a_{1},\dots,a_{l})} \end{aligned} \tag{5} Applying the Naive assumption once more: \begin{aligned} \prod_{i=1}^k\frac{P(a_{1},\dots,a_{l}|q_i)}{P(a_{1},\dots,a_{l})} &= \prod_{i=1}^k\frac{\prod_{j=1}^{l} P(a_j|q_i)}{\prod_{j=1}^{l} P(a_j)} \\ &= \prod_{i=1}^k\prod_{j=1}^{l} \frac{P(q_i,a_j)}{P(q_i)P(a_j)} \end{aligned} \tag{6} This indicates that under the Naive assumption, the relevance of two multivariate variables is equal to the product of the relevance of their individual univariate pairs. If we take the logarithm of both sides, the result is even more elegant: \text{PMI}(Q,A)=\sum_{i=1}^k\sum_{j=1}^{l} \text{PMI}(q_i,a_j) \tag{7} In other words, the mutual information between two multivariate variables is equal to the sum of the mutual information between all pairs of their univariate components. Put differently, mutual information is additive!
Interlude: A Side Story
To help everyone intuitively understand the principle of word vector modeling, let us imagine ourselves as the “Matchmaker” (Yuelao) of the language world. Our goal is to measure the “affinity” (fate) between any two words and pave the way for each word to find its best partner.
As the saying goes, “Fate brings people together from a thousand miles away, while without it, they remain strangers even when meeting face to face.” For every word, the best partner must be its “destined word.” What kind of two words are considered to have “affinity”? Naturally, it is when “you are in my eyes, and I am in yours.” As mentioned earlier, the Skip-gram model focuses on the conditional probability P(w_2|w_1), which results in “w_1 has w_2 in its eyes, but w_2 does not necessarily have w_1 in its eyes.” In other words, w_2 is more like a “playboy” in the word world, such as stop words like “the” or “of.” They can mix with anyone but are not necessarily sincere to anyone. Therefore, for “you to be in me and me to be in you,” one must simultaneously consider P(w_2|w_1) and P(w_1|w_2), or consider a more symmetric quantity—the “relevance” mentioned earlier. Thus, the “Matchmaker” decides to use relevance to quantitatively describe the “affinity” value between two words.
Next, the “Matchmaker” begins his work, calculating the “affinity” between words one by one. As he works, he discovers serious problems.
First, the number is too large to finish. One must realize that the word world contains tens of thousands, hundreds of thousands, or even millions of words. If the affinity between every pair were calculated and recorded, it would require a table with billions or even trillions of entries. The workload is immense; the Matchmaker might be retired before finishing. However, from a responsible perspective, we cannot ignore the possibility of any two words being together!
Second, the N encounters between words are but a drop in the ocean compared to the long river of history. Does the fact that two words have never met really mean they have no affinity? Just because they haven’t met now doesn’t mean they won’t in the future. As a cautious Matchmaker, one cannot be so arbitrary. The relationships between words are complex; therefore, even if two words have never met, one cannot simply cut them off; their affinity value must still be estimated.
When reprinting, please include the original article address: https://kexue.fm/archives/4669
For more detailed reprinting matters, please refer to: Scientific Space FAQ