Loss Function
Now, let us define the loss function to solve for the word vectors. Let \tilde{P} denote the frequency estimate of P. We can directly use the following expression as the loss: \sum_{w_i,w_j}\left(\langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle-\log\frac{\tilde{P}(w_i,w_j)}{\tilde{P}(w_i)\tilde{P}(w_j)}\right)^2 \tag{16} Compared to GloVe, this approach is simpler in terms of both the number of parameters and the model form; hence, we call it "Simpler GloVe." The GloVe model is: \sum_{w_i,w_j}\left(\langle \boldsymbol{v}_i, \boldsymbol{\hat{v}}_j\rangle+b_i+\hat{b}_j-\log X_{ij}\right)^2 \tag{17} In the GloVe model, a distinction is made between the center word vector and the context vector, and the final model suggests outputting the sum of these two sets of word vectors. It is said that this yields better results, which is a somewhat forced trick, though not necessarily a flaw. The biggest problem is that the parameters b_i, \hat{b}_j are also trainable, which makes the model severely ill-posed! We have: \begin{aligned} &\sum_{w_i,w_j}\left(\langle \boldsymbol{v}_i, \boldsymbol{\hat{v}}_j\rangle+b_i+\hat{b}_j-\log \tilde{P}(w_i,w_j)\right)^2\\ =&\sum_{w_i,w_j}\left[\langle \boldsymbol{v}_i+\boldsymbol{c}, \boldsymbol{\hat{v}}_j+\boldsymbol{c}\rangle+\Big(b_i-\langle \boldsymbol{v}_i, \boldsymbol{c}\rangle - \frac{|\boldsymbol{c}|^2}{2}\Big)\right.\\ &\qquad \left.+\Big(\hat{b}_j-\langle \boldsymbol{\hat{v}}_j, \boldsymbol{c}\rangle - \frac{|\boldsymbol{c}|^2}{2}\Big)-\log X_{ij}\right]^2 \end{aligned} \tag{18} This means that if you have a set of solutions, adding any constant vector to all word vectors still results in a valid set of solutions! This is a serious issue because we cannot predict which set of solutions will be obtained. Once a very large constant vector is added, various metrics become meaningless (for example, the cosine similarity between any two words would approach 1). In fact, checking the word vectors generated by GloVe reveals that the norms of stop words are much larger than those of general words. This means that when a group of words is placed together, the influence of stop words is significantly more pronounced, which is clearly detrimental to the optimization of downstream models. (Although, judging from current experimental results on GloVe, this might just be my own perfectionism speaking.)
Mutual Information Estimation
To solve the model, the first problem to address is how to calculate
P(w_i,w_j), P(w_i), P(w_j). P(w_i) and P(w_j) are simple; they can be estimated
directly through statistics. But what about P(w_i,w_j)? When are two words considered to
have co-occurred? Of course, different applications can have different
schemes. For instance, we could consider two words appearing in the same
article as having met once. This scheme is usually helpful for topic
classification, but the computational cost is too high. A more common
scheme is to select a fixed integer, denoted as window; all
words within window positions before and after a word are
considered to have met that word.
A detail worth noting is: Should the
co-occurrence of a center word with itself be counted? The
definition of a window should include words whose distance from the
center word does not exceed window, so it should
technically be included. However, if it is included, it has little
predictive significance because this term always exists. If it is not
included, the mutual information between a word and itself is reduced.
Therefore, we adopt a small trick: do not count identical co-occurrence
terms and let the model learn this itself. That is, even if the center
word appears in the context (excluding the center position itself), it
is not included in the loss. Since the amount of data is far greater
than the number of parameters, this term can always be learned
eventually.
Weighting and Downsampling
The GloVe model defines the following weighting formula: \lambda_{ij}=\Big(\min\{x_{ij}/x_{max}, 1\}\Big)^{\alpha} \tag{19} where x_{ij} represents the co-occurrence frequency of the word pair (w_i,w_j), and x_{max}, \alpha are fixed constants, usually taken as x_{max}=100, \alpha=3/4. This means that word pairs with low co-occurrence frequencies are down-weighted as they are more likely to be noise. Thus, the final GloVe loss is: \sum_{w_i,w_j}\lambda_{ij}\left(\langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle+b_i+b_j-\log \tilde{P}(w_i,w_j)\right)^2 \tag{20}
In our text model, we continue to use this weighting but with some choices. First, raising the frequency to the power of \alpha is equivalent to increasing the weight of low-frequency items, which is basically consistent with the approach in word2vec. What is worth considering is the \min truncation operation. If this truncation is performed, it is equivalent to greatly reducing the weight of high-frequency words, similar to the downsampling of high-frequency words in word2vec. This can improve the learning effect for low-frequency words, but the possible consequence is that the norms of high-frequency words are not learned well. We can see this in the section "The Meaning of Norms." Overall, different scenarios have different requirements; therefore, in the final released source code, we allow users to define whether to truncate this weight.
Adagrad
Like GloVe, we also use the Adagrad algorithm for optimization. The reason for using Adagrad is that it is perhaps the simplest adaptive learning rate algorithm currently available.
However, I discovered that the Adagrad implementation in the GloVe source code is wrong!! I don’t know if the way GloVe is written is a deliberate improvement or a typo (it seems unlikely to be a typo?), but in any case, if I use their iteration process without modification for the Simpler GloVe model in this article, various unsolvable NaNs easily appear! If written as standard Adagrad, NaNs do not occur.
Selecting a word pair w_i, w_j, we obtain the loss: L=\lambda_{ij}\left(\langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle-\log\frac{\tilde{P}(w_i,w_j)}{\tilde{P}(w_i)\tilde{P}(w_j)}\right)^2 \tag{21} Its gradients are: \begin{aligned} \nabla_{\boldsymbol{v}_i} L=\lambda_{ij}\left(\langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle-\log\frac{\tilde{P}(w_i,w_j)}{\tilde{P}(w_i)\tilde{P}(w_j)}\right)\boldsymbol{v}_j\\ \nabla_{\boldsymbol{v}_j} L=\lambda_{ij}\left(\langle \boldsymbol{v}_i, \boldsymbol{v}_j\rangle-\log\frac{\tilde{P}(w_i,w_j)}{\tilde{P}(w_i)\tilde{P}(w_j)}\right)\boldsymbol{v}_i \end{aligned} \tag{22} Then, update according to the Adagrad algorithm formula. The default initial learning rate is chosen as \eta=0.1, and the iteration formula is: \left\{\begin{aligned}\boldsymbol{g}_{\gamma}^{(n)} =& \nabla_{\boldsymbol{v}_{\gamma}^{(n)}} L\\ \boldsymbol{G}_{\gamma}^{(n)} =& \boldsymbol{G}_{\gamma}^{(n-1)} + \boldsymbol{g}_{\gamma}^{(n)}\otimes \boldsymbol{g}_{\gamma}^{(n)}\\ \boldsymbol{v}_{\gamma}^{(n)} =& \boldsymbol{v}_{\gamma}^{(n-1)} - \frac{\boldsymbol{g}_{\gamma}^{(n)}}{\sqrt{\boldsymbol{G}_{\gamma}^{(n-1)}}}\eta \end{aligned}\right.,\,\gamma=i,j \tag{23} According to the formula, the Adagrad algorithm is basically insensitive to the scaling of the loss. In other words, multiplying the loss by 10 results in basically no change in the final optimization effect. However, in stochastic gradient descent, multiplying the loss by 10 is equivalent to multiplying the learning rate by 10.
When reposting, please include the original address of this article: https://kexue.fm/archives/4675
For more detailed reposting matters, please refer to: "Scientific Space FAQ"