For readers familiar with Deep Learning and Natural Language Processing (NLP), Word2Vec is a household name. Although not everyone has used it directly, most have heard of it—the high-efficiency tool for obtaining word vectors released by Google.
Word2Vec is Incredible?
Most people treat Word2Vec as a synonym for word vectors; that is, they use it purely as a tool to obtain word vectors, and few care about the model itself. This might be because the model is so simplified that people assume such a simple model must be inaccurate and thus unusable for modeling language, even if its byproduct—the word vectors—is of good quality. Indeed, if viewed strictly as a language model, Word2Vec is quite crude.
However, why must we view it only as a language model? If we step away from the constraints of language modeling and look at the model itself, we find that the two models of Word2Vec—CBOW and Skip-Gram—are actually very useful. They describe the relationship between surrounding words and the current word from different perspectives, and many basic NLP tasks are built upon this relationship, such as keyword extraction and logical reasoning. This series of articles aims to serve as a starting point by introducing the Word2Vec model itself and several seemingly "incredible" applications to provide new ideas for researching such problems.
Speaking of the "incredible" nature of Word2Vec, when it was first released, perhaps the most surprising feature was its Word Analogy property, such as the linear relationship: \text{king} - \text{man} \approx \text{queen} - \text{woman}. The author, Mikolov, believed this property implied that the word vectors generated by Word2Vec possessed semantic reasoning capabilities. It was this feature, combined with Google’s prestige, that made Word2Vec rapidly popular. Unfortunately, when we train word vectors ourselves, it is actually quite difficult to reproduce this specific result, and there is no solid theoretical basis suggesting that a good set of word vectors must satisfy this Word Analogy property. In contrast, the various uses I will introduce here have very high reproducibility. Readers can even train a Word2Vec model on a small corpus and obtain similar results.
Mathematical Principles: Online Resources
Readers interested in this series should understand the mathematical
principles of Word2Vec. Since Word2Vec has been out for several years,
there are countless articles introducing it. I recommend the blog series
by Peghoty:
http://blog.csdn.net/itplus/article/details/37969519
Additionally, the post "What exactly are Word Vectors and Embeddings?" on this blog is also helpful for understanding the principles of Word2Vec.
For convenience, I have also collected two corresponding PDF
files:
Detailed
Explanation of Mathematical Principles in word2vector.pdf
Deep
Learning in Action: word2vec.pdf
The first one is the PDF version of Peghoty’s recommended blog series. Of course, if your English is good, you can directly read the original Word2Vec papers:
[1] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient Estimation of Word Representations in Vector Space. In Proceedings of Workshop at ICLR, 2013.
[2] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed Representations of Words and Phrases and their Compositionality. In Proceedings of NIPS, 2013.
However, I personally feel that the original papers are not as clear as the Chinese explanations.
Mathematical Principles: A Simple Explanation
Simply put, Word2Vec consists of "two training schemes + two speed-up methods," so strictly speaking, it has four candidate models.
The two training schemes are CBOW and Skip-Gram, as shown in the figure:
In plain language, these are "using surrounding words to predict the current word" (P(w_t|\text{Context})) and "using the current word to predict surrounding words" (P(w_{\text{others}}|w_t)), which are conditional probability modeling problems. The two speed-up methods are Hierarchical Softmax and Negative Sampling. Hierarchical Softmax is a simplification of Softmax that reduces the complexity of predicting probabilities from \mathcal{O}(|V|) to \mathcal{O}(\log_2 |V|), though its accuracy may be slightly lower than native Softmax. Negative Sampling takes a different approach: it treats the combination of input and output as a single input and performs a binary classification to score it. This can be seen as modeling the joint probabilities P(w_t, \text{Context}) and P(w_t, w_{\text{others}}). Positive samples are those that appear in the corpus, while negative samples are randomly sampled. For more details, it is best to read Peghoty’s blog series; that is where I learned the implementation details of Word2Vec.
Finally, it should be noted that the model used in this series is the "Skip-Gram + Hierarchical Softmax" combination. We will be using the P(w_{\text{others}}|w_t) model itself, not just the word vectors. Therefore, readers who wish to follow this series need to have some understanding of the Skip-Gram model and some impression of how Hierarchical Softmax is constructed and implemented.
Please stay tuned!
Original link: https://kexue.fm/archives/4299
For more details on reprinting,
please refer to: "Scientific
Space FAQ"