If you ask me which word vector model is the most convenient and easy to use, I think it should be word2vec. However, if you ask me which word vector model is the most beautiful, I don’t know. I feel that every model has some shortcomings. Setting aside whether the experimental results are good or not (which is merely a matter of evaluation metrics), looking at the theory alone, no model can be called truly "beautiful."
This article discusses several questions about word vectors that people are generally concerned about. Many conclusions are basically discovered through experiments and lack reasonable explanations, including:
How to construct a word vector model?
Why use cosine similarity for synonym search? What is the meaning of the inner product of vectors?
Does the norm (magnitude) of a word vector have any special meaning?
Why do word vectors possess word analogy properties? (King - Man + Woman = Queen)
How to construct sentence vectors after obtaining word vectors? What is the basis for using the sum of word vectors as a simple sentence vector?
These discussions are both specific and general. Some explanations might be directly transferable to the interpretation of the properties of word vectors in the GloVe model and the Skip-gram model; readers can try this for themselves.
Centering on the discussion of these questions, this article proposes a new GloVe-like word vector model, referred to here as simpler glove. Based on modifications to Stanford’s GloVe source code, an implementation is provided. The specific code is available on GitHub.
Why improve GloVe? It is certain that the ideas behind GloVe are very inspiring. However, although it claims to rival or even surpass word2vec, it is inherently a relatively poor model (we will explain why it is poor later). Therefore, there is room for improvement.
Content Overview:
1 Modeling Language
1.1 From Conditional Probability to Mutual Information
1.2 Additivity of Mutual Information
1.3 Interlude: Side Story2 Models Describing Correlation
2.1 Geometric Word Vectors
2.2 Airport - Airplane + Train = Railway Station
2.3 Model Form
2.4 Forgetting Normalization3 Solving the Model
3.1 Loss Function
3.2 Estimating Mutual Information
3.3 Weighting and Downsampling
3.4 Adagrad4 Interesting Results
4.1 Meaning of the Norm
4.2 Word Analogy Experiments
4.3 Ranking Related Words
4.4 Redefining Similarity
4.5 Keyword Extraction
4.6 Sentence Similarity
4.7 Sentence Vectors
When reprinting, please include the original address of this article: https://kexue.fm/archives/4667
For more detailed reprinting matters, please refer to: "Scientific Space FAQ"