A More Unique Word Vector Model (Part 6): Code, Sharing, and Conclusion

List

A More Unique Word Vector Model (1): simpler glove

A More Unique Word Vector Model (2): Modeling Language

A More Unique Word Vector Model (3): Correlation-Describing Models

A More Unique Word Vector Model (4): Solving the Model

A More Unique Word Vector Model (5): Interesting Results

A More Unique Word Vector Model (6): Code, Sharing, and Conclusion

Code

The implementation of this article is located at: https://github.com/bojone/simpler_glove

The source code is modified from the original Stanford GloVe. I have only made minor modifications because the main difficulty lies in counting co-occurrence frequencies. I am grateful to the predecessors at Stanford for providing such a classic and excellent statistical implementation case. In fact, I am not familiar with the C language, so the modifications I made might not be very sophisticated; I hope experts can correct them.

In addition, to implement the “interesting results” from the previous section, I have added simpler_glove.py to GitHub. It encapsulates a class that can directly read the model files (txt format) exported by the C version of simple glove and includes some common functions for easy calling.

Sharing

Here is a set of Chinese word vectors trained using the model in this article. The training corpus is from Baidu Baike, containing a total of 1 million articles, approximately 300,000 words, with a word vector dimension of 128. A special treatment was applied during tokenization: all numbers and English characters were split into individual digits and letters. Friends who need to experiment can download it:

Link: http://pan.baidu.com/s/1jIb3yr8
Password: 1ogw

Conclusion

This article can be considered a relatively complete exploration of word vector models, and it is also a result of my theoretical “obsessive-compulsive disorder” (OCD). Fortunately, I finally obtained a model that looks good theoretically, which has initially cured my OCD. As for experimental results, applications, etc., they remain to be further verified in future use.

Most of the derivations in this article can be imitated to explain the experimental results of the word2vec skip-gram model; readers can try it. In fact, the word2vec skip-gram model does exhibit similar performance to the model in this article, including the properties of the word vectors.

Overall, combining theory with experiment is a wonderful thing, and of course, it is also very hard work, because just the things mentioned above took me several months of thinking time.

When reposting, please include the original link of this article: https://kexue.fm/archives/4681

For more details on reposting, please refer to: Scientific Space FAQ