The Incredible Word2Vec: 2. The Trained Model · English (unofficial) translations of posts at kexue.fm

Since the next few posts will explain how to use Word2Vec, I have already trained a Word2Vec model. To save readers’ time and ensure that the results in subsequent posts can be reproduced, I have decided to share this pre-trained model, which was trained using Gensim. While word vectors alone are not very large, as mentioned in the first post, we will need the complete Word2Vec model. Therefore, I am sharing the full model, which consists of four files, making the total size relatively large.

I would like to remind readers that if you want to obtain the complete Word2Vec model without modifying the source code, Python’s Gensim library is likely your only choice. As far as I know, other versions of Word2Vec only provide word vectors at the end, not the complete model.

For knowledge mining, Word2Vec trained on knowledge base corpora (such as encyclopedia corpora) would obviously perform better. However, I am still in the process of crawling encyclopedia data. Once finished, I will train another model and share it then.

Model Overview

The general details of this model are as follows:

Feature	Description
Training Corpus	WeChat public account articles, multi-domain, balanced Chinese corpus
Corpus Quantity	8 million articles, total word count reaching 65 billion
Model Vocabulary	352,196 words in total, mostly Chinese, including common English words
Model Architecture	Skip-Gram + Huffman Softmax
Vector Dimensions	256 dimensions
Segmentation Tool	Jieba segmentation, with a 500,000-entry dictionary, new word discovery disabled
Training Tool	Gensim’s Word2Vec, trained on a server for 7 days
Other Details	Window size 10, minimum word frequency 64, 10 iterations

It is important to note: Public account articles are relatively “modern” and reflect recent internet hotspots with broad coverage; thus, the content is quite typical. For segmentation, I used Jieba and disabled new word discovery, preferring to segment fewer words to ensure higher accuracy. Of course, the built-in dictionary was insufficient, so I compiled 500,000 entries myself. These entries came from two sources: 1. Merging dictionaries collected from the internet; 2. Performing new word discovery on public account articles and adding them to the dictionary after manual screening. Consequently, the segmentation results are quite reliable, containing many popular words and offering high usability.

Training Code

You can refer to this code for your own modifications. Note that hashlib.md5 is used here for deduplication (originally 10 million articles, reduced to 8 million after deduplication), though this step is not strictly necessary.

#! -*- coding:utf-8 -*-

import gensim, logging
logging.basicConfig(format='%(asctime)s : %(levelname)s : %(message)s', level=logging.INFO)

import pymongo
import hashlib

db = pymongo.MongoClient('172.16.0.101').weixin.text_articles_words
md5 = lambda s: hashlib.md5(s).hexdigest()

class sentences:
    def __iter__(self):
        texts_set = set()
        for a in db.find(no_cursor_timeout=True):
            if md5(a['text'].encode('utf-8')) in texts_set:
                continue
            else:
                texts_set.add(md5(a['text'].encode('utf-8')))
                yield a['words']
        print u'Finally calculated %s articles'%len(texts_set)

word2vec = gensim.models.word2vec.Word2Vec(sentences(), size=256, window=10, min_count=64, sg=1, hs=1, iter=10, workers=25)
word2vec.save('word2vec_wx')

Download Links

Link: https://pan.baidu.com/s/1htC495U Password: 4ff8

Included files: word2vec_wx, word2vec_wx.syn1neg.npy, word2vec_wx.syn1.npy, word2vec_wx.wv.syn0.npy. All four files are necessary for Gensim to load the model. While the specific meaning of each file is not entirely clear, word2vec_wx is likely the model declaration, word2vec_wx.wv.syn0.npy should be the word vector table, word2vec_wx.syn1.npy contains the parameters from the hidden layer to the output layer (Huffman tree parameters), and word2vec_wx.syn1neg.npy is less certain.

If you only care about the word vectors, you can also download the C-version format (compatible with the C version of Word2Vec, containing only word vectors):
Link: https://pan.baidu.com/s/1nv3ANLB Password: dgfw

Demonstrations

Here are some casual demonstrations of the model finding synonyms. Suggestions for improvement are welcome.

>>> import gensim
>>> model = gensim.models.Word2Vec.load('word2vec_wx')

>>> pd.Series(model.most_similar(u'WeChat'))
0        (QQ, 0.752506196499)
1       (Subscription Account, 0.714340209961)
2       (QQ Number, 0.695577561855)
3       (Scan, 0.695488214493)
4     (WeChat Public Account, 0.694692015648)
5        (Private Chat, 0.681655049324)
6    (WeChat Public Platform, 0.674170553684)
7         (Private Message, 0.65382117033)
8       (WeChat Platform, 0.65175652504)
9        (Official, 0.643620729446)

>>> pd.Series(model.most_similar(u'Public Account'))
0       (Subscription Account, 0.782696723938)
1     (WeChat Public Account, 0.760639667511)
2     (WeChat Public Account, 0.73489522934)
3      (Public Platform, 0.716173946857)
4       (Scan, 0.697836577892)
5    (WeChat Public Platform, 0.696847081184)
6        (Pin to Top, 0.666775584221)
7      (Public Account, 0.665741920471)
8      (WeChat Platform, 0.661035299301)
9        (Menu Bar, 0.65234708786)

>>> pd.Series(model.most_similar(u'Awesome'))
0      (Impressive, 0.701575636864)
1      (Great, 0.619165301323)
2      (Reliable, 0.588266670704)
3      (Miserable, 0.586573541164)
4     (Bragging, 0.569260418415)
5     (Amazing, 0.565731525421)
6      (Awesome (slang), 0.563843131065)
7      (Absolutely, 0.549570798874)
8     (Seriously, 0.549259066582)
9    (Got skills, 0.545115828514)

>>> pd.Series(model.most_similar(u'Guangzhou'))
0    (Dongguan, 0.840889930725)
1    (Shenzhen, 0.799216389656)
2    (Foshan, 0.786817133427)
3    (Huizhou, 0.779960036278)
4     (Zhuhai, 0.73523247242)
5     (Xiamen, 0.72509008646)
6    (Wuhan, 0.724122405052)
7    (Shantou, 0.719602584839)
8    (Zengcheng, 0.713532209396)
9    (Shanghai, 0.710560560226)

>>> pd.Series(model.most_similar(u'Zhu Yuanzhang'))
0        (Zhu Di, 0.857951819897)
1        (Prince of Yan, 0.853199958801)
2        (Imperial Court, 0.847517609596)
3    (Emperor Taizu of Ming, 0.837111353874)
4       (Zhao Kuangyin, 0.835654854774)
5        (Proclaimed Emperor, 0.835589051247)
6        (Raised an Army, 0.833530187607)
7       (Ming Taizu, 0.829249799252)
8        (Taizu, 0.826784193516)
9        (Chancellor, 0.826457977295)

>>> pd.Series(model.most_similar(u'Calculus'))
0    (Linear Algebra, 0.808522999287)
1    (Mathematical Analysis, 0.791161835194)
2    (Advanced Mathematics, 0.786414265633)
3      (Mathematics, 0.758676528931)
4     (Probability Theory, 0.747221827507)
5    (Advanced Algebra, 0.737897276878)
6    (Analytic Geometry, 0.730488717556)
7    (Complex Functions, 0.715447306633)
8     (Differential Equations, 0.71503329277)
9    (Calculus (Academic), 0.704192101955)

>>> pd.Series(model.most_similar(u'apple'))
0         (banana, 0.79927945137)
1     (pineapple, 0.789698243141)
2           (pen, 0.779583632946)
3        (orange, 0.769554674625)
4         (sweet, 0.721074819565)
5          (fruit, 0.71402490139)
6           (pie, 0.711439430714)
7    (watermelon, 0.700904607773)
8        (apples, 0.697601020336)
9         (juice, 0.694036960602)

>>> pd.Series(model.most_similar(u'Penguin'))
0     (Seal, 0.665253281593)
1    (Emperor Penguin, 0.645192623138)
2    (Polar Bear, 0.619929730892)
3     (Elephant, 0.618502140045)
4     (Whale, 0.606555819511)
5      (Cat, 0.591019570827)
6     (Lizard, 0.584576964378)
7     (Blue Whale, 0.572826981544)
8     (Dolphin, 0.566122889519)
9     (Gorilla, 0.563284397125)

>>> pd.Series(model.most_similar(u'Football'))
0      (Basketball, 0.842746257782)
1    (Football Sport, 0.819511592388)
2      (Youth Training, 0.793446540833)
3      (Volleyball, 0.774085760117)
4     (Table Tennis, 0.760577201843)
5    (Football Events, 0.758624792099)
6     (Baseball/Softball, 0.750351667404)
7    (Basketball Sport, 0.746055066586)
8      (Football Team, 0.74296438694)
9      (Tennis, 0.742858171463)

>>> pd.Series(model.most_similar(u'Dad'))
0    (Mom, 0.779690504074)
1    (Son, 0.752222895622)
2     (Grandma, 0.70418381691)
3     (Ma, 0.693783283234)
4    (Grandpa, 0.683066487312)
5    (Father, 0.673043072224)
6    (Daughter, 0.670304119587)
7    (Parents, 0.669358253479)
8     (Pa, 0.663688421249)
9    (Maternal Grandma, 0.652905225754)

>>> pd.Series(model.most_similar(u'Taobao'))
0       (Tao, 0.770935535431)
1      (Shop, 0.739198565483)
2     (Mobile Client, 0.728774428368)
3     (Tmall Shop, 0.725838780403)
4      (Password/Code, 0.721312999725)
5    (Login Taobao, 0.717839717865)
6      (Taobao Shop, 0.71473968029)
7     (Search Taobao, 0.697688698769)
8      (Tmall, 0.690212249756)
9        (Online Store, 0.6820114851)

When reprinting, please include the original address of this article: https://kexue.fm/archives/4304

For more detailed reprinting matters, please refer to: "Scientific Space FAQ"