Obtaining and Processing Chinese Wikipedia Corpus

Threshold

While downloading is easy, using the Wikipedia corpus still has a certain threshold. The Wikipedia corpus downloaded directly is a compressed text package containing many HTML and Markdown tags, which basically cannot be used directly. Fortunately, enthusiastic experts have already written processing tools for us, mainly two: 1. Wikipedia Extractor; 2. The wikicorpus library in gensim. Both are based on Python.

However, neither of these two mainstream processing methods satisfies me. First, the results extracted by Wikipedia Extractor remove content marked with {{}}, which leads to situations like the following:

In Western languages, the word "mathematics" ( ; ) is derived from the Ancient Greek ( )

This is because the words inside the parentheses were marked with {{}} and were cleared. According to online tutorials, processing directly with gensim.corpora.wikicorpus.WikiCorpus is even more problematic because it removes all punctuation. For a perfectionist like me who pursues a high-quality corpus, this is unacceptable. Therefore, I combined gensim with my own logic to write a processing script.

Code

from gensim.corpora.wikicorpus import extract_pages, filter_wiki
import bz2file
import re
import opencc
from tqdm import tqdm
import codecs

wiki = extract_pages(bz2file.open('zhwiki-latest-pages-articles.xml.bz2'))

def wiki_replace(d):
    s = d[1]
    s = re.sub(':*{\|[\s\S]*?\|}', '', s)
    s = re.sub('<gallery>[\s\S]*?</gallery>', '', s)
    s = re.sub('(.){{([^{}\n]*?\|[^{}\n]*?)}}', '\\1[[\\2]]', s)
    s = filter_wiki(s)
    s = re.sub('\* *\n|\'{2,}', '', s)
    s = re.sub('\n+', '\n', s)
    s = re.sub('\n[:;]|\n +', '\n', s)
    s = re.sub('\n==', '\n\n==', s)
    s = u'[' + d[0] + u']\n' + s
    return opencc.convert(s).strip()

i = 0
f = codecs.open('wiki.txt', 'w', encoding='utf-8')
w = tqdm(wiki, desc=u'0 articles acquired')
for d in w:
    if not re.findall('^[a-zA-Z]+:', d[0]) and d[0] and not re.findall(u'^#', d[1]):
        s = wiki_replace(d)
        f.write(s+'\n\n\n')
        i += 1
        if i % 100 == 0:
            w.set_description(u'%s articles acquired'%i)

f.close()

Notes

As you can see, the main part of the code consists of regular expressions. First, we use bz2file to read the downloaded corpus directly without decompressing it, and then use extract_pages from gensim to extract each page. After extraction, we first handle some special non-text markers on the page, and then replace some useful {{}} markers with [[]], because [[]] markers are not completely cleared (readers will have to test the specific principles themselves). Then, we use the filter_wiki function from gensim for direct cleaning, followed by handling line break issues. Finally, we convert Traditional Chinese to Simplified Chinese using opencc.

In the subsequent loop, the condition re.findall(’^[a-zA-Z]+:’, d[0]) is used to remove help pages, and re.findall(u’^#’, d[1]) is used to remove redirect pages. In the end, approximately 919,000 pages are obtained. tqdm is used to display progress, which is a must-have. The program ran for about 40 minutes on my machine, resulting in a plain text corpus of about 1.5G. The running time is not important because preprocessing is a one-time task.

It is worth noting that opencc should not be installed using sudo apt-get install opencc, as the default version is too low. You should compile and install it from source, and then use pip install opencc to install the Python interface. At this point, calling opencc in Python might trigger a "segmentation fault." If this happens, you need to run:

cp /usr/lib/libopencc.so.1.0.0 /usr/lib/x86_64-linux-gnu/

By-product

As mentioned above, redirects mean that two words have the same meaning. I extracted all redirects from the Chinese Wikipedia and created a mapping table. In other words, the two words in each row of the word list have the same meaning. This can be considered a by-product.

Synonym table based on Chinese Wikipedia redirects: wiki_cn_mapping.7z

When reposting, please include the original link: https://kexue.fm/archives/4176

For more details on reposting, please refer to: "Scientific Space FAQ"