An NLP Library Based on the Principle of Minimum Entropy: nlp zero

Default Tokenization

The library comes with a built-in dictionary and can be used as a simple tokenization tool:

from nlp_zero import *

t = Tokenizer()
t.tokenize(u'Scan the QR code and follow the official account')

The built-in dictionary includes new words discovered through "new word discovery" algorithms and has been manually optimized by the author, ensuring relatively high quality.

Lexicon Construction

You can build a lexicon using a large amount of raw corpora.

First, we need to write an iterator container so that we don’t have to load all the corpora into memory at once. The implementation of the iterator is very flexible. For example, if the data is stored in MongoDB:

import pymongo
db = pymongo.MongoClient().weixin.text_articles

class D:
    def __iter__(self):
        for i in db.find().limit(10000):
            yield i['text']

If the data is stored in a text file, it would look something like this:

class D:
    def __iter__(self):
        with open('text.txt') as f:
            for l in f:
                yield l.strip() # Python 2.x might need encoding conversion

Then you can execute the following:

from nlp_zero import *
import logging
logging.basicConfig(level = logging.INFO, format = '%(asctime)s - %(name)s - %(message)s')

f = Word_Finder(min_proba=1e-8)
f.train(D()) # Calculate mutual information
f.find(D())  # Construct the lexicon

View the results using Pandas:

import pandas as pd

words = pd.Series(f.words).sort_values(ascending=False)

Create a tokenization tool directly using the statistically constructed lexicon:

t = f.export_tokenizer()

t.tokenize(u'The weather is nice today')

Sentence Template Construction

Similar to the previous section, you need to write an iterator.

Since sentence template construction is based on word statistics, a tokenization function is required. You can use the built-in tokenizer or an external one, such as Jieba.

from nlp_zero import *
import logging
logging.basicConfig(level = logging.INFO, format = '%(asctime)s - %(name)s - %(message)s')

tokenize = Tokenizer().tokenize # Use the built-in tokenizer
# Alternatively, use tokenize = jieba.lcut for Jieba

f = Template_Finder(tokenize, window=3)
f.train(D())
f.find(D())

View the results using Pandas:

import pandas as pd

templates = pd.Series(f.templates).sort_values(ascending=False)
idx = [i for i in templates.index if not i.is_trivial()]
templates = templates[idx] # Filter out trivial templates

Each template has been encapsulated into a class.

Hierarchical Decomposition

Sentence structure parsing based on sentence templates.

from nlp_zero import *

# Create a Trie and add templates
# Templates can be added via tuples,
# or directly via "trie[template_class] = 10"
trie = XTrie()
trie[(None, u'ne')] = 10
trie[(None, u'can', None, u'ma')] = 9
trie[(u'I', None)] = 8
trie[(None, u'of', None, u'is', None)] = 7
trie[(None, u'of', None, u'is', None, u'ne')] = 7
trie[(None, u'of', None)] = 12
trie[(None, u'and', None)] = 12

tokenize = Tokenizer().tokenize # Use built-in tokenizer
p = Parser(trie, tokenize) # Create a parser

p.parse(u'Can I eat the egg?') # Parse the sentence
"""Output:
>>> p.parse(u'Can I eat the egg?')
+---> (egg) can (eat) ma
|     +---> egg
|     |     +---> egg
|     +---> can
|     +---> eat
|     |     +---> eat
|     +---> ma
"""

For convenience in calling and visualizing results, the output is encapsulated as a SentTree class. This class has three attributes: template (the current main template), content (the string covered by the current main template), and modules (a list of semantic blocks, where each block is also described as a SentTree). Overall, it is designed according to the assumptions about language structure discussed in the article "The Principle of Minimum Entropy (III): Sentence Templates and Language Structure".