About Character Tagging Method
The previous article discussed the character tagging method for word segmentation. It is important to note that character tagging has great potential; otherwise, it would not have achieved the best results in open tests. In my view, character tagging is effective for two main reasons. First, it transforms the word segmentation problem into a sequence labeling problem where the labeling is aligned—meaning the input characters and output labels correspond one-to-one. This is a mature problem in sequence labeling. Second, this tagging method is actually a process of summarizing semantic patterns. Taking the 4-tag system as an example: we know that the character "Li" is a common surname and often serves as the first character of a multi-character word (like a name), so it is marked as ’b’; while "Xiang" often appears at the end of words like "Li-Xiang" (Ideal), giving it a high probability of being marked as ’e’. Consequently, if "Li" and "Xiang" are placed together, even if the word "Li-Xiang" is not in the original dictionary, we can correctly output "be," thereby identifying "Li-Xiang" as a single word. It is for this reason that even the HMM model, often considered the least precise, can produce decent results.
Regarding tagging, another topic worth discussing is the number of tags. The 4-tag system is commonly used, but there are also 6-tag and 2-tag systems. The simplest way to mark segmentation results is 2-tag (marking "split/no-split"), but the performance is poor. Why do more tags yield better results? Because more tags actually summarize semantic patterns more comprehensively. For example, using 4-tag labeling, we can summarize which characters form words alone, which are often used at the beginning, and which are used at the end. With only 2-tag, we can only summarize which characters often start a word. From an inductive perspective, this is not comprehensive enough. However, is 6-tag necessarily better than 4-tag? I don’t think so. 6-tag aims to summarize which characters serve as the second or third character, but is this perspective correct? It seems there aren’t many characters fixed specifically to the second or third position; the patterns for these are much weaker than those for the first and last characters (though from a new word discovery perspective, 6-tag makes it easier to find long words).
Bi-directional LSTM
To understand Bi-directional LSTM (Bi-LSTM), the logic is: Bi-LSTM is an improved version of LSTM, and LSTM is an improved version of RNN. Therefore, one must first understand RNN.
In my previous work, "From Boosting Learning to Neural Networks: Is a Mountain a Mountain?", I mentioned that the output of a model can actually be a feature and used as an input for the model. RNN is exactly such a network structure. A standard multi-layer neural network is a unidirectional propagation process from input to output. If high-dimensional input is involved, this can still be done, but with too many nodes, it becomes difficult to train and prone to overfitting. For instance, if an image input is 1000 \times 1000, it is hard to process directly, which led to CNNs. Or for a 1000-word sentence where each word uses a 100-dimensional vector, the input dimension is also large. In such cases, RNN is a solution (CNN can also be used, but RNN is more suitable for sequence problems).
The idea of RNN is that to predict the final result, I first use the first word to predict. Of course, the prediction using only the first word won’t be precise, so I take this result as a feature and use it along with the second word to predict the next result. Then, I combine this new prediction with the third word to make a new prediction, repeating this process until the last word. Thus, if the input has n words, we effectively make n predictions and provide n prediction sequences. Throughout this process, the model shares a single set of parameters. Therefore, RNN reduces the number of parameters, prevents overfitting, and is inherently designed for sequence problems, making it particularly suitable for them.
LSTM improves upon RNN by enabling the capture of longer-distance information. However, whether it is LSTM or RNN, there is an issue: they process from left to right, meaning later words might be treated as more important than earlier ones. For word segmentation, this is inappropriate because every character in a sentence should have equal weight. Thus, Bi-directional LSTM was introduced. It performs one LSTM pass from left to right and another from right to left, then combines the two results.
Application in Word Segmentation Tasks
Deep learning for word segmentation has been attempted by many before, as seen in the following articles:
In these articles, whether using simple neural networks or LSTM, the approach is similar to traditional models: they predict the label of the current character based on context. The context here is a fixed window, such as using the five characters before and after plus the current character. There is nothing wrong with this approach, but it merely replaces traditional probability estimation methods like HMM, ME, or CRF with neural networks. The entire framework remains unchanged; it is essentially still an n-gram model. With LSTM, which can perform sequence-to-sequence (seq2seq) output, why not directly output the sequence for the original sentence? This would truly utilize the full-text information. This is the experiment of this article.
LSTM can output a sequence based on an input sequence, considering contextual connections. Therefore, we can attach a softmax classifier to each output in the sequence to predict the probability of each label. Based on this seq2seq approach, we can directly predict the labels for the entire sentence.
Keras Implementation
Action is more important than words. The word vector dimension is set to 128, and the sentence length is truncated at 32 (samples longer than 32 characters are discarded; these are rare, as sentences separated by natural delimiters like commas and periods seldom exceed 32 characters). This time, I used 5 tags: in addition to the original 4 tags, I added an ’x’ tag to represent padding for sentences shorter than 32 characters. For example, if a sentence is 20 characters long, the 21st to 32nd labels are all ’x’.
For data, I used the portion provided by Microsoft Research (MSR) from the Bakeoff 2005 corpus. The code is as follows:
# -*- coding:utf-8 -*-
import re
import numpy as np
import pandas as pd
# Load data
s = open('msr_train.txt').read().decode('utf-8')
s = s.split('\r\n')
def clean(s): # Clean up data inconsistencies
if u'quote/s' not in s:
return s.replace(u' quote/s', '')
elif u'quote/s' not in s:
return s.replace(u'quote/s ', '')
else:
return s
s = u''.join(map(clean, s))
s = re.split(u'[,.!?\/]/[bems]', s)
data = [] # Generate training samples
label = []
def get_xy(s):
s = re.findall('(.)/(.)', s)
if s:
s = np.array(s)
return list(s[:,0]), list(s[:,1])
for i in s:
x = get_xy(i)
if x:
data.append(x[0])
label.append(x[1])
maxlen = 32
d = pd.DataFrame(index=range(len(data)))
d['data'] = data
d['label'] = label
d = d[d['data'].apply(len) <= maxlen]
d.index = range(len(d))
tag = pd.Series({'s':0, 'b':1, 'm':2, 'e':3, 'x':4})
chars = [] # Count all characters and assign IDs
for i in data:
chars.extend(i)
chars = pd.Series(chars).value_counts()
chars[:] = range(1, len(chars)+1)
# Generate model input format
from keras.utils import np_utils
d['x'] = d['data'].apply(lambda x: np.array(list(chars[x])+[0]*(maxlen-len(x))))
def trans_one(x):
_ = map(lambda y: np_utils.to_categorical(y,5), tag[x].values.reshape((-1,1)))
_ = list(_)
_.extend([np.array([[0,0,0,0,1]])]*(maxlen-len(x)))
return np.array(_)
d['y'] = d['label'].apply(trans_one)
# Design model
word_size = 128
from keras.layers import Dense, Embedding, LSTM, TimeDistributed, Input, Bidirectional
from keras.models import Model
sequence = Input(shape=(maxlen,), dtype='int32')
embedded = Embedding(len(chars)+1, word_size, input_length=maxlen, mask_zero=True)(sequence)
blstm = Bidirectional(LSTM(64, return_sequences=True), merge_mode='sum')(embedded)
output = TimeDistributed(Dense(5, activation='softmax'))(blstm)
model = Model(input=sequence, output=output)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
batch_size = 1024
model.fit(np.array(list(d['x'])), np.array(list(d['y'])).reshape((-1,maxlen,5)), batch_size=batch_size, nb_epoch=50)
# Transition probabilities (simplified to equal probability)
zy = {'be':0.5, 'bm':0.5, 'eb':0.5, 'es':0.5, 'me':0.5, 'mm':0.5, 'sb':0.5, 'ss':0.5}
zy = {i:np.log(zy[i]) for i in zy.keys()}
def viterbi(nodes):
paths = {'b':nodes[0]['b'], 's':nodes[0]['s']}
for l in range(1,len(nodes)):
paths_ = paths.copy()
paths = {}
for i in nodes[l].keys():
nows = {}
for j in paths_.keys():
if j[-1]+i in zy.keys():
nows[j+i]= paths_[j]+nodes[l][i]+zy[j[-1]+i]
if nows:
k = np.argmax(list(nows.values()))
paths[list(nows.keys())[k]] = list(nows.values())[k]
return list(paths.keys())[np.argmax(list(paths.values()))]
def simple_cut(s):
if s:
r = model.predict(np.array([list(chars[list(s)].fillna(0).astype(int))+[0]*(maxlen-len(s))]), verbose=False)[0][:len(s)]
r = np.log(r)
nodes = [dict(zip(['s','b','m','e'], i[:4])) for i in r]
t = viterbi(nodes)
words = []
for i in range(len(s)):
if t[i] in ['s', 'b']:
words.append(s[i])
else:
words[-1] += s[i]
return words
else:
return []
not_cuts = re.compile(u'([\da-zA-Z ]+)|[,.!?\/]')
def cut_word(s):
result = []
j = 0
for i in not_cuts.finditer(s):
result.extend(simple_cut(s[j:i.start()]))
result.append(s[i.start():i.end()])
j = i.end()
result.extend(simple_cut(s[j:]))
return resultWe can use model.summary() to view the model
structure:
_________________________________________________________________ Layer (type) Output Shape Param # ================================================================= input_2 (InputLayer) (None, 32) 0 _________________________________________________________________ embedding_2 (Embedding) (None, 32, 128) 660864 _________________________________________________________________ bidirectional_1 (Bidirectional) (None, 32, 64) 98816 _________________________________________________________________ timedistributed_2 (TimeDistribute) (None, 32, 5) 325 ================================================================= Total params: 760,005 _________________________________________________________________
How are the final results? I don’t intend to compare them with standard evaluation metrics; achieving over 90% accuracy on tests is not difficult for modern models. I am more interested in the recognition of new words and the handling of ambiguity. Below are some test results (randomly selected):
RNN / meaning / is / , / in / order / to / predict / final / result / , / I / first / use / first / word / predict / , / of / course / , / only / use / first / prediction / result / certainly / not / precise / , / I / take / this / result / as / feature / , / with / second / word / together / , / to / predict / result / ; / then / , / I / use / this / new / prediction / result / combine / third / word / , / to / make / new / prediction / ; / then / repeat / this / process / .
Married / and / not / yet / married
Su Jianlin / is / Scientific / Space / blogger / .
Guangdong Province / Yunfu City / Xinxing County
Wei Zexi / is / a / college / student
This / is / truly / an / unbearable / environment
Leo Tolstoy / is / Russia / one / famous / writer
Sofia / capital / of / Bulgaria / is / national / political / , / economic / , / cultural / center / , / located / in / Bulgaria / central / west
Roosevelt / was / World War II / period / Allied / camp / important / leader / one / of / . / 1941 / Pearl Harbor / incident / occurred / after / , / Roosevelt / strongly / advocated / against / Japan / declaration of war / , / and / introduced / price / control / and / rationing / . / Roosevelt / with / Lend-Lease / Act / made / America / transform / into / " / democratic / nation / arsenal / " / , / made / America / become / Allied / main / arms / supplier / and / financier / , / also / made / America / domestic / industry / significantly / expand / , / achieve / full / employment / . / Late / WWII / Allied / gradually / reversed / situation / after / , / Roosevelt / on / shaping / postwar / world / order / played / key / role / , / its / influence / in / Yalta / Conference / and / United Nations / establishment / especially / obvious / . / Later / , / with / US / assistance / , / Allies / defeated / Germany / , / Italy / and / Japan / .
It can be observed that the test results are very optimistic. Whether it is personal names (Chinese or foreign) or place names, the recognition effect is excellent. Regarding this model, I will stop here for now and continue to delve deeper in the future.
Conclusion
In fact, this article provides a framework that can directly label sequences through Bi-directional LSTM, providing a complete labeled sequence. This labeling approach can be used for many tasks, such as POS tagging and entity recognition. Therefore, the seq2seq labeling approach based on Bi-LSTM has broad applications and is worth studying. Even the currently popular deep learning machine translation is implemented using such sequence-to-sequence models.
When reposting, please include the original address: https://kexue.fm/archives/3924
For more detailed reposting matters, please refer to: Scientific Space FAQ