After the Teddy Cup competition last year, I wrote a brief blog post introducing the application of deep learning in sentiment analysis titled "Text Sentiment Classification (II): Deep Learning Models". Although the article was quite rough, it received an unexpected amount of feedback from readers. However, there were some ambiguities in the implementation of that article because: 1. Since that post, Keras has undergone significant changes, making the original code no longer compatible; 2. The code therein might have been modified by me casually, so the version published was not the most appropriate one. Therefore, nearly a year later, I am revisiting this topic to complete some tests that were left unfinished.
Why use deep learning models? Besides reasons like higher precision, another important reason is that it is currently the only model capable of achieving "end-to-end" learning. So-called "end-to-end" means being able to directly input raw data and labels, allowing the model to complete the entire process itself—including feature extraction and model learning. Looking back at our process for Chinese sentiment classification, it generally involves several steps: "segmentation — word vectors — sentence vectors (LSTM) — classification." While such models often achieve state-of-the-art results, some questions still require further testing to resolve. For Chinese, the character is the lowest-granularity unit of text. Therefore, from an "end-to-end" perspective, one should directly input the sentence as characters rather than segmenting it into words first. Is there actually a necessity for word segmentation? This article tests and compares the performance of character one-hot encoding, character embeddings, and word embeddings.
Model Testing
This article tests three models, or rather, three frameworks. The specific code is provided at the end of the text. These three frameworks are:
1. one hot: Character-based, no segmentation. Each sentence is truncated to 200 characters (padded with empty strings if shorter). The sentence is then input into an LSTM model in the form of a "character-one hot" matrix for learning and classification.
2. one embedding: Character-based, no segmentation. Each sentence is truncated to 200 characters (padded with empty strings if shorter). The sentence is then input into an LSTM model in the form of a "character-character embedding" matrix for learning and classification.
3. word embedding: Word-based, with segmentation. Each sentence is truncated to 100 words (padded with empty strings if shorter). The sentence is then input into an LSTM model in the form of a "word-word embedding" matrix for learning and classification.
The LSTM model structures used are similar. The corpus used is the same as in "Text Sentiment Classification (II): Deep Learning Models", with 15,000 samples for training and the remaining approximately 6,000 samples for testing. Surprisingly, all three models achieved similar results.
| Iterations | 90 | 30 | 30 |
| Time per Epoch | 100s | 36s | 18s |
| Training Accuracy | 96.60% | 95.95% | 98.41% |
| Test Accuracy | 89.21% | 89.55% | 89.03% |
As can be seen, in terms of accuracy, the three are similar with little differentiation. Whether using one-hot, character embeddings, or word embeddings, the results are roughly the same. Perhaps using the method from "Text Sentiment Classification (II): Deep Learning Models" to select an appropriate threshold for each model would yield higher test accuracy, but the relative accuracy between models likely wouldn’t change much.
Of course, there may be some unfair conditions in the testing itself that could lead to biased results, and I did not perform repeated tests. For example, the one-hot model iterated 90 times, while the other two models iterated 30 times. This is because the sample dimensions constructed by the one-hot model are too large, requiring a longer time to show convergence. Furthermore, during the training process, the accuracy fluctuated upwards rather than rising steadily like the other two models. In fact, this is a common characteristic of all one-hot models.
Further Discussion
It appears that the one-hot model indeed suffers from the curse of dimensionality, takes longer to train, and shows no significant improvement in performance. Does this mean there is no need to study one-hot representations?
I don’t think so. The reasons people criticized the one-hot model in the past, besides the curse of dimensionality, included the "semantic gap"—meaning there is no correlation between any two words (whether using Euclidean distance or cosine similarity, the result for any two words is the same). However, while this assumption might not hold for words, isn’t it quite reasonable when applied to Chinese "characters"? There are few cases where a Chinese character stands alone as a word; most are two-character words. This means the assumption that any two characters have no correlation is approximately true at the character level! Since we later used LSTM, and LSTM itself has the function of integrating neighboring data, it implicitly includes the process of integrating characters into words.
Furthermore, the one-hot model has another very important feature—it has no information loss. From the one-hot encoding results, we can inversely decode which characters or words composed the original sentence. However, I cannot determine what the original word was from a word vector. These points suggest that in many cases, the one-hot model is very valuable.
And why do we use word vectors? Word vectors essentially make an assumption: each word has a relatively fixed meaning. This assumption is also approximately true at the word level, as polysemous words are relatively rare. Because of this, we can place words into a lower-dimensional real number space, representing a word with a real vector and using the distance or cosine similarity between them to represent the similarity between words. This is also why word vectors can solve "synonymy" (multiple words for one meaning) but cannot solve "polysemy" (one word with multiple meanings).
From this perspective, among the three models above, only one-hot and word embedding are theoretically sound, while one embedding seems to become a bit of a hybrid, as characters themselves cannot be said to have a very fixed meaning. But why did one embedding also perform well? I suspect this might be because binary classification itself is a very coarse classification (0 or 1). If it were a multi-class problem, the performance of the one embedding method might drop. However, I haven’t conducted more tests because it is too time-consuming.
Of course, this can only be considered my subjective speculation, and I welcome corrections. In particular, the evaluation of the one embedding part is open to debate.
The Code
Perhaps you don’t want to hear me ramble and just want to see the code. Here is the code for the three models. It is best to have GPU acceleration, especially for testing the one-hot model; otherwise, it will be painfully slow.
Model 1: one hot
# -*- coding:utf-8 -*-
'''
One-hot test
On GTX960, approx 100s per epoch
After 90 iterations, training accuracy 96.60%, test accuracy 89.21%
Dropout shouldn't be too high, otherwise information loss is too severe
'''
import numpy as np
import pandas as pd
pos = pd.read_excel('pos.xls', header=None)
pos['label'] = 1
neg = pd.read_excel('neg.xls', header=None)
neg['label'] = 0
all_ = pos.append(neg, ignore_index=True)
maxlen = 200 # Truncate length (characters)
min_count = 20 # Drop characters appearing fewer than this many times. Simple dimensionality reduction.
content = ''.join(all_[0])
abc = pd.Series(list(content)).value_counts()
abc = abc[abc >= min_count]
abc[:] = list(range(len(abc)))
word_set = set(abc.index)
def doc2num(s, maxlen):
s = [i for i in s if i in word_set]
s = s[:maxlen]
return list(abc[s])
all_['doc2num'] = all_[0].apply(lambda s: doc2num(s, maxlen))
# Manually shuffle data
idx = list(range(len(all_)))
np.random.shuffle(idx)
all_ = all_.loc[idx]
# Generate data according to Keras input requirements
x = np.array(list(all_['doc2num']))
y = np.array(list(all_['label']))
y = y.reshape((-1,1)) # Adjust label shape
from keras.utils import np_utils
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout
from keras.layers import LSTM
import sys
sys.setrecursionlimit(10000) # Increase stack depth
# Build model
model = Sequential()
model.add(LSTM(128, input_shape=(maxlen,len(abc))))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='rmsprop',
metrics=['accuracy'])
# A single one-hot matrix is maxlen*len(abc), very memory intensive
# Use a generator to produce one-hot matrices for low-memory PCs
# Generate one-hot matrix only when called
# Reducing batch_size lowers memory usage but increases training time
batch_size = 128
train_num = 15000
# Pad with zero rows if insufficient
gen_matrix = lambda z: np.vstack((np_utils.to_categorical(z, len(abc)), np.zeros((maxlen-len(z), len(abc)))))
def data_generator(data, labels, batch_size):
batches = [list(range(batch_size*i, min(len(data), batch_size*(i+1)))) for i in range(len(data)//batch_size+1)]
while True:
for i in batches:
xx = np.zeros((maxlen, len(abc)))
xx, yy = np.array(list(map(gen_matrix, data[i]))), labels[i]
yield (xx, yy)
model.fit_generator(data_generator(x[:train_num], y[:train_num], batch_size), samples_per_epoch=train_num, nb_epoch=30)
model.evaluate_generator(data_generator(x[train_num:], y[train_num:], batch_size), val_samples=len(x[train_num:]))
def predict_one(s): # Prediction function for a single sentence
s = gen_matrix(doc2num(s, maxlen))
s = s.reshape((1, s.shape[0], s.shape[1]))
return model.predict_classes(s, verbose=0)[0][0]Model 2: one embedding
# -*- coding:utf-8 -*-
'''
One embedding test
On GTX960, 36s per epoch
After 30 iterations, training accuracy 95.95%, test accuracy 89.55%
Dropout shouldn't be too high, otherwise information loss is too severe
'''
import numpy as np
import pandas as pd
pos = pd.read_excel('pos.xls', header=None)
pos['label'] = 1
neg = pd.read_excel('neg.xls', header=None)
neg['label'] = 0
all_ = pos.append(neg, ignore_index=True)
maxlen = 200 # Truncate length (characters)
min_count = 20 # Drop characters appearing fewer than this many times.
content = ''.join(all_[0])
abc = pd.Series(list(content)).value_counts()
abc = abc[abc >= min_count]
abc[:] = list(range(1, len(abc)+1))
abc[''] = 0 # Add empty string for padding
word_set = set(abc.index)
def doc2num(s, maxlen):
s = [i for i in s if i in word_set]
s = s[:maxlen] + ['']*max(0, maxlen-len(s))
return list(abc[s])
all_['doc2num'] = all_[0].apply(lambda s: doc2num(s, maxlen))
# Manually shuffle data
idx = list(range(len(all_)))
np.random.shuffle(idx)
all_ = all_.loc[idx]
# Generate data according to Keras input requirements
x = np.array(list(all_['doc2num']))
y = np.array(list(all_['label']))
y = y.reshape((-1,1)) # Adjust label shape
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Embedding
from keras.layers import LSTM
# Build model
model = Sequential()
model.add(Embedding(len(abc), 256, input_length=maxlen))
model.add(LSTM(128))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
batch_size = 128
train_num = 15000
model.fit(x[:train_num], y[:train_num], batch_size = batch_size, nb_epoch=30)
model.evaluate(x[train_num:], y[train_num:], batch_size = batch_size)
def predict_one(s): # Prediction function for a single sentence
s = np.array(doc2num(s, maxlen))
s = s.reshape((1, s.shape[0]))
return model.predict_classes(s, verbose=0)[0][0]Model 3: word embedding
# -*- coding:utf-8 -*-
'''
Word embedding test
On GTX960, 18s per epoch
After 30 iterations, training accuracy 98.41%, test accuracy 89.03%
Dropout shouldn't be too high, otherwise information loss is too severe
'''
import numpy as np
import pandas as pd
import jieba
pos = pd.read_excel('pos.xls', header=None)
pos['label'] = 1
neg = pd.read_excel('neg.xls', header=None)
neg['label'] = 0
all_ = pos.append(neg, ignore_index=True)
all_['words'] = all_[0].apply(lambda s: list(jieba.cut(s))) # Call Jieba segmentation
maxlen = 100 # Truncate length (words)
min_count = 5 # Drop words appearing fewer than this many times.
content = []
for i in all_['words']:
content.extend(i)
abc = pd.Series(content).value_counts()
abc = abc[abc >= min_count]
abc[:] = list(range(1, len(abc)+1))
abc[''] = 0 # Add empty string for padding
word_set = set(abc.index)
def doc2num(s, maxlen):
s = [i for i in s if i in word_set]
s = s[:maxlen] + ['']*max(0, maxlen-len(s))
return list(abc[s])
all_['doc2num'] = all_['words'].apply(lambda s: doc2num(s, maxlen))
# Manually shuffle data
idx = list(range(len(all_)))
np.random.shuffle(idx)
all_ = all_.loc[idx]
# Generate data according to Keras input requirements
x = np.array(list(all_['doc2num']))
y = np.array(list(all_['label']))
y = y.reshape((-1,1)) # Adjust label shape
from keras.models import Sequential
from keras.layers import Dense, Activation, Dropout, Embedding
from keras.layers import LSTM
# Build model
model = Sequential()
model.add(Embedding(len(abc), 256, input_length=maxlen))
model.add(LSTM(128))
model.add(Dropout(0.5))
model.add(Dense(1))
model.add(Activation('sigmoid'))
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
batch_size = 128
train_num = 15000
model.fit(x[:train_num], y[:train_num], batch_size = batch_size, nb_epoch=30)
model.evaluate(x[train_num:], y[train_num:], batch_size = batch_size)
def predict_one(s): # Prediction function for a single sentence
s = np.array(doc2num(list(jieba.cut(s)), maxlen))
s = s.reshape((1, s.shape[0]))
return model.predict_classes(s, verbose=0)[0][0]When reposting, please include the original address of this article: https://kexue.fm/archives/3863
For more detailed reposting matters, please refer to: Scientific Space FAQ