Recording a Semi-Supervised Sentiment Analysis Experiment

Idea

The idea for this article originates from this OpenAI post: "OpenAI Research Finds Unsupervised Sentiment Neuron: Can Directly Control Sentiment of Generated Text".

The article introduces a method for unsupervised (actually semi-supervised) sentiment classification models, and the experimental results were excellent. However, the experiments in that paper were massive and nearly impossible for an individual to reproduce (it took one month to train on 4 Pascal GPUs). Nevertheless, the underlying idea is simple, and based on it, we can create a "simplified version." The logic is as follows:

Generally, when using deep learning for sentiment classification, a conventional approach is Embedding layer + LSTM layer + Dense layer (Sigmoid activation). What we usually call word vectors is equivalent to pre-training the Embedding layer (which has the largest number of parameters and is most prone to overfitting). OpenAI’s idea is: why not pre-train the LSTM layer as well? The pre-training method also uses a language model. Of course, to ensure the pre-training results do not lose sentiment information, the number of hidden nodes in the LSTM needs to be relatively large.

If the LSTM layer is also pre-trained, then the remaining Dense layer has few parameters, so it can be fully trained with a small number of labeled samples. This is the entire logic of the semi-supervised learning approach. As for the "sentiment neuron" mentioned in the OpenAI article, that is merely a descriptive metaphor.

Admittedly, from the perspective of the sentiment analysis task, the 73.48% accuracy in this article is hardly impressive; a simple "dictionary + rules" solution can achieve over 80% accuracy. I am merely verifying the feasibility of this experimental scheme. I believe that if the scale were as large as OpenAI’s, the results would be much better. Moreover, this article aims to describe a modeling strategy, not limited to sentiment analysis; the same idea can be applied to any binary or even multi-class classification problem.

Process

First, load the dataset and re-partition the training and test sets:

from keras.preprocessing import sequence
from keras.models import Model
from keras.layers import Input, Embedding, LSTM, Dense, Dropout
from keras.datasets import imdb
from keras import backend as K
import numpy as np

max_features = 10000 # Keep top max_features words
maxlen = 100 # Pad/truncate to 100 words
batch_size = 1000
nb_grams = 10 # Train a 10-gram language model
nb_train = 1000 # Number of training samples

# Load built-in IMDB dataset
(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)
x_lm_ = np.append(x_train, x_test)

# Construct data for training the language model
# Only existing data is used here; in a real environment, 
# other data can be added for more thorough training.
x_lm = []
y_lm = []
for x in x_lm_:
    for i in range(len(x)):
        x_lm.append([0]*(nb_grams - i + max(0,i-nb_grams))+x[max(0,i-nb_grams):i])
        y_lm.append([x[i]])

x_lm = np.array(x_lm)
y_lm = np.array(y_lm)
x_train = sequence.pad_sequences(x_train, maxlen=maxlen)
x_test = sequence.pad_sequences(x_test, maxlen=maxlen)
x = np.vstack([x_train, x_test])
y = np.hstack([y_train, y_test])

# Re-partition training and test sets
# Merge original train/test sets, randomly pick 1000 samples as new training set, rest as test set
idx = range(len(x))
np.random.shuffle(idx)
x_train = x[idx[:nb_train]]
y_train = y[idx[:nb_train]]
x_test = x[idx[nb_train:]]
y_test = y[idx[nb_train:]]

Then, build the model:

embedded_size = 100 # Word vector dimension
hidden_size = 1000 # LSTM dimension, can be understood as encoded sentence vector dimension.

# Encoder part
inputs = Input(shape=(None,), dtype='int32')
embedded = Embedding(max_features, embedded_size)(inputs)
lstm = LSTM(hidden_size)(embedded)
encoder = Model(inputs=inputs, outputs=lstm)

# Train the encoder part entirely using the n-gram model
input_grams = Input(shape=(nb_grams,), dtype='int32')
encoded_grams = encoder(input_grams)
softmax = Dense(max_features, activation='softmax')(encoded_grams)
lm = Model(inputs=input_grams, outputs=softmax)
lm.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
# Using sparse cross-entropy avoids converting categories to one-hot form beforehand.

# Sentiment analysis part
# Fix the encoder and follow it with a simple Dense layer (equivalent to logistic regression)
# At this point, only hidden_size + 1 = 1001 parameters are being trained
# Therefore, theoretically, a small number of labeled samples can suffice for training
for layer in encoder.layers:
    layer.trainable=False

sentence = Input(shape=(maxlen,), dtype='int32')
encoded_sentence = encoder(sentence)
sigmoid = Dense(10, activation='relu')(encoded_sentence)
sigmoid = Dropout(0.5)(sigmoid)
sigmoid = Dense(1, activation='sigmoid')(sigmoid)
model = Model(inputs=sentence, outputs=sigmoid)
model.compile(loss='binary_crossentropy', optimizer='adam', metrics=['accuracy'])

Now, train the language model. This part is quite time-consuming: # Train the language model; usually 2 or 3 iterations are enough.

lm.fit(x_lm, y_lm,
       batch_size=batch_size,
       epochs=3)

The training results for the language model are:

Epoch 1/3
11737946/11737946 [==============================] - 2400s - loss: 5.0376
Epoch 2/3
11737946/11737946 [==============================] - 2404s - loss: 4.5587
Epoch 3/3
11737946/11737946 [==============================] - 2404s - loss: 4.3968

Next, start training the sentiment analysis model with 1,000 samples. Since it has been pre-trained, there are few parameters to train here. Combined with Dropout, 1,000 samples will not lead to severe overfitting. # Train the sentiment analysis model

model.fit(x_train, y_train,
          batch_size=batch_size,
          epochs=200)

The training results are:

Epoch 198/200
1000/1000 [==============================] - 0s - loss: 0.2481 - acc: 0.9250
Epoch 199/200
1000/1000 [==============================] - 0s - loss: 0.2376 - acc: 0.9330
Epoch 200/200
1000/1000 [==============================] - 0s - loss: 0.2386 - acc: 0.9350

Next, evaluate the model:

# Evaluate model performance
model.evaluate(x_test, y_test, verbose=True, batch_size=batch_size)

The accuracy is 73.04%, which is mediocre. Let’s try transfer learning by training the model using both the training set and the prediction results of the test set (which may contain errors):

# Retrain the model using the training set and test set predictions
y_pred = model.predict(x_test, verbose=True, batch_size=batch_size)
y_pred = (y_pred.reshape(-1) > 0.5).astype(int)
xt = np.vstack([x_train, x_test])
yt = np.hstack([y_train, y_pred])

model.fit(xt, yt,
          batch_size=batch_size,
          epochs=10)

# Evaluate model performance again
model.evaluate(x_test, y_test, verbose=True, batch_size=batch_size)

The training results are:

Epoch 8/10
50000/50000 [==============================] - 27s - loss: 0.1455 - acc: 0.9561
Epoch 9/10
50000/50000 [==============================] - 27s - loss: 0.1390 - acc: 0.9590
Epoch 10/10
50000/50000 [==============================] - 27s - loss: 0.1349 - acc: 0.9600

This time we obtained an accuracy of 73.33%. It is not difficult to see that this process can be iterated. Repeating it once more yielded 73.33%, and a second time yielded 73.47%... It can be expected that this will converge to a stable value. I repeated it 5 more times, and it stabilized at 73.48%.

From the initial 73.04% to 73.48% after transfer learning, there is an improvement of about 0.44%. It might not seem like much, but for students participating in competitions or writing papers, a 0.44% improvement is a noteworthy contribution.

Comments

As stated at the beginning, this was a not-so-successful attempt, as it is a "simplified version," so one should not dwell too much on the low accuracy. From the experimental results, this scheme is reliable. Training a language model through a large volume of mixed-sentiment corpora can indeed extract text features effectively, similar to the autoencoding process in images.

This experiment was kept very simple, without fine-tuning hyperparameters. Potential improvements include: increasing the scale of the language model, adding more sentiment corpora (only sentiment reviews are needed, labels are not), and optimizing training details. I will leave those for now.