Some time ago, I participated in a rather absurd online competition—Aspect-Based Sentiment Analysis in specific domains. The homepage can be found here. The task of the competition was to identify entities in a given text and then determine the sentiment associated with them. For example, in the sentence “I like Honda, but I don’t like Toyota,” one must identify “Honda” and “Toyota.” From the perspective of Honda, the sentiment is positive, while from the perspective of Toyota, the sentiment is negative. In other words, it is equivalent to combining entity recognition with sentiment analysis.
Critique
It sounds high-end, so why do I call it absurd? The task itself is quite good and worth researching. However, the official organization was quite frustrating, mainly reflected in two points: 1. The competition was divided into preliminary, semi-final, and final stages. The preliminary lasted over a month, after which some participants were filtered into the semi-finals. The semi-finals simply involved a slight change in data, while the problem and the domain of the data remained identical. The semi-final also lasted a month. What exactly was the point of this semi-final? 2. Take a look at what the contestants were discussing in the group chat:
User A 17:40:54
128004 [Hangzhou De’ao Audi Certified Pre-owned Car] Audi TT Coupe 45 TFSI Quattro 2015, 536,900 RMB
User A 17:40:57
@Competition Guide
User A 17:41:09
Where should the aspect be extracted for this?
Competition Guide 17:41:19
Audi TTUser B 20:19:47
I haven’t driven many good cars, but I feel Honda’s handling is better than Toyota or Nissan. Should “Toyota” and “Nissan” here be labeled as neg or neu?
User B 20:20:00
It feels like the standards between the preliminary and semi-final are inconsistent for this.
User B 20:20:12
@Competition Guide @Competition Guide 3
Competition Guide 21:29:52
neuUser C 10:15:00
@Competition Guide SAIC Volkswagen, should “SAIC” be deleted?
Competition Guide 10:15:18
NoUser D 20:49:06
Is there an aspect like “Imported Ford”? @Competition Guide
User D 20:49:16
Imported BMW?
Competition Guide 20:54:43
NoUser C 10:57:28
Kia Forte appeared a lot, should I mark “Kia”? @Competition Guide
Competition Guide 11:43:04
No
I don’t have much to say. If the organizers think this is machine learning, then so be it, but to me, it looks more like “administrator learning.”
Anyway, since it’s such a silly competition, I’ll play along. I don’t expect to get a high ranking. Since the competition hasn’t ended yet, I’m making my model public. If your score is lower than mine, you can use this template to boost your results.
Model
In fact, my approach to this task is similar to what I described in “Seq2Seq Core Entity Recognition Based on Bidirectional LSTM and Transfer Learning,” treating it as a sequence labeling problem. However, I replaced the LSTM with a GRU, which has fewer parameters. This time, I used character-level labeling: 0 for non-entity parts, 1 for positive entities, 2 for neutral entities, and 3 for negative entities. That’s it. Since the labeled corpus is in the automotive domain, I crawled some automotive-related text myself and wrote a GRU-based language model to train character embeddings. I felt that the Word2Vec approach for character vectors was too coarse and might not perform well on small corpora.
And then? There is no “then.” The rest is basically a repetition of the Seq2Seq core entity recognition post; even the code is the same. Of course, at the end, they provided a list of entities in the automotive domain, so I used this list for forced alignment in the later Viterbi algorithm. The final transfer learning didn’t provide a significant boost, so you can decide whether to use it.
The part of the process I am most satisfied with is that it is end-to-end. Once the corpus is provided, there is almost no manual intervention. If you change the domain of the corpus, it can be run through quickly just the same.
Results
My accuracy in the preliminary was 0.56, and currently, in the semi-final, it is 0.55. It’s not great; the best score on the leaderboard is 0.67. I don’t know what methods they are using; I hope some experts can provide guidance. Regardless, I don’t plan to continue with it.
Code
#! -*- coding:utf-8 -*-
import numpy as np
import pandas as pd
from tqdm import tqdm
import re
import time
import os
print 'read data ...'
train_data = pd.read_csv('Train.csv', index_col='SentenceId', delimiter='\t', encoding='utf-8')
test_data = pd.read_csv('Test.csv', index_col='SentenceId', delimiter='\t', encoding='utf-8')
train_label = pd.read_csv('Label.csv', index_col='SentenceId', delimiter='\t', encoding='utf-8')
addition_data = pd.read_csv('addition_data.csv', header=None, encoding='utf-8')[0]
train_data.dropna(inplace=True) # drop some empty sentences
neg_data = pd.read_excel('neg.xls', header=None)[0]
pos_data = pd.read_excel('pos.xls', header=None)[0]
script_name = 'shibie.py'
now = int(time.time())
os.system('mkdir %s'%now)
os.system('cp %s %s'%(script_name, now))
os.system('cp addition_data.csv %s'%now)
# some parameters
min_count = 5
maxlen = 100
word_size = 64
print 'making mapping dictionary ...'
word2id = ''.join(train_data['Content']) + ''.join(test_data['Content']) + ''.join(addition_data)
word2id = pd.Series(list(word2id)).value_counts()
word2id = word2id[word2id >= min_count]
word2id[:] = range(1, len(word2id)+1)
print 'keep %s words.'%len(word2id)
def doc2id(s):
return list(word2id[list(s)].fillna(len(word2id)+1).astype(np.int32))
print 'translating texts into id sequences ...'
train_data['doc2id'] = map(lambda i: doc2id(train_data.loc[i, 'Content']), tqdm(iter(train_data.index)))
test_data['doc2id'] = map(lambda i: doc2id(test_data.loc[i, 'Content']), tqdm(iter(test_data.index)))
addition_data[:] = map(lambda i: doc2id(addition_data[i]), tqdm(iter(addition_data.index)))
pos_data[:] = map(lambda i: doc2id(pos_data[i]), tqdm(iter(pos_data.index)))
neg_data[:] = map(lambda i: doc2id(neg_data[i]), tqdm(iter(neg_data.index)))
# make n-grams for train language model
n = 8
def gen_ngrams(s):
s = [0]*(n-1) + s + [0]*(n-1)
return zip(*[s[i:] for i in range(n)])
print 'generating ngrams ...'
from itertools import chain
ngrams = pd.concat([train_data['doc2id'].apply(gen_ngrams),
test_data['doc2id'].apply(gen_ngrams),
addition_data.apply(gen_ngrams),
pos_data.apply(gen_ngrams),
neg_data.apply(gen_ngrams)])
ngrams = np.array(list(chain(*ngrams)))
def findall(sub_string, string):
start = 0
idxs = []
while True:
idx = string[start:].find(sub_string)
if idx == -1:
return idxs
else:
idxs.append(start + idx)
start += idx + len(sub_string)
tags = {'pos':1, 'neu':2, 'neg':3}
def label2tag(i):
s = train_data.loc[i]['Content']
r = np.array([0]*len(s))
try:
l = train_label.loc[[i]].as_matrix()
except:
return r
for i in l:
for j in findall(i[0], s):
r[j:j+len(i[0])] = tags[i[1]]
return r
print 'translating target into tags ...'
train_data['label'] = map(label2tag, tqdm(iter(train_data.index)))
print 'keep %s train sample.'%len(train_data)
from keras.layers import Input, Embedding, GRU, Dense, TimeDistributed, Bidirectional
from keras.models import Model
from keras.utils import np_utils
RNN = GRU # which type of RNN we used, try LSTM or GRU
# in order to gain good word embedding, we use GRU to train a n-grams language model
# it costs more time, but it produces better word embedding.
print 'training language model ...'
lm_input = Input(shape=(n-1,), dtype='int32')
lm_embedded = Embedding(len(word2id)+2,
word_size,
input_length=n-1,
mask_zero=True)(lm_input)
lm_rnn = RNN(64)(lm_embedded)
lm_output = Dense(len(word2id)+2, activation='softmax')(lm_rnn)
language_model = Model(input=lm_input, output=lm_output)
language_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
def lm_generator(ngrams, batch_size):
while True:
np.random.shuffle(ngrams)
for p in np.split(ngrams, range(batch_size, len(ngrams), batch_size)):
yield p[:, :-1], np_utils.to_categorical(p[:, -1], len(word2id)+2)
nb_epoch = 8 # accuracy changes slightly after 5 epoch
batch_size = 4096
lm_history = language_model.fit_generator(lm_generator(ngrams, batch_size), nb_epoch=nb_epoch, samples_per_epoch=len(ngrams))
language_model.save_weights('%s/language_model_weights.model'%now)
structure = open('%s/language_model_structure.model'%now, 'w')
structure.write(language_model.to_json())
structure.close()
# here we use 2 layers of bidirectional GRU to make a sequence tagging model
print 'training ner model ...'
ner_input = Input(shape=(maxlen,), dtype='int32')
ner_embedded = Embedding(len(word2id)+2,
word_size,
input_length=maxlen,
mask_zero=True,
trainable=False,
weights=[language_model.get_weights()[0]])(ner_input)
ner_brnn = Bidirectional(RNN(64, return_sequences=True), merge_mode='sum')(ner_embedded)
ner_brnn = Bidirectional(RNN(32, return_sequences=True), merge_mode='sum')(ner_brnn)
ner_output = TimeDistributed(Dense(5, activation='softmax'))(ner_brnn)
ner_model = Model(input=ner_input, output=ner_output)
ner_model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
ner_data = train_data['doc2id'].apply(lambda s: s[:maxlen] + [0]*(maxlen - len(s[:maxlen])))
ner_data = np.array(list(ner_data))
ner_target = train_data['label'].apply(list).apply(lambda s: s[:maxlen] + [4]*(maxlen - len(s[:maxlen])))
ner_target = np.array(list(ner_target))
ner_target = np.array(map(lambda y:np_utils.to_categorical(y,5), ner_target))
sample_weight = (3/(train_data['label'].apply(lambda s:(np.array(s)==2).sum())+3)).as_matrix()
nb_epoch = 300
batch_size = 1024
ner_history_1 = ner_model.fit(ner_data, ner_target, batch_size=batch_size, nb_epoch=nb_epoch, sample_weight=sample_weight)
ner_model.save_weights('%s/ner_model_weights_1.model'%now)
structure = open('%s/ner_model_structure_1.model'%now, 'w')
structure.write(ner_model.to_json())
structure.close()
test_ner_data = test_data['doc2id'].apply(lambda s: s[:maxlen] + [0]*(maxlen - len(s[:maxlen])))
test_ner_data = np.array(list(test_ner_data))
print 'predicting ...'
train_data['predict'] = list(ner_model.predict(ner_data, batch_size=batch_size, verbose=1))
test_data['predict'] = list(ner_model.predict(test_ner_data, batch_size=batch_size, verbose=1))
def viterbi(nodes):
paths = nodes[0]
for l in range(1,len(nodes)):
paths_ = paths.copy()
paths = {}
for i in nodes[l].keys():
nows = {}
for j in paths_.keys():
if j[-1]+i in zy.keys():
nows[j+i]= paths_[j]+nodes[l][i]+zy[j[-1]+i]
k = np.argmax(nows.values())
paths[nows.keys()[k]] = nows.values()[k]
return paths.keys()[np.argmax(paths.values())]
zy = {'00':1,
'01':1,
'02':1,
'03':1,
'10':1,
'11':1,
'20':1,
'22':1,
'30':1,
'33':1}
zy = {i:np.log(zy[i]) for i in zy.keys()}
from acora import AcoraBuilder
views = pd.read_csv('View.csv', delimiter='\t', encoding='utf-8')['View']
views = AcoraBuilder(*views)
views = views.build()
def predict(i, data):
y_pred = data.loc[i, 'predict']
s = data.loc[i, 'Content'][:maxlen]
nodes = [dict(zip(['0','1','2','3'], k)) for k in np.log(y_pred[:len(s)])]
tags_pred_1 = viterbi(nodes)
for j in views.finditer(s):
for k in range(j[1], j[1]+len(j[0])):
nodes[k]['1'] += 100
nodes[k]['2'] += 100
nodes[k]['3'] += 100
try:
nodes[j[1]-1]['0'] += 50
nodes[k+1]['0'] += 50
except:
pass
tags_pred_2 = viterbi(nodes)
r = []
for j in re.finditer('1+|2+|3+', tags_pred_2):
t = pd.Series(list(tags_pred_1[j.start():j.end()])).value_counts()
t = t[t.index != '0']
if len(t) == 0:
continue
else:
if t.index[0] == '1':
r.append((i, s[j.start():j.end()], 'pos'))
elif t.index[0] == '2':
r.append((i, s[j.start():j.end()], 'neu'))
else:
r.append((i, s[j.start():j.end()], 'neg'))
return r
print 'creating the final export ...'
train_data['pred'] = map(lambda i: predict(i, train_data), tqdm(iter(train_data.index)))
test_data['pred'] = map(lambda i: predict(i, test_data), tqdm(iter(test_data.index)))
result_1 = pd.DataFrame(list(chain(*test_data['pred'])), columns=['SentenceId', 'View', 'Opinion'])
result_1 = result_1.drop_duplicates()
result_1.to_csv('%s/result_1.csv'%now, index=None, encoding='utf-8')
# transfer learning
# we use the train result to train ner model again
result_1['SentenceId'] = result_1['SentenceId'].apply(int)
result = result_1.set_index('SentenceId')
def label2tag(i):
s = test_data.loc[i]['Content']
r = np.array([0]*len(s))
try:
l = result.loc[[i]].as_matrix()
except:
return r
for i in l:
for j in findall(i[0], s):
r[j:j+len(i[0])] = tags[i[1]]
return r
test_data['label'] = map(label2tag, tqdm(iter(test_data.index)))
ner_data = train_data['doc2id'].append(test_data['doc2id']).apply(lambda s: s[:maxlen] + [0]*(maxlen - len(s[:maxlen])))
ner_data = np.array(list(ner_data))
ner_target = train_data['label'].append(test_data['label']).apply(list).apply(lambda s: s[:maxlen] + [0]*(maxlen - len(s[:maxlen])))
ner_target = np.array(list(ner_target))
ner_target = np.array(map(lambda y:np_utils.to_categorical(y, 5), ner_target))
nb_epoch = 100
batch_size = 1024
ner_history_2 = ner_model.fit(ner_data, ner_target, batch_size=batch_size, nb_epoch=nb_epoch)
ner_model.save_weights('%s/ner_model_weights_2.model'%now)
structure = open('%s/ner_model_structure_2.model'%now, 'w')
structure.write(ner_model.to_json())
structure.close()
print 'predicting again ...'
test_data['predict'] = list(ner_model.predict(test_ner_data, batch_size=batch_size, verbose=1))
print 'creating the final export again ...'
test_data['pred'] = map(lambda i: predict(i, test_data), tqdm(iter(test_data.index)))
result_2 = pd.DataFrame(list(chain(*test_data['pred'])), columns=['SentenceId', 'View', 'Opinion'])
result_2 = result_2.drop_duplicates()
result_2.to_csv('%s/result_2.csv'%now, index=None, encoding='utf-8')Download the package: Aspect-Based Domain Sentiment Analysis_Package.7z
Original Address: https://kexue.fm/archives/4118
For more details on reprinting, please refer to: Scientific Space FAQ