Saliency Maps for RNN
Recurrent Neural Networks (RNNs) are the go-to method for many sequence tasks. For instance, a common approach for text classification is the combination of "word vectors + LSTM + fully connected classifier," as shown in the figure below:
If such a model works well, we might consider the following task: how can we measure the importance (Saliency) of the inputs w_1, \dots, w_n to the final classification result? For example, in a sentiment classification task, how do we identify which words have a significant impact on the final classification? This article presents a direct approach to this problem.
The principle is simple. Since we pass the state vector of the last step of the RNN (represented by the green shaded vector in the diagram) to the subsequent classifier, the final state vector \boldsymbol{h}_n serves as the target vector. The RNN is a recursive process where \boldsymbol{h}_0, \boldsymbol{h}_1, \dots, \boldsymbol{h}_{n-1} (where \boldsymbol{h}_0 is usually initialized to all zeros) gradually approaches \boldsymbol{h}_n.
Therefore, we can sequentially consider the distance from each intermediate vector to the target vector: \Vert\boldsymbol{h}_n-\boldsymbol{h}_0\Vert, \Vert\boldsymbol{h}_n-\boldsymbol{h}_1\Vert, \dots, \Vert\boldsymbol{h}_n-\boldsymbol{h}_{n-1}\Vert, 0
The transition from \boldsymbol{h}_i to \boldsymbol{h}_{i+1} occurs because the word w_{i+1} is taken into account. The consequence is: the original distance \Vert\boldsymbol{h}_n-\boldsymbol{h}_i\Vert between the state and the target vector now becomes \Vert\boldsymbol{h}_n-\boldsymbol{h}_{i+1}\Vert. Thus, we can use the difference: \Vert\boldsymbol{h}_n-\boldsymbol{h}_{i}\Vert - \Vert\boldsymbol{h}_n-\boldsymbol{h}_{i+1}\Vert to measure the impact of word w_{i+1} on the final classification. This value can be positive, meaning the introduction of w_{i+1} reduced the distance to the target, thereby promoting the correct classification. Conversely, it can be negative, representing a counterproductive effect on classification. A larger absolute value indicates a greater degree of influence. We can use this metric to sort words in descending order to determine their relative importance. Of course, one can also divide by the norm of the target vector to eliminate the influence of scale: \frac{\Vert\boldsymbol{h}_n-\boldsymbol{h}_{i}\Vert}{\Vert\boldsymbol{h}_n\Vert} - \frac{\Vert\boldsymbol{h}_n-\boldsymbol{h}_{i+1}\Vert}{\Vert\boldsymbol{h}_n\Vert}
A Simple Experiment
Beyond theoretical reasoning, experimental evidence is necessary. Here, we use text sentiment classification as an example. The following code is modified from the article "Text Sentiment Classification (III): To Segment or Not to Segment".
#! -*- coding:utf-8 -*-
# Environment: tensorflow 1.2 + Keras 2.0.6
import numpy as np
import pandas as pd
import jieba
pos = pd.read_excel('pos.xls', header=None)
neg = pd.read_excel('neg.xls', header=None)
pos['words'] = pos[0].apply(jieba.lcut)
neg['words'] = neg[0].apply(jieba.lcut)
words = {}
for l in pos['words'].append(neg['words']): # Statistics to get vocabulary
for w in l:
if w in words:
words[w] += 1
else:
words[w] = 1
min_count = 10 # Discard words with frequency lower than min_count
maxlen = 100 # Truncate sentences to 100 words
words = {i:j for i,j in words.items() if j >= min_count}
id2word = {i+1:j for i,j in enumerate(words)} # ID to word mapping
word2id = {j:i for i,j in id2word.items()} # Word to ID mapping
def doc2num(s):
s = [word2id.get(i,0) for i in s[:maxlen]]
return s + [0]*(maxlen-len(s))
pos['id'] = pos['words'].apply(doc2num)
neg['id'] = neg['words'].apply(doc2num)
x = np.vstack([np.array(list(pos['id'])), np.array(list(neg['id']))])
y = np.array([[1]]*len(pos)+[[0]]*len(neg))
# Manually shuffle data
idx = list(range(len(x)))
np.random.shuffle(idx)
x = x[idx]
y = y[idx]
from keras.models import Model
from keras.layers import Input, Dense, Dropout, Embedding, Lambda
from keras.layers import LSTM
from keras import backend as K
# Build model
input = Input(shape=(None,))
input_vecs = Embedding(len(words)+1, 128, mask_zero=True)(input) # mask_zero=True pads with 0
lstm = LSTM(128, return_sequences=True, return_state=True)(input_vecs) # returns a list
lstm_state = Lambda(lambda x: x[1])(lstm) # second element is the final state
dropout = Dropout(0.5)(lstm_state)
predict = Dense(1, activation='sigmoid')(dropout)
# First element is the sequence of state vectors; prepend a zero vector (h_0)
lstm_sequence = Lambda(lambda x: K.concatenate([K.zeros_like(x[0])[:,:1], x[0]], 1))(lstm)
lstm_dist = Lambda(lambda x: K.sqrt(K.sum((x[0]-K.expand_dims(x[1], 1))**2, 2)/K.sum(x[1]**2,1,keepdims=True)))([lstm_sequence, lstm_state])
model = Model(inputs=input, outputs=predict) # Sentiment classification model
model.compile(loss='binary_crossentropy',
optimizer='adam',
metrics=['accuracy'])
model_dist = Model(inputs=input, outputs=lstm_dist) # Model to calculate weights
model_dist.compile(loss='mse',
optimizer='adam')
batch_size = 128
train_num = 15000
model.fit(x[:train_num], y[:train_num], batch_size = batch_size, epochs=5, validation_data=(x[train_num:],y[train_num:]))
def saliency(s): # Function to output sorted saliency
ws = jieba.lcut(s)[:maxlen]
x_ = np.array([[word2id.get(w,0) for w in ws]])
score = np.diff(model_dist.predict(x_)[0])
idxs = score.argsort()
return [(i,ws[i],-score[i]) for i in idxs] # Output: (position, word, weight)Some Results
After 5 epochs, the model’s validation accuracy is around 90%. We then test our saliency function:
Input: "Fast delivery, timely logistics, water heater well packaged, already called a technician to install it, heats up very quickly, very good water heater."
Saliency Results:
[(1, ’very fast’, 0.2719), (23, ’quite fast’, 0.2428), (5, ’timely’, 0.2125), (17, ’good’, 0.2000), (10, ’good’, 0.1809), (6, ’,’, 0.0910), (27, ’good’, 0.0894), (29, ’water heater’, 0.0759), (9, ’very’, 0.0739), ...]Input: "Evaluated after use, works well, same as the one I bought in the mall, should be genuine, boils water quickly. Five stars."
Saliency Results:
[(13, ’mall’, 0.1848), (20, ’genuine’, 0.1837), (7, ’good’, 0.1058), (6, ’quite’, 0.0911), (27, ’.’, 0.0780), (17, ’,’, 0.0686), (21, ’,’, 0.0609), (8, ’use’, 0.0519), ...]Input: "Newly purchased delicious stuff... the one on the left is Bright yogurt, shelf life is very thoughtful at 150 days... the box is also lovely. Taste is super great... of course the price is a bit expensive..."
Saliency Results:
[(9, ’Bright’, 0.2362), (2, ’delicious’, 0.2097), (0, ’new’, 0.0986), (19, ’..’, 0.0820), (24, ’.’, 0.0516), (10, ’yogurt’, 0.0512), (26, ’super’, 0.0500), (33, ’expensive’, 0.0457), ...]Input: "Installation cost me over 500, Midea is really black-hearted, truly frustrating, the after-sales service is terrible! Bad review!!!!!"
Saliency Results:
[(10, ’black-hearted’, 0.4279), (22, ’terrible’, 0.2783), (3, ’cost’, 0.1957), (23, ’!’, 0.1150), (25, ’!’, 0.0996), (26, ’!’, 0.0864), (27, ’!’, 0.0750), (28, ’!’, 0.0650), ...]Input: "The author’s writing is average, the viewpoints are similar to other books on the market, I do not recommend readers to buy it."
Saliency Results:
[(12, ’book’, 0.2148), (3, ’average’, 0.2100), (2, ’writing’, 0.1983), (15, ’not’, 0.1388), (5, ’viewpoint’, 0.1055), (19, ’.’, 0.1045), (17, ’readers’, 0.1005), ...]Input: "Overall a bit messy feeling. Repeat and repeat."
Saliency Results:
[(2, ’messy’, 0.4870), (1, ’a bit’, 0.2530), (5, ’.’, 0.2274), (9, ’.’, 0.1819), (4, ’feeling’, 0.0914), (6, ’repeat’, 0.0422), (3, ’of’, 0.0355), (8, ’repeat’, 0.0352), ...]Input: "Too ridiculous, what era is this, can’t even download ringtones! I’ve searched everywhere and it’s not supported! Sometimes it suddenly crashes, really regret it!"
Saliency Results:
[(19, ’not’, 0.2435), (1, ’ridiculous’, 0.2287), (11, ’download’, 0.2016), (0, ’too’, 0.1423), (27, ’crash’, 0.1286), (10, ’cannot’, 0.1119), (5, ’what’, 0.0956), ...]
The effectiveness is evident; the words ranked at the top are generally those with strong emotional tendencies. It is worth noting that this evaluation scheme also automatically considers the impact of word position. If a sentiment word appears repeatedly in a sentence, subsequent occurrences generally receive lower weights (because the preceding ones already allowed us to complete the classification). For example:
Input: "Very terrible, nothing is more terrible than this, really too terrible."
Saliency Results:
[(1, ’terrible’, 0.3586), (3, ’nothing’, 0.2013), (7, ’terrible’, 0.2004), (11, ’really too’, 0.1885), (13, ’it’, 0.1544), (6, ’more’, 0.0874), (4, ’than’, 0.0867), (12, ’terrible’, -0.0510), ...]Input: "Doing happy things in a happy place, having a happy mood."
Saliency Results:
[(1, ’happy’, 0.2748), (6, ’happy’, 0.2063), (10, ’having’, 0.1117), (11, ’happy’, 0.0926), (12, ’of’, 0.0653), (9, ’,’, 0.0623), (5, ’doing’, 0.0594), ...]
Self-Evaluation
I feel this is a simple and clear solution for evaluating input importance in RNN models. It does not require extensive mathematical knowledge, and I welcome readers to try it on more tasks.
Original Address: https://kexue.fm/archives/4582