A Brief Analysis and Countermeasures for Exposure Bias in Seq2Seq

Softmax

First, let’s review the content related to Softmax. As everyone knows, for a vector (x_1, x_2, \dots, x_n), its Softmax is: (p_1, p_2, \dots, p_n) = \frac{1}{\sum_{i=1}^n e^{x_i}} \left(e^{x_1}, e^{x_2}, \dots, e^{x_n}\right) Since e^t is a strictly monotonically increasing function of t, if x_k is the maximum among x_1, x_2, \dots, x_n, then p_k is also the maximum among p_1, p_2, \dots, p_n.

For classification problems, the loss we use is generally cross-entropy, which is: -\log p_t = \log\left(\sum_{i=1}^n e^{x_i}\right) - x_t where t is the target class. As described in the article "Seeking a Smooth Maximum Function", the first term in the above equation is actually a smooth approximation of \max(x_1, x_2, \dots, x_n). Therefore, to understand cross-entropy intuitively, we can write: -\log p_t \approx \max(x_1, x_2, \dots, x_n) - x_t In other words, cross-entropy is actually narrowing the gap between the target class score x_t and the global maximum. Obviously, this gap can only be 0 at minimum, and at this point, the target class score is the maximum. Therefore, the effect of Softmax plus cross-entropy is "hoping that the score of the target class becomes the maximum."

Teacher Forcing

Now, let’s look at Seq2Seq, which models the joint probability distribution through conditional decomposition: \begin{aligned} p(\boldsymbol{y}|\boldsymbol{x}) &= p(y_1, y_2, \dots, y_n|\boldsymbol{x}) \\ &= p(y_1|\boldsymbol{x}) p(y_2|\boldsymbol{x}, y_1) \dots p(y_n|\boldsymbol{x}, y_1, \dots, y_{n-1}) \end{aligned} Each term is naturally modeled using Softmax, i.e.: \begin{aligned} &p(y_1|\boldsymbol{x}) = \frac{e^{f(y_1;\boldsymbol{x})}}{\sum_{y_1} e^{f(y_1;\boldsymbol{x})}}, \\ &p(y_2|\boldsymbol{x}, y_1) = \frac{e^{f(y_1, y_2;\boldsymbol{x})}}{\sum_{y_2} e^{f(y_1, y_2;\boldsymbol{x})}}, \\ &\dots, \\ &p(y_n|\boldsymbol{x}, y_1, \dots, y_{n-1}) = \frac{e^{f(y_1, y_2, \dots, y_n;\boldsymbol{x})}}{\sum_{y_n} e^{f(y_1, y_2, \dots, y_n;\boldsymbol{x})}} \end{aligned} Multiplying them together gives: p(\boldsymbol{y}|\boldsymbol{x}) = \frac{e^{f(y_1;\boldsymbol{x}) + f(y_1, y_2;\boldsymbol{x}) + \dots + f(y_1, y_2, \dots, y_n;\boldsymbol{x})}}{\left(\sum_{y_1} e^{f(y_1;\boldsymbol{x})}\right) \left(\sum_{y_2} e^{f(y_1, y_2;\boldsymbol{x})}\right) \dots \left(\sum_{y_n} e^{f(y_1, y_2, \dots, y_n;\boldsymbol{x})}\right)} \label{eq:join-target} The training objective is: -\log p(\boldsymbol{y}|\boldsymbol{x}) = -\log p(y_1|\boldsymbol{x}) - \log p(y_2|\boldsymbol{x}, y_1) - \dots - \log p(y_n|\boldsymbol{x}, y_1, \dots, y_{n-1}) This direct training objective is called Teacher Forcing because when calculating -\log p(y_2|\boldsymbol{x}, y_1), we need to know the true y_1; when calculating -\log p(y_3|\boldsymbol{x}, y_1, y_2), we need to know the true y_1, y_2, and so on. It’s as if an experienced teacher has pre-paved most of the road for us, and we only need to find the next step. This method is simple to train and can achieve parallel training when combined with models like CNN or Transformer, but it may lead to the Exposure Bias problem.

Exposure Bias

In fact, the name Teacher Forcing itself implies that it will have an Exposure Bias problem. Think back to the process of a teacher teaching a student to solve a problem. The general steps are:

1. How should we think about the first step;

2. After the first step is figured out, what choices do we have for the second step;

3. After the second step is determined, what can we do for the third step;

4. ...

5. With these n-1 steps, the last step is not difficult to think of.

This process is actually the same as the assumption of the Teacher Forcing scheme in Seq2Seq. Readers with teaching experience know that usually, students nod frequently and feel they understand everything, but when they are asked to solve problems on their own after class, most are still completely confused. Why is this? One of the reasons is Exposure Bias. Simply put, the problem lies in the fact that the teacher always assumes the student can think of the previous steps and then teaches the next step. But what if one step is thought of incorrectly or cannot be thought of at all? At this point, the process cannot continue, and the correct answer cannot be obtained. This is the Exposure Bias problem.

Beam Search

In fact, when we actually solve problems, it’s not always like this. If we are stuck at a certain step and cannot be sure, we traverse several choices and then continue to deduce, using the subsequent results to help us determine the step we couldn’t decide on earlier. Corresponding to Seq2Seq, this is equivalent to the decoding process based on Beam Search.

For Beam Search, we should find that the beam size is not always the larger the better. In some cases, it is even best when the beam size is 1. This seems a bit unreasonable because, theoretically, the larger the beam size, the closer the found sequence should be to the optimal sequence, so it should be more likely to be correct. In fact, this is also one of the phenomena of Exposure Bias.

From equation [eq:join-target], we can see that the scoring function for the target sequence y_1, y_2, \dots, y_n in Seq2Seq is: f(y_1;\boldsymbol{x}) + f(y_1, y_2;\boldsymbol{x}) + \dots + f(y_1, y_2, \dots, y_n;\boldsymbol{x}) Normally, we hope that the target sequence has the highest score among all candidate sequences. According to the Softmax method introduced at the beginning of this article, the probability distribution we establish should be: p(\boldsymbol{y}|\boldsymbol{x}) = \frac{e^{f(y_1;\boldsymbol{x}) + f(y_1, y_2;\boldsymbol{x}) + \dots + f(y_1, y_2, \dots, y_n;\boldsymbol{x})}}{\sum_{y_1, y_2, \dots, y_n} e^{f(y_1;\boldsymbol{x}) + f(y_1, y_2;\boldsymbol{x}) + \dots + f(y_1, y_2, \dots, y_n;\boldsymbol{x})}} \label{eq:ideal-target} However, the denominator of the above equation requires summing over all paths, which is difficult to implement. Equation [eq:join-target] has been widely used as a compromise. But equation [eq:join-target] and equation [eq:ideal-target] are not equivalent, so even if the model is successfully optimized, the phenomenon of "the optimal sequence is not the target sequence" may occur.

Simple Example

Let’s look at a simple example. Suppose the sequence length is only 2, the candidate sequences are (a, b) and (c, d), and the target sequence is (a, b). After training, the probability distribution of the model is: \begin{array}{c|c} \hline p(a) & p(c)\\ \hline 0.6 & 0.4 \\ \hline \end{array} \qquad \begin{array}{c|c|c|c} \hline p(b|a) & p(d|a) & p(b|c) & p(d|c)\\ \hline 0.55 & 0.45 & 0.1 & 0.9\\ \hline \end{array}

If the beam size is 1, since p(a) > p(c), the first step can only output a. Then, since p(b|a) > p(d|a), the second step can only output b, successfully outputting the correct sequence (a, b). However, if the beam size is 2, the first step outputs (a, 0.6), (c, 0.4). In the second step, traversing all combinations, we get: \begin{array}{c|c|c|c} \hline (a, b) & (a, d) & (c, b) & (c, d)\\ \hline 0.33 & 0.27 & 0.04 & 0.36\\ \hline \end{array} Therefore, the incorrect sequence (c, d) is output.

Is it because the model was not trained well? Not necessarily. As mentioned before, the purpose of Softmax plus cross-entropy is to maximize the score of the target. For the first step, we have p(a) > p(c), so the training goal for the first step has been achieved. For the second step, given a is known, we have p(b|a) > p(d|a), which means the training goal for the second step has also been achieved. Therefore, the model is considered trained, but due to limitations in model capacity or other reasons, the scores are not particularly high, even though the goal of "making the target score the maximum" was completed.

Thinking about Countermeasures

From the above example, readers may see where the problem lies: it is mainly that p(d|c) is too high, and p(d|c) was not trained. There is no explicit mechanism to suppress p(d|c) from becoming large, hence the phenomenon "the optimal sequence is not the target sequence."

Seeing this, readers might think of a naive countermeasure: add an additional optimization objective to reduce the probability of those non-target sequences found by Beam Search. In fact, this is indeed an effective solution, with results published in the 2016 paper "Sequence-to-Sequence Learning as Beam-Search Optimization". However, this almost requires performing a Beam Search for every sample before each training step, which is computationally expensive. There are more recent results, such as the ACL 2019 Best Long Paper "Bridging the Gap between Training and Inference for Neural Machine Translation", which focuses on solving the Exposure Bias problem. Additionally, methods like directly optimizing BLEU through reinforcement learning can also alleviate Exposure Bias to some extent.

However, as far as I know, most of these methods dedicated to solving Exposure Bias involve drastic changes to the training process and may even sacrifice the parallel training capability of the original model (requiring recursive sampling of negative samples). The increase in cost is often much larger than the improvement in performance.

Constructing Negative Samples

Looking at most papers solving Exposure Bias, and combining our previous examples and experiences, it’s not hard to think that the main idea is to construct representative negative samples and then reduce their probability during training. So the question is how to construct "representative" negative samples. Here is a simple strategy I conceived, which experiments have shown can alleviate Exposure Bias to some extent and improve text generation performance. Importantly, this strategy is simple, almost plug-and-play, and hardly loses training performance.

The method is simple: randomly replace some input words of the Decoder (the Decoder’s input words have a special name, called oracle words), as shown below:

The purple [R] represents a randomly replaced word. In fact, many Exposure Bias papers follow this line of thought, but the random word selection schemes differ. My proposed scheme is simple:

1. 50% probability of making no change;

2. 50% probability of replacing 30% of the words in the input sequence, where the replacement targets are any words from the original target sequence.

That is, the probability of random replacement occurring is 50%, the replacement ratio is 30%, and the random sampling space is the set of words in the target sequence. The inspiration for this strategy is that although Seq2Seq may not perfectly generate the target sequence, it usually generates most of the target words (though the order might be wrong or some words might repeat). Therefore, the input sequence after such replacement can often serve as a representative negative sample. (Note: the 50% and 30% ratios were chosen arbitrarily without careful tuning.)

How is the effect? I conducted two experiments on title (summary) generation (the first two from CLGE), where the baseline is task_seq2seq_autotitle_csl.py. The code is open-sourced at:

Github Address: https://github.com/bojone/exposure_bias

The results are shown in the tables below:

It can be seen that in the CSL task, the random replacement strategy consistently improved all metrics. In the LCSTS task, the metrics were mixed, but considering LCSTS is inherently difficult and the baseline metrics were low, the CSL results are more convincing. This indicates that the proposed strategy is indeed a worthwhile approach. (Note: all experiments were repeated twice and averaged.)

Adversarial Training

Thinking further: since one of the ideas to solve Exposure Bias is to construct representative negative inputs—essentially making the model predict correctly even under perturbation—and we recently discussed a method for generating perturbed samples, namely adversarial training. If we directly add adversarial training to the baseline model, can we improve performance? For simplicity, I conducted an experiment adding gradient penalty (a form of adversarial training) to the baseline.

As seen, adversarial training (gradient penalty) further improved all metrics for CSL generation, while on LCSTS it mainly improved Rouge metrics with a slight decrease in BLEU. Therefore, adversarial training can also be included in the list of potential techniques to improve text generation models.

Summary

This article discussed the Exposure Bias phenomenon in Seq2Seq, attempted to analyze its causes intuitively and theoretically, and provided simple and feasible countermeasures. These include a random replacement strategy and an adversarial training strategy. The advantage of these strategies is that they are almost plug-and-play, and experiments show they can improve various metrics of text generation to some extent.

Reprinting please include the original address: https://kexue.fm/archives/7259

For more detailed reprinting matters, please refer to: "Scientific Space FAQ"