``Non-Autoregressive'' is Not Bad: Reading Comprehension QA Based on MLM

Autoregressive Generation

As the name suggests, autoregressive generation refers to a decoding stage where generation happens word-by-word recursively. It models the following probability distribution: p(y_1,y_2,\dots,y_n|x)=p(y_1|x)p(y_2|x,y_1)\dots p(y_n|x,y_1,\dots,y_{n-1}) \label{eq:at} For a more detailed introduction, you can refer to “Playing with Keras: Automatic Title Generation with seq2seq” and “From Language Models to Seq2Seq: Transformer is All About Masking”. I will not go into too much detail about autoregressive generation here.

Non-Autoregressive Generation

Since autoregressive generation requires recursive decoding, it cannot be parallelized, resulting in slower decoding speeds. Therefore, in recent years, much work has been done on non-autoregressive generation, yielding many results. Simply put, non-autoregressive generation aims to make the decoding of each word parallelizable. The simplest non-autoregressive model directly assumes that each word is independent: p(y_1,y_2,\dots,y_n|x)=p(y_1|x)p(y_2|x)\dots p(y_n|x) \label{eq:nat} This is a very strong assumption and is only applicable in specific circumstances. If applied directly to general text generation like automatic summarization, the results would be very poor. For more complex work related to non-autoregressive generation, you can find many papers by searching for “non-autoregressive text generation” on Arxiv or Google.

As mentioned in the title, the method for reading comprehension in this article is “based on MLM.” Readers familiar with the BERT model should know that MLM (Masked Language Model) is actually a special case of Equation [eq:nat]. Therefore, generation models based on MLM fall into the category of non-autoregressive generation.

Model Illustration

First, define a maximum length l_{\max}. Then, concatenate the question and the passage, and insert l_{\max} [MASK] tokens into the sequence. Input this into BERT, and finally, let the parts corresponding to [MASK] predict the answer (this applies to both the training and prediction stages), as shown in the figure below:

Code and Performance

Code link: task_reading_comprehension_by_mlm.py

Ultimately, using the evaluation script provided by SogouQA, the score on the validation set is approximately 0.77 (Accuracy=0.7282149325820084, F1=0.8207266829447049, Final=0.7744708077633566). This is on par with the previous “Universal seq2seq” model, but the prediction speed is significantly improved. The previous seq2seq solution could only predict about 2 items per second, while the current prediction speed reaches 12 items per second—a 6x speedup without any loss in performance.

Training and Prediction

First, the biggest problem with seq2seq is its slowness, which becomes even more pronounced for long text generation. Therefore, if high efficiency is required, one naturally has to abandon the seq2seq approach.

If efficiency is not a concern, is seq2seq always the best? Not necessarily. Although Equation [eq:at] is more accurate from a modeling perspective, seq2seq training is done via teacher forcing, which leads to the “exposure bias” problem: During training, the input at each time step comes from the ground-truth answer text; however, during generation, the input at each time step comes from the output of the previous time step. Thus, if one word is generated poorly, the error may propagate, making the subsequent generation increasingly worse.

In short, there is an inconsistency between training and prediction, and this inconsistency can lead to error accumulation. In contrast, the MLM-based approach behaves consistently during training and prediction because it does not require ground-truth labels as input (the answer positions are also input as [MASK] during prediction). Therefore, there is no error accumulation. Furthermore, because of this characteristic, decoding no longer requires recursion and can be parallelized, improving decoding speed.

Uniqueness of the Correct Answer

Additionally, non-autoregressive generation like MLM is relatively more suitable for short text generation because shorter texts are closer to the independence assumption. At the same time, non-autoregressive generation is suitable for scenarios where “there is only one correct answer.” The reading comprehension task performed in this article is mainly extractive, which corresponds exactly to this scenario; hence, MLM performs quite well.

In fact, sequence labeling models such as frame-by-frame Softmax or CRF can also be regarded as non-autoregressive generation models. I believe the fundamental reason they are effective is that the “correct answer sequence is unique,” rather than the intuitive belief of “input-output alignment.” That is to say, if the condition “there is only one correct answer” is met, then non-autoregressive generation can be considered.

Note that “unique answer” here does not mean that each sample has only one human-annotated answer, but rather that the task is designed such that the answer becomes unique. For example, in word segmentation, once the labeling method is designed, each sentence corresponds to only one correct segmentation scheme. In contrast, for title generation, it is obvious that the same article can have different titles; therefore, the answer for title generation is not unique (even if there is only one title per article in the training data).