Many readers are likely familiar with Machine Reading Comprehension (MRC) tasks. Simply put, it involves finding the answer to a given question from a provided passage, following the process: Passage + Question \to Answer. I have previously written several articles on reading comprehension, such as "DGCNN: A CNN-based Reading Comprehension Question Answering Model". As for Question-Answer (QA) pair construction, it is essentially the inverse task of reading comprehension, following the process: Passage \to Answer + Question. In academia, this is generally referred to as "Question Generation" (QG). Since answers can often be selected through relatively simple rules or random sampling in most cases, many papers focus only on the step: Passage + Answer \to Question.
This article presents a fully end-to-end practice of Passage \to Answer + Question, including a model introduction and implementation code based on bert4keras. Readers are welcome to try it out.
Preview of Results
Input Passage: K2 (Qogir), the second highest mountain in the world, is located on the border of China.
Generated QA:
What is the name of the second highest mountain in the world? K2 (Qogir)
K2 is the world’s how-many-th highest peak? Second
Input Passage: On July 28, Thailand will celebrate the 68th birthday of King Maha Vajiralongkorn, Rama X.
Generated QA:
Who is the Rama X of Thailand? King Maha Vajiralongkorn
How old is King Maha Vajiralongkorn of Thailand? years old
What is the date of King Maha Vajiralongkorn’s birthday? July 28
Which country does King Maha Vajiralongkorn belong to? Thailand
Input Passage: The Water Splashing Festival, also known as the Songkran Festival, has a history of 700 years and is the grandest traditional festival for the Dai (China) and De’ang ethnic groups.
Generated QA:
What is another name for the Water Splashing Festival? Songkran Festival
For which ethnic group in our country is the Water Splashing Festival the grandest festival of the year? Dai
How many years of history does the Water Splashing Festival have? years
Methodology Analysis
The goal of this article is to implement "Passage \to Answer + Question" in a completely end-to-end manner, where even the selection of the answer is completed automatically by the model without manual rules. In fact, the concept is quite simple: use the "BERT + UniLM" approach to build a Seq2Seq model (using UniLM’s Attention Mask combined with BERT’s pre-trained weights). If readers are not yet familiar with UniLM, you are encouraged to read "From Language Models to Seq2Seq: Transformer is All About Masking".
In a previous article, "Universal Seq2Seq: Reading Comprehension QA Based on Seq2Seq", I provided an implementation for reading comprehension using a Seq2Seq model, which directly constructs p(\text{Answer} | \text{Passage}, \text{Question}), as illustrated below:
In fact, by slightly modifying the above model to include the question in the generation target, we can achieve QA pair generation. The model then becomes p(\text{Question}, \text{Answer} | \text{Passage}), as shown here:
However, intuition suggests that "Passage \to Answer" and "Passage + Answer \to Question" should be less difficult than "Passage + Question \to Answer." Therefore, if we swap the generation order of the question and the answer to p(\text{Answer}, \text{Question} | \text{Passage}), the final results will be better:
Implementation Details
The model introduction ends here; there isn’t much more to say other than determining the inputs and outputs and applying "BERT + UniLM." Below is my reference implementation:
What is worth discussing here is the decoding logic. A typical Seq2Seq model stops decoding after reaching one [SEP] token. However, the model in this article needs to decode until two [SEP] tokens are reached. The content up to the first [SEP] is the answer, and the content between the two [SEP] tokens is the question. Theoretically, many QA pairs can be constructed from a given passage. In other words, the target is not unique. Therefore, we cannot use deterministic decoding algorithms like Beam Search; instead, we should use random decoding algorithms (for related concepts, refer to the "Decoding Algorithms" section in "How to Deal with the ’Never-Ending’ Problem in Seq2Seq?").
The problem is that if a purely random decoding algorithm is used, the generated questions might become too "unconstrained"—meaning content unrelated to the passage might appear. For example, if the passage is "China’s Mars probe Tianwen-1 was successfully launched," the generated question might be "What was China’s first artificial satellite?" While related, it is too divergent. Therefore, I suggest a compromise strategy: use random decoding to generate the answer, and then use deterministic decoding to generate the question. This helps ensure the reliability of the question. Of course, if you care more about the diversity of the generated questions, you can use random decoding for everything; it’s up to your own tuning.
Readers should also note that the reference script above does not impose constraints on the answer, so the generated answer might not be a literal span from the passage. Since this is just a reference implementation, it is still some distance from being production-ready. Interested readers are encouraged to understand and modify the code according to their own needs. Furthermore, since QA pair construction has been completely transformed into a Seq2Seq problem, any techniques used to improve Seq2Seq performance can be applied to improve the quality of QA pair generation, such as the previously discussed "Analysis and Countermeasures for Exposure Bias in Seq2Seq". These are left for the readers to explore.
Summary
This article presents an end-to-end practice for QA pair generation, primarily based on the "BERT + UniLM" Seq2Seq model to directly generate answers and questions from passages, and discusses decoding strategies. Overall, there is nothing particularly unique about the model itself, but by leveraging BERT’s pre-trained weights, the quality of the final generated QA pairs is quite noteworthy.
When reposting, please include the original article address: https://kexue.fm/archives/7630
For more detailed reposting matters, please refer to: "Scientific Space FAQ"