Today, I added a new example to bert4keras: Reading Comprehension Question Answering (task_reading_comprehension_by_seq2seq.py), the corpora used are the same as before, namely WebQA and SogouQA. The final score is around 0.77 (single model, without fine-tuning).
Method Overview
Since the main purpose this time is to add a demo to bert4keras, efficiency was not the primary concern. The goal was mainly universality and ease of use, so I adopted the most universal solution—seq2seq—to implement reading comprehension.
When using seq2seq, one basically doesn’t need to worry about model design; you just need to concatenate the passage and the question, and then predict the answer. Furthermore, the seq2seq approach naturally includes a method for judging whether a passage contains an answer and naturally leads to a multi-passage voting strategy. In short, if efficiency is not considered, seq2seq is a quite elegant solution for reading comprehension.
This implementation of seq2seq still uses the UNILM approach. Readers who are not yet familiar with it can first read "From Language Models to Seq2Seq: Transformer is All About Masking" to understand the relevant content.
Model Details
Building a seq2seq model using the UNILM scheme in bert4keras is essentially a one-line task. Therefore, the main work for this example lies not in building the model, but in the processing of inputs and outputs.
Input Format
First is the input. The input format is very simple and can be clearly expressed with a single diagram:
Output Processing
If answering based on a single passage and a single question, one can simply follow the conventional seq2seq processing scheme—namely, beam search—for decoding.
However, WebQA and SogouQA are designed for search scenarios, where multiple passages exist simultaneously to answer the same question. This involves the choice of a voting scheme. A naive idea is to decode each passage combined with the question separately using beam search, assign a confidence score, and then perform voting according to the method in "CNN-based Reading Comprehension QA Model: DGCNN". The difficulty with this approach lies in assigning a reasonable confidence score to each answer; compared to the idea we present later, it appears less natural and is also slightly less efficient.
Here, we present a solution that is more "compatible" with beam search:
First, exclude passages that do not contain an answer, and then, when decoding each character of the answer, directly average the probability values predicted by all passages (in a certain way).
Specifically, all passages are concatenated with the question
separately, and then each provides a probability distribution for the
first character. Those passages that give [SEP] as the first character
imply they have no answer and are excluded. After exclusion, the
probability distributions of the first character from the remaining
passages are averaged, and the top-k are retained (the standard beam
search process). When predicting the second character, each passage is
combined with the top-k candidates to predict their respective
probability distributions for the second character. These are then
averaged across passages before selecting the top-k. This continues
until [SEP] appears.
(Essentially, this is standard beam search with the addition of
averaging across passages; if this is still unclear, you will have to
check the source code!)
Additionally, there should be two ways to generate answers: one is extractive, where the answer must be a segment of the passage; the other is generative, where there is no need to consider whether the answer is a segment of the passage, and the answer is decoded directly. Both modes have corresponding logic in the decoding process of this article.
Experimental Code
Code link: task_reading_comprehension_by_seq2seq.py
Finally, using the evaluation script provided with SogouQA, the score on the validation set is approximately 0.77 (Accuracy=, F1=, Final=). The performance of this single model far exceeds the previous "Open Source DGCNN Reading Comprehension QA Model (Keras Version)". Of course, the improvement comes at a cost—the prediction speed is significantly reduced, only being able to predict about 2 data items per second.
(The model has not been finely tuned; there is likely still room for improvement. Currently, the focus is on the demo.)
Summary
This article mainly provides an example of reading comprehension based on BERT and the seq2seq approach, and introduces a multi-passage voting beam search strategy for readers to reference and test.
If reprinting, please include the
original address of this article:
https://kexue.fm/archives/7115
For more detailed reprinting matters,
please refer to:
"Scientific Space FAQ"