[Corpus] Baidu's Chinese Question Answering Dataset WebQA

Information Extraction

As is well known, there are a large number of questions asked and answered on Baidu Zhidao. However, respondents on Baidu Zhidao often seem to be quite lazy; they frequently prefer to directly copy and paste large sections of text from the internet as their answers. These contents may or may not be relevant to the question. For example:

https://zhidao.baidu.com/question/557785746.html

Question: How high is the altitude of Guangzhou Baiyun Mountain?

Answer: Guangzhou Baiyun Mountain is the first of the new “Eight Sights of Guangzhou,” a national 4A-level scenic spot, and a national key scenic spot. It is located in the northeast of Guangzhou and is one of the famous mountains in Southern Guangdong, known since ancient times as the “First Beauty of Guangzhou.” The mountain body is quite broad, consisting of more than 30 peaks, and is a branch of the Jiulian Mountains, the highest peak in Guangdong. It covers an area of 20.98 square kilometers. The main peak, Moxing Ridge, is 382 meters high (Note: the latest measured height is 372.6 meters — National Administration of Surveying, Mapping and Geoinformation, 2008). The peaks overlap, and streams crisscross. Climbing high, one can overlook the entire city and see the Pearl River in the distance. Whenever the sky clears after rain or in late spring, white clouds linger among the mountains, creating a magnificent sight, hence the name Baiyun (White Cloud) Mountain.

In fact, for this question, only the sentence “The main peak, Moxing Ridge, is 382 meters high” is meaningful. If we were to be even more concise, only “382 meters” is meaningful; the rest is essentially filler. Extracting correct and concise answers from large volumes of relevant text for a given question is a difficult task for both humans and machines. This requires not only good algorithms but also high-quality datasets for training.

WebQA

To this end, Baidu utilized Baidu Zhidao and other resources to construct such a dataset, called WebQA, which is currently at version 1.0:

http://idl.baidu.com/WebQA.html

The paper published by Baidu using this dataset:
Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. 2016. Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering. arXiv:1607.06275.

Thanks to Baidu! Open-source datasets are precious!

Data Overview

This section is reproduced from http://idl.baidu.com/WebQA.html.

The released file is 267MB, but for our purposes, it seems to contain a bit too much, including word segmentation results, sequence labeling results, and word vector results. It appears to be the data used directly by the internal research group for their experiments. For us, we clearly only need the pure Q&A corpus. Therefore, I have streamlined it, retaining only the most basic corpus content:

Clean Version

Link: https://pan.baidu.com/s/1pLXEYtd Password: 6fbf

File List:
WebQA.v1.0/readme.txt
WebQA.v1.0/me_test.ann.json (One question paired with one piece of evidence; evidence contains the answer)
WebQA.v1.0/me_test.ir.json (One question paired with multiple pieces of evidence; evidence may or may not contain the answer)
WebQA.v1.0/me_train.json (Mixed training corpus)
WebQA.v1.0/me_validation.ann.json (One question paired with one piece of evidence; evidence contains the answer)
WebQA.v1.0/me_validation.ir.json (One question paired with multiple pieces of evidence; evidence may or may not contain the answer)

The difference between test and validation is that, theoretically, the distribution of validation is closer to the distribution of train. Generally, validation is used to verify model accuracy, while test is used to verify the model’s generalization ability. The difference between ann and ir is that, because ir provides multiple pieces of evidence for each question, a more reliable answer can be obtained through voting across different segments; whereas ann is in a one-question-one-evidence format, which is a true test of reading comprehension ability.

The organized data format is as follows, using me_train.json as an example:

1. If read using Python’s json library, you get a dictionary me_train, where the keys are question identifiers like Q_TRN_010878.

2. A single record is obtained via me_train[’Q_TRN_010878’], which is also a dictionary with two keys: question and evidences.

3. me_train[’Q_TRN_010878’][’question’] retrieves the text of the question, such as “Who played the father of Huo Xiaolin in ’Brave Heart’?”

4. evidences contains the evidence materials and corresponding answers for the question, also as a dictionary, with keys like Q_TRN_010878#06.

5. me_train[’Q_TRN_010878’][’evidences’][’Q_TRN_010878#05’] retrieves a single record, which is a dictionary with two keys: evidence and answer.

6. evidence is the corresponding material, e.g., “Ans: ’Brave Heart’ Huo Shaochang and Madam Hua’s son Yang Zhigang plays Huo Xiaolin... Kou Zhenhai plays Huo Shaochang... Huo Xiaolin’s father...” answer is a list of answers (as there may be multiple), such as [u’Kou Zhenhai’]. If the evidence does not contain an answer, the answer is [u’no_answer’].

These correspond one-to-one with the original dataset.