Baidu’s "2020 Language and Intelligence Challenge" has officially started. This year, there are five tracks: Machine Reading Comprehension, Recommended Dialogue, Semantic Parsing, Relation Extraction, and Event Extraction. For each track, the organizers have provided baseline models based on PaddlePaddle. Here, I am providing my personal baselines for three of these tracks based on bert4keras. From these, we can see how convenient, fast, and concise it is to build baseline models using bert4keras.
Brief Analysis of Ideas
Here is a brief analysis of the task characteristics of these three tracks and the corresponding baseline designs.
Reading Comprehension
Sample example:
{
"context": "Hello friend, pregnancy reactions in women generally start from around 6-12 weeks, which means reactions begin about a month after a woman becomes pregnant. By the third month, pregnancy reactions basically end. Most women experience nausea and vomiting in the early stages of pregnancy; these symptoms vary from person to person. Unless the nausea and vomiting are very severe, medical attention is not required; otherwise, these are normal symptoms of early pregnancy. Between 1-3 months, you can observe your skin; generally, early pregnancy may cause skin pigmentation or stretch marks on the abdominal wall, which become more obvious in the later stages of pregnancy. Many women also experience fatigue and sleepiness in early pregnancy. At three months, the bladder is compressed by the growing uterus, reducing its capacity, so frequent urination also occurs during pregnancy. The cessation of menstruation is also the most common symptom of early pregnancy. For women with regular periods, if the period is more than two weeks late after sexual activity, pregnancy is possible. If you want to judge whether you are pregnant, you can see if you have these reactions. Of course, these are just the manifestations for most people; some women's experiences are not exactly like this. If you cannot determine if you are pregnant, it is best to go to the hospital for an examination.",
"qas": [
{
"question": "How long into pregnancy do reactions occur",
"id": "f2843cffb845aad0100062841222023e",
"answers": [
{
"text": "Around 6-12 weeks",
"answer_start": -1
},
{
"text": "6-12 weeks",
"answer_start": -1
},
{
"text": "More than 1 month",
"answer_start": -1
}
]
}
]
}There isn’t much to say about this baseline. After passing through BERT, it connects to two fully connected layers + Softmax to predict the start and end of the answer respectively. Some training samples are labeled with multiple answers, but only one answer needs to be predicted during inference. Therefore, during the training phase, one answer is randomly selected for training each time.
Relation Extraction
Sample example:
{
"text": "The 40-episode TV series 'The Proud Twins' (Jue Dai Shuang Jiao), directed by the famous director Wong Jing in 2004 and starring Nicholas Tse as Hua Wuque, was released as a revised version titled 'The Little Fish and the Proud Twins'. The editor is not targeting the director or actors, but this TV series can indeed be described as a 'thunderous' magical adaptation. Fortunately, all the leading actors managed to hold the audience with their looks and acting skills, even though the plot was a mess.",
"spo_list": [
{
"predicate": "Director",
"object": {
"@value": "Wong Jing"
},
"subject": "The Little Fish and the Proud Twins"
},
{
"predicate": "Starring",
"object": {
"@value": "Nicholas Tse"
},
"subject": "The Little Fish and the Proud Twins"
},
{
"predicate": "Protagonist",
"object": {
"@value": "Hua Wuque"
},
"subject": "The Proud Twins"
},
{
"predicate": "Plays",
"object": {
"inWork": "The Little Fish and the Proud Twins",
"@value": "Hua Wuque"
},
"subject": "Nicholas Tse"
}
]
}Relation extraction is essentially the triplet extraction task from last year, but with some upgrades this year. The upgrade lies in considering the polysemy of the same predicate. For example, "Plays" might refer to which TV series someone acted in, or it might refer to which character they played in a TV series. If a single sentence contains multiple different objects being played, all of them must be extracted to be considered correct. Although it’s called an upgrade, there is no fundamental change. We only need to concatenate the predicate with its corresponding object prefix to treat them as different predicates. This reduces the problem to a conventional triplet extraction task. For instance, "Plays_@value" and "Plays_inWork" are treated as two different predicates to be extracted separately. My baseline model is still based on last year’s "semi-pointer semi-annotation" design. For details, please refer to "A Lightweight Information Extraction Model Based on DGCNN and Probabilistic Graphs".
Event Extraction
Sample example:
{
"text": "Nestle lays off 4,000 people: When the times abandon you, they won't even say goodbye!",
"id": "409389c96efe78d6af1c86e0450fd2d7",
"event_list": [
{
"event_type": "Organizational Relationship - Layoff",
"trigger": "Layoff",
"trigger_start_index": 2,
"arguments": [
{
"argument_start_index": 0,
"role": "Layoff Party",
"argument": "Nestle",
"alias": []
},
{
"argument_start_index": 4,
"role": "Number of Layoffs",
"argument": "4,000 people",
"alias": []
}
],
"class": "Organizational Relationship"
}
]
}Event extraction is a relatively new task that requires extracting event types and the elements describing the event. The same sentence may contain multiple events, and the same entity can simultaneously describe multiple events (for example, "Month X, Day Y" might be the occurrence time for multiple events). Event extraction itself is a complex task, but the organizers of this competition only evaluate triplets consisting of (event_type, role, argument). That is, if such a triplet matches, 1 point is awarded. Since event_type and role are discrete categories and the argument is an entity from the original text, this evaluation metric reduces the task to a standard entity labeling problem. Therefore, it can be solved using conventional sequence labeling models. Both my baseline and the official baseline are provided by converting the task into a sequence labeling task.
Matching the Original Sequence
The three competitions mentioned above are essentially extraction problems, meaning the output entities are fragments of the original text. However, after the original text passes through the BERT tokenizer, it might not perfectly match the original text due to small "additions," "deletions," or "modifications" (such as converting to lowercase, changes in the number of spaces, or character transliterations). These small changes are insignificant for engineering evaluations but are very important for competitions or academic evaluations because even if different characters look the same, they will result in a matching error. For example, readers can try running the following code in Python:
u'a\u0300' == u'\u00e0'To map the tokenized results back to the original sequence, I spent
some time adding a rematch method to the
Tokenizer in bert4keras. By passing the original text and
the tokenized results, it returns the mapping relationship from tokens
to the original text. With this mapping, you can slice directly from the
original text. For the specific implementation, please refer directly to
the baseline code.
Summary
I wrote three baselines and "watered" another blog post
Reprinting notice: Please include the original address of this article: https://kexue.fm/archives/7321
For more detailed reprinting matters, please refer to: "Scientific Space FAQ"