A few months ago, I participated in the Baidu Entity Linking Competition, which was one of the evaluation tasks of CCKS2019. The official name for it is “Entity Mention-to-Entity Linking.” The competition concluded completely a few weeks ago. My final F1 score was around 0.78 (the champion’s was 0.80), ranking 14th. The results were not particularly outstanding (the only unique feature was that the model was very lightweight and could easily run on a GTX 1060). Therefore, this article is purely a record of the process. Experts, please feel free to take it with a grain of salt.
Task Introduction
Entity linking primarily refers to predicting the knowledge base ID corresponding to a specific entity in an input query, given an existing knowledge base. In other words, the knowledge base records many entities. There may be multiple interpretations for an entity with the same name, each identified by a unique ID. Our task is to predict which interpretation (ID) the entity in the query actually corresponds to. This is a necessary step for Question Answering systems based on Knowledge Graphs.
Data Format
Entity linking is prepared for Knowledge Graph-based QA, so first, we
need a knowledge base (kb_data). An example is as
follows:
{"alias": ["Victory"], "subject_id": "10001", "subject": "Victory", "type": ["Thing"], "data": [{"predicate": "Abstract", "object": "The League of Legends Victory series skins are one of the commemorative limited series skins produced by Riot Games..."}, {"predicate": "Producer", "object": "Riot Games"}, {"predicate": "Foreign Name", "object": "Victorious"}, {"predicate": "Source", "object": "League of Legends"}, {"predicate": "Chinese Name", "object": "Victory"}, {"predicate": "Attribute", "object": "Virtual"}, {"predicate": "Sense Description", "object": "Game 'League of Legends' Victory series limited skins"}]}
{"alias": ["Song of Zhang San"], "subject_id": "10002", "subject": "Song of Zhang San", "type": ["CreativeWork"], "data": [{"predicate": "Abstract", "object": "The classic old song 'Song of Zhang San', written and composed by Zhang Zishi..."}, {"predicate": "Original Singer", "object": "Li Shouquan"}, {"predicate": "Composer", "object": "Zhang Zishi"}, {"predicate": "Duration", "object": "3 min 58 sec"}, {"predicate": "Language", "object": "Mandarin"}, {"predicate": "Music Style", "object": "Folk"}, {"predicate": "Record Company", "object": "UFO Records"}, {"predicate": "Cover", "object": "Qi Qin, Su Rui, Southern Duet, etc."}, {"predicate": "Lyricist", "object": "Zhang Zishi"}, {"predicate": "Release Time", "object": "1986-08-01"}, {"predicate": "Chinese Name", "object": "Song of Zhang San"}, {"predicate": "Album", "object": "8 and a Half"}, {"predicate": "Sense Description", "object": "Song performed by Li Shouquan"}, {"predicate": "Tag", "object": "Single"}, {"predicate": "Tag", "object": "Musical Work"}]}
...The knowledge base includes many entities. Each entity’s information includes a unique subject_id, aliases, and entity-related predicates and objects. Simply put, this is a Knowledge Graph. A characteristic of the knowledge base is that “entities” do not necessarily refer to specific proper nouns; they also include common nouns, verbs, adjectives, etc., such as “Victory” or “Beautiful.” Furthermore, there are many homonymous entities (which is why the entity linking task exists). For example, in the knowledge base provided for this competition, there are 15 entities named “Victory,” as shown below:
{"alias": ["Victory"], "subject_id": "10001", "subject": "Victory", "type": ["Thing"], "data": [{"predicate": "Abstract", "object": "The League of Legends Victory series skins..."}, ... , {"predicate": "Sense Description", "object": "Game 'League of Legends' Victory series limited skins"}]}
{"alias": ["Victory"], "subject_id": "19044", "type": ["Vocabulary"], "data": [{"predicate": "Abstract", "object": "Victory, a Chinese vocabulary. Pinyin: sheng li. Victory refers to achieving the intended purpose..."}, ... , {"predicate": "Sense Description", "object": "Chinese word"}], "subject": "Victory"}
{"alias": ["Victory"], "subject_id": "37234", "type": ["Thing"], "data": [{"predicate": "Abstract", "object": "'Victory' is an allegorical novel written by Joseph Conrad..."}, ... , {"predicate": "Sense Description", "object": "Novel written by Joseph Conrad"}], "subject": "Victory"}
...In addition to the knowledge base, we have a set of labeled samples in the following format:
{"text_id": "1", "text": "Nanjing South Railway Station: Take the high-speed rail and get off at Nanjing South Railway Station. Nanjing South Railway Station", "mention_data": [{"kb_id": "311223", "mention": "Nanjing South Railway Station", "offset": "0"}, {"kb_id": "341096", "mention": "High-speed rail", "offset": "6"}, {"kb_id": "311223", "mention": "Nanjing South Railway Station", "offset": "9"}, {"kb_id": "311223", "mention": "Nanjing South Railway Station", "offset": "15"}]}
{"text_id": "2", "text": "Bitcoin attracts countless fans, but the Central Bank's heart belongs elsewhere | Jiemian News - jmedia", "mention_data": [{"kb_id": "278410", "mention": "Bitcoin", "offset": "0"}, {"kb_id": "199602", "mention": "Central Bank", "offset": "9"}, {"kb_id": "215472", "mention": "Jiemian News", "offset": "18"}]}
{"text_id": "3", "text": "Interpreting '1587, a Year of No Significance'", "mention_data": [{"kb_id": "131751", "mention": "1587, a Year of No Significance", "offset": "3"}]}
{"text_id": "4", "text": "'The Time in Between Season 1' Xunlei Download_Full Version Online Watch_US TV Series...", "mention_data": [{"kb_id": "NIL", "mention": "The Time in Between Season 1", "offset": "1"}, {"kb_id": "57067", "mention": "Xunlei", "offset": "10"}, {"kb_id": "394479", "mention": "US TV Series", "offset": "23"}]}
...This training data labels the entities (mentions) in the query text, their positions (offsets), and the corresponding entity IDs (kb_id) in the knowledge base. Each query text may have multiple identified entities. Since only the query text is provided during prediction, it is necessary to perform both entity recognition and entity linking.
Basic Approach
As mentioned above, in this Baidu competition, we not only need to find the knowledge base ID corresponding to the entity but also need to identify the entity first. In other words, we need to perform entity recognition followed by entity linking. In this section, we analyze these two tasks based on the competition data to derive the strategy for solving the problem.
Entity recognition technology is clearly mature. The standard solution is BiLSTM+CRF, and recently, fine-tuning with BERT+CRF has become popular. In my model, the entity recognition model uses an “LSTM + semi-pointer semi-annotation structure” plus some manual features. This was done both for speed considerations and to align with the characteristics of the labeled data.
For the entity linking step, we see that each entity in the knowledge base corresponds to multiple “predicate-object” pairs. Processing these pairs separately would be quite tedious, so I simply concatenated all “predicate-object” pairs into a single string to serve as the complete description of the entity. For example, here is the concatenated description for a certain entity named “Victory”:
Abstract: The League of Legends Victory series skins are one of the commemorative limited series skins produced by Riot Games...
Producer: Riot Games
Foreign Name: Victorious
Source: League of Legends
Chinese Name: Victory
Attribute: Virtual
Sense Description: Game ’League of Legends’ Victory series limited skins
Name: Victory
In this way, each entity corresponds to a (usually long) text description. To perform entity linking, we essentially need to match the query text and the entity span with this entity description. Overall, it is quite similar to a text matching problem. Therefore, my approach was to encode the query text and the entity description text separately, mark the position of a specific entity in the query text, fuse the query and description encodings via Attention, and finally treat it as a binary classification problem.
The advantage of this is low training cost. The disadvantage is that each iteration processes only one entity from the query text and traverses only a single entity from the knowledge base at a time. Overall, the sampling efficiency is low, and training time is relatively long. During prediction, all homonymous entities are traversed, binary classification is performed for each against the query text and entity span, and the one with the highest probability is output.
Model Details
Here, I will introduce my processing and modeling process step by step. In my implementation, entity labeling and entity linking are trained jointly and share some modules. The overall philosophy of the model (including the training scheme) is similar to the article “A Lightweight Information Extraction Model Based on DGCNN and Probabilistic Graphs”. Readers can refer to that for comparison.
Entity Recognition
First is the entity recognition part, introduced in two parts: the “basic model” and “manual features.”
Basic Model
The “basic model” refers to the neural network part, which performs character labeling through mixed word-character embeddings and manual features. The labeling structure is still the “semi-pointer semi-annotation structure” I previously conceived (refer to here and here).
Unlike my usual habit of using full CNNs, this model primarily uses BiLSTM because I did not intend to make the model very deep. When the model is very shallow (only one or two layers), BiLSTM is often better than CNNs or Attention.
Manual Features
Since the knowledge base is provided and the identified entity names
all appear in the knowledge base (the alias field; if they
don’t appear, it can be considered a labeling error), a baseline for
entity recognition is to directly take all aliases from the
knowledge base to form a vocabulary and then build a maximum matching
model based on this vocabulary. Doing this yields an entity
recall of about 92%, but the precision is low, only about 30%, with a
total F1 of about 40%.
Next, we can observe that in the training data, the labeling of entities is quite “capricious.” Overall, subjective factors are quite significant. It can be said that this is not semantic-based entity recognition at all, but rather “modeling the trajectory of the annotators.” In other words, we are primarily trying to model the annotators’ habits rather than performing entity recognition based on semantic understanding.
For example, in the query “’Dark Cop’ Xunlei Download / Online Watch - Crime / History” in the diagram, only “Dark Cop,” “Xunlei,” and “History” were labeled. In fact, “Download,” “Online,” “Watch,” and “Crime” are all entities in the knowledge base, and there are entity IDs that could fit this query. Why weren’t they labeled? It can only be said that the annotators didn’t like to label them, didn’t want to, or ran out of energy. Another case is “HD Video”: in some queries, “HD Video” is labeled as a single entity, while in others, it is labeled as two entities, “HD” and “Video” (because “HD Video,” “HD,” and “Video” are all entities in the knowledge base).
Therefore, many entity recognition results have no logical explanation; they are just the habits of the annotators. To better fit these habits, we can perform statistics on the training set to see which entities in the knowledge base are labeled frequently and which are labeled rarely. This allows us to perform a basic filtering of the knowledge base entities (refer to the open-source code for filtering details). Using the filtered entity set as a dictionary for maximum matching, the final entity recall is around 91.8%, but the precision can reach 60%, and the F1 can reach 70%+.
In other words, through simple statistics and maximum matching, the F1 of the entity recognition step can reach 70%+. We convert the results of this maximum matching into 0/1 features and pass them into the basic model, which acts as a further filter on the maximum matching results. The final F1 for entity recognition was approximately 81% (I don’t remember the exact value).
Entity Linking
Now for the entity linking step, also introduced in two parts: the “basic model” and “manual features.” It is still based on a quite simple baseline model combined with manual features for improvement.
Basic Model
In the entity linking model, our task is to “determine whether a certain entity in the query matches a homonymous entity in the knowledge base.” To do this, we need random sampling: randomly sample one entity from all identified entities in the query, and then sample one from all homonymous entities in the knowledge base.
For the query, we continue to use the encoding sequence of the query text from the entity recognition step. Then, we mark the sampled entity with a 0/1 sequence and concatenate it to this encoding sequence, along with some manual features. After concatenation, another BiLSTM is applied to obtain the final encoding sequence \boldsymbol{Q}. For the homonymous entity, we take its description, pass it through an Embedding layer (shared with the query’s Embedding), and then a BiLSTM to complete the encoding \boldsymbol{D} of the entity description.
With the encoding sequences for both, we can apply Attention. As mentioned in “A Quick Read of ’Attention is All You Need’ (Introduction + Code)”, the three elements of Attention are query, key, and value. Here, we first use \boldsymbol{Q} as the query and \boldsymbol{D} as the key and value for one Attention operation, and then use \boldsymbol{D} as the query and \boldsymbol{Q} as the key and value for another. The two Attention results are processed via MaxPooling to obtain fixed-length vectors, which are then concatenated and passed through a fully connected layer for binary classification.
Manual Features
The manual features used in entity linking are query-centric, ultimately generating a vector sequence of the same length as the query, which is concatenated to the query’s encoding result. In this model, three manual features are used:
Whether each character in the query appears in the entity description (a 0/1 sequence).
Both the query and the entity description are segmented into words, and then it is checked whether each word in the query appears in the entity description (a 0/1 sequence; the label for each word is repeated for the length of the word to ensure a sequence of the same length).
Whether each word/segment in the query is a certain
objectof the entity (a 0/1 sequence; refer to the knowledge base example at the beginning for the meaning ofobject).
I recall that these three features significantly improved the accuracy of entity linking (though I forget the exact magnitude).
Additionally, during prediction, a statistical result was used. As emphasized earlier, the entity recognition step is more like “modeling labeling behavior” than “semantic understanding,” and it contains a lot of randomness. In fact, the same applies to the entity linking step. We can observe through statistics that for some entities, there may be many homonymous entities in the knowledge base, but only a few are ever labeled. The annotators might not have even looked at the others. In other words, what might have been a 1-out-of-50 problem becomes a 1-out-of-5 problem in the annotators’ habits.
Thus, we can perform statistics on all entities and their corresponding entity IDs appearing in the training set. For each entity name, we get a distribution describing which IDs are labeled for that name. If the distribution is highly concentrated on a few IDs, we simply keep those IDs and discard the remaining homonymous entities. It turned out that this improved both prediction speed and accuracy.
Code Sharing
Github Address: https://github.com/bojone/el-2019
The code testing environment is Python 2.7 + Keras 2.2.4 + Tensorflow 1.8. The entire model only requires a single GTX 1060 6G version to run, which is very easy. Other parts are similar to the experimental training of the previous information extraction model; in fact, the code was adapted from that, so there isn’t much difference.
Conclusion
This article shared the process of participating in an entity linking competition. From a model perspective, the overall structure is quite simple; the main performance improvement came from the proposed manual features. These manual features are primarily statistical features derived from observing data characteristics, which are competition tricks. In fact, these tricks might be considered opportunistic and may not be applicable in a real production environment, but they are closely tied to the characteristics of the competition data.
Again, I tend to believe that this was more of a labeling behavior modeling task than a true semantic understanding competition, especially in the entity recognition step, where subjectivity was too strong. It was one of the main bottlenecks of the competition. To improve the final score, we had to put a lot of effort into entity recognition. However, doing well in entity recognition only means you have better fitted the annotators’ behavior, not that you have achieved the best semantic entity recognition, and it doesn’t help much with the original intent of the competition—entity linking.
Furthermore, in a real production environment, there are usually standard methods for entity recognition, so one typically only needs to focus on entity linking. This is what I consider a slight oversight in the design of this competition.
Of course, labeled data is hard to come by. I still want to thank the folks at Baidu for hosting the competition and providing the labeled data!
When reposting, please include the original address of this article: https://kexue.fm/archives/6919
For more detailed reposting matters, please refer to: "Scientific Space FAQ"