Nearly three years ago, in Baidu’s "2019 Language and Intelligence Technology Competition" (hereafter LIC2019), I proposed a new relation extraction model (refer to "A Lightweight Information Extraction Model Based on DGCNN and Probabilistic Graphs"), which was later published and named "CasRel." At that time, it was considered SOTA (State-of-the-Art) for relation extraction. However, when CasRel was proposed, I was actually new to the field, so in hindsight, CasRel still has many imperfections. I had thought about improving it further, but I hadn’t come up with a particularly good design.
Later, I proposed GlobalPointer and the more recent Efficient GlobalPointer. I felt I finally had enough "materials" to build a new relation extraction model. Starting from the idea of probabilistic graphical models and referencing some SOTA designs that followed CasRel, I eventually arrived at a model similar to TPLinker.
Basic Idea
At first glance, relation extraction is the extraction of triplets (s, p, o) (i.e., subject, predicate, object). However, in terms of concrete implementation, it is actually the extraction of "quintuplets" (s_h, s_t, p, o_h, o_t), where s_h, s_t are the head and tail positions of s, and o_h, o_t are the head and tail positions of o.
From the perspective of a probabilistic graphical model, we can construct the model as follows:
1. Design a scoring function for quintuplets S(s_h, s_t, p, o_h, o_t);
2. During training, ensure that for annotated quintuplets S(s_h, s_t, p, o_h, o_t) > 0, while for all other quintuplets S(s_h, s_t, p, o_h, o_t) < 0;
3. During prediction, enumerate all possible quintuplets and output those where S(s_h, s_t, p, o_h, o_t) > 0.
However, directly enumerating all quintuplets is computationally prohibitive. Assuming the sentence length is l and the total number of predicates is n, even with the constraints s_h \leq s_t and o_h \leq o_t, the total number of quintuplets is: n \times \frac{l(l+1)}{2} \times \frac{l(l+1)}{2} = \frac{1}{4}nl^2(l+1)^2 This is a computational complexity of the fourth power of the length (O(l^4)), which is difficult to implement in practice. Therefore, some simplification is necessary.
Simplification and Decomposition
Given our current computing power, we can generally only accept computational complexity at the level of the square of the length (O(l^2)). Thus, we can at most identify "one pair" of heads or tails at a time. To this end, we can use the following decomposition: S(s_h, s_t, p, o_h, o_t) = S(s_h, s_t) + S(o_h, o_t) + S(s_h, o_h | p) + S(s_t, o_t | p) \label{eq:factor} It should be noted that this equation is a model assumption based on our understanding of the task and hardware constraints, rather than a theoretical derivation. Each term has an intuitive meaning: S(s_h, s_t) and S(o_h, o_t) are the head-tail scores for the subject and object, respectively. By requiring S(s_h, s_t) > 0 and S(o_h, o_t) > 0, we extract all subjects and objects. The latter two terms represent predicate matching. S(s_h, o_h | p) uses the head features of the subject and object as their respective representations for matching. If we can ensure there are no nested entities within subjects and objects, then theoretically S(s_h, o_h | p) > 0 would be sufficient to extract all predicates. However, considering the possibility of nested entities, we also perform matching on the tails of the entities, which is the S(s_t, o_t | p) term.
At this point, the training and prediction processes become:
1. During training, ensure that for annotated quintuplets S(s_h, s_t) > 0, S(o_h, o_t) > 0, S(s_h, o_h | p) > 0, and S(s_t, o_t | p) > 0. For others, the scores should be < 0.
2. During prediction, enumerate all possible pairs, outputting parts where S(s_h, s_t) > 0, S(o_h, o_t) > 0, S(s_h, o_h | p) > 0, and S(s_t, o_t | p) > 0, and then take their intersection as the final output (i.e., satisfying all four conditions simultaneously).
In implementation, since S(s_h, s_t) and S(o_h, o_t) are used to identify the entities corresponding to subjects and objects, this is equivalent to an NER task with two entity types. Thus, we can use a single GlobalPointer to complete this. As for S(s_h, o_h | p), it is used to identify (s_h, o_h) pairs for a given predicate p. Unlike NER, it does not require the s_h \leq o_h constraint. We also use GlobalPointer for this, but to identify cases where s_h > o_h, we remove the default lower triangular mask of GlobalPointer. Finally, S(s_t, o_t | p) is handled identically to S(s_h, o_h | p).
To recap: as an NER module, GlobalPointer can unifiedly identify nested and non-nested entities because it is based on token-pair identification. Therefore, we should further understand GlobalPointer as a token-pair identification model rather than limiting it to the scope of NER. Once this is recognized, it becomes clear that S(s_h, s_t), S(o_h, o_t), S(s_h, o_h | p), and S(s_t, o_t | p) can all be implemented with GlobalPointer. Whether to add a lower triangular mask can be set according to the specific task context.
Loss Function
Now that we have designed the scoring functions, we need a loss function to train the model. Here, we continue to use the multi-label cross-entropy proposed in "Generalizing ’Softmax + Cross Entropy’ to Multi-label Classification", which is the default for GlobalPointer. Its general form is: \log \left(1 + \sum\limits_{i\in \mathcal{P}} e^{-S_i}\right) + \log \left(1 + \sum\limits_{i\in \mathcal{N}} e^{S_i}\right) \label{eq:loss-1} where \mathcal{P} and \mathcal{N} are the sets of positive and negative categories, respectively. In previous articles, we used "multi-hot" vectors to mark positive and negative categories. If the total number of categories is K, we use a K-dimensional vector where positive positions are 1 and negative positions are 0. However, in the context of S(s_h, o_h | p) and S(s_t, o_t | p), we each need an n \times l \times l matrix for labeling. Adding them together and accounting for the batch size b, the total dimension is 2bnl^2. For example, if b=64, n=50, l=128, then 2bnl^2 \approx 100 \text{ million}. This means that if we insist on using the "multi-hot" form for labels, we would have to create a matrix with 100 million parameters at each training step and transfer it to the GPU, which is costly in terms of both creation and transmission.
Therefore, to improve training speed, we need to implement a "sparse version" of multi-label cross-entropy. That is, we only transmit the indices corresponding to the positive classes. Since positive classes are far fewer than negative classes, the size of the label matrix is greatly reduced. A "sparse version" of multi-label cross-entropy means we need to implement Equation [eq:loss-1] knowing only \mathcal{P} and \mathcal{A} = \mathcal{P} \cup \mathcal{N}. To do this, we use the following implementation: \begin{aligned} & \log \left(1 + \sum\limits_{i\in \mathcal{N}} e^{S_i}\right) = \log \left(1 + \sum\limits_{i\in \mathcal{A}} e^{S_i} - \sum\limits_{i\in \mathcal{P}} e^{S_i}\right) \\ =& \log \left(1 + \sum\limits_{i\in \mathcal{A}} e^{S_i}\right) + \log \left(1 - \left(\sum\limits_{i\in \mathcal{P}} e^{S_i}\right) \Bigg/ \left(1 + \sum\limits_{i\in \mathcal{A}} e^{S_i}\right)\right) \end{aligned} If we let a = \log \left(1 + \sum\limits_{i\in \mathcal{A}} e^{S_i}\right) and b = \log \left(\sum\limits_{i\in \mathcal{P}} e^{S_i}\right), it can be written as: \log \left(1 + \sum\limits_{i\in \mathcal{N}} e^{S_i}\right) = a + \log\left(1 - e^{b - a}\right) In this way, the loss for the negative classes is calculated using \mathcal{P} and \mathcal{A}, while the loss for the positive classes remains unchanged.
Finally, in general multi-label classification tasks, the number of
positive classes is variable. In such cases, we can start the class
indices from 1 and use 0 as a padding label to ensure the label matrix
size is consistent for each sample, then mask the 0 class in the loss
implementation. The corresponding implementation is already built into
bert4keras; for details, refer to sparse_multilabel_categorical_crossentropy.
Experimental Results
For convenience, we will temporarily call the above model GPLinker
(GlobalPointer-based Linking). A reference implementation based on
bert4keras is as follows:
Script Link: task_relation_extraction_gplinker.py
The experimental results on LIC2019 are as follows (the code for CasRel is task_relation_extraction.py):
| Model | F1 |
|---|---|
| CasRel | 0.8220 |
| GPLinker (Standard) | 0.8272 |
| GPLinker (Efficient) | 0.8268 |
The pre-trained model used is BERT-base. The difference between Standard and Efficient is the use of the Standard GlobalPointer and Efficient GlobalPointer, respectively. These results demonstrate two things: first, GPLinker is indeed more effective than CasRel; second, the design of Efficient GlobalPointer can match the performance of the standard version with significantly fewer parameters. In the LIC2019 task, using the Standard GlobalPointer results in nearly 10 million parameters for GPLinker, whereas using Efficient GlobalPointer reduces this to about 300,000.
Furthermore, on a 3090 GPU, compared to the "multi-hot" version of multi-label cross-entropy, the model using the sparse version can increase training speed by 1.5 times without losing accuracy. Compared with CasRel, GPLinker using sparse multi-label cross-entropy is only 15% slower in training but nearly twice as fast in decoding, making it both fast and effective.
Summary
This article shared a joint entity and relation extraction model based on GlobalPointer—"GPLinker"—and provided a "top-down" derivation for reference.
Please include the original address when reprinting: https://kexue.fm/archives/8888
For more detailed reprinting matters, please refer to: "Scientific Space FAQ"