Efficient GlobalPointer: Fewer Parameters, Better Performance

A Large Number of Parameters

Let’s briefly review GlobalPointer (for a detailed introduction, please refer to "GlobalPointer: A Unified Way to Handle Nested and Non-Nested NER"). Simply put, GlobalPointer is a token-pair recognition module based on inner products. It can be used for NER because, for NER, we only need to identify token-pairs like "(start, end)" for each type of entity.

Suppose an input t of length n is encoded into a sequence of vectors [\boldsymbol{h}_1, \boldsymbol{h}_2, \dots, \boldsymbol{h}_n]. In the original GlobalPointer, through transformations \boldsymbol{q}_{i,\alpha} = \boldsymbol{W}_{q,\alpha}\boldsymbol{h}_i and \boldsymbol{k}_{i,\alpha} = \boldsymbol{W}_{k,\alpha}\boldsymbol{h}_i, we obtain sequences of vectors [\boldsymbol{q}_{1,\alpha}, \boldsymbol{q}_{2,\alpha}, \dots, \boldsymbol{q}_{n,\alpha}] and [\boldsymbol{k}_{1,\alpha}, \boldsymbol{k}_{2,\alpha}, \dots, \boldsymbol{k}_{n,\alpha}]. We then define: s_{\alpha}(i,j) = \boldsymbol{q}_{i,\alpha}^{\top}\boldsymbol{k}_{j,\alpha} as the score for a continuous segment from i to j being an entity of type \alpha. Here, we temporarily omit the bias term; if you feel it is necessary, you can add it yourself.

Consequently, for every type of entity, there are corresponding \boldsymbol{W}_{q,\alpha} and \boldsymbol{W}_{k,\alpha}. Let \boldsymbol{W}_{q,\alpha}, \boldsymbol{W}_{k,\alpha} \in \mathbb{R}^{d \times D}. For every new entity type added, we must add 2Dd parameters. In contrast, if using CRF + BIO tagging, each new entity type only requires adding 2D parameters (the transition matrix parameters are few and can be ignored). For BERT-base, common choices are D=768, d=64. It is evident that the parameter count of GlobalPointer is significantly larger than that of CRF.

Recognition and Classification

In fact, it is not hard to imagine that for any type \alpha, the scoring matrix s_{\alpha}(i,j) must share many similarities. For most token-pairs, they represent "non-entities," and the correct scores for these non-entities are all negative. This implies that we do not need to design independent s_{\alpha}(i,j) for every entity type; they should share more commonalities.

How can we highlight the commonalities of s_{\alpha}(i,j)? Taking NER as an example, we know that NER can actually be decomposed into two steps: "extraction" and "classification." "Extraction" involves extracting segments that are entities, and "classification" involves determining the type of each entity. Thus, the "extraction" step is equivalent to NER with only one entity type, which can be completed with a single scoring matrix, i.e., (\boldsymbol{W}_q\boldsymbol{h}_i)^{\top}(\boldsymbol{W}_k\boldsymbol{h}_j). For the "classification" step, we can use "feature concatenation + Dense layer," i.e., \boldsymbol{w}_{\alpha}^{\top}[\boldsymbol{h}_i; \boldsymbol{h}_j]. We can then combine these two terms as a new scoring function: s_{\alpha}(i,j) = (\boldsymbol{W}_q\boldsymbol{h}_i)^{\top}(\boldsymbol{W}_k\boldsymbol{h}_j) + \boldsymbol{w}_{\alpha}^{\top}[\boldsymbol{h}_i; \boldsymbol{h}_j] \label{eq:EGP-1} In this way, the parameters for the "extraction" part are shared across all entity types. Therefore, for each new entity type, we only need to add the corresponding \boldsymbol{w}_{\alpha} \in \mathbb{R}^{2D}, meaning the parameter increase per entity type is only 2D. Furthermore, we denote \boldsymbol{q}_i = \boldsymbol{W}_q\boldsymbol{h}_i, \boldsymbol{k}_i = \boldsymbol{W}_k\boldsymbol{h}_i. To further reduce the parameter count, we can use [\boldsymbol{q}_i; \boldsymbol{k}_i] to replace \boldsymbol{h}_i. At this point: s_{\alpha}(i,j) = \boldsymbol{q}_i^{\top}\boldsymbol{k}_j + \boldsymbol{w}_{\alpha}^{\top}[\boldsymbol{q}_i; \boldsymbol{k}_i; \boldsymbol{q}_j; \boldsymbol{k}_j] \label{eq:EGP} Here \boldsymbol{w}_{\alpha} \in \mathbb{R}^{4d}. Since usually d \ll D, the parameter count of Equation [eq:EGP] is often less than that of Equation [eq:EGP-1]. This is the final scoring function used by Efficient GlobalPointer.

Surprising Experiments

Efficient GlobalPointer is already built into bert4keras>=0.10.9. Readers only need to change one line of code to switch to Efficient GlobalPointer.

# from bert4keras.layers import GlobalPointer
from bert4keras.layers import EfficientGlobalPointer as GlobalPointer

Below we compare the results of GlobalPointer and Efficient GlobalPointer:

As can be seen, the experimental results for Efficient GlobalPointer are quite good. Except for a slight decrease in the People’s Daily task, the other two tasks achieved certain improvements. Overall, the magnitude of improvement is greater than the magnitude of decrease. Thus, Efficient GlobalPointer not only saves parameters but also improves performance. In terms of speed, Efficient GlobalPointer is almost identical to the original GlobalPointer.

Analysis and Commentary

Considering that People’s Daily NER has only 3 entity types, while CLUENER and CMeEE have 10 and 9 entity types respectively, and the scores show that People’s Daily is higher than the other two, this indicates that CLUENER and CMeEE are more difficult. On the other hand, since Efficient GlobalPointer achieved improvements on CLUENER and CMeEE, we can tentatively infer: the more entity categories and the more difficult the task, the more effective Efficient GlobalPointer becomes.

This is not difficult to understand. The original GlobalPointer has too many parameters, so on average, each parameter is updated more sparsely, making it relatively easier to overfit. Efficient GlobalPointer shares the "extraction" parameters, distinguishing different entity types only through "classification" parameters. Thus, the learning of the entity extraction step is more sufficient, and the entity classification step is easier to learn because it has fewer parameters. Conversely, the good experimental results of Efficient GlobalPointer indirectly prove that the decomposition in Equation [eq:EGP] is reasonable.

Of course, it is not ruled out that the original GlobalPointer might achieve better results when there is sufficient training data. However, even so, when the number of categories is large, the original GlobalPointer may consume too much video memory to be usable. Taking the base version D=768, d=64 as an example, if there are 100 categories, the parameter count for the original GlobalPointer would be 2 \times 768 \times 64 \times 100, which is nearly 10 million. It must be said that this is indeed not friendly enough.

Final Summary

This article pointed out the issue of low parameter utilization in the original GlobalPointer and proposed a corresponding improved version, Efficient GlobalPointer. Experimental results show that while reducing the number of parameters, Efficient GlobalPointer basically does not lose performance and may even achieve improvements.

When reposting, please include the original address of this article: https://kexue.fm/archives/8877

For more detailed reposting matters, please refer to: "Scientific Space FAQ"