bert4keras in Hand, Baseline I Have: CLUE Benchmark Code

Text Classification

First are the IFLYTEK and TNEWS tasks, which are standard text classification problems. The approach is simple: the conventional “[CLS] + sentence + [SEP]” is passed into BERT, and then the hidden vector of [CLS] is extracted for classification.

Additionally, the pronoun disambiguation task WSC can also be converted into a single-text classification task. The original task is to determine whether two fragments in a sentence (one of which is a pronoun) refer to the same object. The baseline approach is to mark these two fragments with different symbols in the text and then pass the marked text directly into BERT for binary classification.

Text Matching

Next, AFQMC, CMNLI, and OCNLI are three text matching tasks. Text matching can be simply understood as a classification task for sentence pairs. For example, similarity matching determines whether two sentences are similar; Natural Language Inference (NLI) determines the logical relationship between two sentences (entailment, neutral, contradiction), etc. In the pre-training era, the standard practice for sentence matching tasks is to connect the two sentences with [SEP] and treat them as a single-text classification task.

It should be noted that in the original BERT, the SegmentIDs (originally called token type IDs) of the two sentences are different. However, considering that models like RoBERTa do not have an NSP task, SegmentID 1 might not have been pre-trained. Therefore, in this implementation, SegmentIDs are all set to 0. Experimental results show that this treatment does not reduce the effectiveness of text matching.

Similarly, for the CSL task, which determines whether an abstract description matches four given keywords, we connect the four keywords with semicolons “;” and treat them as a single sentence, thus converting it into a conventional text matching problem.

Reading Comprehension

Reading comprehension refers to the CMRC2018 task, which is an extractive reading comprehension task with the same format as SQuAD. A paragraph is paired with multiple questions, and each question must have an answer that is a fragment of the paragraph. The general approach is to concatenate the question and the paragraph with [SEP], pass them into BERT, and use two fully connected layers to predict the start and end positions respectively. The problem with this is that it severs the connection between the start and end and causes inconsistency between training and prediction behavior.

The baseline here uses GlobalPointer as the output structure, which treats the start-end combination as a whole for classification. For specific details, please refer to “GlobalPointer: A Unified Way to Handle Nested and Non-Nested NER”. Using GlobalPointer ensures that training and prediction behaviors are completely consistent and significantly improves decoding speed.

Furthermore, in both SQuAD and CMRC2018, paragraph lengths often significantly exceed 512, and some answers are indeed located at later positions. Directly truncating the front part might result in no answer being present. While models like NEZHA or RoFormer can directly handle text longer than 512, models like BERT cannot. To maintain code versatility, this implementation follows the sliding window design of the original BERT baseline, splitting the paragraph into multiple sub-paragraphs with a stride of 128. Each sub-paragraph is then combined with the question and passed into the model. After this segmentation, some paragraphs may be “answerless” for a given question; in such cases, the answer is pointed directly to the [CLS] position (0,0). During the prediction phase, long paragraphs are segmented using the same method, each answering the same question, and the answer with the highest score is finally selected.

Multiple Choice

Multiple choice here refers to the C3 task, which is also a type of reading comprehension task. A paragraph is followed by multiple questions, and the answer to each question is one of four given candidate answers, though not necessarily a fragment from the paragraph. The baseline approach for this multiple-choice task might surprise many: it is converted into a text matching problem. Each candidate answer is matched with the paragraph and the question, and the one with the highest score is chosen during prediction.

Consequently, a single original question needs to be split into four samples for processing, requiring four predictions to reach an answer, which greatly increases the computational load. However, surprisingly, this approach is generally the most effective among all intuitively conceived baselines, performing much better than concatenating all candidate answers together for a 4-way classification. A similar task in the English domain is DREAM, and the models on its leaderboard are basically variants of this idea.

Idiom Understanding

The idiom reading comprehension task, CHID, is essentially a multiple-choice reading comprehension problem, but its form is considerably more complex, so it is introduced separately.

Specifically, each sample in CHID has 10 candidate idioms and several questions. Each question has several blanks, and we must decide which candidate idiom is most suitable for each blank. If each question had only one blank, we could directly apply the multiple-choice approach from the previous section. However, here a question may have multiple blanks. Using the multiple-choice approach, we can only identify one blank at a time. Therefore, we use [unused0] to represent the blank we want to identify, while other unidentified blanks (if any) are replaced by four [MASK] tokens. For example:

[CLS] “This is actually an absurd farce. After Apple discovered that the owner of the iPad trademark in mainland China belonged to Shenzhen Proview rather than Taiwan Proview, it began to worry and [unused1].” Xiao Caiyuan stated. In fact, two dramatic factors made the case appear even more [MASK] [MASK] [MASK] [MASK]. Materials submitted by Apple in the lawsuit filed in the Hong Kong court showed that IPADL was actually a special purpose company established under the operation of Apple’s lawyers, aimed at acquiring the i-Pad trademark rights from Proview. [SEP] Final Word [SEP]

In other words, a question with multiple blanks will be split into multiple sub-questions. According to the aforementioned multiple-choice approach, each sub-question needs to be concatenated with candidate answers for prediction. Thus, the computational cost of each sub-question is equivalent to 10 samples of ordinary classification. This is indeed a bit laborious, but necessary for performance. To achieve the effect of a larger batch size, gradient accumulation is usually required. Additionally, some questions are quite long, so we still need to truncate. The truncation method centers on the blank to be identified, extending as far as possible to the left and right by equal distances.

Finally, according to the task design, each sample has several questions, and every blank in every question shares the same 10 candidate idioms, but the answers for each blank are non-repetitive. If the prediction for each blank were made independently by taking the answer with the maximum value, duplicate prediction results might occur, which contradicts the problem design.

To ensure non-repetitive prediction results, we need to use the “Hungarian algorithm”: assuming there are m blanks and each blank has n > m candidate answers, we will obtain m \times n scoring sentences. We need to select a unique answer for each blank such that the total score is maximized. In mathematics, this is known as the “assignment problem,” and the standard solution is the “Hungarian algorithm.” We can solve it directly using scipy.optimize.linear_sum_assignment. This post-processing algorithm improves accuracy by about 6% compared to simply taking the maximum for each item (which might lead to duplicate answers).

Entity Recognition

The final task is CLUENER, a conventional non-nested named entity recognition task. Common baselines are BERT+Softmax or BERT+CRF. The one used here is BERT+GlobalPointer, which can also be referenced in “GlobalPointer: A Unified Way to Handle Nested and Non-Nested NER”. When GlobalPointer is used for NER, it can handle both nested and non-nested cases in a unified way. My multiple experiments show that in non-nested scenarios, GlobalPointer can achieve results comparable to CRF, with faster training and prediction speeds. Therefore, using GlobalPointer as the NER baseline is a natural choice.