T5 PEGASUS: Open-sourcing a Chinese Generative Pre-trained Model

Tokenizer

First, we introduce our improvements to the Tokenizer. The Tokenizer used by mT5 is sentencepiece, a subword tokenization library written in C++. It is efficient and lightweight, but unfortunately, it is not particularly friendly to Chinese, mainly manifested in:

1. Sentencepiece forcibly converts certain full-width symbols into half-width symbols, which is unacceptable in some cases and may affect the evaluation results of tasks;

2. Although the built-in algorithm of sentencepiece has the ability to segment Chinese words, it is still not intelligent enough for Chinese word segmentation;

3. Sentencepiece is written in C++. Although it is open-source, for those accustomed to Python, C++ is equivalent to a black box, making it difficult to read the source code or modify it.

These characteristics led us to switch the Tokenizer back to the BERT Tokenizer. However, simply replacing it with the original Chinese BERT Tokenizer is insufficient. Firstly, our previous work "Speeding up without dropping points: Chinese WoBERT based on word granularity" has shown that using words as units in generative models yields better results. Secondly, even looking only at characters, the vocab.txt of the original Chinese BERT is quite incomplete, missing common punctuation marks (such as double quotes) and Chinese characters (such as "Ya", etc.). To this end, we chose to add word segmentation functionality to the BERT tokenizer and further improve vocab.txt.

Specifically, we added the top 200,000 words from Jieba segmentation to the original Chinese BERT token_dict, and then modified the Tokenizer logic so that it can segment words. These changes are already built into bert4keras and can be called directly. Next, we used this modified Tokenizer to traverse and segment the pre-training corpus we prepared, counted the frequency of each token, and finally kept only the top 50,000 most frequent tokens to obtain a vocab.txt with a size of 50,000 to build our final Tokenizer.

In addition to using this new Tokenizer to train T5 PEGASUS, we also used it to re-train a version of the WoBERT model (WoBERT^+), which readers are welcome to try (Link).

Pre-training Task

For the pre-training task, we hoped it would be closer to natural language generation (rather than just predicting masked parts like T5) and have as much practical value as possible. To this end, we focused on PEGASUS, from the paper "PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization". PEGASUS claims in its paper to be a pre-training model specifically tailored for summarization, but in our view, it can also serve as a general generative pre-training task. The general idea of PEGASUS is to construct summary-like data pairs using the Longest Common Subsequence (LCS) method. T5 PEGASUS did not fully replicate the PEGASUS approach but borrowed the idea for corpus construction.

Specifically, assuming a document has n sentences, we select approximately n/4 sentences (not necessarily contiguous) such that the text formed by these n/4 sentences has the longest possible LCS with the text formed by the remaining 3n/4 sentences. We then treat the 3n/4 sentences as the source text and the n/4 sentences as the summary, thus forming a "(source, summary)" pseudo-summary data pair. We use these data pairs to train the Seq2Seq model. Note that if there are no duplicate sentences in the document, the sentences in the source and summary will not overlap, so this generation task is not a simple copy of the source text and thus possesses a certain level of difficulty.

The search algorithm uses the following greedy approach to step-by-step search until the length requirement is met:

1. First, find 1 sentence such that its LCS with the remaining n-1 sentences is the longest;

2. Assuming k sentences have been found, continue to find the (k+1)-th sentence such that the text formed by these k+1 sentences has the longest LCS with the text formed by the remaining n-k-1 sentences.

Parameters and Configuration

The currently open-sourced T5 PEGASUS is the base version, with a total of 275 million parameters. During training, the maximum length was 512, the batch size was 96, and the learning rate was 10^{-4}. It was trained for 1 million steps using 6 RTX 3090 GPUs, taking about 13 days. The data consists of over 30GB of finely processed general corpus. The training accuracy was approximately 47%, and the training loss was about 2.97. The model was written, trained, and tested using bert4keras.

Github Address: https://github.com/ZhuiyiTechnology/t5-pegasus

Experiments and Evaluation

On the CSL and LCSTS text generation tasks, T5 PEGASUS is the SOTA among all models known to us:

More importantly, T5 PEGASUS has very impressive few-shot learning capabilities:

Even if the number of labeled samples is reduced to 10, T5 PEGASUS can still fine-tune a summary (title) generation model, with performance significantly exceeding other models. On LCSTS, T5 PEGASUS has similar few-shot learning effects, but since the performance of non-T5 PEGASUS models was too poor, the table is not included here.

Few-shot Demonstration

Below is a demonstration of the generation results of the model trained with only 10 labeled samples:

Input: Aiming at the problem of accurate measurement of reliability and fault tolerance of multi-processor systems based on hypercube networks, combined with the characteristics of structural failures often occurring when multi-processor systems are attacked by computer viruses, the structural connectivity and sub-structural connectivity evaluation of n-dimensional hypercube networks were studied. First, a 3-way structural cut method for constructing n-dimensional hypercube networks was used to obtain an upper bound for its 3-way structural connectivity; then, using equivalent or reduction transformation methods for constructing 3-way sub-structural sets of n-dimensional hypercube networks, a lower bound for its 3-way structural sub-connectivity was obtained; finally, using the property that the 3-way structural connectivity of any network is not less than the 3-way sub-structural connectivity, it was confirmed that both the 3-way structural connectivity and sub-structural connectivity of hypercube networks are equal to the dimension of the hypercube network.
Title: 3-way Structural Connectivity and Sub-structural Connectivity of Hypercube Networks
Prediction: Research on Evaluation of Structural Connectivity and Sub-structural Connectivity Based on n-dimensional Hypercube Networks

Input: Aiming at the problems of low prediction accuracy, large calculation volume, and high energy consumption of traditional Wireless Body Area Network (WBAN) prediction models for sensing data, an adaptive cubic exponential smoothing algorithm based on a penalty error matrix is proposed. First, a lightweight prediction model is established between sensing nodes and routing nodes; second, a carpet search method is used for parameter optimization of the prediction model; finally, a penalty error matrix is used for further fine-grained processing of the prediction model parameters. Experimental results show that compared with the ZigBee protocol, within a 1000 time-slot range, the proposed method can save about 12% of energy; compared with the carpet search method using a penalty error matrix, the prediction accuracy is improved by 3.306%. The proposed method can effectively reduce computational complexity while further reducing the energy consumption of WBAN.
Title: Energy-saving Method for Synchronous Prediction in Wireless Body Area Networks Based on Penalty Error Matrix
Prediction: Adaptive Cubic Exponential Smoothing Algorithm Based on Penalty Error Matrix

Input: Aiming at the problems of low message transmission efficiency and large network resource overhead in the Internet of Vehicles (IoV) environment, a routing algorithm based on cognitive interaction of vehicle nodes suitable for urban traffic scenarios is proposed. First, the concept of node cognitive interaction degree is proposed based on trust theory, and vehicle nodes in the IoV are classified and assigned different initial cognitive interaction values; at the same time, influence factors such as vehicle node interaction time, interaction frequency, physical distance between vehicle nodes, interval hop count, and message survival time are introduced to construct a vehicle node cognitive interaction evaluation model. Based on this model, the cognitive interaction degree of nodes is calculated and updated, and neighbor nodes with relatively high cognitive interaction degrees are selected as relay nodes for message forwarding by comparing the cognitive interaction values between corresponding vehicle nodes.
Title: Routing Algorithm Based on Node Cognitive Interaction in Internet of Vehicles Environment
Prediction: Routing Algorithm Based on Vehicle Node Cognitive Interaction

Input: Aiming at the joint estimation problem of Direction of Arrival (DOA) and distance of near-field sources, a Near-field Iterative Adaptive Algorithm (NF-IAA) is proposed. First, all possible positions of signal sources in the near-field region are represented by dividing a two-dimensional grid. Each position is regarded as a potential signal source incident on the array, and the data model of the array output is represented; then, the results of the previous spectrum estimation are used through cyclic iteration to construct the signal covariance matrix, and the inverse of the covariance matrix is used as a weighting matrix to estimate the potential signal source energy corresponding to each position; finally, a three-dimensional energy spectrum is plotted. Since only the energy of real signal sources is non-zero, the positions corresponding to the spectral peaks are the positions of the real signal sources. Simulation experiments show that under the condition of 10 snapshots, the DOA resolution probability of NF-IAA reaches 9...
Title: Joint Estimation of Near-field Source Two-dimensional Parameters Based on Iterative Adaptive Method
Prediction: Near-field Iterative Adaptive Algorithm Based on NF-IAA

Input: Aiming at the problem that existing software crowdsourcing worker selection mechanisms do not sufficiently consider collaborative development among workers, a software crowdsourcing worker selection mechanism based on active time grouping is proposed on the basis of the bidding model. First, crowdsourcing workers are divided into multiple collaborative development groups based on active time; then, the weight of the collaborative work group is calculated according to the development ability of workers in the group and the collaboration factor; finally, the collaborative work group with the largest weight is selected as the optimal work group, and the most suitable worker is selected from the group for each task module according to the module complexity. Experimental results show that compared with the ability-priority selection method, this mechanism has only a 0.57% difference in average worker ability, while reducing project risk by an average of 32% by ensuring collaboration among workers, which can effectively guide crowdsourcing software tasks requiring multi-person collaboration...
Title: Software Crowdsourcing Worker Selection Mechanism Based on Active Time Grouping
Prediction: Software Crowdsourcing Worker Selection Mechanism Based on Active Time Grouping

As can be seen, even with very few labeled samples, readable generation results can still be obtained. This is thanks to the fact that PEGASUS-style pseudo-summary pre-training is very close to downstream tasks.

Simple Summary

This article mainly shared our Chinese generative pre-trained model T5 PEGASUS. Based on mT5, it uses PEGASUS-style pseudo-summary pre-training on Chinese corpora, resulting in good text generation performance, especially outstanding few-shot learning capabilities. Readers with text generation needs are welcome to use it.

Original URL: https://kexue.fm/archives/8209

For more details on reprinting, please refer to: Scientific Space FAQ