SPACES: An ``Extract-Generate'' Framework for Long Text Summarization (A Summary of the CAIL Challenge)

Statistics

The official competition provided 9,484 annotated samples in the form of (original text, summary) pairs. While the original training data included other auxiliary annotations, we did not use them to maintain model generality. Therefore, our model is theoretically applicable to any supervised summarization task where the sample format is (original text, summary).

Below are some statistics of the training data:

1. Total count: 9,484;

2. Input: Average 2,568 characters, standard deviation 1,122, maximum 13,064, minimum 866;

3. Output: Average 283 characters, standard deviation 36, maximum 474, minimum 66;

4. Metric: Word-based weighted ROUGE.

In short, this is a text generation task with an input of roughly 3,000 characters and an output of 300 characters. The difficulty lies in the fact that the average length of over 2,000 characters far exceeds the text lengths we typically handle. Although 9,484 is the total released dataset, the first-stage data currently available online consists of 4,047 samples, which is actually enough to achieve decent results. The model’s dependence on data volume is not particularly severe.

Sample Preview

The figure above demonstrates a sample from the training set. The top part is the input (original judgment document), and the bottom is the output (human-annotated summary). The green highlights indicate the “Longest Common Subsequence” (LCS) between the two. As seen, the output overlaps significantly with the input.

Modeling Approach

Given these data characteristics, it is natural to adopt an “Extraction + Generation” approach, combined with new methods to ensure summary faithfulness and improve final performance. We named the final model SPACES:

• S: Sparse Softmax (a newly designed Softmax alternative);

• P: Pretrained Language Model;

• A: Abstractive (generative);

• C: Copy Mechanism (a newly designed Copy mechanism);

• E: Extractive;

• S: Special Words (adding specific words to the pretrained model).

Admittedly, this acronym was “painstakingly” forced (facepalm) to match one of this blog’s domains, “spaces.ac.cn.” However, these abbreviations do indeed list the main technical points of our model. Below, we will introduce SPACES in detail.

Corpus Conversion

First, it is important to remember that the extraction model is a process, not the final result. The extracted results will be fed into a Seq2Seq model for optimization. Therefore, the principle for the extraction model is “comprehensiveness”—trying to cover as much information as possible required for the final summary. To this end, we converted the training corpus using the following rules:

1. Construct a custom sentence segmentation function to achieve finer granularity;

2. For each sentence in the human summary, find the sentence in the original text with the highest similarity (allowing duplicate matches);

3. Use all matched original sentences as extraction labels;

4. Delete some matched sentences to maximize the ROUGE score against the human summary.

Note that we removed step 4 in the final model, although it was the default choice in our initial version. While step 4 improves the extraction model’s metrics, the final score after combining with the generation model actually decreased. This is understandable: the generation model inherently has the ability to delete and modify, and it does so better than the extraction model. If the extraction model accidentally deletes a key sentence, the generation model can hardly recover it. Thus, step 4 violates the “comprehensiveness” principle; we should leave the editing work to the generation model.

Metric Issues

The conversion process involves choosing a “similarity” metric. As mentioned, the competition uses “word-based weighted ROUGE.” Initially, we used this, but during debugging, we found it was not the best choice. We ultimately chose “character-based weighted ROUGE.”

What is the difference? The official word-based metric aims to ensure precise matching of proper nouns. For example, if the target is “Law of the People’s Republic of China on the Protection of Minors” and you predict “Law of the People’s Republic of China on the Protection of Cultural Relics,” a character-based metric would still give a high score for the overlapping parts. A word-based metric treats them as different words, resulting in a total mismatch. Thus, word-based metrics favor precise proper noun matching.

However, word-based metrics have a serious side effect: they reduce the weight of long words. For instance, in “According to the relevant provisions of the <Law of the People’s Republic of China on the Protection of Minors>,” the core term has a weight of only 1, while trivial words like “According to,” “the,” “<,” and “>” also have weights of 1. Consequently, the model might prioritize fitting these trivial words over the core legal term. Simply put, a summary with a high word-based score might not necessarily contain key information.

How to reconcile this? The best solution would be word-based but weighted by character count (e.g., giving 14 points for a 14-character word instead of 1). However, this requires implementing a custom ROUGE function. We ultimately chose character-based weighted ROUGE, which was sufficient because the summary and original text describe the same case, making it unlikely to confuse similar-sounding but different legal terms.

Model Structure

For the model, we used a sentence-level sequence labeling model. Sentence vectors were generated using “BERT + Mean Pooling” (fixed), and the labeling model was built using DGCNN. For more on DGCNN, refer to my previous posts on CNN-based reading comprehension and information extraction.

One detail: during training, we used a threshold of 0.3 for EarlyStopping, but for the final generation model data, we used a threshold of 0.2 to adhere to the “comprehensiveness” principle.

Output Data

We use the original text as input, output an extracted summary via the extraction model, and then use that as input for the generation model. However, if we train the extraction model on the whole set and then extract from the same set, the scores will be artificially high due to overfitting, leading to inconsistency between training and prediction.

The solution is cross-validation. We split the data into n folds, train the extraction model on n-1 folds, and predict on the remaining fold. Repeating this n times provides extracted summaries for all data while minimizing training-prediction inconsistency.

Model Overview

The generation model can be summarized in the following diagram:

Basic Architecture

We chose the classic UniLM for the Seq2Seq architecture. Since the total length of “input + output” often exceeds 512, we used Huawei’s NEZHA as the base model because it uses relative position encoding and is not limited by length.

Today, there are other options:

1. Extending absolute position encoding (as in my post on hierarchical decomposition) to handle sequences up to 260,000 tokens.

2. Using the multilingual T5 (mT5), which also uses relative position encoding, though one must be careful with its tokenizer’s handling of full-width punctuation.

Additionally, we were the first to add specific words to the NEZHA model, moving away from the character-only approach of standard Chinese pretrained models. This improved both performance and speed (see my post on WoBERT).

BIO Copy

The Copy mechanism is a standard feature in generative summarization. The conventional approach is Pointer Networks, but it has two drawbacks: 1) it copies one token at a time, failing to guarantee continuous n-gram segments; 2) it is complex to implement. We devised a simpler “BIO Copy” mechanism that handles continuous segments easily.

As shown in the diagram, we add a sequence prediction task to the Decoder. Instead of just modeling p(y_t|y_{< t}, x), we predict a label distribution: p(y_t, z_t|y_{< t}, x) = p(y_t|y_{< t}, x) p(z_t|y_{< t}, x) where z_t \in \{\text{B, I, O}\}:

• B: Token is copied;

• I: Token is copied and continues a segment;

• O: Token is not copied.

During training, labels are derived from the LCS between the summary and original text. During prediction, if z_t is B, we mask tokens not in the original text; if z_t is I, we mask tokens that don’t form a valid n-gram from the original text. This ensures the output remains faithful to the source. While it only improved the score by about 0.5%, it significantly prevents factual errors.

Sparse Softmax

We discovered a Softmax alternative called Sparse Softmax, which improves performance in various classification and generation tasks.

Sparse Softmax is inspired by Sparsemax research, but we used a simpler version:

Here, \Omega_k is the set of indices of the top k logits. We used k=10. This prevents “over-learning.” In standard cross-entropy, to reduce loss to a small \varepsilon, the gap between the maximum and minimum logit must be very large: s_{\max} - s_{\min} \geq \log (n-1) - \log (e^{\varepsilon} - 1) For large n, this creates an unnecessarily large margin, leading to overfitting. Sparse Softmax truncates this, providing a roughly 2% boost in this competition. Note: it is best suited for fine-tuning pretrained models; training from scratch with it may lead to underfitting.

Other Details

We used EMA (Exponential Moving Average) for weights, which stabilizes training. We also applied BIO prediction to both the Encoder and Decoder, which improved results by enhancing synchronization between the two.