P-tuning: Automatically Constructing Templates to Unleash the Potential of Language Models

Rethinking Templates

First, let’s think about "what a template is." Intuitively, a template is a prefix/suffix composed of natural language. Through these templates, we align downstream tasks with pre-training tasks to fully utilize the original pre-trained model for better zero-shot and few-shot learning.

Wait, do we really care if the template is made of "natural language"?

Not really. Essentially, we don’t care what the template looks like. We only need to know which tokens the template consists of, where they should be inserted, whether they can complete our downstream task after insertion, and what the output candidate space is. Whether the template is natural language has no fundamental impact on us. The "natural language" requirement is only to better achieve "consistency," but it is not mandatory. Thus, P-tuning considers templates of the following form:

Here, [u1] to [u6] represent [unused1] to [unused6] in the BERT vocabulary. That is, a few tokens never seen before are used to form the template. The number of tokens is a hyperparameter, and their placement (front or back) can be adjusted. Then, to make the "template" work, we use labeled data to solve for this template.

How to Optimize

Depending on the amount of labeled data, we discuss two cases.

Case 1: Labeled data is scarce. In this case, we fix the weights of the entire model and only optimize the Embeddings of the [unused1] to [unused6] tokens. In other words, we are learning six new Embeddings that function as a template. Since the model weights are mostly fixed, training is fast. Because there are very few parameters to learn, the template can be learned even with very few labeled samples without easily overfitting.

Case 2: Labeled data is abundant. If we still follow the first scheme, underfitting will occur because six tokens provide too few optimizable parameters. Therefore, we can release all weights for fine-tuning, which is what the original paper did for the SuperGLUE experiments. Readers might wonder: what is the difference between this and directly adding a fully connected layer for fine-tuning? The original paper’s results show that this approach performs better, likely because it remains more consistent with the pre-training task.

Furthermore, in the examples above, the target tokens (such as "very" or "sports") are manually selected. Can they also be replaced by [unused*] tokens? The answer is yes, but it also depends on two situations: 1. When labeled data is scarce, manually selecting appropriate target tokens often works better; 2. When labeled data is abundant, using [unused*] tokens for target tokens works better because the model has a larger optimization space.

Enhancing Correlation

In the original paper, P-tuning does not randomly initialize a few new tokens and train them directly. Instead, it calculates these Embeddings through a small LSTM model and sets this LSTM model as learnable. What is the benefit of this extra step? The original paper suggests: tokens appearing in an LSTM have stronger correlation, making them somewhat more like "natural language" (since natural language tokens are not independent), and it also helps prevent local optima. I further confirmed this with the authors on GitHub (see here); the difference in effect is that the LSTM approach allows the model to converge faster and achieve better results.

However, adding an LSTM feels a bit awkward and slightly complicates implementation. According to the authors, the LSTM helps the template tokens stay closer to natural language (to some extent), but this doesn’t necessarily require an LSTM, nor does an LSTM guarantee it. I believe a more natural method is to not only predict the target tokens of the downstream task (like "very" or "news") during training but also simultaneously perform other token predictions.

For example, if it is an MLM model, randomly mask other tokens to predict them; if it is an LM model, predict the entire sequence instead of just the target word. The reasoning is: since our MLM/LM are pre-trained on natural language, we (perhaps overconfidently) believe that a sequence that can be reconstructed well must be close to natural language. Thus, adding these training objectives can also make the model stay closer to natural language. Based on my tests, adding such auxiliary objectives indeed improves performance compared to purely optimizing the downstream task target.

Stopped Gradients

How should the P-tuning algorithm be implemented? If all weights are released for training, it is simple and no different from standard BERT fine-tuning. The key is how to implement "optimizing only a few tokens" in few-shot scenarios.

There are several ways to implement this, such as building a new Embedding layer for the tokens to be optimized, concatenating it with the BERT Embedding layer, and only releasing the weights of the new layer during training. However, this requires significant changes to the original model code. The best way is to minimize code changes so that it is almost transparent to the user. To this end, I devised a scheme using stop_gradient to modify the Embedding layer:

class PtuningEmbedding(Embedding):
    """New Embedding layer that only optimizes specific tokens"""
    def call(self, inputs, mode='embedding'):
        embeddings = self.embeddings
        embeddings_sg = K.stop_gradient(embeddings)
        mask = np.zeros((K.int_shape(embeddings)[0], 1))
        mask[1:9] += 1  # Only optimize tokens with IDs 1-8
        self.embeddings = embeddings * mask + embeddings_sg * (1 - mask)
        return super(PtuningEmbedding, self).call(inputs, mode)

After a variable passes through the stop_gradient operator, its gradient becomes zero during backpropagation, but the forward pass remains unchanged. Therefore, in the code above, the forward result is unaffected, but during backpropagation, the tokens where the mask is non-zero receive gradients, while others do not. This achieves the goal of updating only specific tokens.

The complete code is available at:

GitHub: https://github.com/bojone/P-tuning

The original authors also open-sourced their code:

GitHub: https://github.com/THUDM/P-tuning

Testing and Performance

The original authors’ results on SuperGLUE showed that with P-tuning: 1. Both GPT and BERT performance improved compared to direct fine-tuning; 2. GPT’s performance could even exceed BERT’s. This indicates that GPT has NLU capabilities as well as NLG capabilities, effectively "squeezing out" GPT’s potential. BERT also improves with P-tuning, suggesting that P-tuning is a general method for releasing the potential of language models.

The original paper contains rich experiments; I recommend reading it carefully. Notably, in Table 2’s last column, when the pre-trained model is large enough, our hardware might not be able to fine-tune the entire model. P-tuning allows optimizing only a few token parameters, significantly reducing the memory and computation required. Thus, P-tuning provides a way to utilize large-scale pre-trained models under limited computing power.

Of course, my consistent view is that "an algorithm that hasn’t been tested on Chinese has no soul." Therefore, I also tested it on Chinese tasks. The test task is the same as in the PET article: few-shot sentiment classification. The models tested include BERT and GPT, with their respective candidate templates shown below:

Note that for LM models, introducing a prefix is very important; performance drops significantly if only a suffix is used. For MLM models, prefixes also generally perform better than suffixes. The overall results are as follows:

In the table, "Few-shot" uses only a "small amount of labeled samples," "Unsupervised" uses "a large amount of unlabeled samples," and "Semi-supervised" uses "a small amount of labeled samples + a large amount of unlabeled samples." P-tuning results are all few-shot. The PET results reported are for the best manual templates (some manual templates performed much worse). From a few-shot perspective, P-tuning indeed achieves the best results. From a template construction perspective, P-tuning is much better than manual templates. From a model perspective, P-tuning brings GPT’s classification performance close to BERT’s, revealing GPT’s strong NLU potential.

Discrete vs. Continuous

Before P-tuning, there were already efforts in automatic template construction, such as "How Can We Know What Language Models Know?" and "AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts". However, these searched for natural language templates in a discrete space, which limited their effectiveness and prevented them from achieving breakthrough results.

In contrast, P-tuning abandons the "natural language" requirement, turning it into a continuous parameter problem solvable by simple gradient descent, which yields better results. This change highlights the essence of templates—the key to a template is how it is used, not what it is made of. This "stripping away the fluff" approach is truly enlightening.

(Note: As pointed out by reader @brotherb, the paper "Prefix-Tuning: Optimizing Continuous Prompts for Generation" published earlier this year is quite similar to P-tuning. Both design non-natural language templates, though Prefix-Tuning focuses on NLG while P-tuning focuses on NLU.)

Adapter

We can also understand P-tuning from the perspective of Adapters. Shortly after BERT was released, Google proposed a fine-tuning method called Adapter in the paper "Parameter-Efficient Transfer Learning for NLP". Instead of fine-tuning the entire model, it fixes the original BERT weights and adds residual modules (Adapters) on top of BERT, optimizing only these modules. Since Adapters have fewer parameters, the fine-tuning cost is lower. The idea originated from CV’s "Learning multiple visual domains with residual adapters". However, it has been seen less frequently in the last two years, perhaps because while it speeds up training, it slows down inference and often incurs a loss in accuracy.

In P-tuning, if we don’t view the newly inserted tokens as a "template" but as part of the model, it is essentially an Adapter-like approach. It fixes the original weights and inserts new optimizable parameters, optimizing only those. The difference is that the parameters are inserted into the Embedding layer. Thus, P-tuning and Adapters share many similarities.

Why is it Effective?

Finally, a question worth considering: Why is P-tuning better? For example, in a full-data scenario where all weights are released, P-tuning still outperforms direct fine-tuning. Why?

In fact, readers asking this might be "too used" to the standard practice of adding a fully connected layer to BERT. Clearly, whether it’s PET or P-tuning, they are closer to the pre-training task. The practice of adding a fully connected layer is actually further from the pre-training task. So, in a sense, the effectiveness of P-tuning is "obvious," while why adding a fully connected layer works is the question that deserves more scrutiny.

A paper last year, "A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks", attempted to answer this. The general line of reasoning is:

1. Pre-training models are a type of language model task;
2. Downstream tasks can be expressed as special cases of this language model task;
3. When the output space is limited, it approximates adding a fully connected layer;
4. Therefore, fine-tuning with a fully connected layer is effective.

As we can see, the core assumption is point 2—that downstream tasks can be expressed in a PET-like form. This further suggests that PET and P-tuning are more natural ways to use pre-trained models, and direct fine-tuning is just a corollary. In other words, PET and P-tuning are the "return to essence" solutions, which is why they are more effective.