Among the various designs for pre-training tasks, NSP (Next Sentence Prediction) is generally considered one of the poorer choices. This is because its difficulty is relatively low, and its inclusion in pre-training does not significantly benefit downstream task fine-tuning; in fact, the RoBERTa paper even suggested it could have a negative effect. Consequently, subsequent pre-training research has generally followed two paths: either removing the NSP task entirely, as RoBERTa did, or finding ways to increase its difficulty, as ALBERT did. In other words, NSP has long been somewhat "shunned."
However, a reversal has arrived, and NSP might be making a "comeback." A recent paper titled "NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task–Next Sentence Prediction" (hereinafter referred to as NSP-BERT) shows that NSP can actually achieve very impressive Zero-Shot results! This is another classic case of Prompt-based Few/Zero-Shot learning, but this time, NSP is the protagonist.
Background Review
We used to believe that pre-training was purely for pre-training—that it merely provided a better initialization for downstream task training. For instance, BERT’s pre-training tasks include MLM (Masked Language Model) and NSP (Next Sentence Prediction). For a long time, researchers did not care about these pre-training tasks themselves but focused on how to achieve better performance on downstream tasks through fine-tuning. Even when T5 scaled model parameters to 11 billion, it still followed the "pre-train + fine-tune" paradigm.
The first to powerfully break this mindset was GPT-3, released last year. It demonstrated that with a sufficiently large pre-trained model, we can design specific templates (Prompts) to achieve excellent Few/Zero-Shot effects without any fine-tuning. Wherever there is GPT, BERT is never absent. Since GPT could do it, BERT should be able to as well. This led to the subsequent PET work, which also constructed special templates to utilize pre-trained MLM models for Few/Zero-Shot learning. (For readers unfamiliar with this, you can refer to "Is GPT-3 Necessary? No, BERT’s MLM Model Can Also Do Few-Shot Learning").
Since then, work on "pre-training + prompts" has gradually increased and is now even showing an "explosive" trend. This series of works is now generally referred to as "Prompt-based Language Models," and many can be found with a simple search. Today, a consensus has formed: constructing appropriate Prompts to make the form of downstream tasks closer to the pre-training tasks usually yields better results. Therefore, how to construct Prompts has become one of the focuses of this series of work, with P-tuning being a classic example (refer to "P-tuning: Automatically Constructing Templates to Release the Potential of Language Models").
NSP Enters the Stage
If you look closely at Prompt-based research, you will find that the current focus is primarily on how to better utilize pre-trained GPT, MLM, or Encoder-Decoder models, with little attention paid to other pre-training tasks. The NSP-BERT work, however, fully explores the potential of the NSP task and inspires us that even within the scope of Prompt-based methods, there is still significant room for divergent research ideas.
The so-called NSP task is not actually about predicting the next sentence in a generative sense, but rather, given two sentences, determining whether they are adjacent. Accordingly, the idea behind NSP-BERT is quite simple: taking a classification problem as an example, it treats the input as the first sentence, then adds specific Prompts to each candidate category as the second sentence, and judges which second sentence is most coherent with the first. It can be seen that the logic of NSP-BERT is very similar to PET; in fact, Prompt-based works are all quite easy to understand—the difficulty lies in being the first to think of doing it this way.
The figure below demonstrates the reference Prompt schemes for NSP-BERT on some common NLU tasks. It can be seen that NSP-BERT can handle quite a few tasks:
In fact, after looking at this diagram, you already understand most of the ideas behind NSP-BERT. The rest of the paper merely expands on the details of this diagram. Students who want to delve deeper can read the original paper carefully.
While the NSP-BERT mode is not the first of its kind—someone previously proposed using NLI models for Zero-Shot classification (refer to "NLI Models as Zero-Shot Classifiers")—the format is basically consistent with NSP. However, NLI requires supervised fine-tuning on labeled corpora, whereas the utilization of purely unsupervised NSP is a first-time attempt.
Experimental Results
Interestingly, for us, NSP-BERT is a very "down-to-earth" and conscientious piece of work. For example, it was written by Chinese researchers, its experimental tasks are all in Chinese (FewCLUE and DuEL2.0), and the code has been open-sourced. Below is the author’s open-source address:
Most importantly, the performance of NSP-BERT is truly impressive:
Overall, after seeing these experimental results, I can only say "my apologies for the oversight" to NSP. Such a heavyweight in the model world was right in front of us, yet we never realized its potential. Kudos to the authors of NSP-BERT for their keen observation.
Summary
This article shared a paper that uses BERT’s pre-training task NSP for Zero-Shot learning. The results show that using NSP for Zero-Shot can achieve excellent performance. Perhaps in time, NSP will truly "rise."
When reposting, please include the original address of this article: https://kexue.fm/archives/8671
For more detailed reposting matters, please refer to: "Scientific Space FAQ"