Starting from this article, we will apply the sampling algorithms introduced previously to specific text generation examples. As our first example, we will introduce how to use BERT for random text sampling. Random text sampling refers to the process of randomly generating natural language sentences from a model. It is commonly believed that such random sampling is a feature exclusive to unidirectional autoregressive language models like GPT-2 and GPT-3, and that bidirectional Masked Language Models (MLM) like BERT cannot achieve this.
Is that really the case? Of course not. Using BERT’s MLM model, one can actually perform text sampling; in fact, it is exactly the Gibbs sampling introduced in the previous article. This fact was first explicitly pointed out in the paper "BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model". The title of the paper is quite interesting: "BERT also has a mouth, so it must say something." Now, let’s see what BERT can actually say.
Sampling Process
First, let us review the Gibbs sampling process introduced in the previous article:
Gibbs Sampling
The initial state is \boldsymbol{x}_0=(x_{0,1},x_{0,2},\cdots,x_{0,l}), and the state at time t is \boldsymbol{x}_t=(x_{t,1},x_{t,2},\cdots,x_{t,l}).
Sample \boldsymbol{x}_{t+1} through the following process:
Uniformly sample an index i from 1,2,\cdots,l;
Calculate p(y|\boldsymbol{x}_{t,-i})=\frac{p(x_{t,1},\dots,x_{t,i-1},y,x_{t,i+1},\cdots,x_{t,l})}{\sum\limits_y p(x_{t,1},\dots,x_{t,i-1},y,x_{t,i+1},\cdots,x_{t,l})};
Sample y \sim p(y|\boldsymbol{x}_{t,-i});
\boldsymbol{x}_{t+1} = {\boldsymbol{x}_t}_{[x_{t,i}=y]} (i.e., replace the i-th position of \boldsymbol{x}_t with y to obtain \boldsymbol{x}_{t+1}).
The most critical step is the calculation of p(y|\boldsymbol{x}_{-i}). Its specific meaning is "predicting the probability of the i-th element by using all l-1 elements except the i-th one." Readers familiar with BERT should immediately realize: isn’t this exactly what BERT’s MLM model does? Therefore, by combining the MLM model with Gibbs sampling, random text sampling can be achieved.
Thus, implementing the above Gibbs sampling process for MLM-based text sampling results in the following workflow:
MLM Model Random Sampling
The initial sentence is \boldsymbol{x}_0=(x_{0,1},x_{0,2},\cdots,x_{0,l}), and the sentence at time t is \boldsymbol{x}_t=(x_{t,1},x_{t,2},\cdots,x_{t,l}).
Sample a new sentence \boldsymbol{x}_{t+1} through the following process:
Uniformly sample an index i from 1,2,\cdots,l, and replace the token at the i-th position with [MASK] to obtain the sequence \boldsymbol{x}_{t,-i}=(x_{t,1},\dots,x_{t,i-1},\text{[MASK]},x_{t,i+1},\cdots,x_{t,l});
Feed \boldsymbol{x}_{t,-i} into the MLM model and calculate the probability distribution at the i-th position, denoted as p_{t+1};
Sample a token from p_{t+1}, denoted as y;
Replace the i-th token of \boldsymbol{x}_t with y to serve as \boldsymbol{x}_{t+1}.
Readers might notice that this sampling process can only sample sentences of a fixed length and does not change the sentence length. This is indeed true because Gibbs sampling can only perform sampling from a specific distribution, and sentences of different lengths actually belong to different distributions. Theoretically, they have no intersection. However, when we build language models, we usually use autoregressive models to uniformly model the distribution of sentences of different lengths, so we often do not notice the fact that "different sentences actually belong to different probability distributions."
Of course, solving this is not impossible. The original paper "BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model" points out that the initial sentence can be set as a sequence consisting entirely of [MASK] tokens. In this way, we can first randomly sample a length l, and then start the Gibbs sampling process with l [MASK] tokens as the initial sentence, thereby obtaining sentences of different lengths.
Reference Code
With an existing MLM model, implementing the Gibbs sampling described
above is quite simple. Below is the reference code based on
bert4keras:
Gibbs Sampling Reference Code: basic_gibbs_sampling_via_mlm.py
Here are some examples:
Initial Sentence:
Science and technology are the primary productive forces.
Sampling Results:
How is the unboxing of the Honor laptop?
What to do if WeChat records are useless?
What to do if the browser cannot be installed?
How to use the epf converter a7l?
What to do if the browser is not installed?
How to use the Honor laptop charging?
What to do if asp.net cannot be opened?
What to do if the browser is not installed?
What to do if the browser cannot be restarted?
How to use the ro Hanba exchange mac tv version?
Initial Sentence:
Beijing adds 3 new local confirmed cases and 1 asymptomatic carrier.
Sampling Results:
Macau recorded 233 cases of H1N1 infection and 13 cases of radioactive infection.
The celebration ceremony is a grand event where academy painting and steel workers participate in creation.
After the celebration ceremony, other games on the Ghibli platform also participated in the celebration.
Clinical trials found that the g-chromosome in the findings mostly comes from gastrointestinal infections.
Clinical trials found that people usually truly enjoy the pleasure of the clitoris.
The celebration mode is more integrated with celebrations in other games on the Ghibli platform.
The celebration mode is updated and celebrated on Ghibli or other games.
Macau recorded 20 cases of H1N1 infection and 2 cases of radioactive infection.
Clinical trials found that female chromosomes often come from gastrointestinal infections.
Clinical trials found that 90% of infection cases are m-type gastrointestinal infections.
Initial Sentence:
9 consecutive [MASK] tokens
Sampling Results:
You learn from your mother every day!
That night, everything was white before my eyes.
Layers of verdant green are full of life.
The kindergarten wants to do business.
How exactly can one enter the officialdom?
Hello, teacher and classmates!
The mountains are heavy, and both are vast.
Plum rains, the fog is thick outside the window.
At that time, everything was white before my eyes.
It’s still a great cake cutting!
The author used the Chinese BERT-base released by Google for the experiments. It can be seen that the sampled sentences are quite rich and possess a certain degree of readability, which is quite good for a base-version model.
For consecutive [MASK] tokens as initial values, repeated experiments may yield very different results:
Initial Sentence:
17 consecutive [MASK] tokens
Sampling Results:
What to eat for other facial paralysis? What is good to eat for other facial paralysis?
How to treat pediatric facial paralysis? What medicine to take for facial paralysis?
How to treat infant facial paralysis? What is good to eat for facial paralysis?
What is the cause of headache in children? What kind of disease is urticaria?
What to eat for other facial paralysis \cdot What is good to eat for other facial paralysis?
How to install the sanitary ware? How to connect the faucet?
What is good to eat for other facial paralysis? What is good to eat for other facial paralysis?
What is the cause of headache in children? What kind of disease is urticaria?
The kitchen cabinets can’t fit in, how to plug in the faucet?
Otherwise, the kitchen cupboard can’t be found, what to do with the hot water faucet?
Initial Sentence:
17 consecutive [MASK] tokens
Sampling Results (Originally sampled as Japanese):
Please see the link below for the appendix tweets.
Please use the system you want to operate in Amakata-cho.
There are 2 areas, guidance from the Clocker specialty store!
This site adopts a system that fits the discount!
Simultaneous works use the surface system.
Manufacturer’s products use the system until seriousness.
Please use the system on the bulletin board.
Those living in the center use the production system.
Please see the list of Airware address levels.
Please see the link below for appendix support.
Amazingly, Japanese was also sampled, and the author checked with Baidu Translate—these Japanese sentences are actually readable. On one hand, this reflects the diversity of the random sampling results; on the other hand, it also shows that the Google version of Chinese BERT did not perform denoising well, and the training corpus must have been mixed with a significant amount of non-Chinese and non-English text.
Reflections
Recently, Google, Stanford, OpenAI, and others co-published a paper "Extracting Training Data from Large Language Models", pointing out that language models like GPT-2 can completely reproduce (expose) training data. This is not hard to understand, as language models are essentially memorizing sentences. Gibbs sampling based on MLM shows that this problem exists not only in explicit language models like GPT-2 but also in bidirectional language models like MLM. We can see some clues from the sampling examples above; for instance, sampling Japanese indicates that the original corpus was not perfectly denoised. Starting from "Beijing adds 3 new local confirmed cases...", we sampled results related to H1N1, which reflects the era of the training corpus. These results imply that if you do not want your open-source model to expose privacy, you must do a thorough job of cleaning the pre-training corpus.
Furthermore, regarding the paper "BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model", there is another point of interest: the original paper claimed that the MLM model is a Markov Random Field (MRF), but this is actually incorrect. The authors later clarified this on their homepage; interested readers can see "BERT has a Mouth and must Speak, but it is not an MRF". In general, using MLM for random sampling is fine, but it does not strictly correspond to a Markov Random Field.
Summary
This article introduced random text sampling based on BERT’s MLM, which is essentially a natural application of Gibbs sampling. Overall, this is a fairly simple example. For readers who already understand Gibbs sampling, this article presents almost no technical difficulty; for those who do not yet understand it, this specific example serves as a good way to further understand the Gibbs sampling process.
Reprinting: Please include the original address of this article: https://kexue.fm/archives/8119
More details on reprinting: Please refer to the "Scientific Space FAQ".