I believe many readers have seen the "Qingyuan Project" launched by Tsinghua University and the Beijing Academy of Artificial Intelligence (BAAI) over the past few days (related link: "Is the Chinese version of GPT-3 here? BAAI releases Qingyuan CPM — a large-scale pre-trained model centered on Chinese"). It open-sources the largest Chinese GPT-2 model to date, CPM-LM (2.6 billion parameters). It is said that models with 20 billion or even 100 billion parameters will be open-sourced in the future, aiming to create a "Chinese version of GPT-3."
We know that GPT-3 can achieve Few-Shot learning without fine-tuning. In the current demonstration examples of CPM-LM, the Few-Shot performance is also quite impressive, making one eager to try it out. Naturally, to experiment with it, I wanted to adapt it to my own bert4keras framework for convenience. Thus, the adaptation work began. I initially thought it would be a simple task, but I ended up encountering pitfalls for nearly three days before getting it right. Here, I will briefly record the process of troubleshooting and testing.
Model Introduction
The first model released under this project is called CPM-LM, with approximately 2.6 billion parameters and pre-trained on 100GB of Chinese data. It is a unidirectional language model. For other details, you can read more at the links below. With such a large number of parameters, we generally use it directly without considering fine-tuning. Its capability lies in unconditionally generating random text; of course, we can also provide some prompts to implement text continuation. Applications like Few-Shot learning are essentially variations of text continuation.
Homepage: https://cpm.baai.ac.cn/
GitHub: https://github.com/TsinghuaAI/CPM-Generate
Official Account: https://mp.weixin.qq.com/s/oI2Ak-M57MSuycLVpVEiHw
Regarding the model structure, this was the first pitfall I encountered during adaptation. The architecture of CPM-LM is identical to OpenAI’s GPT-2. So, strictly speaking, this is a 2.6-billion-parameter Chinese GPT-2 model. Initially, I didn’t look closely and was slightly misled by the CPM-LM-TF2 project, leading me to believe its structure was the same as GPT2_ML (GPT2_ML is neither GPT nor GPT-2; it sits somewhere in between). Consequently, I couldn’t get reasonable results for a long time. Once I realized this, rebuilding the GPT-2 model and adapting the corresponding weights was not difficult. Converting the weights to TensorFlow format was also straightforward, thanks to the reference provided by the CPM-LM-TF2 project.
Tokenizer
The second pitfall I encountered during adaptation concerned the tokenizer. I must say, the tokenizer written for CPM-LM is, in my view, quite unrefined, and it still bothers me.
The tokenizer is essentially a wrapper around Google’s SentencePiece,
but it is wrapped in a very inelegant way—a nightmare for anyone with a
penchant for clean code. Specifically, tools like the BERT tokenizer or
SentencePiece default to removing spaces, newlines, and other
delimiters. However, CPM-LM wants to preserve spaces and newlines, so it
replaces them with other symbols before feeding them into the tokenizer
(currently, spaces are replaced by U+2581 "▂" and newlines
by U+2583 "▃"), and then replaces them back before the
final output. This is a common practice and is understandable. What I
find most incomprehensible is that the newline replacement symbol "▃" is
actually not in its SentencePiece model’s vocabulary! To prevent "▃"
from becoming an <unk> token, CPM-LM replaces it
again with <cls>. In other words, it performs a
double replacement just to get the ID for a newline character...
When I first saw this design, I was on the verge of a breakdown: how
hard could it be to just add a few characters to SentencePiece? Why
write it like this? Regardless, the open-source model creators are the
bosses, so I had to find a way to adapt to it. After much thought and
some patching of the original SpTokenizer in
bert4keras, I finally managed to get it working.
Usage and Testing
Enough complaining. In short, after more than two days of effort,
starting from version 0.9.3, bert4keras can now load the
CPM-LM model. Running inference likely requires more than 16GB of VRAM
(I am using a 22GB RTX). The weight conversion process and basic loading
scheme can be found here:
Some Few-Shot results (the output has some randomness; if you only care about Few-Shot performance, consider switching the decoding method to beam search):
# Common Sense Reasoning
# Example output: Beijing
query = u"""
The capital of the USA is Washington
The capital of France is Paris
The capital of Japan is Tokyo
The capital of China is
"""
print(text_expansion.generate(query[1:-1], 1)[0])
# Word Translation
# Example output: bird
query = u"""
dog dog
cat cat
pig pig
bird
"""
print(text_expansion.generate(query[1:-1], 1)[0])
# Subject Extraction
# Example output: Yang Zhenning
query = u"""
Since 1931, Hua Luogeng studied and worked at Tsinghua University. Hua Luogeng
In a simple room, Chen Jingrun conquered the "Goldbach Conjecture." Chen Jingrun
Here, Shing-Tung Yau received the IBM Scholarship. Shing-Tung Yau
Yang Zhenning made milestone contributions in fields such as particle physics, statistical mechanics, and condensed matter physics.
"""
print(text_expansion.generate(query[1:-1], 1)[0])
# Triplet Extraction
# Example output: Zhang Hong, weight, 140 jin
query = u"""
Yao Ming's height is 211cm, he is an idol to many. -> Yao Ming, height, 211cm
Although Jay Chou held his wedding in Europe, he is a native Chinese. -> Jay Chou, nationality, China
Xiao Ming was born in Wuhan, but he doesn't like living there; he moved to Beijing when he grew up. -> Xiao Ming, birthplace, Wuhan
Kris Wu is an idol to many, but he is Canadian, which disappoints many. -> Kris Wu, nationality, Canada
Wu Yao's birthday is May 8th; everyone celebrated for him on that day. -> Wu Yao, birthday, May 8th
"Blue and White Porcelain" is Jay Chou's most proud song. -> Jay Chou, work, "Blue and White Porcelain"
Beijing is the capital of China. -> China, capital, Beijing
Jiang Bi's hometown is in Panlong City; she went to Shenzhen to work after graduation. -> Jiang Bi, native place, Panlong City
Last week we went to Wang Li's hometown in Yunnan and only returned to Wuhan yesterday. -> Wang Li, native place, Yunnan
Yesterday, November 17th, I went to Haidilao with friends, and the waiter celebrated my friend Liu Zhang's birthday. -> Liu Zhang, birthday, November 17th
Zhang Hong's weight reached 140 jin, and she is very distressed. ->
"""
print(text_expansion.generate(query[1:-1], 1)[0])Summary
This article briefly introduced the 2.6-billion-parameter GPT-2
model, CPM-LM, recently open-sourced by Tsinghua University, and its
adaptation into the bert4keras framework. I shared some of
the hurdles encountered during the conversion process and finally
demonstrated the impressive Few-Shot capabilities of CPM-LM.
When reprinting, please include the original address of this article: https://kexue.fm/archives/7912
For more detailed reprinting matters, please refer to: "Scientific Spaces FAQ"