Recently, I noticed that an expert developer open-sourced a Chinese GPT2 model. It is the largest version with 1.5 billion parameters. Looking at the demo provided by the author, the generation effect is quite impressive. I wanted to load it into my bert4keras to play with it. However, the overall architecture of the early version of bert4keras was written in a rather "rigid" way, making it very inconvenient to integrate multiple different models. Two weeks ago, I finally couldn’t stand it anymore and rewrote the entire structure of bert4keras. Now, bert4keras can be considered flexible enough to implement various Transformer-based models, such as GPT2 and T5, which have already been integrated.
GPT2 Introduction
GPT is a name many readers have likely heard of. Simply put, it is a language model based on the Transformer structure, originating from the paper "Improving Language Understanding by Generative Pre-Training". However, it was not born just to be a language model; it uses the language model for pre-training and then fine-tunes on downstream tasks to improve performance. It is a pioneer of the "Transformer + Pre-training + Fine-tuning" paradigm. Relatively speaking, BERT can be considered its "successor." GPT2 is the upgraded version of GPT—larger model, more training data—with the largest version reaching 1.5 billion parameters.
Chinese Version
Most readers who have seen popular science posts about GPT2 have been amazed by its generation results. However, no matter how good it is, it was designed for other languages; OpenAI did not help train a Chinese version. The good news is that a project called GPT2_ML has open-sourced a Chinese version of GPT2, and it is the largest 1.5-billion-parameter model.
Currently, the GPT2 integrated into bert4keras is the one provided by the GPT2_ML project, not the OpenAI version, as bert4keras prioritizes serving Chinese tasks. It is worth noting that the model structure of GPT2_ML is different from OpenAI’s GPT2 and also different from BERT. A comparison of the blocks for the three is shown below:
Testing it out
First, download the model weights from the following address:
Link: https://pan.baidu.com/s/1OXBd16o82SpIzu57kwA8Mg Extraction code: q79r
The main file "model.ckpt-100000.data-00000-of-00001" can also be downloaded from Google Drive. After downloading, please check the SHA256 of model.ckpt-100000.data-00000-of-00001 (4a6e5124df8db7ac2bdd902e6191b807a6983a7f5d09fb10ce011f9a073b183e).
Then, install bert4keras version 0.6.0 or higher (the current latest version), and you can run the following test code (if the code becomes outdated due to version iterations, please check basic_language_model_gpt2_ml.py for the latest version):
#! -*- coding: utf-8 -*-
# Basic test: Chinese GPT2 model
# Intro link: https://kexue.fm/archives/7292
import numpy as np
from bert4keras.models import build_transformer_model
from bert4keras.tokenizers import Tokenizer
from bert4keras.snippets import AutoRegressiveDecoder
from bert4keras.snippets import uniout
config_path = '/root/gpt2/config.json'
checkpoint_path = '/root/gpt2/model.ckpt-100000'
dict_path = '/root/gpt2/vocab.txt'
tokenizer = Tokenizer(dict_path,
token_start=None,
token_end=None,
do_lower_case=True) # Build tokenizer
model = build_transformer_model(config_path=config_path,
checkpoint_path=checkpoint_path,
model='gpt2_ml') # Build model, load weights
class ArticleCompletion(AutoRegressiveDecoder):
"""Article continuation based on random sampling
"""
@AutoRegressiveDecoder.set_rtype('probas')
def predict(self, inputs, output_ids, step):
token_ids = np.concatenate([inputs[0], output_ids], 1)
return model.predict(token_ids)[:, -1]
def generate(self, text, n=1, topk=5):
token_ids, _ = tokenizer.encode(text)
results = self.random_sample([token_ids], n, topk) # Based on random sampling
return [text + tokenizer.decode(ids) for ids in results]
article_completion = ArticleCompletion(start_id=None,
end_id=511, # 511 is the Chinese period
maxlen=256,
minlen=128)
print(article_completion.generate(u'The weather is nice today'))Partial results (translated from Chinese):
>>> article_completion.generate(u’The weather is nice today’)
[’The weather is nice today, perfect for a run. Last night I watched a documentary about running. The heroine tells the story of a girlś growth, which is very inspiring and beautiful. I also want to run, but I didnt́ know I needed sports shoes to run, so I bought a pair. This documentary is about sports shoes. One episode tells the story of a girl who hadnt́ worn sports shoes since elementary school and only started trying to run in high school.’]
>>> article_completion.generate(u’Double 11’)
[’Double 11 is coming soon! Are you still worried about the logistics delivery for Double 11? Are you still worried about not having time to pick up goods from the warehouse? Are you still worried about not knowing how to buy cheap goods? Are you still annoyed that you cant́ buy the products you like? Well, Double 11 is here! Today, the editor will take you to see how these couriers deliver goods! 1. Logistics delivery: The delivery of courier companies is mainly handled by the courier company, which is responsible for dispatching items and logistics services.’]
>>> article_completion.generate(u’Scientific Space’)
[’Scientific Space Station (English: science space station) is a space station of the Peopleś Republic of China. The space station was developed by the Dalian Institute of Chemical Physics, Chinese Academy of Sciences, and mainly develops and uses the nuclear-powered space station of the Dalian Institute of Chemical Physics. The Scientific Space Station is located in Haidian District, Beijing, 393 meters above the ground, with a total construction area of about 10,000 square meters and a total investment of about 500 million yuan. Construction of the Scientific Space Station began on December 26, 2018, and it was completed and put into use in June 2021.’]
Doesn’t it feel quite good?
Want to Fine-tune?
Seeing these results, many readers’ first thought might be: how can I use this for my own model? Can I fine-tune it on my own tasks?
Unfortunately, I have to share a somewhat pessimistic result: I
tested fine-tuning this GPT2 model, which has nearly 1.5 billion
parameters, using the Adam optimizer on a TITAN RTX with 22G of VRAM. I
found that it couldn’t even run with a batch_size=1...
Finally, I discovered that it can only be fine-tuned using the AdaFactor
optimizer.
Regarding AdaFactor, I will have the opportunity to write an article to discuss it later.
When reposting, please include the original address of this article: https://kexue.fm/archives/7292
For more detailed reposting matters, please refer to: "Scientific Space FAQ"