Building Your Own DialoGPT: A Generative Multi-turn Dialogue Model Based on LM

Corpus Introduction

Here is a brief introduction to the LCCC dataset (Large-scale Cleaned Chinese Conversation). For specific details, you can visit the GitHub page; the download links are also provided there. LCCC is divided into two versions: base and large. The base version primarily originates from Weibo conversations, while the large version integrates other open-source dialogue corpora on top of the base. According to the authors, LCCC underwent a rigorous cleaning process, so the overall quality appears to be very high.

\begin{array}{c|c|c} \hline \text{LCCC-base} & \text{Single-turn Dialogue} & \text{Multi-turn Dialogue} \\ \hline \text{Total Dialogue Turns} & \text{3,354,382} & \text{3,466,607} \\ \hline \text{Total Utterances} & \text{6,708,554} & \text{13,365,268} \\ \hline \end{array}

\begin{array}{c|c|c} \hline \text{LCCC-large} & \text{Single-turn Dialogue} & \text{Multi-turn Dialogue} \\ \hline \text{Total Dialogue Turns} & \text{7,273,804} & \text{4,733,955} \\ \hline \text{Total Utterances} & \text{14,547,608} & \text{18,341,167} \\ \hline \end{array}

To simplify the task, all samples were processed into two-person dialogues. Below are some sample examples:

A: When we go back for the New Year, let’s buy some rabbit heads and have a good hotpot meal.
B: I haven’t seen any good rabbit heads in Taiyuan.
A: I’ll bring one back for you from Hongqiao; I saw an authentic one the other day.
B: Love you the most.
A: That’s a must.

A: Yeah, I’ll wait a bit longer! You’re in Shanghai now, right? The wind in Shanghai seems even stronger than in Nanjing. Try to stay indoors.
B: Right, I’m at home, it’s fine. You must be careful!

A: I also went for a stroll last year and even ran into my old PE teacher; we took a photo together.
B: Haha, I also went to find my English teacher from 10th grade, but I couldn’t find her—she happened to have something to do and wasn’t at school 
A: You’re really looking for memories.
B: Haha, I haven’t been back since graduation and wanted to take a look.

Model Design

After understanding what the data looks like, we need to design the model. Obviously, we need to train a model to predict what the next response should be. Since the corpus contains multi-turn dialogues, we also require the model to support multi-turn conversations. The simplest way to consider dialogue history is to concatenate all historical dialogues up to the current sentence into a single text string to serve as the model’s input.

Given some input to predict an output, it seems formally like we should use a Seq2Seq model. Using Seq2Seq directly isn’t a big problem, but standard Seq2Seq is generally used for inputs and outputs with relatively fixed forms—for instance, the input text length should be concentrated within a certain range and shouldn’t vary too much. However, when considering multi-turn dialogues, we theoretically don’t know how many previous turns there are, so in principle, the input text length is unlimited. Using Seq2Seq also presents training efficiency issues: for each dialogue turn, we can only train one response at a time. If a multi-turn dialogue has n responses, it would have to be split into n samples for training.

Therefore, we need a model whose length can vary quite freely and that can predict an entire multi-turn dialogue simultaneously. A suitable choice for this requirement is a unidirectional Language Model (LM, GPT). The approach is shown in the figure below:

As shown in the figure, we choose the currently mainstream Transformer model. Following the conventional BERT input format, we concatenate each dialogue sentence using [SEP] and then train a left-to-right unidirectional language model. To distinguish between different speaking roles, we use different Segment IDs for different speakers. Furthermore, considering that both BERT and GPT use absolute position encodings, there is an upper limit to the text length they can process, while the number of dialogue turns is theoretically infinite. Therefore, we adopted NEZHA, which uses relative position encoding, as the basic structure and used NEZHA’s pre-trained weights as the initial weights for the model.

In short, we added a lower-triangular Attention Mask to NEZHA to turn it into a language model. For a related introduction, please refer to "From Language Models to Seq2Seq: Transformer is All About the Mask".

Training Details

First, here is my reference implementation and the trained model:

https://github.com/bojone/nezha_gpt_dialog

The code is implemented based on bert4keras, and the basic structure is NEZHA Base. To make the training more "accessible," I used only one card (a 22G RTX) combined with gradient accumulation. Readers with smaller video memory can reduce the batch_size. The open-sourced model was trained using the Adam optimizer with a learning rate of 2 \times 10^{-5} and a weight decay rate of 0.01. It was trained for 10 million steps (over 20 days), with a single-step batch_size of 16 and gradient accumulation for 16 steps, which is equivalent to training for 625,000 steps with a batch_size of 256. The final loss was approximately 2.83 (average cross-entropy per token).

Let’s test our training results.

>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
Hotpot is delicious.
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
Me too.
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
Hahahaha, I’ve had hotpot twice this past week.
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
Me too, I get stuffed every time.
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
Eating hotpot won’t make you fat.
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
Hotpot is great.
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
Do you have that kind of hotpot over there?
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
Is hotpot good? I want some too.
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
Go ahead and eat; you’ll be happy once you do.
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
I want to eat hotpot every day too!
>>> print(chatbot.response([u’I’ve been eating hotpot every day lately’]))
You can’t eat hotpot for that long.

>>> print(chatbot.response([u’Is the weather good where you are?’]))
It’s okay, not hot, a bit cold.
>>> print(chatbot.response([u’Is the weather good where you are?’, u’It’s okay, not hot, a bit cold.’, u’Then how thick are the clothes you have to wear?’]))
I’m wearing two layers.

Comparative Analysis

CDial-GPT also open-sourced its own pre-trained model, and I have converted it into a format that bert4keras can load (CDial-GPT-tf); readers can also test and compare it. In terms of training, CDial-GPT used a model implemented in PyTorch with a GPT Base structure. It used four 2080Ti cards, with a total batch_size of 32 and gradient accumulation for 64 steps. The paper states it was trained for 30 epochs, totaling about 21 million steps (twice mine), which is roughly equivalent to training for 330,000 steps with a batch_size of 2048.

The input design of CDial-GPT is also different, as shown below:

As illustrated, the main difference between CDial-GPT and our design is the concatenation method for multi-turn dialogues. We previously used [SEP] directly for connection, whereas CDial-GPT uses role markers like [speaker1] and [speaker2] (abbreviated as S1 and S2 in the figure) for connection, using [SEP] only at the end to indicate the conclusion of the response. Consequently, because the format of the predicted part differs from the historical format, only one response can be trained at a time. Multi-turn dialogues must be split into multiple samples for training, which theoretically increases training complexity (it takes multiple steps to finish training one multi-turn dialogue sample).

As for the results, my personal testing suggests there is no significant difference between the two. Interested readers can perform their own comparative tests.

Conclusion

This article shared a practice in dialogue modeling. Based on the LCCC chit-chat corpus open-sourced by CDial-GPT, we used a Language Model (GPT) to perform generative modeling on multi-turn dialogues, resulting in a relatively general chit-chat dialogue model. Finally, we compared the approach in this article with the original model open-sourced by CDial-GPT.