Sohu Text Matching: A Multi-task Baseline Based on Conditional LayerNorm

Task Introduction

As the name suggests, the task of the competition is text matching—determining whether two texts are similar. While this is a relatively conventional task, what makes it interesting is that it is divided into multiple subtasks. Specifically, it is categorized into two major classes, A and B. Class A has looser matching standards, while Class B has stricter ones. Each major class is further divided into three subcategories: "Short-Short Matching," "Short-Long Matching," and "Long-Long Matching." Therefore, although the task type is the same, strictly speaking, there are six different subtasks.

# Example of Class A sample
{
    "source": "London, UK, 20/21 Premier League Round 20, Tottenham Hotspur vs Liverpool. Tottenham has 9 wins, 6 draws, and 3 losses in 18 league rounds this season, currently ranking 5th with 33 points. Liverpool has 9 wins, 7 draws, and 3 losses in 19 league rounds, currently ranking 4th with 34 points. From the current trend, this match is very passive from Tottenham's perspective. Ultimately, the score of this match was Tottenham 1-3 Li",
    "target": "At 4:00 AM on January 29, Beijing time, the 20th round of the Premier League ushered in a strong dialogue, with Tottenham hosting Liverpool at home. Tottenham vs Liverpool, match highlights are as follows: First: Can Tottenham successfully take revenge? In the first round between the two sides, Tottenham was killed 1-2 by Liverpool away. After the game, Mourinho said the best team lost. This round Tottenham hosts Liverpool, can they successfully take revenge given the Reds' 5-round winless slump? Second: Liverpool's recent",
    "labelA": "1"
}

# Example of Class B sample
{
    "source": "London, UK, 20/21 Premier League Round 20, Tottenham Hotspur vs Liverpool. Tottenham has 9 wins, 6 draws, and 3 losses in 18 league rounds this season, currently ranking 5th with 33 points. Liverpool has 9 wins, 7 draws, and 3 losses in 19 league rounds, currently ranking 4th with 34 points. From the current trend, this match is very passive from Tottenham's perspective. Ultimately, the score of this match was Tottenham 1-3 Li",
    "target": "At 4:00 AM on January 29, Beijing time, the 20th round of the Premier League ushered in a strong dialogue, with Tottenham hosting Liverpool at home. Tottenham vs Liverpool, match highlights are as follows: First: Can Tottenham successfully take revenge? In the first round between the two sides, Tottenham was killed 1-2 by Liverpool away. After the game, Mourinho said the best team lost. This round Tottenham hosts Liverpool, can they successfully take revenge given the Reds' 5-round winless slump? Second: Liverpool's recent",
    "labelB": "0"
}

Generally speaking, completing this task would require at least two models, as the classification standards for Class A and Class B are different. If one wanted to be more precise, six models might be necessary. However, training six models independently is often laborious, and different tasks cannot learn from each other to improve performance. Therefore, it is natural to think of sharing a portion of the parameters to turn this into a multi-task learning problem.

Model Introduction

Of course, treating it as a conventional multi-task learning problem would be too generic. Given the characteristic that these tasks have the "same form but different standards," I conceived a way to implement these six subtasks using a single model through Conditional LayerNorm (Conditional Layer Normalization).

Regarding Conditional LayerNorm, we previously introduced it in the article "Conditional Text Generation based on Conditional Layer Normalization". Although the example then was text generation, its applicable scenarios are not limited to that. Simply put, Conditional LayerNorm is a scheme for incorporating condition vectors into a Transformer to control the output results. It integrates the conditions into the \beta and \gamma parameters of the LayerNorm layer.

For the six tasks in this competition, we only need to pass the task type as a condition into the model. This allows a single model to handle six different tasks. The schematic diagram is as follows:

In this way, the entire model is shared; we simply input the task type ID alongside the sentences, achieving maximum parameter sharing.

Code Reference

The implementation of Conditional LayerNorm has long been embedded in bert4keras. Once this design was conceived, implementing it with bert4keras was a natural progression. Reference code is available here:

Github Link: https://github.com/bojone/sohu2021-baseline

The code uses RoFormer as the base model. This is mainly because, in "Long-Long Matching," the total length of the two concatenated texts is quite long. Using RoFormer, which is based on sub-words, can shorten the sequence length, allowing the processing of longer texts with the same computational power. Furthermore, the RoPE (Rotary Position Embedding) used by RoFormer can theoretically handle texts of any length. After several tests, the offline F1 score was around 0.74, and the online F1 score after submission to the test set was around 0.73. Testing on a 3090, a single epoch takes about an hour, and 4 to 5 epochs are sufficient.

The current code mixes all data together for random training. This has a minor drawback: samples with shorter original sequence lengths are padded to the maximum length, which slows down the training speed for short-sequence samples (though it is certainly faster than training six models independently). An optimization would be to ensure that samples within the same batch have similar lengths during batching, but I was too lazy to implement that—I’ll leave it to you to optimize!

Summary

This article shares a baseline for the Sohu text matching competition. It primarily uses Conditional LayerNorm to increase model diversity, enabling a single model to process different types of data and produce different outputs.

Original Address: https://kexue.fm/archives/8337

For more details regarding reposting, please refer to: "Scientific Space FAQ"