Before Using ALBERT and ELECTRA, Make Sure You Really Understand Them

Characteristics

Simply put, ALBERT is essentially a BERT with parameter sharing. It is equivalent to changing the function y=f_n(f_{n-1}(\cdots(f_1(x)))) to y=f(f(\cdots(f(x)))), where f represents each layer of the model. In this way, what originally had n layers of parameters now has only 1 layer. Consequently, the number of parameters is greatly reduced, or rather, the volume of the saved model weights is very small. This is the first meaning of “Lite.” Furthermore, because the total number of parameters is reduced, the time and GPU memory required for model training are also reduced accordingly; this is the second meaning of “Lite.” Additionally, when the model is very large, parameter sharing acts as a strong regularization method, making it less prone to overfitting compared to BERT. Ultimately, its large-scale models achieve certain performance improvements, which is the highlight of ALBERT.

Prediction

It is important to note that we did not mention prediction speed. Obviously, parameter sharing does not bring acceleration during the inference phase. The model still performs forward calculations step-by-step and does not care whether the current parameters are the same as the previous ones; even if they are the same, it cannot be accelerated (because the input has changed). Therefore, the prediction speed of ALBERT and BERT of the same specification is the same. In fact, if we were to be pedantic, ALBERT might even be slightly slower because it uses matrix decomposition for the Embedding layer, a step that introduces additional computation, although this computation is generally imperceptible.

Training

As for training speed, while there is an improvement, it is not as significant as imagined. Reducing the number of parameters to 1/n does not mean the training speed increases by n times. In my previous experiments, the base version of ALBERT was only about 10% to 20% faster in training than the corresponding BERT-base, and the reduction in GPU memory usage was similar. If the model is smaller (tiny/small versions), this gap narrows further. In other words, ALBERT’s training advantage is only obvious in large models; for smaller models, this advantage remains difficult to perceive significantly.

Performance

Regarding performance, the original ALBERT paper explains it quite clearly, as shown in the table below. Parameter sharing limits the model’s representational capacity. Therefore, the xlarge version of ALBERT is required to match the BERT-large version, and the xxlarge version is needed to consistently surpass it. In other words, as long as the version specification is smaller than xlarge, the performance of ALBERT is inferior to BERT of the same specification. Evaluation results on Chinese tasks are similar; you can refer to here and here. Moreover, I previously conducted a more extreme experiment: loading ALBERT weights but relaxing the parameter sharing constraint (treating ALBERT as BERT), which also led to performance improvements! (Refer to “Lifting Constraints, Enhancing Models: One Line of Code to Improve ALBERT Performance”). Thus, it is a solid fact that small-specification ALBERT is inferior to BERT.

Conclusion

Therefore, the summarized suggestion is: Unless you are using the xlarge version, there is no need to use ALBERT. At the same speed, ALBERT performs worse than BERT; at the same performance level, ALBERT is slower than BERT. Now that BERT also has tiny/small versions, such as those open-sourced by our company, they are basically just as fast and perform better, unless you truly need the characteristic of small file size.

What does the xlarge version imply? Some readers haven’t even tried BERT because their machines can’t run it; most readers have limited GPU memory and have only run BERT-base, never having run or being unable to run the large version. Since xlarge is even larger than large, the hardware requirements are higher. So, frankly speaking, for most readers, there is no need to use ALBERT.

Why then did the idea that ALBERT is both fast and good spread? Aside from some non-standard promotion by self-media, I think it is largely due to the promotion by user “brightmart.”

It must be said that brightmart made an indelible contribution to the popularization of ALBERT in China. Even before the English version of the ALBERT model was released, brightmart trained and open-sourced the Chinese version of ALBERT (albert_zh) and went on to train multiple versions: tiny, small, base, large, and xlarge. At that time, BERT only had base and large versions, while ALBERT had tiny and small versions. When people tried them, they found them much faster than BERT, so many were left with the impression that ALBERT is very fast. In fact, ALBERT being fast had nothing to do with ALBERT itself; the key was the tiny/small specifications. Corresponding BERT tiny/small models are also very fast...

Of course, you can think about the more fundamental reasons why ALBERT’s parameter sharing works, or research how to improve prediction speed after parameter sharing. These are all valuable questions, but it is not recommended to use ALBERT versions lower than xlarge.

Characteristics

The starting point of ELECTRA is the belief that BERT’s MLM (Masked Language Model) operation of randomly selecting a portion of tokens to mask is too simple. It aims to increase the difficulty. Therefore, it borrows the idea of GANs, using a standard method to train an MLM model (generator), then sampling and replacing tokens in input sentences based on the MLM model. The processed sentences are fed into another model (discriminator) to determine which parts of the sentence were replaced and which were not. The generator and discriminator are trained simultaneously. As the generator improves, the difficulty for the discriminator increases, which intuitively helps the model learn more valuable content. Finally, only the discriminator’s encoder is kept for use; the generator is generally discarded.

Because this progressive mode makes the training process more targeted, the main highlight of ELECTRA is higher training efficiency. According to the paper, it can achieve the effect of a BERT of the same specification in 1/4 of the time or less. This is the primary highlight of ELECTRA.

Theory

However, in my view, ELECTRA is a model that is theoretically shaky.

Why say this? The idea of ELECTRA originates from GANs, but in Computer Vision (CV), do we have examples of taking a trained GAN’s discriminator and fine-tuning it for downstream tasks? At least I haven’t seen any. In fact, this is theoretically untenable. Taking the original GAN as an example, the optimal solution for its discriminator is D(x)=\frac{p(x)}{p(x)+q(x)}, where p(x) and q(x) are the distributions of real and fake samples, respectively. Assuming the training is stable and the generator has sufficient fitting capability, as the model trains, fake samples will gradually approach real samples, so q(x) approaches p(x), and D(x) approaches the constant 1/2. That is to say, theoretically, the final discriminator is just a constant function. How can you guarantee it extracts good features?

Although ELECTRA is not exactly a GAN, it is consistent on this point. Therefore, ELECTRA emphasizes that the MLM model acting as the generator cannot be too complex (otherwise, as mentioned, the discriminator would degenerate into a constant). The paper states that the best results are achieved when the generator’s size is between 1/4 and 1/2 of the discriminator’s size. This starts to become “metaphysical”. We only argued that being too good is bad; we cannot prove why being slightly worse works, nor can we prove how much worse it should be, or why simultaneous training of the generator and discriminator is better. Now these have all become pure “alchemy.”

Performance

Of course, saying it is theoretically shaky does not mean its performance is bad, nor does it mean the relevant evaluations were faked. Because constraints were placed on the generator’s capability, the results of ELECTRA training still have some significance, and its performance is acceptable. It’s just that we went through a process of “the greater the expectation, the greater the disappointment.”

The ELECTRA paper first appeared in the ICLR 2020 submissions, and the results at that time shocked everyone: the small version of ELECTRA far surpassed the small version of BERT, even approaching the base version, while the base version of ELECTRA reached the level of BERT-large. However, when the code and weights were released, the results shown on GitHub were “eye-opening”—they basically dropped by 2 percentage points. Later, the authors clarified that the paper reported results on the dev set while GitHub reported the test set. People understood a bit more, but in that case, ELECTRA lost its highlight in terms of performance compared to BERT. (Refer to the transition from “ELECTRA: Surpassing BERT, the Best NLP Pre-training Model of 2019” to “My Thoughts on the Release of ELECTRA Source Code”).

In fact, evaluations on Chinese tasks reflect this more accurately. For example, in the Chinese-ELECTRA open-sourced by HFL, ELECTRA’s performance across various tasks is almost identical to BERT of the same level. It has advantages in a few tasks, but no “crushing” results appeared.

Trade-offs

Some readers might think that even if the performance is similar, the pre-training is faster, which is still an advantage. This is undeniable. However, a recent Arxiv paper indicates that ELECTRA’s “similar performance” might be an illusion on simple tasks; if more complex tasks are constructed, it still gets “crushed” by BERT.

The paper is titled “Commonsense knowledge adversarial dataset that challenges ELECTRA”. The authors constructed a new dataset, QADS, based on SQuAD 2.0 using synonym substitution. According to the authors’ tests, an ELECTRA-large model that achieves 88% on SQuAD 2.0 drops to only 22% on QADS, while interestingly, BERT can still achieve over 60%. Of course, this paper still looks a bit rough and hasn’t received authoritative confirmation, so it shouldn’t be fully trusted, but its results already prompt us to reflect on ELECTRA. Previously, the paper “Probing Neural Network Comprehension of Natural Language Arguments” showed that a single “not” could pull BERT off its pedestal; it seems ELECTRA might have similar issues, perhaps even more severe.

Setting aside other evidence, I personally feel that ELECTRA’s ultimate abandonment of MLM is a “lose the big picture for the small” operation. If your starting point is that MLM is too simple, you should find ways to increase the difficulty of MLM. Why replace MLM with a discriminator? Improving the MLM model directly using a generator network (rather than replacing it with a discriminator) is possible. A recent Microsoft paper, “Variance-reduced Language Pretraining via a Mask Proposal Network”, provides such a reference scheme. It lets the generator choose the positions to mask instead of selecting them randomly. Although I haven’t replicated its experiments, its entire reasoning process is very convincing, unlike the “shot in the dark” feeling of ELECTRA. Furthermore, I want to emphasize again that MLM is very useful. It is not just a pre-training task but also a model with practical value, such as in “Is GPT-3 Necessary? No, BERT’s MLM Model Can Also Do Few-shot Learning”.

Conclusion

So, after all that, the conclusion is: ELECTRA’s pre-training speed is indeed faster, but current experiments show it has no prominent advantage over BERT of the same level in downstream tasks. You can try it, but don’t be too disappointed if the performance worsens. Additionally, if you need to use the weights of the MLM part (for example, for text generation in UniLM, refer to here), you cannot use ELECTRA because its main body is a discriminator, not an MLM model. The MLM model acting as the generator in ELECTRA is simplified compared to the discriminator and may suffer from insufficient fitting or inadequate learning, making it a poor pre-trained MLM model.

As for the idea behind ELECTRA—improving upon the simplicity of random masking in MLM—the direction seems correct. However, the effectiveness of replacing a generative model with a discriminative one still requires further verification. Readers interested in in-depth analysis could further research this.