A Review of Some Recent Non-Transformer Works

Synthesizer

To interpret the MLP-related work mentioned above, one must mention the Synthesizer published by Google in May last year: "Synthesizer: Rethinking Self-Attention in Transformer Models". In fact, if you already understand Synthesizer, several papers in the list above can be glossed over.

In a previous blog post, we provided a simple interpretation of Synthesizer. Setting aside the scaling factor, the Attention operation can be decomposed as: \boldsymbol{O}=\boldsymbol{A}\boldsymbol{V},\quad \boldsymbol{A}=\text{softmax}(\boldsymbol{B}),\quad \boldsymbol{B}=\boldsymbol{Q}\boldsymbol{K}^{\top} where \boldsymbol{Q}, \boldsymbol{K}, \boldsymbol{V} are transformations of the input sequence. Readers familiar with Self-Attention should be clear on this. Synthesizer experimented with several new algorithms for \boldsymbol{B}, the most impressive of which is called "Random," where the entire \boldsymbol{B} is treated as a parameter matrix (updated or not updated after random initialization).

In the Random case, the Attention matrix no longer changes with the sample; that is, all samples share the same Attention matrix. However, it still achieves good results, which indeed dealt a strong blow to the inherent understanding of Attention at the time. Synthesizer’s experiments were quite extensive, including machine translation, automatic summarization, dialogue generation, and pre-training + fine-tuning. It can be said that most of the papers listed above do not have experiments as rich as Synthesizer.

MLP-Mixer

Synthesizer probably didn’t expect that a year later, it would change its name and become popular.

The MLP-Mixer proposed in the paper "MLP-Mixer: An all-MLP Architecture for Vision" is actually the Random mode of Synthesizer with the softmax activation removed. That is, it sets \boldsymbol{B} as a trainable parameter matrix and then directly sets \boldsymbol{A}=\boldsymbol{B}. The model description ends here. The only other difference is that MLP-Mixer performs CV tasks while Synthesizer performs NLP tasks.

By the way, why is this model called MLP-Mixer? Because the authors gave this direct trainable Attention mode the name "token-mixing MLP" and renamed the original FFN to "channel-mixing MLP" (formerly called Position-wise FC). Regardless of the name, it claims to be just MLP, so the model is called MLP-Mixer.

In fact, I believe a more standard name for this is a 1D convolution with a window size of 1. However, whether in this paper or the previous "Attention Is All You Need", the authors would rather invent new names for these conventional operations and selectively reduce or even ignore the connection with convolution. It seems they have gone to great lengths for the sake of "A Good Title Is All You Need."

This point was also criticized by Yann LeCun: if it were truly a standard MLP, the input should be flattened into a one-dimensional vector before being connected to a transformation matrix.

External Attention

From an analogical perspective, Synthesizer’s Random mode or MLP-Mixer is equivalent to setting both \boldsymbol{Q} and \boldsymbol{K} in Attention as parameter matrices. The External Attention proposed in "Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks" sets \boldsymbol{K} and \boldsymbol{V} as (fixed-size) parameter matrices, also for CV tasks.

Normally this wouldn’t be an issue, as deep learning is results-oriented; if the results are good, it can be published. However, personally, I find many claims in External Attention to be questionable.

First, it calls itself "two linear layers," deliberately downplaying its connection with Attention (is it embarrassing to say it’s a special case of Attention?). Then it says that "by introducing two external memory units (the \boldsymbol{K} and \boldsymbol{V} set as parameters), it implicitly learns the features of the entire dataset." This statement isn’t wrong, but in fact, any parameter of any model can be explained this way; it is not a unique characteristic of External Attention. Furthermore, it claims to achieve linear complexity, but that requires fixing the lengths of \boldsymbol{K} and \boldsymbol{V}. In this case, it would be more convincing to compare it with LinFormer, which also has linear complexity (the paper compares it with Performer, but Performer’s approach to reducing complexity is different; LinFormer is more comparable).

Setting aside the terminology, the working mechanism of External Attention seems a bit confusing. It’s not hard to imagine that External Attention’s encoding of each feature is isolated. In NLP terms, this would mean each word is encoded independently without any connection to the context, which certainly wouldn’t work. So why does it work in CV?

Stack of FFN

As for the paper "Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet", it overlaps significantly with MLP-Mixer, but it is written much more straightforwardly. It simply passes the input through a regular FFN, transposes the output, passes it through another FFN, and finally transposes it back. If you are already familiar with Transformers, it becomes clear very quickly what was done.

The paper itself is very short, only 4 pages in total, including 1 page of code and half a page of references. The main text is only about 2.5 pages, making it more like a brief report. Perhaps the authors intended to dig deeper into this area, but Google’s MLP-Mixer came out first, making further work less meaningful, so they hurriedly finished and published it. (This part of the story is purely my own speculation.)

Pre-trained CNN

In fact, CNNs were the first to attempt to replace RNNs (in Seq2Seq tasks). Facebook’s "Convolutional Sequence to Sequence Learning" was published earlier, but it was quickly overshadowed by Google’s "Attention Is All You Need". After the release of models like GPT and BERT, Transformer-like models became the mainstream, and CNNs were rarely studied deeply.

The paper "Are Pre-trained Convolutions Better than Pre-trained Transformers?" helps us verify the effectiveness of "CNN + Pre-training." The results show that whether using downstream data for supervised training or pre-training followed by fine-tuning, CNN models based on dilated convolutions or dynamic convolutions are slightly superior to Transformer models and are even faster. By the way, this paper has already been accepted by ACL 2021, so it was actually written earlier and only released this month.

The main inspiration from this paper is: improvements in pre-training and improvements in models should not be conflated. Pre-training technology itself often brings improvements to various models. One should not immediately think of Transformers when pre-training is mentioned, nor should pre-training only be combined with Transformers. In fact, I used to prefer CNNs and achieved good results in multiple tasks through the design of "Dilated Gated Convolutional Neural Networks" (DGCNN). This paper reaffirms the value of CNNs. Nevertheless, I will likely not devote my primary energy to CNN research.

First, theoretically, CNNs cannot capture sufficiently long-range dependencies, which is a fundamental flaw. Although the receptive field of a CNN can be quickly increased through methods like dilated convolution, it is only "relatively large" and not the "one-step" reach of Transformers in theory. Second, from the perspective of improving efficiency, Transformers themselves have a lot of room for optimization. If the switch to CNNs is purely for execution efficiency, that reason seems less convincing. Furthermore, the \mathcal{O}(n^2) complexity of Transformers also brings more room for experimentation (like UniLM) and more variety (like K-BERT).

In summary, we cannot deny the value of CNNs, but if one is already focused on Transformers, there is no need to divert too much energy toward CNNs.

ResMLP

As for the ResMLP proposed by Facebook in "ResMLP: Feedforward networks for image classification with data-efficient training", there is no essential difference from the aforementioned MLP-Mixer and Stack of FFN. Its textual description is also very similar to Stack of FFN. Ignoring minor details, one could even consider the three to be the same model. Finally, ResMLP’s experimental tasks are also in CV.

FNet

In my view, FNet proposed in "FNet: Mixing Tokens with Fourier Transforms" is the most interesting paper among the seven listed. In a sense, FNet is also a special case of MLP-Mixer, but it is a very interesting one: while the attention matrix of MLP-Mixer is obtained through direct parameter optimization, the parameter matrix of FNet is obtained directly through the Fourier Transform! Therefore, the "attention layer" of FNet has no optimization parameters at all!

We can also understand FNet from the perspective of attention. Setting aside the normalization factor, the attention operation can be roughly written as: \boldsymbol{O}=\boldsymbol{A}\boldsymbol{V},\quad \boldsymbol{A}=\exp(\boldsymbol{B}),\quad \boldsymbol{B}=\boldsymbol{Q}\boldsymbol{K}^{\top} Here \boldsymbol{Q}, \boldsymbol{K} were originally n \times d matrices. FNet says: \boldsymbol{Q}, \boldsymbol{K} can be replaced by n \times 1 matrices: \boldsymbol{Q}=\boldsymbol{K}=\begin{pmatrix}0 \\ 1 \\ 2 \\ \vdots \\ n - 1\end{pmatrix} Yes, you read that right; it crudely replaces them with an n \times 1 matrix composed of 0 \sim n-1. Of course, this would cause \exp(\boldsymbol{B}) to explode exponentially. To avoid this, FNet changes it to: \boldsymbol{A}=\exp(\text{i}\boldsymbol{B}) In other words, by making it an imaginary exponent, it won’t explode! It’s that crude, and that’s how the Fourier-based FNet is obtained. The original paper performs Fourier transforms in both the sequence length and feature dimension directions and then keeps only the real part, using this operation to replace self-attention. For the implementation of the Fourier transform, we have an algorithm called the "Fast Fourier Transform (FFT)" with an efficiency of \mathcal{O}(n \log n), so FNet can also effectively handle long sequences.

Some results of FNet are shown in the table below. In fact, from the perspective of pre-training and downstream task performance, FNet does not have much of an advantage. However, its performance on the Long-Range Arena (a benchmark for testing long-range capabilities of models) is quite good.

Of course, the fact that FNet’s crude approach works at all is already a miracle. The biggest impact it brings is undoubtedly: even this works? Why does the Fourier transform work? I don’t know the answer. Some comments online say that this shows the attention mechanism is actually a transformation of coordinate bases, and the Fourier transform is also a transformation of bases, so their roles are similar. This statement feels somewhat fundamental. In ICLR 2021, there was also a paper "Is Attention Better Than Matrix Decomposition?" which used SVD instead of Attention and achieved good results. This indicates that the idea of base transformation indeed exists (SVD is also a base transformation), but how to maintain sequentiality while transforming bases and which base transformation is more suitable are questions for which we have no clues.

gMLP / aMLP

Finally, the gMLP and aMLP introduced in "Pay Attention to MLPs" are relatively conventional explorations of new structures, essentially enhanced versions of MLP-Mixer. The "g" in gMLP stands for "gate." Simply put, gMLP combines MLP-Mixer with a gating mechanism, while the "a" in aMLP stands for "attention," combining attention with gMLP.

Specifically, gMLP operates roughly as follows: \begin{aligned} &[\boldsymbol{X}_1, \boldsymbol{X}_2] = \boldsymbol{X} \\ &\boldsymbol{Y} = \boldsymbol{W}\boldsymbol{X}_2 + \boldsymbol{b} \\ &\boldsymbol{O} = \boldsymbol{X}_1 \otimes \boldsymbol{Y} \end{aligned} In short, the input is split into two halves along the feature dimension, and one half is passed through an MLP-Mixer to serve as a gate for the other half. aMLP combines MLP-Mixer with a simple single-head Self-Attention to serve as the gate: \begin{aligned} &[\boldsymbol{X}_1, \boldsymbol{X}_2] = \boldsymbol{X} \\ &\boldsymbol{Y}_1 = \boldsymbol{W}\boldsymbol{X}_2 + \boldsymbol{b} \\ &\boldsymbol{Y}_2 = \text{SelfAttention}(\boldsymbol{X}) \\ &\boldsymbol{O} = \boldsymbol{X}_1 \otimes (\boldsymbol{Y}_1 + \boldsymbol{Y}_2) \end{aligned}

The paper conducts comprehensive experiments, including both CV and NLP. From the reported results, gMLP is slightly worse than standard Self-Attention, while aMLP is generally superior to Self-Attention, further affirming the value of the gating mechanism. However, whether it is gMLP or aMLP, the sense of artificial stacking is too strong. It’s fine for publishing a paper, but personally, I don’t think it brings any new inspiration to the development direction of models.