Transformer Positional Encodings that Rack Researchers' Brains

Trainable Type

Obviously, the most primitive scheme for absolute positional encoding is not to design anything specific, but to directly treat positional encodings as trainable parameters. For example, if the maximum length is 512 and the encoding dimension is 768, then a 512 \times 768 matrix is initialized as position vectors, which are updated during the training process. Current models like BERT and GPT use this type of positional encoding. In fact, it can be traced back even earlier; for instance, Facebook’s 2017 paper "Convolutional Sequence to Sequence Learning" already utilized it.

For this trainable absolute positional encoding, it is generally believed that it lacks extrapolation capabilities. That is, if the pre-training maximum length is 512, it can only handle sentences up to length 512 and nothing longer. Of course, one could randomly initialize position vectors beyond 512 and then continue fine-tuning. However, the author’s recent research indicates that through hierarchical decomposition, absolute positional encoding can be extrapolated to a sufficiently long range while maintaining decent performance. For details, please refer to the author’s previous blog post "Hierarchical Decomposition of Positional Encodings: Enabling BERT to Handle Ultra-Long Text". Therefore, extrapolation is not necessarily a distinct disadvantage of absolute positional encoding.

Sinusoidal Type

Sinusoidal positional encoding is an explicit solution proposed in Google’s paper "Attention is All You Need": \left\{\begin{aligned} &\boldsymbol{p}_{k,2i}=\sin\Big(k/10000^{2i/d}\Big)\\ &\boldsymbol{p}_{k, 2i+1}=\cos\Big(k/10000^{2i/d}\Big) \end{aligned}\right. where \boldsymbol{p}_{k,2i}, \boldsymbol{p}_{k,2i+1} are the 2i-th and (2i+1)-th components of the encoding vector for position k, and d is the dimension of the position vector.

Clearly, the characteristic of sinusoidal positional encoding is that it has an explicit generation rule, so it can be expected to have some extrapolation capability. Another reason for using it is: since \sin(\alpha+\beta)=\sin\alpha\cos\beta+\cos\alpha\sin\beta and \cos(\alpha+\beta)=\cos\alpha\cos\beta-\sin\alpha\sin\beta, this indicates that the vector at position \alpha+\beta can be represented as a combination of vectors at positions \alpha and \beta, providing the possibility of expressing relative positional information. Curiously, we rarely see work directly using this form of absolute positional encoding nowadays, for reasons unknown.

Recursive Type

In principle, RNN models do not require positional encoding; they inherently possess the possibility of learning positional information through their structure (because recursion implies we can train a "counting" model). Therefore, if an RNN layer is connected after the input and before the Transformer, positional encoding is theoretically unnecessary. Similarly, we can use an RNN model to learn an absolute positional encoding. For example, starting from a vector \boldsymbol{p}_0, we can obtain encoding vectors for each position through a recursive format \boldsymbol{p}_{k+1}=f(\boldsymbol{p}_k).

The ICML 2020 paper "Learning to Encode Position for Transformer with Continuous Dynamical Model" pushed this idea to the extreme. It proposed modeling positional encoding using an Ordinary Differential Equation (ODE) d\boldsymbol{p}_t/dt=\boldsymbol{h}(\boldsymbol{p}_t,t), a scheme called FLOATER. Obviously, FLOATER also belongs to recursive models. The function \boldsymbol{h}(\boldsymbol{p}_t,t) can be modeled by a neural network; thus, such differential equations are also known as Neural ODEs, a field that has seen increasing work recently.

Theoretically, positional encoding based on recursive models also possesses good extrapolation capabilities and offers better flexibility than sinusoidal positional encoding (for instance, it is easy to prove that sinusoidal encoding is a specific solution of FLOATER). However, recursive positional encoding clearly sacrifices some parallelism, which may lead to speed bottlenecks.

Multiplicative Type

We just mentioned that the combination of input \boldsymbol{x}_k and absolute positional encoding \boldsymbol{p}_k is generally \boldsymbol{x}_k + \boldsymbol{p}_k. Is there an "unusual" combination method? For example, \boldsymbol{x}_k \otimes \boldsymbol{p}_k (element-wise multiplication)? When we build models, there are multiple ways to fuse two vectors: addition, multiplication, or even concatenation can be considered. Why does everyone default to only considering addition when doing absolute positional encoding?

Regrettably, the author does not know the answer either. Perhaps everyone defaults to addition because vector addition has a distinct geometric meaning, but for deep learning models, this geometric meaning actually has little practical value. A recent experiment seen by the author suggests that replacing "addition" with "multiplication"—that is, \boldsymbol{x}_k \otimes \boldsymbol{p}_k—seems to yield better results than \boldsymbol{x}_k + \boldsymbol{p}_k. The author has not performed a complete comparison of the specific effects, but merely provides this as a possibility. For the source of the experiment, refer to "Research on Chinese Language Models: (1) Multiplicative Positional Encoding".

Classic Type

Relative positional encoding originated from Google’s paper "Self-Attention with Relative Position Representations". The NEZHA model open-sourced by Huawei also uses this positional encoding. Subsequent variants of relative positional encoding are basically simple modifications following this pattern.

It is generally believed that relative positional encoding was inspired by absolute positional encoding. Consider a general Attention with absolute positional encoding: \left\{\begin{aligned} \boldsymbol{q}_i =&\, (\boldsymbol{x}_i + \boldsymbol{p}_i)\boldsymbol{W}_Q \\ \boldsymbol{k}_j =&\, (\boldsymbol{x}_j + \boldsymbol{p}_j)\boldsymbol{W}_K \\ \boldsymbol{v}_j =&\, (\boldsymbol{x}_j + \boldsymbol{p}_j)\boldsymbol{W}_V \\ a_{i,j} =&\, \text{softmax}\left(\boldsymbol{q}_i \boldsymbol{k}_j^{\top}\right)\\ \boldsymbol{o}_i =&\, \sum_j a_{i,j}\boldsymbol{v}_j \end{aligned}\right. where \text{softmax} normalizes over the j dimension, and the vectors here refer to row vectors. We first expand \boldsymbol{q}_i \boldsymbol{k}_j^{\top}: \boldsymbol{q}_i \boldsymbol{k}_j^{\top} = \left(\boldsymbol{x}_i + \boldsymbol{p}_i\right)\boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\left(\boldsymbol{x}_j + \boldsymbol{p}_j\right)^{\top} = \left(\boldsymbol{x}_i \boldsymbol{W}_Q + \boldsymbol{p}_i \boldsymbol{W}_Q\right)\left(\boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{W}_K^{\top}\boldsymbol{p}_j^{\top}\right) To introduce relative positional information, Google removed the first position term and changed the second term \boldsymbol{p}_j \boldsymbol{W}_K to a binary position vector \boldsymbol{R}_{i,j}^{K}, resulting in: a_{i,j} = \text{softmax}\left(\boldsymbol{x}_i \boldsymbol{W}_Q\left(\boldsymbol{x}_j\boldsymbol{W}_K + \textcolor{green}{\boldsymbol{R}_{i,j}^K}\right)^{\top}\right) And in \boldsymbol{o}_i =\sum\limits_j a_{i,j}\boldsymbol{v}_j = \sum\limits_j a_{i,j}(\boldsymbol{x}_j\boldsymbol{W}_V + \boldsymbol{p}_j\boldsymbol{W}_V), \boldsymbol{p}_j \boldsymbol{W}_V is replaced by \boldsymbol{R}_{i,j}^{V}: \boldsymbol{o}_i = \sum_j a_{i,j}\left(\boldsymbol{x}_j\boldsymbol{W}_V + \textcolor{green}{\boldsymbol{R}_{i,j}^{V}}\right) The so-called relative position involves changing the vectors \boldsymbol{R}_{i,j}^{K}, \boldsymbol{R}_{i,j}^{V}, which originally depended on binary coordinates (i,j), to depend only on the relative distance i-j, and usually, they are clipped to adapt to arbitrary distances: \begin{aligned} \boldsymbol{R}_{i,j}^{K} = \boldsymbol{p}_K\left[\text{clip}(i-j, p_{\min}, p_{\max})\right]\\ \boldsymbol{R}_{i,j}^{V} = \boldsymbol{p}_V\left[\text{clip}(i-j, p_{\min}, p_{\max})\right] \end{aligned} \label{eq:rp-clip} In this way, only a finite number of positional encodings are needed to express relative positions of any length (due to clipping). Whether \boldsymbol{p}_K, \boldsymbol{p}_V are chosen to be trainable or sinusoidal, they can meet the requirement of processing text of any length.

XLNET Type

XLNET-style positional encoding actually originates from the Transformer-XL paper "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context". It was only after the XLNET model, which used the Transformer-XL architecture and surpassed BERT to some extent, that Transformer-XL became widely known. Therefore, this positional encoding is often attributed to XLNET.

XLNET-style positional encoding stems from the complete expansion of \boldsymbol{q}_i \boldsymbol{k}_j^{\top} mentioned above: \boldsymbol{q}_i \boldsymbol{k}_j^{\top} = \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{p}_j^{\top} + \boldsymbol{p}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{p}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{p}_j^{\top} \label{eq:qk-exp} Transformer-XL’s approach is simple: directly replace \boldsymbol{p}_j with a relative position vector \boldsymbol{R}_{i-j}, and as for the two \boldsymbol{p}_i terms, they are replaced by two trainable vectors \boldsymbol{u} and \boldsymbol{v}: \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\textcolor{green}{\boldsymbol{R}_{i-j}^{\top}} + \textcolor{red}{\boldsymbol{u}}\boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \textcolor{red}{\boldsymbol{v}} \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\textcolor{green}{\boldsymbol{R}_{i-j}^{\top}} In this encoding method, \boldsymbol{R}_{i-j} is not clipped like in Equation [eq:rp-clip], but directly uses the sinusoidal generation scheme. Since the encoding space of \boldsymbol{R}_{i-j} is not necessarily the same as \boldsymbol{x}_j, the \boldsymbol{W}_K^{\top} in front of \boldsymbol{R}_{i-j} is replaced by another independent matrix \boldsymbol{W}_{K,R}^{\top}. Also, \textcolor{red}{\boldsymbol{u}}\boldsymbol{W}_Q and \textcolor{red}{\boldsymbol{v}} \boldsymbol{W}_Q can be directly merged into single vectors \textcolor{red}{\boldsymbol{u}} and \textcolor{red}{\boldsymbol{v}}. Thus, the final formula used is: \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_{K,R}^{\top}\textcolor{green}{\boldsymbol{R}_{i-j}^{\top}} + \textcolor{red}{\boldsymbol{u}}\boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \textcolor{red}{\boldsymbol{v}} \boldsymbol{W}_{K,R}^{\top}\textcolor{green}{\boldsymbol{R}_{i-j}^{\top}} Furthermore, the positional bias on \boldsymbol{v}_j is simply removed, i.e., \boldsymbol{o}_i = \sum\limits_j a_{i,j}\boldsymbol{x}_j\boldsymbol{W}_V. It seems that starting from this work, subsequent relative positional encodings are only added to the Attention matrix and no longer to \boldsymbol{v}_j.

T5 Type

The T5 model comes from the article "Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer", which uses an even simpler relative positional encoding. The idea still stems from the expansion in Equation [eq:qk-exp]. If we must analyze the meaning of each term, they can be understood as combinations of four types of attention: "input-input", "input-position", "position-input", and "position-position". If we believe that input information and positional information should be independent (decoupled), then they should not have excessive interaction. Thus, the "input-position" and "position-input" attention terms can be deleted. Meanwhile, \boldsymbol{p}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{p}_j^{\top} is actually a scalar that depends only on (i,j), which we can directly train as a parameter, simplifying it to: \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \textcolor{green}{\boldsymbol{\beta}_{i,j}} To put it plainly, it merely adds a trainable bias term to the Attention matrix. Like the XLNET style, the positional bias on \boldsymbol{v}_j is removed. A similar idea is found in the TUPE positional encoding proposed by Microsoft in the ICLR 2021 paper "Rethinking Positional Encoding in Language Pre-training".

What is quite "unique" is that, unlike conventional positional encoding which treats \boldsymbol{\beta}_{i,j} as a function of i-j and clips it, T5 performs a "bucketing" process on relative positions. That is, a relative position of i-j actually corresponds to position f(i-j). The mapping relationship is as follows:

For the specific mapping code, readers can check the source code themselves. The idea behind this design is also very intuitive: for nearby positions (0–7), we need to be more precise, so each is assigned an independent positional encoding. For slightly further positions (e.g., 8–11), we don’t need to distinguish them as clearly, so they can share a positional encoding. The further the distance, the larger the sharing range, until it reaches a specified range and is clipped.

DeBERTa Type

DeBERTa was also developed by Microsoft and released last June. The paper is "DeBERTa: Decoding-enhanced BERT with Disentangled Attention". It has recently gained some popularity, partly because it was officially accepted into ICLR 2021, and partly because it topped the SuperGLUE leaderboard, slightly outperforming T5.

In fact, DeBERTa’s main improvement is also in positional encoding. Again, starting from the expansion in Equation [eq:qk-exp], T5 simply removed the 2nd and 3rd terms, keeping only the 4th term and replacing it with relative positional encoding. DeBERTa does the exact opposite: it throws away the 4th term, keeps the 2nd and 3rd terms, and replaces them with relative positional encoding (as it turns out, research is about enumerating all permutations and combinations to see which is optimal): \boldsymbol{q}_i \boldsymbol{k}_j^{\top} = \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} + \boldsymbol{x}_i \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\textcolor{green}{\boldsymbol{R}_{i,j}^{\top}} + \textcolor{green}{\boldsymbol{R}_{j,i}} \boldsymbol{W}_Q \boldsymbol{W}_K^{\top}\boldsymbol{x}_j^{\top} As for the design of \boldsymbol{R}_{i,j}, it is clipped like in Equation [eq:rp-clip], with nothing particularly special.

However, the interesting part of DeBERTa is that it provides a new perspective on using relative and absolute positional encodings. It points out that most NLP tasks may only need relative positional information, but in certain scenarios, absolute positional information is more helpful. Thus, it understands the entire model in two parts. Taking the Base version of the MLM pre-training model as an example, it has 13 layers in total. The first 11 layers use only relative positional encoding; this part is called the Encoder. The last 2 layers add absolute positional information; this part is called the Decoder, also abbreviated as EMD (Enhanced Mask Decoder). For fine-tuning downstream tasks, the first 11 layers of the Encoder plus 1 layer of the Decoder are used.

The performance on SuperGLUE certainly affirms the value of DeBERTa, but the various naming conventions in the paper are truly uncomfortable. For instance, calling parts "Encoder" and "Decoder" easily leads to the misunderstanding that it is a Seq2Seq model. The abbreviation EMD also shares its name with Earth Mover’s Distance. While name collisions are sometimes unavoidable, colliding with objects well-known in the ML community is quite confusing. I wonder what the authors were thinking...

CNN Type

Although the classic work using CNNs for NLP, "Convolutional Sequence to Sequence Learning", added positional encoding, we know that general CNN models, especially those in computer vision, do not have additional positional encodings. So how does a CNN model capture positional information?

If the author were to answer, the answer might be that the anisotropy of the convolution kernel allows it to distinguish relative positions in different directions. However, the ICLR 2020 paper "How Much Position Information Do Convolutional Neural Networks Encode?" gives a perhaps surprising answer: Positional information in CNN models is leaked by Zero Padding!

We know that to maintain the size of features during the convolutional encoding process, we usually pad the input with zeros. This paper shows that this operation enables the model to identify positional information. That is to say, while the anisotropy of the convolution kernel is important, the existence of zero padding is fundamental. One can imagine that what is actually extracted is the relative distance between the current position and the padding boundary.

However, this capability depends on the locality of CNNs. Global structures without priors like Attention are not applicable. For readers only interested in Transformer positional encoding schemes, this can be considered an expansion of horizons.

Complex Type

Complex-valued positional encoding is perhaps the most idiosyncratic positional encoding scheme. It comes from the ICLR 2020 paper "Encoding word order in complex embeddings". The main idea of the paper is to combine the properties of complex numbers with some basic principles to derive its positional encoding form (Complex Order) as: \left[r_{j, 1} e^{\text{i}\left(\omega_{j, 1} k+\theta_{j, 1}\right)}, \ldots, r_{j, 2} e^{\text{i}\left(\omega_{j, 2} k+\theta_{j, 2}\right)}, \cdots, r_{j, d} e^{\text{i}\left(\omega_{j, d} k+\theta_{j, d}\right)}\right] \label{eq:complex} Here \text{i} is the imaginary unit, j represents a certain word, k represents the position of that word, and \begin{aligned} \boldsymbol{r}_j =&\, [r_{j, 1},r_{j, 2},\cdots,r_{j, d}]\\ \boldsymbol{\omega}_j =&\, [\omega_{j, 1},\omega_{j, 2},\cdots,\omega_{j, d}]\\ \boldsymbol{\theta}_j =&\, [\theta_{j, 1},\theta_{j, 2},\cdots,\theta_{j, d}]\\ \end{aligned} represent three sets of word vectors for word j. You read that right; it indeed assumes each word has three sets of position-independent word vectors (which can, of course, be parameter-shared in some form to degenerate into two or even one set), and then the word vector related to position k is calculated according to the formula above.

You think introducing multiple sets of word vectors is its most idiosyncratic part? No! We see that Equation [eq:complex] is still in complex form. Guess what it does next? Realize it? No, it directly uses it in a complex-valued model! That is to say, it follows a complex-valued model route. Not only is the input Embedding layer complex, but every layer of the Transformer inside is complex. It also implemented and compared complex-valued versions of Fasttext, LSTM, CNN, and other models! The first author of this article is Benyou Wang. You can find that his related work basically revolves around complex-valued models; he is truly a die-hard fan of complex-valued models.

Fusion Type

Coincidentally, using the form of complex numbers, the author also conceived a clever positional encoding that can fuse absolute and relative positional encoding into one. It is shared here for interested readers to exchange and study.

For simplicity, let’s assume \boldsymbol{q}_m, \boldsymbol{k}_n are two-dimensional row vectors at positions m and n. Since they are two-dimensional, we can treat them as complex numbers for calculation. We know that the key to Attention lies in the inner product of vectors, which in complex representation is: \langle \boldsymbol{q}_m, \boldsymbol{k}_n\rangle = \text{Re}\left[\boldsymbol{q}_m \boldsymbol{k}_n^*\right] where ^* denotes the complex conjugate, the multiplication on the right is ordinary complex multiplication, and \text{Re}[] denotes taking the real part of the result. The above equation means:

The inner product of two two-dimensional vectors is equal to the real part of the product of one complex number and the conjugate of the other, when they are viewed as complex numbers.

If we multiply \boldsymbol{q}_m, \boldsymbol{k}_n by e^{\text{i}m\theta}, e^{\text{i}n\theta} respectively to become \boldsymbol{q}_m e^{\text{i}m\theta}, \boldsymbol{k}_n e^{\text{i}n\theta}, then it is equivalent to equipping them with absolute positional encoding (because they explicitly depend on absolute positions m, n). Putting them into the inner product, we have: \langle \boldsymbol{q}_m e^{\text{i}m\theta}, \boldsymbol{k}_n e^{\text{i}n\theta}\rangle = \text{Re}\left[\left(\boldsymbol{q}_m e^{\text{i}m\theta}\right) \left(\boldsymbol{k}_n e^{\text{i}n\theta}\right)^*\right] = \text{Re}\left[\boldsymbol{q}_m \boldsymbol{k}_n^* e^{\text{i}(m-n)\theta}\right] Quite interestingly, the inner product depends only on the relative position m-n! This cleverly fuses absolute and relative positions together.

Note that we are not as "crazy" as Complex Order; the above calculation is essentially within the realm of real numbers, we just used complex numbers to complete some derivations. From the above result, for a two-dimensional real vector [x, y] at position n, we treat it as a complex number and multiply it by e^{\text{i}n\theta}, obtaining the identity: (x + y\text{i})e^{\text{i}n\theta} = (x \cos n\theta - y\sin n\theta) + \text{i} (x \sin n\theta + y\cos n\theta) This means that by: \begin{pmatrix}x \\ y\end{pmatrix} \to \begin{pmatrix}x \cos n\theta - y\sin n\theta \\ x \sin n\theta + y\cos n\theta \end{pmatrix} = \begin{pmatrix}x \\ y \end{pmatrix}\cos n\theta + \begin{pmatrix}-y \\ x \end{pmatrix}\sin n\theta to endow [x, y] with absolute positional information, it is equivalent to relative positional encoding during the Attention calculation. For vectors with more than two dimensions, one can consider performing the same operation on every group of two dimensions, where \theta for each group can be different.

In this way, we obtain a positional encoding scheme that fuses absolute and relative positions. Formally, it looks a bit like multiplicative absolute positional encoding. By implementing this positional encoding in \boldsymbol{q} and \boldsymbol{k}, the effect is equivalent to relative positional encoding. If explicit absolute positional information is still needed, this positional encoding can also be applied to \boldsymbol{v} simultaneously. Overall, through absolute position operations, we can achieve the effects of both absolute and relative positions. Preliminary experiments show that it can work, but it has not been fully verified. Everyone is welcome to try and exchange ideas.