GELU, which stands for Gaussian Error Linear Unit, can be considered a variant of ReLU. It is an activation function in the form of a non-elementary function. It was proposed in the paper "Gaussian Error Linear Units (GELUs)", later adopted in GPT, then in BERT, and subsequently used in many other pre-trained language models. With the rise of pre-trained models like BERT, GELU has gained significant popularity and has unexpectedly become a mainstream activation function.
In the original GELU paper, the authors provided not only the exact form of GELU but also two approximation forms using elementary functions. This article discusses how these approximations were derived.
The GELU Function
The form of the GELU function is: \text{GELU}(x) = x \Phi(x) where \Phi(x) is the cumulative distribution function (CDF) of the standard normal distribution: \Phi(x) = \int_{-\infty}^x \frac{e^{-t^2/2}}{\sqrt{2\pi}}dt = \frac{1}{2}\left[1 + \text{erf}\left(\frac{x}{\sqrt{2}}\right)\right] Here, \text{erf}(x) = \frac{2}{\sqrt{\pi}}\int_0^x e^{-t^2}dt. The original paper mentions two approximations: x\Phi(x) \approx x\sigma(1.702 x) \label{eq:x-sigma} and x\Phi(x) \approx \frac{1}{2} x \left[1 + \tanh\left(\sqrt{\frac{2}{\pi}}\left(x + 0.044715 x^3\right)\right)\right] \label{eq:x-phi} Currently, many implementations of Transformer architectures still use the approximation in Eq. [eq:x-phi] as the implementation of the GELU function. However, since many frameworks now include accurate \text{erf} calculation functions, the value of elementary function approximations may not be as significant as it once was. Thus, one might treat this as an exercise in mathematical analysis.
What to Use for Approximation?
Clearly, finding an approximation for GELU is equivalent to finding an approximation for \Phi(x), which in turn is equivalent to finding an approximation for \text{erf}\left(\frac{x}{\sqrt{2}}\right).
First, we must address the question: what function should we use for the approximation? From the plot of \text{erf}(x), we can observe its characteristics:
It is an odd function, i.e., \text{erf}(x) = -\text{erf}(-x);
It is monotonically increasing, with \lim_{x\to -\infty}\text{erf}(x) = -1 and \lim_{x\to +\infty}\text{erf}(x) = 1.
We have many odd functions, such as x^{2n+1}, \sin x, \tan x, \tanh x, etc. Furthermore, the superposition and composition of odd functions remain odd functions, such as \sin(x + x^3). For a function that is odd, monotonically increasing, and bounded, the most intuitive choice is \tanh x. In fact, \tanh x is very similar to \text{erf}(x).
Therefore, we can start from \tanh x and construct potential fitting forms, such as: \left\{\begin{aligned} &\tanh\left(a x + b x^3 + c x^5\right)\\ &a\tanh x + b \tanh^3 x + c \tanh^5 x\\ &a\tanh bx + c \tanh dx + e \tanh fx\\ &\vdots \end{aligned}\right.
How to Approximate?
Once we have the candidate forms, the next consideration is how to perform the fitting and what criteria to use. Simply put, we need a way to solve for the coefficients. Generally, there are two approaches: local fitting and global fitting.
Local Fitting
Local fitting is based on Taylor expansion. For example, considering the approximation form \tanh(a x + b x^3), we expand it at x=0: \text{erf}\left(\frac{x}{\sqrt{2}}\right) - \tanh\left(a x + b x^3\right) = \left(\sqrt{\frac{2}{\pi}} - a\right) x + \left(\frac{a^3}{3} - b - \frac{1}{3 \sqrt{2 \pi}}\right)x^3 + \dots Setting the first two terms to zero gives us two equations. Solving them yields: a = \sqrt{\frac{2}{\pi}}, \quad b = \frac{4-\pi}{3 \sqrt{2} \pi^{3/2}} Substituting these into x\Phi(x) and converting to numerical form, we get: x\Phi(x) \approx \frac{1}{2} x\left[1 + \tanh\left(\sqrt{\frac{2}{\pi}}\left(x + 0.0455399 x^3\right)\right)\right] \label{eq:x-phi-local}
Global Fitting
Equation [eq:x-phi-local] is already very close to Eq. [eq:x-phi], but the second coefficient is slightly off. This is because Eq. [eq:x-phi-local] is purely a result of local approximation. As the name suggests, local approximation is very accurate locally (in this case, near x=0), but the error increases as we move further away from 0. Therefore, we also need to consider the global error.
A common global error metric is the integral form. For instance, when using g(x, \theta) to approximate f(x), we could calculate: \min_{\theta} \int [f(x)-g(x,\theta)]^2 dx \quad \text{or} \quad \min_{\theta} \int |f(x)-g(x,\theta)| dx However, the importance of the error at each x might differ. To maintain generality, one might multiply by a weight \lambda(x): \min_{\theta} \int \lambda(x)[f(x)-g(x,\theta)]^2 dx \quad \text{or} \quad \min_{\theta} \int \lambda(x)|f(x)-g(x,\theta)| dx Different choices of \lambda(x) lead to different solutions, and choosing the most suitable \lambda(x) is not trivial.
Instead of optimizing an integral error, we can optimize a more intuitive min-max error: \min_{\theta} \max_x |f(x)-g(x,\theta)| This objective is easy to understand: "find an appropriate \theta such that the maximum possible error |f(x)-g(x,\theta)| is as small as possible." This goal aligns with our intuitive understanding and avoids the selection of weights.
Hybrid Fitting
Based on this idea, we fix a =
\sqrt{\frac{2}{\pi}} and re-solve for \tanh(a x + b x^3). We fix a because it represents the first-order local
approximation; we want to preserve some local accuracy while hoping
b can help reduce the global error as
much as possible, thereby achieving a hybrid of local and global
approximation. Thus, we want to solve: \min_{b} \max_x
\left|\text{erf}\left(\frac{x}{\sqrt{2}}\right) - \tanh\left(a x + b
x^3\right)\right| This can be easily solved using
scipy:
import numpy as np
from scipy.special import erf
from scipy.optimize import minimize
def f(x, b):
a = np.sqrt(2 / np.pi)
return np.abs(erf(x / np.sqrt(2)) - np.tanh(a * x + b * x**3))
def g(b):
return np.max([f(x, b) for x in np.arange(0, 4, 0.001)])
options = {'xtol': 1e-10, 'ftol': 1e-10, 'maxiter': 100000}
result = minimize(g, 0, method='Powell', options=options)
print(result.x)The result is b = 0.035677337314877385, which corresponds to the form: x\Phi(x) \approx \frac{1}{2} x\left[1 + \tanh\left(\sqrt{\frac{2}{\pi}}\left(x + 0.04471491123850965 x^3\right)\right)\right] \label{eq:x-phi-global} While there might be slight differences in the last few significant digits, the leading part perfectly matches Eq. [eq:x-phi]. As a side note, Eq. [eq:x-phi] was proposed in the paper "Approximations to the Cumulative Normal Function and its Inverse for Use on a Pocket Calculator", which is a result from over 40 years ago.
As for the first approximation, it comes from the paper "A logistic approximation to the cumulative normal distribution". It is the result of a global approximation of \Phi(x) using \sigma(\lambda x): \min_{\lambda} \max_{x} \left|\Phi(x) - \sigma(\lambda x)\right| Solving this yields \lambda = 1.7017449256323682, i.e., \Phi(x) \approx \sigma(1.7017449256323682 x) This is also very consistent with Eq. [eq:x-sigma].
Summary
In this article, we worked through a mathematical analysis problem together—introducing the GELU activation function and attempting to explore the origins of its two approximation forms.
Original article address: https://kexue.fm/archives/7309
For more details regarding reprinting, please refer to: "Scientific Space FAQ"