The ReLU function, defined as \max(x,0), is one of the most common activation functions. However, its non-differentiability at x=0 is often considered a "drawback." Consequently, many smooth approximations have been proposed, such as SoftPlus, GeLU, and Swish. However, without exception, these smooth approximations use at least the exponential operation e^x (SoftPlus even uses logarithms). From a "computational efficiency" perspective, the computational cost is not negligible (although with current GPU acceleration, we rarely perceive this cost). Recently, a paper titled "Squareplus: A Softplus-Like Algebraic Rectifier" proposed a simpler approximation called SquarePlus. Let’s discuss it here.
It should be pointed out in advance that I do not recommend spending too much time on the selection and design of activation functions. Therefore, while I am sharing this paper, it is mainly to provide a reference result and serve as an exercise for everyone to "practice their skills."
Definition
The form of SquarePlus is very simple, using only addition, multiplication, division, and square root: \text{SquarePlus}(x) = \frac{x + \sqrt{x^2 + b}}{2} where b > 0. When b=0, it degenerates exactly into \text{ReLU}(x) = \max(x,0). The inspiration for SquarePlus roughly comes from: \max(x,0) = \frac{x + |x|}{2} = \frac{x + \sqrt{x^2}}{2} Therefore, to provide differentiability at x=0, a constant b > 0 is added inside the square root (to prevent division-by-zero issues in the derivative).
The original paper points out that since it only uses addition, multiplication, division, and square root, the speed of SquarePlus (primarily on CPUs) is faster than functions like SoftPlus:
Of course, if you don’t care about this slight speed improvement, then as mentioned at the beginning of this article, just treat it as a mathematical exercise.
Properties
Like the SoftPlus function (\log(e^x + 1)), SquarePlus is globally monotonically increasing and is always greater than ReLU, as shown in the figure below (where b=1 for SquarePlus):
The monotonicity can also be seen by directly calculating its derivative: \frac{d}{dx}\text{SquarePlus}(x) = \frac{1}{2}\left(1 + \frac{x}{\sqrt{x^2 + b}}\right) > 0 As for the second derivative: \frac{d^2}{dx^2}\text{SquarePlus}(x) = \frac{b}{2(x^2 + b)^{3/2}} which is also always greater than 0, meaning SquarePlus is a convex function.
Approximation
Now there are two exercises to work on:
1. For what value of b is SquarePlus always greater than SoftPlus?
2. For what value of b is the error between SquarePlus and SoftPlus minimized?
For the first question, solving \text{SquarePlus}(x) \geq \text{SoftPlus}(x) directly yields: b \geq 4\log(e^x + 1)\left[\log(e^x + 1) - x\right] = 4\log(e^x + 1)\log(e^{-x} + 1) For this to hold globally, b must be greater than or equal to the maximum value of the right-hand side. We can prove that the maximum value of the right-hand side occurs at x=0, so b \geq 4\log^2 2 = 1.921812\dots. Thus, the first question is solved.
Proof: Note that \frac{d^2}{dx^2}\log\log(e^x + 1) = \frac{e^x(\log(e^x + 1) - e^x)}{(e^x + 1)^2\log^2(e^x + 1)} < 0 So \log\log(e^x + 1) is a concave function. Then, by Jensen’s inequality: \frac{1}{2}\left(\log\log(e^x + 1) + \log\log(e^{-x} + 1)\right) \leq \log\log(e^{(x + (-x))/2} + 1) = \log\log 2 Which means \log\left(\log(e^x + 1)\log(e^{-x} + 1)\right) \leq 2\log\log 2, or \log(e^x + 1)\log(e^{-x} + 1) \leq \log^2 2. Multiplying both sides by 4 gives the desired conclusion. The equality holds when x = -x, i.e., x = 0.
As for the second question, we need a criterion for "error." Similar to the previous article "How the Two Elementary Function Approximations of GELU Were Derived", we transform this into a \min-\max problem without additional parameters: \min_{b} \max_x \left| \frac{x + \sqrt{x^2 + b}}{2} - \log(e^x + 1) \right| I cannot obtain an analytical solution for this problem; currently, it can only be solved numerically:
import numpy as np
from scipy.special import erf
from scipy.optimize import minimize
def f(x, a):
return np.abs((x + np.sqrt(x**2 + a**2)) / 2 - np.log(np.exp(x) + 1))
def g(a):
return np.max([f(x, a) for x in np.arange(-2, 4, 0.0001)])
options = {'xtol': 1e-10, 'ftol': 1e-10, 'maxiter': 100000}
result = minimize(g, 0, method='Powell', options=options)
b = result.x**2
print(b)The final calculated result is b = 1.52382103\dots, with a maximum error of 0.075931\dots. The comparison is as follows:
Summary
There doesn’t seem to be much to summarize; I’ve just introduced a smooth approximation of ReLU and provided two simple function exercises to go with it.
When reposting, please include the original address of this article: https://kexue.fm/archives/8833
For more detailed reposting matters, please refer to: "Scientific Space FAQ"