Activation functions are the source of non-linearity in neural networks. If these functions were removed, the entire network would consist only of linear operations. Since the composition of linear operations is still linear, the final effect would be equivalent to a single-layer linear model.
So, what are the common activation functions? Or rather, what are the guiding principles for choosing an activation function? Can any non-linear function serve as an activation function?
The activation functions explored here are those for the hidden layers, not the output layer. The final output generally requires specific activation functions that cannot be changed arbitrarily. For example, binary classification typically uses the sigmoid function, and multi-class classification typically uses softmax. In contrast, there is more room for choice regarding the activation functions of the hidden layers.
Even Floating-Point Errors Work!
Theoretically, any non-linear function has the potential to be an
activation function. A very convincing example is OpenAI’s recent
success in utilizing floating-point errors as activation functions. For
details, please read the OpenAI blog:
https://blog.openai.com/nonlinear-computation-in-linear-networks/
Or read the introduction by "Ji Qi Zhi Xin" (Heart of the
Machine):
https://mp.weixin.qq.com/s/PBRzS4Ol_Zst35XKrEpxdw
Nevertheless, different activation functions have different training costs. Although OpenAI’s exploration shows that even floating-point errors can serve as activation functions, the non-differentiability of this operation led them to use "Evolutionary Strategies" to train the model. These so-called "Evolutionary Strategies" are time-consuming and labor-intensive algorithms, such as genetic algorithms.
The Precedent Set by ReLU
If we add differentiability so that we can train using gradient descent, is everything fine? Not necessarily. When neural networks were first invented, the Sigmoid function was generally used as the activation function: \text{sigmoid}(x) = \sigma(x) = \frac{1}{1+e^{-x}} The characteristic of this function is that the left end approaches 0 and the right end approaches 1. Both ends are saturated, as shown below:
Because of this, its derivatives at both ends approach 0. Since we optimize using gradient descent, a derivative approaching zero means the update amount each time is very small (proportional to the gradient), making updates difficult. Especially as the number of layers increases, due to the chain rule of differentiation, the update amount becomes proportional to the n-th power of the gradient, making optimization even more difficult. Consequently, early neural networks could not be made very deep.
A landmark activation function is the ReLU function, which has a very simple definition: \text{relu}(x) = \max(x, 0) Its graph is:
This is a piecewise linear function. Its derivative is 1 on the positive axis and 0 on the negative axis. Thus, half of the space across the entire real domain is unsaturated. In contrast, the sigmoid function is saturated in almost all regions (the proportion of the saturated interval approaches 1; saturation is defined as the derivative being very close to 0).
ReLU is a piecewise linear function, and its non-linearity is weak; therefore, networks generally need to be very deep. However, this happens to meet our needs because, given the same performance, depth is often more important than width—deeper models tend to have better generalization capabilities. Since the introduction of the ReLU activation function, various very deep models have been proposed. A landmark event was likely the VGG model and its success on ImageNet; I will not go into detail about subsequent developments.
The Better Swish
Despite ReLU’s brilliant track record, some felt that having half of the region saturated was a significant drawback. Consequently, variants like LeakyReLU and PReLU were proposed, though these changes are largely similar.
A few days ago, the Google Brain team proposed a new activation
function called Swish. News about it can be found here:
http://mp.weixin.qq.com/s/JticD0itOWH7Aq7ye1yzvg
Its definition is: \text{swish}(x) = x \cdot \sigma(x) = \frac{x}{1+e^{-x}} Its graph is as follows:
The team’s test results indicate that this function outperforms ReLU in many models.
From the graph, the Swish function is similar to ReLU, with the only major difference being the negative axis region close to 0. In hindsight, even I have considered this activation function because it is similar to the GLU (Gated Linear Unit) activation function proposed by Facebook. The GLU activation function is: (\boldsymbol{W}_1\boldsymbol{x}+\boldsymbol{b}_1) \otimes \sigma(\boldsymbol{W}_2\boldsymbol{x}+\boldsymbol{b}_2) In other words, two sets of parameters are trained separately; one set is activated by sigmoid and then multiplied by the other set. Here, \sigma(\boldsymbol{W}_2\boldsymbol{x}+\boldsymbol{b}_2) is called the "gate," which is what the "G" in GLU stands for. The Swish function is equivalent to taking the same values for both sets of parameters and training only one set.
Ideas for Improvement
The Swish function has sparked some controversy. Some believe Google Brain is making a mountain out of a molehill—that a small team could play around with improving an activation function, while large teams like Google Brain should focus on more high-end directions. Regardless, Google Brain conducted many experiments, and the results all showed that Swish is superior to ReLU. So, we need to think: what is the reason behind this?
The following analysis is purely my subjective speculation and currently lacks theoretical or experimental proof; please read with discretion. I believe a very important reason why Swish is superior to ReLU is related to initialization.
Swish is not saturated near the origin; it is only saturated in the region of the negative axis far from the origin. In contrast, ReLU has half of its space saturated even near the origin. When we train models, we generally use uniform or normal distribution initialization. Regardless of the type, the mean is generally 0. This means that half of the initialized parameters fall into the saturated region of ReLU, causing half of the parameters to be unused at the very beginning. Especially due to strategies like BN (Batch Normalization), outputs automatically approximate a normal distribution with a mean of 0, so in these cases, half of the parameters are located in the ReLU saturation zone. In comparison, Swish is better because it has a certain unsaturated zone on the negative axis, leading to higher parameter utilization.
As mentioned earlier, even I had thought about the Swish activation function but did not study it deeply. One reason was that it wasn’t concise or beautiful enough—I even thought it was a bit ugly. Seeing that Swish’s experimental results are so good, I wondered if there are similar, more attractive activation functions. I thought of one: x \cdot \min(1, e^x) Its graph is:
Actually, it looks similar to Swish. The idea was roughly: maintain x on the positive axis, and for the negative axis, think of a function that first decreases, then increases, and approaches 0. I thought of xe^{-x}, and with a slight adjustment, I got this function. In some of my models, its effect is even better than Swish (specifically in my Q&A models). Of course, I only did a few experiments, as I don’t have the energy or computing power to conduct extensive comparative experiments.
Comparison with Swish (orange is Swish):
It should be noted that if you want to use this function, you cannot write it directly in this form because the calculation of e^x may overflow. A version that will not overflow is: \max(x, x \cdot e^{-|x|}) Or, using the ReLU function, it can be written as: x + \text{relu}(x \cdot e^{-|x|} - x)
Could it be that all effective activation functions look like the check marks (\checkmark) on our homework?
When reposting, please include the original address of this article: https://kexue.fm/archives/4647
For more detailed reposting matters, please refer to: "Scientific Space FAQ"