[Memo] On Dropout

What Does It Do?

How does dropout operate? Generally speaking, for an input tensor x, dropout sets some elements to zero and then performs a scaling transformation on the result. Specifically, taking Keras’s Dropout(0.6)(x) as an example, it is essentially equivalent to the following operation performed in NumPy:

import numpy as np

x = np.random.random((10,100)) # Simulating an input with batch_size=10 and dimension=100
def Dropout(x, drop_proba):
    return x * np.random.choice(
                              [0, 1], 
                              x.shape,  
                              p=[drop_proba, 1 - drop_proba]
                             ) / (1. - drop_proba)

print(Dropout(x, 0.6))

In other words, 60% of the elements are set to 0, and the remaining 40% are scaled up to 1/40\% = 2.5 times their original value. It is worth noting that in Keras, the 0.6 in Dropout(0.6)(x) represents the dropout rate (the proportion to discard), whereas in TensorFlow, the 0.6 in tf.nn.dropout(x, 0.6) represents the keep probability (the proportion to retain). The specific meaning depends on the framework being used (of course, if the dropout rate is 0.5, you don’t need to worry about the distinction ^_^).

What Is It For?

Dropout is generally understood as "a low-cost ensemble strategy," which is correct. The specific process can be understood as follows:

After the zeroing operation described above, we can consider the zeroed-out portion as being discarded, resulting in a loss of information. However, even though information is lost, life must go on—or rather, training must continue. This forces the model to use the remaining information to fit the target. Since each dropout mask is random, the model cannot rely too heavily on specific nodes. Therefore, the overall effect is that every time the model is forced to learn using a small subset of features, and since the features being learned change every time, every feature should contribute to the model’s prediction (rather than focusing on a subset of features, which leads to overfitting).

Finally, during prediction, dropout is not applied. This is equivalent to averaging all local features (finally using all the information), which theoretically improves performance and reduces overfitting because the risk is spread across every feature rather than being concentrated on a few.

Being More Flexible

Sometimes we want to impose constraints on how dropout is applied. For example:

• I want all samples within the same batch to be dropped out in the same way, rather than the first sample dropping the first node and the second sample dropping the second node.

• When using LSTM for text classification, I want to dropout according to word vectors—meaning either the entire word vector is dropped or the entire word vector is kept, rather than dropping only certain dimensions of a word vector.

• When using CNN for RGB image classification, I want to dropout by channel—dropping any one of the R, G, or B channels for each image (similar to color transformation or RGB perturbation), rather than dropping individual pixels (which is equivalent to adding noise to the image).

To implement these requirements, one needs to use the noise_shape parameter. This parameter exists in both the Keras Dropout layer and tf.nn.dropout (the two frameworks I am familiar with), and its meaning is the same in both. However, this parameter is rarely mentioned online, and even when it is, the explanation is often vague.

Taking tf.nn.dropout(x, 0.5, noise_shape) as an example: first, noise_shape is a one-dimensional tensor (essentially a 1D array, list, or tuple) whose length must be the same as x.shape. Furthermore, the elements in noise_shape can only be either 1 or the corresponding dimension size in x.shape. For example, if x.shape = (3, 4, 5), then there are only 8 allowed configurations for noise_shape:

(3, 4, 5), (1, 4, 5), (3, 1, 5), (3, 4, 1), (1, 1, 5), (1, 4, 1), (3, 1, 1), (1, 1, 1)

What does each one mean? It can be understood this way: Whichever axis is set to 1, that axis will be dropped consistently. For instance, (3, 4, 5) is standard dropout with no constraints. (1, 4, 5) means every sample in the batch will be dropped in the same way (you can think of it as the same noise being added to every sample). If (3, 4, 5) represents (number of sentences, words per sentence, word vector dimension), then (3, 4, 1) means dropping out by word vectors (which can be understood as randomly skipping certain words).

For some readers, understanding this through NumPy code might be more intuitive:

def Dropout(x, drop_proba, noise_shape):
    return x * np.random.choice(
                              [0, 1], 
                              noise_shape, 
                              p=[drop_proba, 1 - drop_proba]
                             ) / (1. - drop_proba)

Please include the original link when reposting: https://kexue.fm/archives/4521

For more details on reposting, please refer to: "Scientific Space FAQ"