In supervised machine learning, we often talk about the training set (train), validation set (validation), and test set (test). The distinction between these three sets can be confusing; in particular, some readers are unclear about the difference between the validation set and the test set.
Partitioning
If we already have a large labeled dataset and want to complete a test of a supervised model, we typically use uniform random sampling to divide the dataset into a training set, a validation set, and a test set. These three sets must not overlap. A common ratio is 8:1:1, though the ratio is arbitrary. From this perspective, all three sets are from the same distribution.
If participating in a competition where the organizers provide only one labeled dataset (as the training set) and an unlabeled test set, we usually manually partition a validation set from the provided training set. In this case, we generally do not partition an additional test set for two possible reasons: 1. Competition organizers are usually quite stingy, and the samples in the training set are already scarce; 2. We cannot guarantee whether the test set to be submitted follows exactly the same distribution as the training set, so partitioning another test set that follows the training distribution becomes less meaningful.
Parameters
Once a model is established, the training set is used to train the parameters. To be more precise, it is generally used for gradient descent. The validation set is basically used to test the current model’s accuracy after each epoch is completed. Since the validation set has no overlap with the training set, this accuracy is reliable. So why do we still need a test set?
This requires a distinction between the various types of parameters in a model. In fact, for a model, parameters can be divided into ordinary parameters and hyperparameters. Without introducing reinforcement learning, ordinary parameters are those that can be updated by gradient descent—that is, the parameters updated by the training set. Additionally, there is the concept of hyperparameters, such as the number of network layers, the number of network nodes, the number of iterations, the learning rate, etc. These parameters are not within the scope of gradient descent updates. Although there are now some algorithms available to search for model hyperparameters, in most cases, we still manually tune them based on the validation set.
Therefore
That is to say, in a narrow sense, the validation set does not participate in the gradient descent process, meaning it is untrained; but in a broad sense, the validation set participates in a process of “manual parameter tuning.” We adjust the number of iterations, the learning rate, and so on based on the results of the validation set to make the results optimal on that set. Therefore, we can also consider that the validation set participates in training.
Thus, it becomes clear: we still need a set that has not undergone any training at all, which is the test set. We use the test set neither for gradient descent nor for controlling hyperparameters. It is only used to test the final accuracy after the model training is completely finished.
However
Clever readers will realize by analogy that this is actually an endless process. If the accuracy on the test set is very poor, we will still go back and adjust various parameters of the model. At that point, one could argue that the test set has also participated in training. Well, perhaps we need a “test-test set,” and maybe even a “test-test-test set”...
Forget it; let’s just stop at the test set.