Whether in competitions, experiments, or engineering projects, we often encounter situations where the distribution of the training set and the test set are inconsistent. Generally, we partition a validation set from the training set to adjust hyperparameters (refer to "The Meaning of Training, Validation, and Test Sets"), such as controlling the number of training epochs to prevent overfitting. However, if the validation set itself differs significantly from the test set, a model that performs well on the validation set does not necessarily perform well on the test set. Therefore, how to partition a validation set so that its distribution difference from the test set is minimized is a topic worth researching.
Two Cases
First, let us clarify that this article considers scenarios where we can obtain the test set data itself but do not know the test set labels. If it is a closed evaluation scenario where we submit a model and cannot see the test set at all, there is not much we can do. Why does the phenomenon of distribution inconsistency between the test set and the training set occur? There are mainly two cases.
The first case is the inconsistency in the distribution of labels. That is to say, if we only look at the input x, the distribution is basically the same, but the corresponding y distribution is different. A typical example is information extraction tasks; the training set is often constructed through "distant supervision + manual coarse labeling," which results in a large volume but may contain many errors or omissions. Meanwhile, the test set might be constructed through "repeated manual fine labeling," resulting in very few errors. In this case, it is impossible to construct a better validation set simply by partitioning the data.
The second case is the inconsistency in the distribution of inputs. Simply put, the distribution of x is inconsistent, but the labeling of y is basically correct. For example, in a classification problem, the category distribution of the training set may differ from that of the test set; or in a reading comprehension problem, the proportion of factual vs. non-factual question types in the training set may differ from the test set, and so on. In this case, we can appropriately adjust the sampling strategy to make the validation set distribution more consistent with the test set, so that the validation results can better reflect the test results.
The Discriminator
To achieve our goal, we set the labels of the training set to 0 and the labels of the test set to 1, and train a binary discriminator D(x): -\mathbb{E}_{x\sim p(x)}[\log (1 - D(x))] - \mathbb{E}_{x\sim q(x)}[\log D(x)] where p(x) represents the distribution of the training set and q(x) represents the distribution of the test set. It should be noted that we are not directly mixing the training and test sets for sampling; instead, we sample an equal number of samples from the training set and the test set to form each batch. In other words, we need to oversample to achieve class balance.
Some readers might worry about overfitting, i.e., the discriminator completely separating the training set from the test set. In fact, when training the discriminator, we should also partition a validation set just like in ordinary supervised training and determine the number of training epochs based on that validation set to avoid severe overfitting. Alternatively, as seen in some online cases, one could directly use Logistic Regression as the discriminator because Logistic Regression is simple enough and carries a lower risk of overfitting.
Similar to the discriminator in a GAN, it is not difficult to derive that the theoretical optimal solution for D(x) is: D(x) = \frac{q(x)}{p(x)+q(x)} \label{eq:d} That is to say, after the discriminator is trained, it can be considered equivalent to the relative density of the test set distribution.
Importance Sampling
Whether optimizing a model or calculating metrics, we actually hope to perform these actions on the test set. That is, for a given objective f(x) (such as the model’s loss), we want to calculate: \mathbb{E}_{x\sim q(x)}[f(x)] = \int q(x) f(x) dx However, to calculate the objective f(x), we usually need to know the true label of x. Since we do not know the labels for the test set, we cannot calculate it directly. However, we do know the labels for the training set, so we can use importance sampling to solve this: \int q(x) f(x) dx = \int p(x)\frac{q(x)}{p(x)} f(x) dx = \mathbb{E}_{x\sim p(x)}\left[\frac{q(x)}{p(x)} f(x)\right] According to equation [eq:d], we know that \frac{q(x)}{p(x)} = \frac{D(x)}{1-D(x)}, so the final expression becomes: \mathbb{E}_{x\sim q(x)}[f(x)] = \mathbb{E}_{x\sim p(x)}\left[\frac{D(x)}{1-D(x)} f(x)\right] \label{eq:w} Simply put, the idea of importance sampling is to "pick out" samples from the training set that are similar to the test set and assign them higher weights.
Final Strategy
From equation [eq:w], we can derive two strategies:
The first is to apply weights directly according to the formula. That is, still partition the training and validation sets using random shuffling, but assign a weight w(x) = \frac{D(x)}{1-D(x)} to each sample. It is worth noting that some contestants in competitions have used similar approaches, though the weight often circulated is simply D(x). I cannot assert which one is better, but from a theoretical derivation perspective, \frac{D(x)}{1-D(x)} should be more reasonable.
The other strategy is to actually sample the corresponding validation set. This is also not difficult. Suppose all samples in the training set are x_1, x_2, \dots, x_N. We normalize the weights: p_i = \frac{w(x_i)}{\sum_{i=1}^N w(x_i)} Then, perform independent repeated sampling according to the distribution p_1, p_2, \dots, p_N until the specified number of samples is reached. Note that this must be sampling with replacement; therefore, the same sample may be sampled multiple times and should be kept multiple times in the validation set. You cannot remove duplicates, as the distribution would no longer be consistent after deduplication.
Summary
This article compares the differences between the training set and the test set from the perspective of training a discriminator. By combining this with importance sampling, we can obtain a validation set that is closer to the test set or weight the training samples, thereby reducing the discrepancy between the training set optimization process and the test set.
Reprinting: Please include the original address of this article: https://kexue.fm/archives/7805
For more details regarding reprinting, please refer to: "Scientific Space FAQ"