For most readers (including the author), the first biased estimator they encounter is likely the variance: \hat{\sigma}^2_{\text{biased}} = \frac{1}{n}\sum_{i=1}^n \left(x_i - \hat{\mu}\right)^2,\quad \hat{\mu} = \frac{1}{n}\sum_{i=1}^n x_i\label{eq:biased_variance} Then they learn that the corresponding unbiased estimator should be: \hat{\sigma}^2_{\text{unbiased}} = \frac{1}{n-1}\sum_{i=1}^n \left(x_i - \hat{\mu}\right)^2\label{eq:unbiased_variance} In the eyes of many, formula [eq:biased_variance] seems perfectly reasonable, so how is it biased? And why does formula [eq:unbiased_variance], which replaces n with the counter-intuitive n-1, become unbiased?
The following is an attempt to discuss the concepts of unbiased and biased estimation in the clearest language possible.
If we could sample an infinite number of data points, then theoretically the following estimates would be exact: \begin{aligned}\sigma^2 =&\, \mathbb{E}\left[(x - \mu)^2\right]=\lim_{n\to\infty}\frac{1}{n}\sum_{i=1}^n \left(x_i - \hat{\mu}\right)^2\\ \mu =&\, \mathbb{E}[x]=\lim_{n\to\infty}\frac{1}{n}\sum_{i=1}^n x_i\end{aligned} This can also be understood as: when the number of samples tends to infinity, biased and unbiased estimations are equivalent.
The problem is that in practical calculations, we can only sample a batch of data to perform calculations. That is, n is a fixed number. For example, in Stochastic Gradient Descent (SGD), we use the average gradient of a batch of samples as an estimate of the gradient for the entire population. On the other hand, we don’t just estimate once; we might estimate many times. That is, we first sample n points and calculate once to get \hat{\mu}_{1} and \hat{\sigma}^2_{\text{biased},1}, then randomly sample another n points to get \hat{\mu}_{2} and \hat{\sigma}^2_{\text{biased},2}, and so on, obtaining \left(\hat{\mu}_{3},\hat{\sigma}^2_{\text{biased},3}\right),\left(\hat{\mu}_{4},\hat{\sigma}^2_{\text{biased},4}\right),\dots. What we want to know is: \begin{aligned}\mu & \xlongequal{?} \mathbb{E}\left[\hat{\mu}\right] = \lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^N \hat{\mu}_{i}\\ \sigma^2 & \xlongequal{?} \mathbb{E}\left[\hat{\sigma}^2_{\text{biased}}\right]=\lim_{N\to\infty}\frac{1}{N}\sum_{i=1}^N \hat{\sigma}^2_{\text{biased},i} \end{aligned} In other words, does the "infinite average" of "finite averages" equal the final average we are seeking?
As mentioned earlier, this article focuses on discussion and understanding rather than derivation, so I do not intend to complete a general proof. Here, we only use the simplest example: assume n=2, i.e., when using [eq:biased_variance] or [eq:unbiased_variance] for estimation, we only sample two points each time. At this point, the question we need to answer is: \begin{aligned}\mu & \xlongequal{?} \mathbb{E}_{x_1,x_2}\left[\frac{x_1 + x_2}{2}\right]\\ \sigma^2 & \xlongequal{?} \mathbb{E}_{x_1,x_2}\left[\frac{1}{2}\left(\left(x_1 - \frac{x_1 + x_2}{2}\right)^2 + \left(x_2 - \frac{x_1 + x_2}{2}\right)^2\right)\right] \end{aligned} Since this case is relatively simple, we can easily verify it. For example: \mathbb{E}_{x_1,x_2}\left[\frac{x_1 + x_2}{2}\right] = \mathbb{E}_{x_1}\left[\frac{x_1}{2}\right] + \mathbb{E}_{x_2}\left[\frac{x_2}{2}\right] = \frac{\mu}{2} + \frac{\mu}{2} = \mu Therefore, the mean estimated using two samples is an unbiased estimate of the mean, and the same applies to multiple samples.
However, the variance is different: \begin{aligned}&\mathbb{E}_{x_1, x_2} \left[\frac{1}{2}\left(\left(x_1 - \frac{x_1 + x_2}{2}\right)^2 + \left(x_2 - \frac{x_1 + x_2}{2}\right)^2\right)\right]\\ =&\frac{1}{4}\mathbb{E}_{x_1, x_2} \left[\left(x_1 - x_2\right)^2\right]\\ =&\frac{1}{4}\mathbb{E}_{x_1, x_2} \left[x_1^2 + x_2^2 - 2 x_1 x_2\right]\\ =&\frac{1}{4}\Big(\mathbb{E}_{x_1} \left[x_1^2\right] + \mathbb{E}_{x_2} \left[x_2^2\right] - 2 \mathbb{E}_{x_1} \left[x_1\right] \mathbb{E}_{x_2} \left[x_2\right]\Big)\\ =&\frac{1}{4}\Big(2\mathbb{E}_{x} \left[x^2\right] - 2 \mu^2\Big)\\ =&\frac{1}{2}\Big(\mathbb{E}\left[x^2\right] - \mu^2\Big) \end{aligned} Note that the exact expression for variance should be \mathbb{E}\left[x^2\right] - \mu^2. Thus, \hat{\sigma}^2_{\text{biased}} with two samples is a biased estimate of the variance; after repeated estimation and averaging, it still underestimates the true variance. If we analyze the estimate for n samples, the preceding factor is (n-1)/n. Therefore, we only need to multiply by n/(n-1) to obtain the unbiased estimate of the variance, and the final result is formula [eq:unbiased_variance].
Intuitively, using formula [eq:biased_variance] with finite samples to estimate variance results in smaller fluctuations because there are fewer samples, so the variance estimate will also be smaller. This is what is meant by "biased." In an extreme case, what if we only sample one point for estimation? The variance estimated by formula [eq:biased_variance] would be 0. No matter how many times we repeat the experiment, the result remains 0. We certainly cannot say that the variance of the entire population must be 0, can we? This is the simplest example of biased estimation.
Theoretically, the mechanism behind biased estimation is also easy to understand because the formula for calculating variance is equivalent to: \mathbb{E}\left[x^2\right] - \mathbb{E}\left[x\right]^2 The expectation operator \mathbb{E} is a linear operator, so the above expression is non-linear with respect to \mathbb{E} (it is quadratic, specifically the \mathbb{E}\left[x\right]^2 term). As long as an estimator is non-linear with respect to the expectation operator \mathbb{E} (note: the emphasis here is on non-linearity with respect to the expectation operator, not non-linearity of the random variable), direct estimation from finite samples is likely to produce bias. This is because the composition of linear operations remains linear, but the composition of linear and non-linear operations does not preserve the original non-linear operation.
Not all biased estimates can be turned into unbiased estimates simply by replacing n with n-1 like the variance. In general cases, the quantity we want to estimate might be difficult to estimate itself, let alone determining whether it is biased or unbiased. Therefore, eliminating bias for general estimators requires case-by-case analysis.