Biased Estimators and Sample Statistics

Variance in Terms of Expected Differences

Variance is equal to the difference between the expected square value and the expected value squared:

$Var (X) E [Var (X)] ⟹ E [\frac{1}{n} i = 1 \sum n [X_{i} - mean (X)]] = \frac{1}{n} i = 1 \sum n [(X_{i} - mean (X))^{2}] = E [\frac{1}{n} i = 1 \sum n [(X - E [X])^{2}]] The mean value is an unbiased estimator i.e. expected value of mean is the value = E [(X - E [X])^{2}] = E [X^{2} - 2 X \cdot E [X] + (E [X])^{2}] = E [X^{2}] - E [2 X \cdot E [X]] + (E [X])^{2} = E [X^{2}] - 2 (E [X])^{2} + E [X]^{2} = E [X^{2}] - E [X]^{2} = E [X^{2}] - (E [X])^{2}$

Summing Variances

$Var (X \pm Y) = E [(X \pm Y)^{2}] - E [X \pm Y]^{2} = E [X^{2} \pm 2 X Y + Y^{2}] - (E [X] \pm E [Y])^{2} = E [X^{2}] \pm 2 E [X Y] + E [Y^{2}] - (E [X] \pm E [Y])^{2} If X and Y are Independent, the Expected Value is the Product = E [X^{2}] \pm 2 E [X] E [Y] + E [Y^{2}] - (E [X] \pm E [Y])^{2} Expand the Square = E [X^{2}] \pm 2 E [X] E [Y] + E [Y^{2}] - E [X] \mp 2 E [X] E [Y] - E [Y]^{2} Cancel out the \pm and \mp = E [X^{2}] + E [Y^{2}] - E [X] - E [Y]^{2} = (E [X^{2}] - E [X]^{2}) + (E [Y^{2}] - E [Y]^{2}) = Var (X) + Var (Y)$

Note however, if $X$ and $Y$ are not independent:

$Var (X \pm Y) = Var (X) \pm 2 Cov (X, Y) + Var (Y)$

Multiplying Variance by a Constant}

$Var (a X) = E [(a X)^{2}] - E [a x]^{2} Take the constant from the left term = a^{2} E [X^{2}] - E [a x]^{2} Take the constant from the right term = a^{2} E [X^{2}] - (a E [X])^{2} = a^{2} E [X^{2}] - a^{2} E [X]^{2} a^{2} (E [X^{2}] - E [X]^{2}) a^{2} Var (X)$

Variance of a Sample Mean

Note

You may recall this:

$σ_{\overset{x}{ˉ}} = \frac{σ}{n}$

from the Central Limit Theorem ¹, we develop this idea further in the section on t-distributions.

https://en.wikipedia.org/wiki/Central_limit_theorem

Consider a sample of size $n$ , the population will have a variance $σ$ and a mean $μ$ , the sample will have a variance $s$ and a mean $\overset{x}{ˉ}$ . Many different samples could be taken, so there is a distribution of $\overset{x}{ˉ}$ values that could occur. The variance of sample means $var (\overset{x}{ˉ})$ is given by $var (x) = \frac{σ ^{2}}{n}$ , this is a part of the CLT and is shown here.

$Var (\overset{ˉ}{X}) = Var (\frac{1}{n} i = 1 \sum n [X_{i}]) Applying the product rule from earlier = (\frac{1}{n})^{2} Var (i = 1 \sum n [X_{i}]) Applying the sum rule = \frac{1}{n ^{2}} i = 1 \sum n [Var (X_{i})] The variance is already known = \frac{1}{n ^{2}} i = 1 \sum n [σ^{2}] = \frac{1}{n ^{2}} n σ^{2} = \frac{σ ^{2}}{n}$

Bessel's Correction

Suppose that the sample variance $(s)$ was given by the typical formula:

$s ⟹ E [s] ? = \frac{1}{n} i = 1 \sum n [(X_{i} - E [X])] = E [X^{2}] - E [X]^{2}$

The expected value should be $σ$ , in that sense it would be an un-biased estimator of the population mean.

The key here is to recognise that $s$ corresponds to a sample, by introducing $\overset{ˉ}{X}$ we can solve for variance in terms of expected values and the sample mean. The sample introduces the bias so it is necessary to use the sample mean.

$s^{2} ⟹ E [s^{2}] ⋆ = \frac{1}{n} i = 1 \sum n [(X_{i} - \overset{ˉ}{X})^{2}] = E [\frac{1}{n} i = 1 \sum n [(X_{i} - \overset{ˉ}{X})^{2}]] Expand the square = E [\frac{1}{n} i = 1 \sum n [X_{i}^{2} - 2 X \overset{ˉ}{X} + \overset{ˉ}{X}^{2}]]$

Note in particular, that $\overset{ˉ}{X}$ is now a form of the expression, by the CLT this will give a $Var (X) = \frac{σ ^{2}}{n}$ , this is the key insight that pulls this together, it's that very $/ n$ that accounts for the $(n - 1)$ :

$Distribute the sum = E [\frac{1}{n} i = 1 \sum n [X_{i}^{2}] - 2 \overset{ˉ}{X} i = 1 \sum n [X_{i}] + \frac{1}{n} i = 1 \sum n [\overset{ˉ}{X}^{2}]] Distribute the expectation = E [\frac{1}{n} i = 1 \sum n [X_{i}^{2}]] - 2 E [\overset{ˉ}{X}^{2}] + E [\frac{1}{n} i = 1 \sum n [\overset{ˉ}{X}^{2}]] Simplify the right-most term = E [\frac{1}{n} i = 1 \sum n [X_{i}^{2}]] - E [2 \overset{ˉ}{X}^{2}] + E [\frac{1}{n} n \overset{ˉ}{X}^{2}] = E [\frac{1}{n} i = 1 \sum n [X_{i}^{2}]] - E [2 \overset{ˉ}{X}^{2}] + E [\overset{ˉ}{X}^{2}] = E [\frac{1}{n} i = 1 \sum n [X_{i}^{2}]] - E [\overset{ˉ}{X}^{2}] The mean value is an unbiased estimator of the mean and the values are i.i.d. = E [X^{2}] - E [\overset{ˉ}{X}^{2}]$

Expected Value Squared

The expected value of $X$ is the mean value:

$E [X]^{2} = μ^{2}$

Expected Square Value

Recall the definition of Variance from earlier:

$Var (X) ≜ ⟹ E [X^{2}] E [X^{2}] - E [X]^{2} = Var (X) + E [X]^{2} = σ^{2} + μ^{2}$

Applying this to the sample mean:

$Var (\overset{ˉ}{X}) ≜ ⟹ E [\overset{ˉ}{X}^{2}] E [\overset{ˉ}{X}^{2}] - E [\overset{ˉ}{X}]^{2} = Var (\overset{ˉ}{X}) + E [\overset{ˉ}{X}]^{2} The variance of the sample mean was derived earlier = \frac{σ ^{2}}{n} + μ$

This step provides the key insight, if variance is taken on a population, there is no $n$ in the denominator

because the variance of a sample mean is different to the variance of a population statistic, the expected value of a squared sample mean is different to the expected value of a squared observation. The same $n$ from $\frac{σ}{n}$ is the same one that leands to $\frac{1}{n - 1} .$ The expected value of a squared sample mean is less than the expected value of a squared observation, because the sample mean is going to be more central.

Solving The Expected Sample Variance

$s^{2} ⟹ E [s^{2}] ? = \frac{1}{n} i = 1 \sum n [(X_{i} - \overset{ˉ}{X})^{2}] = E [X^{2}] - E [\overset{ˉ}{X}^{2}] = (σ^{2} + μ^{2}) - (\frac{σ ^{2}}{n} + μ^{2}) = \frac{n σ ^{2} - σ ^{2}}{n} = σ^{2} \frac{n - 1}{n}$

This shows that the variance formula, applied to a sample is biased. In order to correct that bias define $s_{b} = \frac{n}{n - 1} s$ like so:

$s_{b}^{2} = \frac{n}{n - 1} \frac{1}{n} i = 1 \sum n [(X_{i} - \overset{ˉ}{X})^{2}] = \frac{1}{n - 1} i = 1 \sum n [(X_{i} - \overset{ˉ}{X})^{2}]$

The expected value of this is:

$E [s_{b}^{2}] = E [\frac{n}{n - 1} s] = \frac{n}{n - 1} E [s] = \frac{n}{n - 1} \frac{n - 1}{n} σ^{2}$

Environmental Informatics (MATH3005)