Estimations

Bias

Let $X_1,\dots ,X_n$ be an i.i.d. sample from some distribution $F_{\theta }(x)$. Estimation $\hat{\theta }=\hat{\theta }(X_1,\dots ,X_n)$ of $\theta$ is called unbiased if $\mathbb{E}\hat{\theta }=\theta$. Otherwise $\hat{\theta }$ is called biased, and its bias equals to

$$\mathrm{bias}(\hat{\theta })=\mathbb{E}\hat{\theta }-\theta .$$

For example, sample average $\hat{\theta }={\bar{X}}_n$ is unbiased estimate of mean $\theta$ since

$$\mathbb{E}{\bar{X}}_n=\frac{1}{n}\sum _{k=1}^n\mathbb{E}{X}_k=\frac{1}{n}\cdot n\theta =\theta .$$

Sometimes estimation ${\hat{\theta }}_n=\hat{\theta }(X_1,\dots ,X_n)$ is biased, but this bias vanishes as $n$ becomes large. If $\underset{n\to \mathrm{\infty }}{lim}\mathbb{E}{\hat{\theta }}_n=\theta$, then estimation ${\hat{\theta }}_n$ is called asymptotically unbiased.

Consistency

Estimation ${\hat{\theta }}_n=\hat{\theta }(X_1,\dots ,X_n)$ is called consistent if it converges to $\theta$ in probability: ${\hat{\theta }}_n\stackrel{P}{\to }\theta$, i.e.,

$$\underset{n\to \mathrm{\infty }}{lim}\mathbb{P}(|{\hat{\theta }}_n-\theta |>\epsilon )=0\text{ for all }\epsilon >0.$$

Due to the law of large numbers $\hat{\theta }={\bar{X}}_n$ is a consistent estimation for expectation $\theta =\mathbb{E}{X}_1$ for any i.i.d. sample $X_1,\dots ,X_n$.

Bias-variance decomposition

Mean squared error (MSE) of $\hat{\theta }$ is

$$\mathrm{MSE}(\hat{\theta })=\mathbb{E}(\hat{\theta }-\theta {)}^2.$$

Bias-variance decomposition:

$$\mathrm{MSE}(\hat{\theta })={\text{bias}}^2(\hat{\theta })+\mathbb{V}(\hat{\theta }).$$

Proof

$$\begin{array}{r}\begin{array}{c}\mathrm{MSE}(\hat{\theta })=\mathbb{E}(\hat{\theta }-\theta {)}^2=\mathbb{E}(\hat{\theta }-\mathbb{E}\hat{\theta }+\mathbb{E}\hat{\theta }-\theta {)}^2=\hfill \\ =\mathbb{E}(\hat{\theta }-\mathbb{E}\hat{\theta }{)}^2+2\mathbb{E}(\hat{\theta }-\mathbb{E}\hat{\theta })(\mathbb{E}\hat{\theta }-\theta )+\mathbb{E}(\mathbb{E}\hat{\theta }-\theta {)}^2=\\ =\mathbb{V}(\hat{\theta })+2(\underset{=0}{\underbrace{\mathbb{E}\hat{\theta }-\mathbb{E}\hat{\theta }}})(\mathbb{E}\hat{\theta }-\theta )+{\mathrm{bias}}^2(\hat{\theta }).\end{array}\end{array}$$

If $\underset{n\to \mathrm{\infty }}{lim}\mathrm{MSE}({\hat{\theta }}_n)=0$, then estimation ${\hat{\theta }}_n$ of $\theta$ asymptotically unbiased and consistent.

In machine learning bias-variance decomposition is also called bias-variance tradeoff:

Asymptotic normality

Estimation ${\hat{\theta }}_n$ is asymptotically normal if $\frac{{\hat{\theta }}_n-\theta }{\mathrm{se}({\hat{\theta }}_n)}\stackrel{D}{\to }\mathcal{N}(0,1)$, i.e.,

$$\underset{n\to \mathrm{\infty }}{lim}\mathbb{P}(\frac{{\hat{\theta }}_n-\theta }{\mathrm{se}({\hat{\theta }}_n)}⩽z)=\mathbb{\Phi }(z),\mathrm{se}({\hat{\theta }}_n)=\sqrt{\mathbb{V}{\hat{\theta }}_n}.$$

If $X_1,\dots ,X_n$ is an i.i.d. sample from some distribution with finite expection $\mu$ and variance ${\sigma }^2$, then according to the central limit theorem ${\bar{X}}_n$ is asymptotically normal estimation of $\mu$.

Maximum likelihood estimation (MLE)

Let i.i.d. sample $X_1,\dots ,X_n\sim F_{\theta }(x)$. Правдоподобие (функция правдоподобия, likelihood) выборки $X_1,\dots ,\dots X_n$ — это просто её совместная pmf или pdf. Вне зависимости от типа распределения будем обозначать правдоподобие как

$$\mathcal{L}(\theta )\equiv L(X_1,\dots ,X_n|\theta )=p(X_1,\dots ,X_n|\theta ).$$

Если выборка i.i.d., то функция правдоподобия распадается в произведение одномерных функций:

$$L(X_1,\dots ,X_n|\theta )=\prod _{k=1}^{n}p(X_k|\theta ).$$

Оценка максимального правдоподобия (maximum likelihood estimation, MLE) максимизирует правдоподобие:

$${\hat{\theta }}_{\mathrm{ML}}=\arg \underset{\theta }{max}\mathcal{L}(\theta )$$

Поскольку максимизировать сумму проще, чем произведение, обычно переходят к логарифму правдоподобия (log-likelihood). Это особенно удобно в случае i.i.d. выборки, тогда

$${\hat{\theta }}_{\mathrm{ML}}=\arg \underset{\theta }{max}\log \mathcal{L}(\theta )=\arg \underset{\theta }{max}\sum _{k=1}^n\log p(X_k|\theta ).$$

Properties of MLE

• consistency: ${\hat{\theta }}_{\mathrm{ML}}\stackrel{P}{\to }\theta$;

• equivariance: if ${\hat{\theta }}_{\mathrm{ML}}$ — MLE for $\theta$ then $\phi (\theta )$ — MLE for $\phi (\theta )$;

• asymptotic normality: $\frac{{\hat{\theta }}_{\mathrm{ML}}-\theta }{\widehat{\mathrm{se}}}\stackrel{D}{\to }\mathcal{N}(0,1)$;

• асимптотическая оптимальность: при достаточно больших $n$ оценка ${\hat{\theta }}_{\mathrm{ML}}$ имеет минимальную дисперсию.

Exercises

1. Let $X_1,\dots ,X_n$ be an i.i.d. sample from $U[0,\theta ]$ and $\hat{\theta }=X_{(n)}$. Is this estimation unbiased? Asymptotically unbiased? Consistent?

2. Show that estimation ${\hat{\theta }}_n$ is consistent if it is asymptotically unbiased and $\underset{n\to \mathrm{\infty }}{lim}\mathbb{V}({\hat{\theta }}_n)=0$.

3. Let $X_1,\dots ,X_n$ be an i.i.d. sample from $U[0,2\theta ]$. Show that sample median $\mathrm{med}(X_1,\dots ,X_n)$ is unbiased estimation of $\theta$. See also ML Handbook.

4. Let $X_1,\dots ,X_n$ be an i.i.d. sample from a distribution with finite moments $\mathbb{E}{X}_1$ and $\mathbb{E}{X}_1^2$. Is sample variance ${\bar{S}}_n$ unbiased estimation of $\theta =\mathbb{V}{X}_1$? Asymptotically unbiased?

5. There are $k$ heads and $n-k$ tails in $n$ independent Bernoulli trials. Find MLE of the probability of heads.

6. Find MLE estimation of $\lambda$ if $X_1,\dots ,X_n$ is an i.i.d. sample from $\mathrm{Pois}(\lambda )$.

7. Let $X_1,\dots ,X_n$ be i.i.d. sample from $\mathcal{N}(\mu ,\tau )$. Find MLE of $\mu$ and $\tau$.

8. Find MLE estimation of $a$ and $b$ if $X_1,\dots ,X_n\sim U[a,b]$.