Maximum Likelihood Estimator (MLE)

Powerful ideas in Statistical Inference.

By Vamshi Jandhyala in mathematics

November 5, 2020

Let $X_1, \dots, X_n$ be $IID$ with PDF $f(x;\theta)$. The likelihood function is defined by

$$ \mathcal{L_n}(\theta) = \prod_{i=1}^n f(X_i;\theta) $$

The likelihood function is just the joint density of the data, except that we treat it is as a function of the parameter $\theta$.

The maximum likelihood estimator (MLE), denoted by $\hat{\theta_n}$ is the value of $\theta$ that maximizes $\mathcal{L_n}(\theta)$.

The maximum likelihood method is one of the most common methods for estimating parameters in a parametric model.

Problem - 1

Let $f(x) = \left(\frac{\alpha m^{\alpha}}{x^{\alpha+1}} \right)\mathbb{I}{x ≥ m}$ where α and m are two unknown parameters. Estimate the values of these parameters using maximum likelihood method.

Solution

Given observed values $X_1 = x_1, X_2 = x_2, \dots , X_n = x_n$, the likelihood of $\alpha, m$ is the function $\mathcal{L}(\alpha, m) = f(x_1, x_2, \dots , x_n|\alpha, m)$. As the $X_i$ are iid, we have

$$ \mathcal{L}(\alpha, m) = \Pi_{i=1}^n f(x_i|\alpha,m) = \frac{\alpha^n m^{n\alpha}}{x_1^{\alpha+1} \dots x_n^{\alpha+1}} $$

For the PDF to be nonzero $m = \min {x_1,\dots,x_n}$.

The logarithm of the likelihood function is given by

$$ \begin{align*} \log \mathcal{L}(\alpha, m) &= n\log \alpha + n\alpha \log m \\
&- (\alpha+1)(\log x_1 + \dots + \log x_n) \end{align*} $$

Differentiating log likelihood function wrt $\alpha$ and equating it to zero, we have

$$ \begin{align*} \frac{\partial \log \mathcal{L}(\alpha, m)}{\partial \alpha} &= \frac{n}{\alpha} + n \log m \\
&- (\log x_1 + \dots + \log x_n) = 0 \end{align*} $$

Therefore, we have

$$ \alpha = \frac{n}{\sum_{i=1}^n\log x_i - n \log \min{x_1,\dots,x_n}} $$

Problem - 2

Let $X_1, \dots , X_n$ be iid random variables with uniform distribution on the interval $[θ − 1/2, θ + 1/2]$.Find the maximum likelihood estimate of θ.

Solution

Given observed values $X_1 = x_1, X_2 = x_2, \dots , X_n = x_n$, the likelihood of θ is the function $\mathcal{L}(θ) = f(x_1, x_2, \dots , x_n|θ)$. As the $X_i$ are iid, we have

$$ \mathcal{L}(θ) = \Pi_{i=1}^n f(x_i|θ) = \left(\frac{1}{θ + 1/2 - θ + 1/2}\right)^n = 1 $$

when $ \max {x_1,\dots,x_n} - 1/2 \leq \theta \leq \min{x_1,\dots,x_n} + 1/2$ .

Thus, we can select any value in the interval $[\max {x_1, \dots , x_n} − 1/2, \min {x_1, \dots , x_n} + 1/2]$ as the MLE for θ. The MLE is not uniquely specified in this case.