Maximum Likelihood Estimation (MLE)

$$$$

Derivation

$\mathbb{X}=\{{\mathit{x}}^{(1)},\dots ,{\mathit{x}}^{(m)}\}$ is a set of $m$ examples drawn independently from the true (but unknown) data distribution $p_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}(\mathit{x})$. ${\hat{p}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}$ is the empirical distribution of the training set $\mathbb{X}$.

$$\begin{array}{rl}{\theta }_{\mathrm{ML}}& =\underset{\theta }{\arg \max }{p}_{\theta }(\mathbb{X})\\ & =\underset{\theta }{\arg \max }{p}_{\theta }({\mathit{x}}^{(1)},\dots ,{\mathit{x}}^{(m)})\\ & =\underset{\theta }{\arg \max }\prod _{i=1}^m{p}_{\theta }({\mathit{x}}^{(i)})\\ & =\underset{\theta }{\arg \max }\log \prod _{i=1}^m{p}_{\theta }({\mathit{x}}^{(i)})\\ & =\underset{\theta }{\arg \max }\sum _{i=1}^m\log p_{\theta }({\mathit{x}}^{(i)})\\ & =\underset{\theta }{\arg \max }{\mathbb{E}}_{\mathbf{x}\sim {\hat{p}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}(\mathit{x})}[\log p_{\theta }(\mathbf{x})]\\ & =\underset{\theta }{\arg \min }{\mathbb{E}}_{\mathbf{x}\sim {\hat{p}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}(\mathit{x})}[-\log p_{\theta }(\mathbf{x})]\end{array}$$

Optimization

The following training goal: $${\theta }_{\mathrm{ML}}=\underset{\theta }{\arg \max }{\mathbb{E}}_{\mathbf{x}\sim {\hat{p}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}(\mathit{x})}[\log p_{\theta }(\mathbf{x})]$$

corresponds to the Negative Log-Likelihood (NLL) loss function: $$L(\theta )={\mathbb{E}}_{\mathbf{x}\sim {\hat{p}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}(\mathit{x})}[-\log p_{\theta }(\mathbf{x})]$$

which can be optimized based on its gradients: $${\mathrm{\nabla }}_{\theta }L(\theta )={\mathbb{E}}_{\mathbf{x}\sim {\hat{p}}_{\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}}(\mathit{x})}[-{\mathrm{\nabla }}_{\theta }\log p_{\theta }(\mathbf{x})]$$

Community Resources

•  Deep Learning
  •  Chapter 5.5 Maximum Likelihood Estimation

Comments