Probabilistic models

Probabilistic models#

In probabilistic approach all quantities are considered as random variables. Each training sample $(x, y)$ comes from a joint probability distribution with density $p (x, y)$ . If we are using some model of machine learning with parameters $w$ , then this density is conditioned on $w$ :

(x, y) \sim p (x, y | w) .

The parametric family $p (x, y | w)$ is called a probabilistic model of a machine learning problem.

Maximum likelihood estimation#

The likelihood of the dataset $D = (X, y) = {(x_{i}, y_{i})}_{i = 1}^{n}$ is

p (y | X, w) .

If the samples $(x_{i}, y_{i})$ are i.i.d., then

p (y | X, w) = \prod_{i = 1}^{n} p (y_{i} | x_{i}, w) .

The optimal weights $\hat{w}$ maximize the likelihood, or, equivalently, log-likelihood:

(33)#

\log p (y | X, w) = \log \prod_{i = 1}^{n} p (y_{i} | x_{i}, w) = \sum_{i = 1}^{n} \log p (y_{i} | x_{i}, w) \to max_{w} .

Alternatively, one can minimize negative log-likelihood (NLL):

- \log p (y | X, w) = - \sum_{i = 1}^{n} \log p (y_{i} | x_{i}, w) \to min_{w} .

The optimal estimation of weights $\hat{w}$ maximizing log-likelihood (33) is called maximum likelihood estimation (MLE).

Bayesian approach#

From (33) we obtain a point estimation ${\hat{w}}_{MLE}$ . In Bayesian framework we estimate not points but distributions!

Assume that parameters $w$ have prior distribution $p (w)$ . Observing the dataset $D$ , we can apply the Bayes formula and obtain posterior distribution

p (w | D) = \frac{p (D | w) p (w)}{p (D)} .

Maximum a posteriori estimation (MAP) maximizes posterior distribution:

{\hat{w}}_{MAP} = \arg max_{w} p (w | D) .

Probabilistic models

Contents

Probabilistic models#

Maximum likelihood estimation#

Bayesian approach#