General linear model

The linear regression model (18) is linear on both features $\mathit{x}$ and weights $\mathit{w}$, since its predictions are given by the inner product

$$\hat{y}={\mathit{x}}^{\mathrm{\top }}\mathit{w}=\sum _{j=0}^d{x}_j{w}_j.$$

Of course, the target variable $y$ can have nonlinear dependence on the predictors $\mathit{x}$. We can easily sacrifise the linearity on $\mathit{x}$ and consider general linear model

(26)$$\hat{y}=\sum _{j=0}^M{\varphi }_j(\mathit{x})w_j.$$

The functions ${\varphi }_j(\mathit{x})$ are called basis functions.

Note

As before, the bias is included in (26) by putting ${\varphi }_0(\mathit{x})=1$.

Note that

• if $M=d$ and ${\varphi }_j(\mathit{x})=x_j$, then (26) turns into multiple linear regression;

• if $d=1$ and ${\varphi }_j(x)=x^j$ then (26) becomes polynomial regression.

The popular choices of ${\varphi }_j(\mathit{x})$ are

• ${\varphi }_j(x)=\exp (-\frac{(x-{\mu }_j{)}^2}{2s^2})$ (Gaussian basis functions);

• ${\varphi }_j(x)=\sigma (\frac{x-{\mu }_j}{s})$ (sigmoidal basis functions).

Sigmoid function

$$\sigma (x)=\frac{1}{1+e^{-x}}$$

TODO

A lot of things…