Deriving Multiple Linear Regression

Introduction

The goal here, is to maximize the probability of seeing some observation $y_i$ given a mean value ${\hat{y}}_i$ and a constant variance $\sigma$

Deriving Multiple Linear Regression

Consider a multiple linear regression of the form:

$$\begin{array}{r}\underset{N\times 12}{\hat{\mathbf{Y}}}=\underset{N\times 3N\times 12}{\mathbf{XW}}\end{array}$$

The bias for this model is included in the weights matrix, as shown in Bias Terms in Multiple Linear Regression.

Consider the probability of seeing an observation under the assumption:

$$\begin{array}{rl}{y}_i& \sim \mathcal{N}\left({\hat{y}}_i,\sigma \right)\\ ⟹\mathcal{P}\left(y_i\right)& \propto e^{{\left(\frac{y_i-\widehat{y_i}}{\sigma }\right)}^2}\end{array}$$

The goal in linear regression is to find $\hat{\mathbf{Y}}=\mathbf{XW}$ such that:

$$\mathbf{W}=\underset{\mathbf{W}}{\mathrm{argmax}}\left[\prod _{i=1}^n\left(\mathcal{P}\left({\mathbf{Y}}_i\right)\right)\right]$$

The argument $\mathbf{W}$ which maximises the likelhood of this expression is equivalent to the the argument $\mathbf{W}$ which minimises the rss, see Maximum Likelihood is equal to Minimum RSS

$$\mathbf{W}=\underset{\mathbf{W}}{\mathrm{argmin}}\left[\epsilon \left(\mathbf{W}\right)\right]$$

where:

$$\begin{array}{r}\epsilon \left(\mathbf{W}\right)=\sum _{i=1}^N\sum _{j=12}^{12}\left[{\left(\mathbf{XW}-\mathbf{Y}\right)}^{\circ 2}\right]\end{array}$$

Note that the squaring operation is elment-wise, so more conventionally:

$$\begin{array}{r}{\left(\epsilon \left(\mathbf{W}\right)\right)}_{\left[ij\right]}=\sum _{i=1}^N\sum _{j=12}^{12}\left[{\left({\left(\mathbf{XW}-\mathbf{Y}\right)}_{\left[ij\right]}\right)}^2\right]\end{array}$$

because $\epsilon$ is a quadratic function:

$$\begin{array}{r}\frac{\partial \epsilon }{\partial \mathbf{W}}=0⟺\mathbf{W}=\underset{\mathbf{W}}{\mathrm{argmin}}\left(\epsilon \right)\end{array}$$

Recall the Matrix Chain Rule

$$\begin{array}{rl}\frac{\partial \epsilon }{\partial \mathbf{W}}& =\frac{\partial \hat{\mathbf{Y}}}{\partial \mathbf{W}}{\left(\frac{\partial \epsilon }{\partial \hat{\mathbf{Y}}}\right)}^{\mathrm{T}}\\ & \\ \frac{\partial \epsilon }{\partial \hat{\mathbf{Y}}}& =2\left(\hat{\mathbf{Y}}-\mathbf{Y}\right)\\ \frac{\partial \hat{\mathbf{Y}}}{\partial \mathbf{W}}& =\mathbf{X}\\ & \\ ⟹\frac{\partial \epsilon }{\partial \mathbf{W}}& =2{\mathbf{X}}^{\mathrm{T}}\left(\hat{\mathbf{Y}}-\mathbf{Y}\right)\end{array}$$

Setting the derivative to 0:

$$\begin{array}{rl}0& ={\mathbf{X}}^{\mathrm{T}}\left(\mathbf{XW}-\mathbf{Y}\right)\\ & ={\mathbf{X}}^{\mathrm{T}}\mathbf{XW}-{\mathbf{X}}^{\mathrm{T}}\mathbf{Y}\\ {\mathbf{X}}^{\mathrm{T}}\mathbf{XW}& ={\mathbf{X}}^{\mathrm{T}}\mathbf{Y}\\ \mathbf{W}& ={\left({\mathbf{X}}^{\mathrm{T}}\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\mathrm{T}}\mathbf{Y}\end{array}$$

Conclusion

So, in conclusion, if $\mathbf{Y}=\mathbf{WX}$, the value for $\mathbf{W}$ that minimises the l2 norm is ^{1 2} :

$$\begin{array}{rl}\mathbf{W}& ={\left({\mathbf{X}}^{\mathrm{T}}\mathbf{X}\right)}^{-1}{\mathbf{X}}^{\mathrm{T}}\mathbf{Y}\end{array}$$

References