RSS as a Maximum Likelihood Estimator

If the residuals of a model are expected to be normally distributed, then the paramaters should be chosen to minimize the RSS ¹.

Otherwise the choice of loss function is arbitrary.

Consider a linear regression:

$$\begin{array}{r}\hat{y}=mx+b+\epsilon \end{array}$$

If we presume that $\epsilon \sim \mathcal{N}\left(0,\sigma \right),\exists \sigma$ we could view this from a perspective of choosing a $y_i$ value that is normally distributed around ${\hat{y}}_i$ with a variance of ${\sigma }^2$.

% If we treat $\hat{y}$ as a a function of $x$ the probability of seeing a value $y$, given that this model is true and the residuals are indeed normal, is given by $\mathtt{pnorm}\left(\hat{y}(x),y,\sigma \right)$, this would correspond to:

The probability of seeing any such value is:

$$\begin{array}{rl}p\left(y_i\right)& =\frac{1}{\sigma \sqrt{2\pi }}\exp \left(\frac{{\left(y_i-{\hat{y}}_i\right)}^2}{2{\sigma }^2}\right)\\ & =\frac{1}{\sigma \sqrt{2\pi }}\exp \left(\frac{{\left(y_i-m{x}_i-b\right)}^2}{2{\sigma }^2}\right)\end{array}$$

The likelihood of seeing all the observations would be given by:

$$\begin{array}{rl}\mathcal{L}\left(\mathbf{y}\right)& =\prod _{i\in N}\left[\frac{1}{\sigma \sqrt{2\pi }}\exp \left(-\frac{{\left(y_i-{\hat{y}}_i\right)}^2}{2{\sigma }^2}\right)\right]\end{array}$$

This function only has three parameters ($\sigma$, $m$ and $b$), everything else is either an observed value or a constant.

What we want to do is choose values of $m$ and $b$ that maximize this likelihood for any given $\sigma$:

$$\begin{array}{rl}m,b& =\underset{m,b}{\mathrm{argmax}}\left(\prod _{i=1}^N\left[p\left(y_i\right)\right]\right)\\ \text{Because log is a monotone transform:}& \\ & =\underset{m,b}{\mathrm{argmax}}\left(\log \left(\prod _{i=1}^N\left[p\left(y_i\right)\right]\right)\right)\\ \text{By log laws:}& \\ & =\underset{m,b}{\mathrm{argmax}}\left(\sum _{i=1}^N\left[\log \left(p\left(y_i\right)\right)\right]\right)\\ \text{Substituting in the probability:}& \\ & =\underset{m,b}{\mathrm{argmax}}\left(\sum _{i=1}^N\left[\log \left(\frac{1}{\sigma \sqrt{2\pi }}\exp \left(-\frac{{\left(y_i-{\hat{y}}_i\right)}^2}{2{\sigma }^2}\right)\right)\right]\right)\\ \text{Drop the negative}& \\ & =\underset{m,b}{\mathrm{argmax}}\left(\sum ^N\left[\log \left(\frac{1}{\sigma \sqrt{2\pi }}\exp \left(\frac{{\left(y_i-{\hat{y}}_i\right)}^2}{2{\sigma }^2}\right)\right)\right]\right)\end{array}$$

$$\begin{array}{rl}\text{Constants can be dropped from argmin}& \\ & =\underset{m,b}{\mathrm{argmin}}\left(\sum ^N\left[\log \left(\exp \left(\frac{{\left(y_i-{\hat{y}}_i\right)}^2}{2{\sigma }^2}\right)\right)\right]\right)\\ & =\underset{m,b}{\mathrm{argmin}}\left(\sum ^N\left[\frac{{\left(y_i-{\hat{y}}_i\right)}^2}{2{\sigma }^2}\right]\right)\end{array}$$

$$\begin{array}{rl}\text{Dropping constants:}& \\ & =\underset{m,b}{\mathrm{argmin}}\left(\sum _{i=1}^N\left[{\left(y_i-{\hat{y}}_i\right)}^2\right]\right)\end{array}$$

Thus, in order to maximize the likelihood of seeing any $y_i$ with normally distributed residuals, it is sufficient to choose values $m$ and $b$ that minimize the RSS.

Footnotes