Environmental Informatics (MATH3005)

Bias Terms in Multiple Regression

Consider a matrix multiplication as shown below.

$$\begin{array}{r}\underset{n\times R}{\hat{\mathbf{Y}}}=\underset{n\times FF\times R}{\mathbf{XW}}\end{array}$$

Where:

• $n$ is the sample size
• $R$ is the number of columns in the output, i.e. the number of response variables (or dependent variables) being modelled
• $F$ is the number of columns in the input, i.e. the number of *features (or input/independent variables)

In this representation, each of the $n$ rows of $\mathbf{X}$ and $\mathbf{Y}$ is an observation and the columns represent input and response variables respectively.

However, I have elected to use a column-major so let's instead represent this:

$$\begin{array}{r}\underset{R\times n}{\hat{\mathbf{Y}}}=\underset{R\times FF\times n}{\mathbf{WX}}\end{array}$$

Now each observation is a column-vector instead of a row vector.

This corresponds to a matrix form thusly:

$$\begin{array}{rl}\left[\begin{array}{ccc}{w}_{11}& w_{12}& w_{13}\\ w_{21}& w_{22}& w_{23}\\ w_{31}& w_{32}& w_{33}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]& =\left[\begin{array}{c}{w}_{\left[1,:\right]}\cdot \mathbf{x}\\ w_{\left[2,:\right]}\cdot \mathbf{x}\\ w_{\left[3,:\right]}\cdot \mathbf{x}\end{array}\right]\end{array}$$

note the following points:

1. This is a column-major representation, the $\mathbf{x}$ vector represents a single observation and subsequent observations would become additional columns of $\mathbf{X}$.
   • A row major representation can be acheived by transposing ${}^{\mathrm{T}}$ the matrices.
   • R, Julia, Octave, Wolfram and Fortran all use column-major representations
   • Python, C(++), Go and Rust use a row-major representation
   • It's important to get this right, languages store values in a certain pattern in memory, cutting against the grain will be less performant.
2. The use of sympy/octave/julia notation, whereby $w_{\left[1,:\right]}$ represents a vector composed of the first row of $\mathbf{W}$.

Let's now add a bias term (i.e. an intercept) so that the model reflects $y=mx+b$ from simple linear regression:

$$\begin{array}{rl}\left[\begin{array}{ccc}{w}_{11}& w_{12}& w_{13}\\ w_{21}& w_{22}& w_{23}\\ w_{31}& w_{32}& w_{33}\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right]+\left[\begin{array}{c}{k}_1\\ k_2\\ k_3\end{array}\right]& =\left[\begin{array}{c}{w}_{\left[1,:\right]}\cdot \mathbf{x}\\ w_{\left[2,:\right]}\cdot \mathbf{x}\\ w_{\left[3,:\right]}\cdot \mathbf{x}\end{array}\right]+\left[\begin{array}{c}{k}_1\\ k_2\\ k_3\end{array}\right]\\ & =\left[\begin{array}{c}{w}_{\left[1,:\right]}\cdot \mathbf{x}+k_1\\ w_{\left[2,:\right]}\cdot \mathbf{x}+k_2\\ w_{\left[3,:\right]}\cdot \mathbf{x}+k_3\end{array}\right]\end{array}$$

The bias term could equally be expressed inside the matrix thusly:

$$\begin{array}{rl}\left[\begin{array}{cccc}{w}_{11}& w_{12}& w_{13}& k_1\\ w_{21}& w_{22}& w_{23}& k_2\\ w_{31}& w_{32}& w_{33}& k_3\end{array}\right]\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\\ 1\end{array}\right]& =\left[\begin{array}{c}{w}_{\left[1,:\right]}\cdot \mathbf{x}+k_1\\ w_{\left[2,:\right]}\cdot \mathbf{x}+k_2\\ w_{\left[3,:\right]}\cdot \mathbf{x}+k_3\end{array}\right]\end{array}$$

That's why we don't often include a bias term when performing multiple linear regression, as it can be included as a component of the weights matrix.

More Observations

If there were $n$ observations:

$$\begin{array}{rl}\left[\begin{array}{cccc}{w}_{11}& w_{12}& w_{13}& k_1\\ w_{21}& w_{22}& w_{23}& k_2\\ w_{31}& w_{32}& w_{33}& k_3\end{array}\right]\left[\begin{array}{ccccc}{x}_{11}& x_{12}& & x_{1n}& 1\\ x_{21}& x_{22}& \cdots & x_{2n}& 1\\ x_{n1}& x_{n2}& & x_{nn}& 1\end{array}\right]& =\left[\begin{array}{c}{w}_{\left[1,:\right]}\cdot \mathbf{x}+k_1\\ w_{\left[2,:\right]}\cdot \mathbf{x}+k_2\\ w_{\left[3,:\right]}\cdot \mathbf{x}+k_3\end{array}\right]\end{array}$$

Row Major

If we transformed this to a row-major representation, the 1s would now be a row of the input and the bias a row of the weights: $$\begin{array}{rl}{\left(\left[\begin{array}{cccc}{w}_{11}& w_{12}& w_{13}& k_1\\ w_{21}& w_{22}& w_{23}& k_2\\ w_{31}& w_{32}& w_{33}& k_3\end{array}\right]\left[\begin{array}{ccccc}{x}_{11}& x_{12}& & x_{1n}& 1\\ x_{21}& x_{22}& \cdots & x_{2n}& 1\\ x_{n1}& x_{n2}& & x_{nn}& 1\end{array}\right]\right)}^{\mathrm{T}}& ={\left[\begin{array}{c}{w}_{\left[1,:\right]}\cdot \mathbf{x}+k_1\\ w_{\left[2,:\right]}\cdot \mathbf{x}+k_2\\ w_{\left[3,:\right]}\cdot \mathbf{x}+k_3\end{array}\right]}^{\mathrm{T}}\\ {\left[\begin{array}{ccccc}{x}_{11}& x_{12}& & x_{1n}& 1\\ x_{21}& x_{22}& \cdots & x_{2n}& 1\\ x_{n1}& x_{n2}& & x_{nn}& 1\end{array}\right]}^{\mathrm{T}}{\left[\begin{array}{cccc}{w}_{11}& w_{12}& w_{13}& k_1\\ w_{21}& w_{22}& w_{23}& k_2\\ w_{31}& w_{32}& w_{33}& k_3\end{array}\right]}^{\mathrm{T}}& ={\left[\begin{array}{c}{w}_{\left[1,:\right]}\cdot \mathbf{x}+k_1\\ w_{\left[2,:\right]}\cdot \mathbf{x}+k_2\\ w_{\left[3,:\right]}\cdot \mathbf{x}+k_3\end{array}\right]}^{\mathrm{T}}\end{array}$$

A note on Memory Layout in languages

Column major matrices are rooted in mathematical notation and conventions of linear algebra. In many textbooks and papers, 1D vectors are often treated as column vectors where convenient. This makes column-major order a natural and intuitive choice for mathematical languages where matrix operations are common. The reader may have already noted that many column-major languages are also 1-indexed, this is for a smiliar reason.

The C language was developed for systems programming, presumably row-major representation was chosen because it is consistent with how people usually layout data, there was no need to cater towards mathematical programming.