10.4 Multioutput Linear Model

In the standard linear model the responses \(y_i\) are univariate. In the multivariate linear model, the data are \(\{\mathbf x_1, \mathbf y_1\}, \ldots, \{\mathbf x_n, \mathbf y_n\}\) with the responses \(\mathbf y_i \in \mathbb{R}^q\). Here, the linear model takes the form \[\begin{equation} \mathbf y_i= \mathbf B^\top \mathbf x_i +{\pmb \epsilon}_i, \qquad i=1, \ldots , n, \tag{10.5} \end{equation}\] where \(\mathbf B\) is now a \(p \times q\) parameter matrix, and we have an error vector \({\pmb \epsilon}_i\in \mathbb{R}^q\) are \(q \times 1\). The model (10.5) may be written in matrix form as \[\begin{equation} \mathbf Y= \mathbf X\mathbf B+\mathbf E, \tag{10.6} \end{equation}\] where \[\mathbf Y= \begin{pmatrix} - & \mathbf y_1^\top &-\\ &\vdots&\\ -&\mathbf y_n^\top&-\end{pmatrix}\] is the \(n \times q\) data matrix for the \(y\)-variables, \(\mathbf X\) is the \(n \times p\) data matrix as defined before, and \[\stackrel{n \times p}{\mathbf E}=\begin{pmatrix} - & \boldsymbol \epsilon_1^\top &-\\ &\vdots&\\ -&\boldsymbol \epsilon_n^\top&-\end{pmatrix}.\]

We will assume that \[\begin{equation} {\pmb \epsilon_1}, \ldots , {\pmb \epsilon}_n \quad \text{are IID}\quad N_p({\mathbf 0}_p, \boldsymbol{\Sigma}). \tag{10.7} \end{equation}\]

Proposition 10.1 The maximum likelihood estimator of \(\mathbf B\) is

\[\begin{equation} \hat{\mathbf B}= (\mathbf X^\top \mathbf X)^{-1}\mathbf X^\top \mathbf Y \tag{10.8} \end{equation}\]

Remarks:

  1. note how similar (10.8) is to its univariate counterpart (10.3). The only thing that is different is that \(\mathbf Y\) is now an \(n \times q\) matrix rather than an \(n \times 1\) vector.

  2. The coefficients for the \(k^{th}\) output, i.e., the \(k^{th}\) column of \(\mathbf B\), are just the OLS estimates from regressing the \(k^{th}\) column of \(\mathbf Y\) on \(\mathbf X\). I.e., the multivariate linear regression estimates are just the univariate estimates applied to each output in turn: the different outputs do not affect each others least squares estimates.

You will see how to fit multivariate linear models in R in the computer tasks.