1.3 Random vectors and matrices

Definition 1.4 The population mean vector of the random vector \(\mathbf x\) is \[{\boldsymbol{\mu}}= {\mathbb{E}}(\mathbf x).\]

The population covariance matrix of \(\mathbf x\) is \[ \boldsymbol{\Sigma}= {\mathbb{V}\operatorname{ar}}(\mathbf x) = {\mathbb{E}}\left((\mathbf x-{\mathbb{E}}(\mathbf x))(\mathbf x-{\mathbb{E}}(\mathbf x))^\top \right).\]

The covariance between \(\mathbf x\) (\(p \times 1\)) and \(\mathbf y\) (\(q \times 1\)) is \[ {\mathbb{C}\operatorname{ov}}(\mathbf x,\mathbf y) = {\mathbb{E}}\left((\mathbf x- {\mathbb{E}}(\mathbf x))(\mathbf y- {\mathbb{E}}(\mathbf y))^\top \right). \]

Let \(\mathbf A\) denote a \(q \times p\) constant matrix, and let \(\mathbf b\) a constant vector of size \(q \times 1\). Expectation is a linear operator in the sense that

\[{\mathbb{E}}(\mathbf A\mathbf x+ \mathbf b) = \mathbf A{\mathbb{E}}(\mathbf x) + \mathbf b=\mathbf A{\boldsymbol{\mu}}+\mathbf b.\]

The following properties follow:

• \({\mathbb{V}\operatorname{ar}}(\mathbf x) = {\mathbb{E}}(\mathbf x\mathbf x^\top) - {\boldsymbol{\mu}}{\boldsymbol{\mu}}^\top\).
• \({\mathbb{V}\operatorname{ar}}(\mathbf A\mathbf x+ \mathbf b) = \mathbf A\boldsymbol{\Sigma}\mathbf A^\top\)
• \({\mathbb{C}\operatorname{ov}}(\mathbf x,\mathbf y) = {\mathbb{E}}(\mathbf x\mathbf y^\top) - {\mathbb{E}}(\mathbf x) {\mathbb{E}}(\mathbf y)^\top\).
• \({\mathbb{C}\operatorname{ov}}(\mathbf x,\mathbf x) = \boldsymbol{\Sigma}\).
• \({\mathbb{C}\operatorname{ov}}(\mathbf x,\mathbf y) = {\mathbb{C}\operatorname{ov}}(\mathbf y,\mathbf x)^\top\).
• \({\mathbb{C}\operatorname{ov}}(\mathbf A\mathbf x,\mathbf B\mathbf y) = \mathbf A{\mathbb{C}\operatorname{ov}}(\mathbf x,\mathbf y)\mathbf B^\top\)
• If \(p=q\) then \[ {\mathbb{V}\operatorname{ar}}(\mathbf x+ \mathbf y) = {\mathbb{V}\operatorname{ar}}(\mathbf x) + {\mathbb{V}\operatorname{ar}}(\mathbf y) + {\mathbb{C}\operatorname{ov}}(\mathbf x,\mathbf y) + {\mathbb{C}\operatorname{ov}}(\mathbf y,\mathbf x). \]

Finally, note that if \(\mathbf x\) and \(\mathbf y\) are independent (in which case I will write \(\mathbf x\perp \!\!\! \perp\mathbf y\)) then \({\mathbb{C}\operatorname{ov}}(\mathbf x,\mathbf y) = {\mathbf 0}_{p,q}\), i.e., a \(p\times q\) matrix of zeros.

1.3.1 Estimators

The population mean vector \({\boldsymbol{\mu}}\) and population covariance matrix \(\boldsymbol{\Sigma}\) will usually be unknown. We can use data to estimate these quantities.

• The sample mean \(\bar{\mathbf x}\) is often used as an estimator of \({\boldsymbol{\mu}}\).
• The sample covariance matrix \(\mathbf S\) is often used as an estimator of \(\boldsymbol{\Sigma}\).

Equation (1.1) gives an unbiased estimator of the sample mean. The sample covariance matrix (1.2) is a biased estimator of the population covariance matrix. An unbiased estimate is obtained by dividing by \(n-1\) rather than \(n\) in Equation (1.2).