2.6 Exercises

  1. Are the vectors \(\left( \begin{array}{c}2\\1\end{array}\right)\), \(\left( \begin{array}{c}1\\1\end{array}\right)\), and \(\left( \begin{array}{c}-1\\2\end{array}\right)\) linearly independent?

    • Give two different bases for \(\mathbb{R}^2\)
    • Describe three different subspaces of \(\mathbb{R}^3\)

  2. Let \[\mathbf A= \left(\begin{array}{ccc}1&0&3\\ 0&2&-4\\ 1&1&1 \end{array}\right)\]

    • What is \(\operatorname{rank}(\mathbf A)\)?
    • Write the product \(\mathbf A\mathbf x\) where \(\mathbf x^\top=(x_1, x_2, x_3)\) as both the inner product of the rows of \(\mathbf A\) and \(\mathbf x\), and as a linear combination of the columns of \(\mathbf A\) (see section 2.2.2)
    • Describe the column space of \(\mathbf A\). What is its dimension?
    • Find a vector in the kernel of \(\mathbf A\).
    • Describe the kernel of \(\mathbf A\) as a vector space and give a basis for the space.
    • Is \(\mathbf A\) invertible? What is \(\det(\mathbf A)\)?



  1. Let’s consider the inner product space \(\mathbb{R}^3\) with the Euclidean inner product \[\langle \mathbf x, \mathbf y\rangle = \mathbf x^\top \mathbf y\] Let \[\mathbf x_1 = \left(\begin{array}{c}1\\2\\0\end{array}\right), \quad \mathbf x_2 = \left(\begin{array}{c}-1\\0\\-2\end{array}\right). \quad \mathbf x_3 = \left(\begin{array}{c}2\\-1\\1\end{array}\right) \]

    • What is the angle between \(\mathbf x_1\) and \(\mathbf x_2\)? Which pairs of vectors are orthogonal to each other?
    • What is the norm associated with this inner-product space, and compute the norm of \(\mathbf x_1, \ldots, \mathbf x_3\). What is the geometric interpretation of the norm?



  1. Prove the following statements:
    • The determinant of an orthogonal matrix must be either \(1\) or \(-1\).
    • If \(\mathbf A\) and \(\mathbf B\) are orthogonal matrices, then \(\mathbf A\mathbf B\) must also be orthogonal.
    • Let \(\mathbf A\) be an \(n\times n\) matrix of the form \[\mathbf A= \mathbf Q\mathbf B\mathbf Q^\top.\] where \(\mathbf Q\) is an \(n\times n\) matrix, and \(\mathbf B\) is an \(n\times n\) diagonal matrix. Prove that \(\mathbf A\) is symmetric.



  1. Consider the matrix \[\mathbf P=\left( \begin{array}{cc}1&0\\ 0&0\end{array}\right).\]

    • Show that \(\mathbf P\) is a projection matrix.
    • Describe the subspace \(\mathbf P\) projects onto.
    • Describe the image and kernel of \(\mathbf P\).
    • Repeat the above questions using \(\mathbf I-\mathbf P\) and check proposition 2.4.

  2. Let \(W=\mathbb{R}^3\) with the usual inner product. Consider the orthogonal projection from \(W\) onto the subspace \(U\) defined by \[U=\operatorname{span}\left\{\left(\begin{array}{c} 1\\1\\1\end{array}\right)\right\}.\]

    • What is projection of the vector \[\mathbf v=\left(\begin{array}{c} 1\\1\\0\end{array}\right)\] onto \(U\)? Show that this vector does minimize \(||\mathbf v-\mathbf u||\) for \(\mathbf u\in U\).
    • Write down the orthogonal projection matrix for the projection \(W\) onto \(U\) and check it is a projection matrix. Check your answer to the previous part of the question.

  3. The centering matrix will be play an important role in this module, as we will use it to remove the column means from a matrix (so that each column has mean zero), centering the matrix.

    Define \[\mathbf H=\mathbf I_n - n^{-1}{\bf 1}_n {\bf 1}_n^\top\] to be the \(n \times n\) centering matrix (see 2.4).
    Let \(\mathbf x=(x_1, \ldots , x_n)^\top\) denote a vector and let \(\mathbf X=[\mathbf x_1 ,\ldots , \mathbf x_n]^\top\) denote an \(n \times p\) data matrix.

    Define the scalar sample mean to be \(\bar{x}=n^{-1}\sum_{i=1}^n x_i\) and the sample mean vector to be \(\bar{\mathbf x}=n^{-1} \sum_{i=1}^n \mathbf x_i\).

    1. Show by direct calculation that \(\mathbf H\) is a projection matrix, i.e. \(\mathbf H^\top = \mathbf H\) and \(\mathbf H^2 =\mathbf H\).

    2. Show that \[ \mathbf H\mathbf x= \mathbf x- \bar{x}{\bf 1}_n = ( x_1 - \bar{x}, \ldots , x_n -\bar{x})^\top. \] Hint: first show that \(n^{-1}{\bf 1}_n^\top \mathbf x=\bar{x}\).

    3. Show that \[ \mathbf x^\top \mathbf H\mathbf x= \sum_{i=1}^n (x_i-\bar{x})^2. \] Hint: use the fact that \(\mathbf H\) is a projection matrix and hence express \(\mathbf x^\top \mathbf H\mathbf x\) as a scalar product of \(\mathbf H\mathbf x\) with itself.
    4. Assuming \(\mathbf X\) is an \(n \times p\) matrix, show that \[ \mathbf H\mathbf X=[\mathbf x_1 - \bar{\mathbf x}, \ldots , \mathbf x_n -\bar{\mathbf x}]^\top. \] Hint: first show that \(n^{-1} {\bf 1}_n^\top \mathbf X=\bar{\mathbf x}^\top\).

    5. Using \(\mathbf S\) to denote the sample covariance matrix, show that \[\begin{equation} \mathbf X^\top \mathbf H\mathbf X= \sum_{i=1}^n (\mathbf x_i -\bar{\mathbf x})(\mathbf x_i -\bar{\mathbf x})^\top = n\mathbf S, \tag{2.3} \end{equation}\] Hint: using the fact that \(\mathbf H\) is a projection matrix, show that \(\mathbf X^\top \mathbf H\mathbf X=(\mathbf H\mathbf X)^\top (\mathbf H\mathbf X)\).

    Comment: Equation (2.3) provides a convenient way to calculate the sample covariance matrix directly in R, given the data matrix \(\mathbf X\).