2.6 Exercises

  1. Are the vectors \(\left( \begin{array}{c}2\\1\end{array}\right)\), \(\left( \begin{array}{c}1\\1\end{array}\right)\), and \(\left( \begin{array}{c}-1\\2\end{array}\right)\) linearly independent?

    • Give two different bases for \(\mathbb{R}^2\)
    • Describe three different subspaces of \(\mathbb{R}^3\)

  2. Let \[\mathbf A= \left(\begin{array}{ccc}1&0&3\\ 0&2&-4\\ 1&1&1 \end{array}\right)\]

    • What is \(\operatorname{rank}(\mathbf A)\)?
    • Write the product \(\mathbf A\mathbf x\) where \(\mathbf x^\top=(x_1, x_2, x_3)\) as both the inner product of the rows of \(\mathbf A\) and \(\mathbf x\), and as a linear combination of the columns of \(\mathbf A\) (see section 2.2.2)
    • Describe the column space of \(\mathbf A\). What is its dimension?
    • Find a vector in the kernel of \(\mathbf A\).
    • Describe the kernel of \(\mathbf A\) as a vector space and give a basis for the space.
    • Is \(\mathbf A\) invertible? What is \(\det(\mathbf A)\)?



  1. Let’s consider the inner product space \(\mathbb{R}^3\) with the Euclidean inner product \[\langle \mathbf x, \mathbf y\rangle = \mathbf x^\top \mathbf y\] Let \[\mathbf x_1 = \left(\begin{array}{c}1\\2\\0\end{array}\right), \quad \mathbf x_2 = \left(\begin{array}{c}-1\\0\\-2\end{array}\right). \quad \mathbf x_3 = \left(\begin{array}{c}2\\-1\\1\end{array}\right) \]

    • What is the angle between \(\mathbf x_1\) and \(\mathbf x_2\)? Which pairs of vectors are orthogonal to each other?
    • What is the norm associated with this inner-product space, and compute the norm of \(\mathbf x_1, \ldots, \mathbf x_3\). What is the geometric interpretation of the norm?



  1. Prove the following statements:
    • The determinant of an orthogonal matrix must be either \(1\) or \(-1\).
    • If \(\mathbf A\) and \(\mathbf B\) are orthogonal matrices, then \(\mathbf A\mathbf B\) must also be orthogonal.
    • Let \(\mathbf A\) be an \(n\times n\) matrix of the form \[\mathbf A= \mathbf Q\mathbf B\mathbf Q^\top.\] where \(\mathbf Q\) is an \(n\times n\) matrix, and \(\mathbf B\) is an \(n\times n\) diagonal matrix. Prove that \(\mathbf A\) is symmetric.



  1. Consider the matrix \[\mathbf P=\left( \begin{array}{cc}1&0\\ 0&0\end{array}\right).\]

    • Show that \(\mathbf P\) is a projection matrix.
    • Describe the subspace \(\mathbf P\) projects onto.
    • Describe the image and kernel of \(\mathbf P\).
    • Repeat the above questions using \(\mathbf I-\mathbf P\) and check proposition 2.4.

  2. Let \(W=\mathbb{R}^3\) with the usual inner product. Consider the orthogonal projection from \(W\) onto the subspace \(U\) defined by \[U=\operatorname{span}\left\{\left(\begin{array}{c} 1\\1\\1\end{array}\right)\right\}.\]

    • Write down the orthogonal projection matrix for the projection \(W\) onto \(U\) and check it is a projection matrix.
    • What is the projection of the vector \[\mathbf v=\left(\begin{array}{c} 1\\1\\0\end{array}\right)\] onto \(U\)? Show that this vector does minimize \(||\mathbf v-\mathbf u||\) for \(\mathbf u\in U\).

  3. The centering matrix will be play an important role in this module, as we will use it to remove the column means from a matrix (so that each column has mean zero), centering the matrix.

    Define \[\mathbf H=\mathbf I_n - n^{-1}{\bf 1}_n {\bf 1}_n^\top\] to be the \(n \times n\) centering matrix (see 2.4).
    Let \(\mathbf x=(x_1, \ldots , x_n)^\top\) denote a vector and let \(\mathbf X=[\mathbf x_1 ,\ldots , \mathbf x_n]^\top\) denote an \(n \times p\) data matrix.

    Define the scalar sample mean to be \(\bar{x}=n^{-1}\sum_{i=1}^n x_i\) and the sample mean vector to be \(\bar{\mathbf x}=n^{-1} \sum_{i=1}^n \mathbf x_i\).

    1. Show by direct calculation that \(\mathbf H\) is a projection matrix, i.e. \(\mathbf H^\top = \mathbf H\) and \(\mathbf H^2 =\mathbf H\).

    2. Show that \[ \mathbf H\mathbf x= \mathbf x- \bar{x}{\bf 1}_n = ( x_1 - \bar{x}, \ldots , x_n -\bar{x})^\top. \] Hint: first show that \(n^{-1}{\bf 1}_n^\top \mathbf x=\bar{x}\).

    3. Show that \[ \mathbf x^\top \mathbf H\mathbf x= \sum_{i=1}^n (x_i-\bar{x})^2. \] Hint: use the fact that \(\mathbf H\) is a projection matrix and hence express \(\mathbf x^\top \mathbf H\mathbf x\) as a scalar product of \(\mathbf H\mathbf x\) with itself.

    4. Assuming \(\mathbf X\) is an \(n \times p\) matrix, show that \[ \mathbf H\mathbf X=[\mathbf x_1 - \bar{\mathbf x}, \ldots , \mathbf x_n -\bar{\mathbf x}]^\top. \] Hint: first show that \(n^{-1} {\bf 1}_n^\top \mathbf X=\bar{\mathbf x}^\top\).

    5. Using \(\mathbf S\) to denote the sample covariance matrix, show that \[\begin{equation} \mathbf X^\top \mathbf H\mathbf X= \sum_{i=1}^n (\mathbf x_i -\bar{\mathbf x})(\mathbf x_i -\bar{\mathbf x})^\top = n\mathbf S, \tag{2.3} \end{equation}\] Hint: using the fact that \(\mathbf H\) is a projection matrix, show that \(\mathbf X^\top \mathbf H\mathbf X=(\mathbf H\mathbf X)^\top (\mathbf H\mathbf X)\).

    Comment: Equation (2.3) provides a convenient way to calculate the sample covariance matrix directly in R, given the data matrix \(\mathbf X\).