3.1 Matrix-matrix products

Before we can introduce the SVD, we first need to recap some basic material on matrix multiplication and eigenvalues. We saw in section 2.2.2 that we can think about matrix-vector products in two ways: \(\mathbf A\mathbf x\) is rows of \(\mathbf A\) times \(\mathbf x\); or as a linear combination of the columns of \(\mathbf A\). We can similarly think about matrix-matrix products in two ways.

The usual way to think about the matrix product \(\mathbf A\mathbf B\) is as the rows of \(\mathbf A\) times the columns of \(\mathbf B\):

\[\left[ \begin{array}{ccc} \cdot & \cdot &\cdot\\ a_{21}&a_{22}&a_{23}\\ \cdot & \cdot &\cdot \end{array} \right]\left[\begin{array}{ccc} \cdot & b_{12} &\cdot\\ \cdot&b_{22}&\cdot\\ \cdot & b_{32} &\cdot \end{array} \right]\]

A better way (for this module) to think of \(\mathbf A\mathbf B\) is as the columns of \(\mathbf A\) times the rows of \(\mathbf B\). If we let \(\mathbf a_i\) denote the columns of \(\mathbf A\), and \(\mathbf b^*_i\) the rows of \(\mathbf B\) then

\[\left[ \begin{array}{ccc} | & | &|\\ \mathbf a_{1}&\mathbf a_{2}&\mathbf a_{3}\\ | & | &| \end{array} \right]\left[\begin{array}{ccc} - & \mathbf b_{1}^* &-\\ -&\mathbf b_{2}^*&-\\ - & \mathbf b_{3}^* &- \end{array} \right]=\sum_{i=1}^3\mathbf a_i \mathbf b_i^*\]

i.e., \(\mathbf A\mathbf B\) is a sum of the columns of \(\mathbf A\) times the rows of \(\mathbf B\).

Note that if \(\mathbf a\) is a vector of length \(n\) and \(\mathbf b\) is a vector of length \(p\) then \(\mathbf a\mathbf b^\top\) is an \(n\times p\) matrix.

Example 3.1 \[\left( \begin{array}{c} 1\\ 2\end{array} \right) \left(2 \; 3\; 1\right)= \left(\begin{array}{ccc} 2&3&1\\ 4&6&2 \end{array} \right). \]

Note that \(\mathbf a\mathbf b^\top\) is a rank-1 matrix as its columns are all multiples of \(\mathbf a\), or in other words, its column space is just multiples of \(\mathbf a\). \[\mathcal{C}(\mathbf a\mathbf b^\top) = \{\lambda \mathbf a: \lambda \in \mathbb{R}\}.\] We sometimes call \(\mathbf a\mathbf b^\top\) the outer product of \(\mathbf a\) with \(\mathbf b\).

By thinking of matrix-matrix multiplication in this way \[\mathbf A\mathbf B=\sum_{i=1}^k \mathbf a_i \mathbf b_i^*\] (where \(k\) is the number of columns of \(\mathbf A\) and the number of rows of \(\mathbf B\)) we can see that the product is a sum of rank-1 matrices. We can think of rank-1 matrices as the building blocks of matrices.

This chapter is about ways of decomposing matrices into their most important parts, and we will do this by thinking about the most important rank-1 building blocks.

Firstly though, we need a recap on eigenvectors.