2.1 Matrix-matrix products

Before we can introduce the SVD, we first need to recap some basic material on matrix multiplication and eigenvalues. We saw in section 1.2.2 that we can think about matrix-vector products in two ways: \(\bA\bx\) is rows of \(\bA\) times \(\bx\); or as a linear combination of the columns of \(\bA\). We can similarly think about matrix-matrix products in two ways.

The usual way to think about the matrix product \(\bA \bB\) is as the rows of \(\bA\) times the columns of \(\bB\):

\[\left[ \begin{array}{ccc} \cdot & \cdot &\cdot\\ a_{21}&a_{22}&a_{23}\\ \cdot & \cdot &\cdot \end{array} \right]\left[\begin{array}{ccc} \cdot & b_{12} &\cdot\\ \cdot&b_{22}&\cdot\\ \cdot & b_{32} &\cdot \end{array} \right]\]

A better way (for this module) to think of \(\bA\bB\) is as the columns of \(\bA\) times the rows of \(\bB\). If we let \(\ba_i\) denote the columns of \(\bA\), and \(\bb^*_i\) the rows of \(\bB\) then

\[\left[ \begin{array}{ccc} | & | &|\\ \ba_{1}&\ba_{2}&\ba_{3}\\ | & | &| \end{array} \right]\left[\begin{array}{ccc} - & \bb_{1}^* &-\\ -&\bb_{2}^*&-\\ - & \bb_{3}^* &- \end{array} \right]=\sum_{i=1}^3\ba_i \bb_i^*\]

i.e., \(\bA\bB\) is a sum of the columns of \(\bA\) times the rows of \(\bB\).

Note that if \(\ba\) is a vector of length \(n\) and \(\bb\) is a vector of length \(p\) then \(\ba \bb^\top\) is an \(n\times p\) matrix.

Example 2.1 \[\left( \begin{array}{c} 1\\ 2\end{array} \right) \left(2 \; 3\; 1\right)= \left(\begin{array}{ccc} 2&3&1\\ 4&6&2 \end{array} \right). \]

Note that \(\ba \bb^\top\) is a rank-1 matrix as its columns are all multiples of \(\ba\), or in other words, its column space is just multiples of \(\ba\). \[\mathcal{C}(\ba \bb^\top) = \{\lambda \ba: \lambda \in \mathbb{R}\}.\] We sometimes call \(\ba \bb^\top\) the outer product of \(\ba\) with \(\bb\).

By thinking of matrix-matrix multiplication in this way \[\bA\bB=\sum_{i=1}^k \ba_i \bb_i^*\] (where \(k\) is the number of columns of \(\bA\) and the number of rows of \(\bB\)) we can see that the product is a sum of rank-1 matrices. We can think of rank-1 matrices as the building blocks of matrices.

This chapter is about ways of decomposing matrices into their most important parts, and we will do this by thinking about the most important rank-1 building blocks.

Firstly though, we need a recap on eigenvectors.