2.1 Basics

In this section, we recap some basic definitions and notation. Hopefully this material will largely be familiar to you.

2.1.1 Notation

The matrix \({\mathbf A}\) will be referred to in the following equivalent ways: \[\begin{eqnarray*} {\mathbf A}=\stackrel{n\times p}{\mathbf A} &=& \left(\begin{array}{cccc} a_{11}&a_{12}&\dots&a_{1p}\\ a_{21}&a_{22}&\dots&a_{2p}\\ \vdots&\vdots&&\vdots\\ a_{n1}&a_{n2}&\dots&a_{np} \end{array} \right) \\ &=&[a_{ij}: i=1, \ldots , n; j=1, \ldots , p]\\ &=&(a_{ij})\\ &=& \left( \begin{array}{c}\mathbf a_1^\top\\ \vdots\\ \mathbf a_n^\top\end{array}\right) \end{eqnarray*}\] where the \(a_{ij}\) are the individual entries, and \(\mathbf a_i^\top=(a_{i1}, a_{i2}, \ldots, a_{ip})\) is the \(i^{th}\) row.

A matrix of order \(1\times 1\) is called a scalar.

A matrix of order \(n\times 1\) is called a (column) vector.

A matrix of order \(1\times p\) is called a (row) vector.

e.g. \(\stackrel{n\times 1}{\mathbf a}=\left( \begin{array}{c} a_1\\\vdots\\a_n \end{array} \right)\)is a column vector.

The \(n\times n\) identity matrix \({\mathbf I}_n\) has diagonal elements equal to 1 and off-diagonal elements equal to zero.

A diagonal matrix is an \(n \times n\) matrix whose off-diagonal elements are zero. Sometimes we denote a diagonal matrix by \(\text{diag}\{a_1,\ldots, a_n\}\).

\[\mathbf I_3 = \left(\begin{array}{ccc} 1&0&0\\ 0&1&0\\ 0&0&1\end{array}\right),\quad \text{diag}\{1,2,3\}=\left(\begin{array}{ccc} 1&0&0\\ 0&2&0\\ 0&0&3\end{array}\right)\quad\]

2.1.2 Elementary matrix operations

1. Addition/Subtraction. If \(\stackrel{n\times p}{\mathbf A}=[a_{ij}]\) and \(\stackrel{n\times p}{\mathbf B}=[b_{ij}]\) are given matrices then \[ {\mathbf A}+{\mathbf B}=[a_{ij}+b_{ij}] \qquad \text{and} \qquad {\mathbf A}-{\mathbf B}=[a_{ij}-b_{ij}].\]

2. Scalar Multiplication. If \(\lambda\) is a scalar and \({\mathbf A}=[a_{ij}]\) then \[\lambda {\mathbf A}=[\lambda a_{ij}].\]

3. Matrix Multiplication. If \(\stackrel{n\times p}{\mathbf A}\) and \(\stackrel{p\times q}{\mathbf B}\) are matrices then \(\mathbf A\mathbf B=\stackrel{n\times q}{\mathbf C}=[c_{ij}]\) where \[c_{ij}=\sum _{k=1}^p a_{ik}b_{kj}, \qquad i=1,\dots,n, \qquad j=1,\dots ,q.\]

4. Matrix Transpose. If \(\stackrel{m \times n}{\mathbf A}=[a_{ij}: i=1, \ldots , m; j=1, \ldots , n]\), then the transpose of \(\mathbf A\), written \(\mathbf A^\top\), is given by the \(n \times m\) matrix \[ \mathbf A^\top =[a_{ji}: j=1, \ldots , n; i=1, \ldots, m]. \] Note from the definitions that \((\mathbf A\mathbf B)^\top={\mathbf B}^\top {\mathbf A}^\top\).

5. Matrix Inverse. The inverse of a matrix \(\stackrel{n\times n}{\mathbf A}\) (if it exists) is a matrix \(\stackrel{n\times n}{\mathbf B}\) such that \({\mathbf A}\mathbf B=\mathbf B\mathbf A={\mathbf I}_n.\) We denote the inverse by \({\mathbf A}^{-1}\). Note that if \({\mathbf A}_1\) and \({\mathbf A}_2\) are both invertible, then \(({\mathbf A}_1 {\mathbf A}_2)^{-1}={\mathbf A}_2^{-1}{\mathbf A}_1^{-1}\).

6. Trace. The trace of a matrix \(\stackrel{n\times n}{\mathbf A}\) is given by \[ \text{tr}({\mathbf A})=\sum _{i=1}^n a_{ii}.\]

Lemma 2.1 For any matrices \(\mathbf A\) (\(n \times m\)) and \(\mathbf B\) (\(m \times n\)), \[ \text{tr}(\mathbf A\mathbf B) = \text{tr}(\mathbf B\mathbf A). \]

7. The determinant of a square matrix \(\stackrel{n\times n}{\mathbf A}\) is defined as \[ \text{det}({\mathbf A})=\sum (-1)^{|\tau |} a_{1\tau(1)}\dots a_{n\tau (n)} \] where the summation is taken over all permutations \(\tau\) of \(\{1,2,\dots ,n\}\), and we define \(|\tau |=0\) or \(1\) depending on whether \(\tau\) can be written as an even or odd number of transpositions.

E.g. If \({\mathbf A}=\left[ \begin{array}{cc} a_{11}&a_{12}\\ a_{21}&a_{22} \end{array} \right]\), then \(\text{det}({\mathbf A})=a_{11}a_{22}-a_{12}a_{21}\).

Proposition 2.1 Matrix \(\stackrel{n\times n}{\mathbf A}\) is invertible if and only if \(\det(\mathbf A)\not = 0\). If \(\mathbf A^{-1}\) exists then \[\det(\mathbf A)=\frac{1}{\det(\mathbf A^{-1})}\]

Proposition 2.2 For any matrices \(\stackrel{n\times n}{\mathbf A}\), \(\stackrel{n\times n}{\mathbf B}\), \(\stackrel{n\times n}{\mathbf C}\) such that \({\mathbf C}={\mathbf{AB}}\), \[ \text{det}({\mathbf C})=\text{det}({\mathbf A}) \cdot \text{det}({\mathbf B}).\]

2.1.3 Special matrices

Definition 2.1 An \(n\times n\) matrix \(\mathbf A\) is symmetric if \[\mathbf A= \mathbf A^\top.\] An \(n\times n\) symmetric matrix \(\mathbf A\) is positive-definite if \[\mathbf x^\top \mathbf A\mathbf x>0 \mbox{ for all } \mathbf x\in \mathbb{R}^n, \mathbf x\not = {\boldsymbol 0}\] and is positive semi-definite if \[\mathbf x^\top \mathbf A\mathbf x\geq 0 \mbox{ for all } \mathbf x\in \mathbb{R}^n.\]

\(\mathbf A\) is idempotent if \(\mathbf A^2=\mathbf A\).

2.1.4 Vector Differentiation

Consider a real-valued function \(f: \mathbb{R}^p \rightarrow \mathbb{R}\) of a vector variable \(\mathbf x=(x_1, \ldots , x_p)^\top\). Sometimes we will want to differentiate \(f\). We define the partial derivative of \(f(\mathbf x)\) with respect to \(\mathbf x\) to be the vector of partial derivatives, i.e. \[\begin{equation} \frac{\partial f}{\partial \mathbf x}(\mathbf x)=\left [ \begin{array}{c} \frac{\partial f}{\partial x_1}(\mathbf x)\\ ..\\ ..\\ ..\\ \frac{\partial f}{\partial x_p}(\mathbf x) \end{array} \right ] \tag{2.1} \end{equation}\] The following examples can be worked out directly from the definition (2.1), using the chain rule in some cases.

Example 2.1 If \(f(\mathbf x)=\mathbf a^\top \mathbf x\) where \(\mathbf a\in \mathbb{R}^p\) is a constant vector, then \[ \frac{\partial f}{\partial \mathbf x}(\mathbf x)=\mathbf a. \]

Example 2.2 If \(f(\mathbf x)=(\mathbf x-\mathbf a)^\top \mathbf A(\mathbf x-\mathbf a)\) for a fixed vector \(\mathbf a\in \mathbb{R}^p\) and \(\mathbf A\) is a symmetric constant \(p \times p\) matrix, then \[ \frac{\partial f}{\partial \mathbf x}(\mathbf x)=2\mathbf A(\mathbf x-\mathbf a). \]

Example 2.3 Suppose that \(g: \, \mathbb{R} \rightarrow \mathbb{R}\) is a differentiable function with derivative \(g^\prime\). Then, using the chain rule for partial derivatives, \[ \frac{\partial g(\mathbf a^\top \mathbf x)}{\partial \mathbf x}=g^{\prime}(\mathbf a^\top\mathbf x)\frac{\partial}{\partial \mathbf x}\left \{\mathbf a^\top \mathbf x\right \}=g^{\prime}(\mathbf a^\top\mathbf x) \mathbf a. \]

Example 2.4 If \(f\) is defined as in Example 2.2 and \(g\) is as in Example 2.3 then, using the chain rule again, \[ \frac{\partial }{\partial \mathbf x} g\{f(\mathbf x)\}=g^{\prime} \{f(\mathbf x)\}\frac{\partial f}{\partial \mathbf x}(\mathbf x) =2 g^{\prime}\{(\mathbf x- \mathbf a)^\top \mathbf A(\mathbf x- \mathbf a)\}\mathbf A(\mathbf x-\mathbf a). \]

If we wish to find a maximum or minimum of \(f(\mathbf x)\) we should search for stationary points of \(f\), i.e. solutions to the system of equations \[ \frac{\partial f}{\partial \mathbf x}(\mathbf x)\equiv \left [ \begin{array}{c} \frac{\partial f}{\partial x_1}(\mathbf x)\\ ..\\ ..\\ ..\\ \frac{\partial f}{\partial x_p}(\mathbf x) \end{array} \right ]={\mathbf 0}_p. \] ::: {.definition #hessian} The Hessian matrix of \(f\) is the \(p \times p\) matrix of second derivatives. \[ \frac{\partial^2f}{\partial \mathbf x\partial \mathbf x^\top}(\mathbf x) =\left \{ \frac{\partial^2 f(\mathbf x)}{\partial x_j \partial x_k}\right \}_{j,k=1}^p. \] :::

The nature of a stationary point is determined by the Hessian

If the Hessian is positive (negative) definite at a stationary point \(\mathbf x\), then the stationary point is a minimum (maximum).

If the Hessian has both positive and negative eigenvalues at \(\mathbf x\) then the stationary point will be a saddle point.