Orthogonal matrices

Definition

Two non-zero vectors $x$ and $y$ are said to be orthogonal if $x^T y = 0$. A set $\{v_1,v_2,\ldots,v_m\}$ of vectors is said to be orthogonal if $v_j^T v_i = 0$ for all $i \neq j$.

Definition

Vectors $\{v_1,v_2,\ldots,v_m\}$ are said to be linearly independent if for scalars $c_1,c_2,\ldots,c_m$

$$c_1 v_1 + c_2 v_2 + \cdots + c_m v_m = 0$$

if and only if $c_j = 0$ for all $j$.

The Gram-Schmidt procedure

Let $\{x_1,x_2,\ldots,x_m\}$ be a set of linearly independent vectors in $\mathbb R^n$, $m \leq n$. Then $\{v_1, v_2, \ldots, v_m\}$ defined by

$$ v_1 = x_1,$$$$ v_2 = x_2 - \left( \frac{v_1^Tx_2}{v_1^Tv_1} \right) v_1,$$$$ v_2 = x_3 - \left( \frac{v_1^Tx_3}{v_1^Tv_1} \right) v_1- \left( \frac{v_2^Tx_3}{v_2^Tv_2} \right) v_2,$$$$\vdots$$$$ v_k = x_k - \sum_{i-1}^{m-1} \left( \frac{v_i^Tx_k}{v_i^Tv_i} \right) v_i.$$

is an orthogonal set of vectors.

Usually, we then define $q_i = v_i/\|v_i\|_2$ to get unit vectors. The vectors $\{q_i\}$ form an orthonormal set.

Consider the $n \times m$ matrix

$$Q = [q_1,q_2, \ldots, q_m],$$

whose columns are the orthonormal vectors $q_j$. It the follows that

$$(Q^TQ)_{ij} = q_i^Tq_j = \begin{cases} 1, & i = j,\\ 0, & i \neq j.\end{cases}$$

Or, $Q^TQ = I$, the $m\times m$ identity matrix.

Definition

A matrix $Q$ is said to be orthogonal if its columns form an orthonormal set in $\mathbb R^n$.

For an $n\times n$ orthogonal matrix, $Q^TQ = I$, and the following theorem is a direct consequence of this.

Theorem (Properties of orthogonal matrices)

If $Q$ is an $n \times n$ orthogonal matrix then

1. $Q$ is invertible and $Q^T = Q^{-1}$.
2. For any $x$ and $y$ in $\mathbb R^n$, $(Qx)^TQy = x^Ty$.
3. For any $x \in \mathbb R^n$, $\|Qx\|_2 = \|x\|_2$.

This last point shows that $Q$ preserves $l_2$ distances.

Similarity transformations

Definition

Two matrices $A$ and $B$ are said to be similar if there exists an invertible matrix $S$ such that $A = S^{-1} B S$.

Theorem

If $A$ and $B$ are similar then they have the same eigenvalues.

Proof

If $\lambda$ is an eigenvalue of $A$ then $Av= \lambda v$ for a non-zero vector $v$. If we multiply this equation by $S^{-1}$ we obtain

$$ S^{-1}A v = \lambda S^{-1} v,$$$$ S^{-1}AS S^{-1} v = \lambda S^{-1} v,$$$$ B w = \lambda w, ~~ w = S^{-1}v.$$

It follows that $w = S^{-1}v \neq 0$ and $\lambda$ is an eigenvalue of $B$. Conversely, if $\lambda$ is an eigenvalue of $B$, $Bw = \lambda w$ for a non-zero vector $w$. We multiply this equation by $S$

$$ SB w = \lambda S w,$$$$ ASw = \lambda S w,$$$$ A v = \lambda v, ~~ v = Sw.$$

And so, $\lambda$ is an eigenvalue of $A$. This shows that the eigenvalues coincide.

Theorem

An $n\times n$ matrix is similar to a diagonal matrix $D$ if and only if $A$ has $n$ linearly independent eigenvectors. In this case, the matrix $S$ in $A = S D S^{-1}$ has its columns being the eigenvectors of $A$.

Definition

The transformation $A \mapsto S A S^{-1}$ is called a similarity transformation of $A$.

Recall that a triangular matrix has its eigenvalues on the diagonal. The following theorem is of great importance.

Theorem (Schur)

For any matrix $A$ there exits a non-singular matrix $U$ such that

$$T = U^{-1} A U,$$

where $T$ is an upper-triangular matrix.

Note that $U$ will be, in general, a complex matrix. Define $U^* = \bar{U}^T$ where the $\bar U$ denotes complex conjugation. It a component of Schur's theorem is that $U^{-1}= U^*$ and $U$ is called unitary (the complex analogue of an orthogonal matrix).

Recall that the Spectral Theorem (from Lecture 17) states that $T$ can be chosen to be diagonal.