Orthogonality
Orthogonal Vectors
When we refer to an inner product on these pages (for numerical linear algebra), we will always mean the standard Euclidean inner product.
Proposition. $\langle x, \alpha y + \beta z\rangle = \alpha\langle x, y\rangle + \beta\langle x, z \rangle$ for all $\alpha, \beta \in \mathbb{F}$ and $x, y, z \in \mathbb{F}^m$.
Proof. Follows from distributivity of matrix multiplication.
Proposition. $\langle \alpha x + \beta y, z\rangle = \overline{\alpha}\langle x, z\rangle + \overline{\beta}\langle y, z \rangle$ for all $\alpha, \beta \in \mathbb{F}$ and $x, y, z \in \mathbb{F}^m$.
Proof. Follows from distributivity of matrix multiplication.
Proposition. $\overline{\langle x, y \rangle} = \langle y, x \rangle$.
Proof. $\overline{x^y} = y^x$.
As always, the norm induced by the standard inner product is the usual $2$-norm and the angle $\theta$ between two vectors $x$ and $y$ is related to the inner product by
\[\cos\theta = \mathrm{Re}\frac{\langle x, y \rangle}{\|x\|\|y\|}.\]As usual, we say that vectors $x$ and $y$ are orthogonal if their inner product is 0. We may write $x \perp y$ to denote this relationship. More generally, we say that a sets $X$ and $Y$ are orthogonal if $\langle x, y\rangle = 0$ for any $x \in X$ and $y\in Y$.
There is also a notion of an orthogonal set $S$ of (nonzero) vectors whose elements are pairwise orthogonal. If, in addition to orthogonality, every element of $S$ has unit norm, then $S$ is orthonormal.
Theorem. If $S \subseteq \mathbb{F}^m$ is an orthogonal set, then it is linearly independent.
Proof. If $S$ is linearly dependent, then we can write some $v \in S$ as a (finite) linear combination
\[v = \sum_{i} c_i v_i\]where each $c_i \in \mathbb{F}$ is nonzero and $v_i \in S \setminus {v}$. Then
\[0 = \langle v_j, v\rangle = \sum_{j} c_i \langle v_j, v_i \rangle = c_j \|v_j\|\]where $v_j$ is one of the $v_i$’s that occurs in the linear combination making up $v$. Recall that $|v_j| = 0$ happens only if $v_j = 0$. Thus, each $v_j = 0$. This implies that one of the vectors in $S$ is zero, contradiction.
Corollary. If $S \subseteq \mathbb{F}^m$ is an orthogonal set with $m$ vectors, then it forms a basis for $\mathbb{F}^m$.
Corollary. If $S \subseteq \mathbb{F}^m$ is an orthogonal set, then it must have at most $m$ vectors.
Orthogonal Projections
Let ${q_1, \ldots, q_n} \subseteq \mathbb{F}^m$ be orthonormal and $S$ be the subspace spanned by this set. The theorem above implies that the $q_i$’s form a basis for $S$. Our goal is to find the projection of some $v \in \mathbb{F}^m$ onto $S$. Define
\[r = v - \langle q_1, v\rangle q_1 - \cdots - \langle q_n, v\rangle q_n.\]This vector is orthogonal to the $q_j$’s since
\[\langle q_i, r\rangle = \langle q_i, v\rangle - \langle q_i, v\rangle\langle q_i, q_i\rangle = \langle q_i, v\rangle - \langle q_i, v\rangle = 0,\]where the second equality follows from normality of the $q_j$’s. So, then, we see that
\[v = r + \sum_{i=1}^n \langle q_i, v\rangle q_i.\]If we allow for the abuse of notation with obvious intent that $x^* y$ is a scalar, then we can say that
\[v = r + \sum_{i=1}^n (q_i^*v) q_i = r + \sum_{i=1}^n (q_iq_i^*) v.\]Each outer product in the sum is a projection matrix which gives the orthogonal projection of $v$ along $q_i$. Accordingly, this tells us that the projection matrix that gives the orthogonal projection of $v$ onto the subspace $S$ is given by
\[\sum_{i=1}^n (q_iq_i^*).\]Notice that if we write $Q = [q_1 \mid \cdots \mid q_n] \in \mathbb{F}^{m\times n}$, then the above matrix is just $QQ^*$.
Unitary Matrices
We say that a square matrix $Q \in \mathbb{F}^{m\times m}$ is unitary (if $\mathbb{F} = \mathbb{R}$, then we say orthogonal) provided that $Q^*Q = I$.
Proposition. If $Q$ is orthogonal, then its adjoint is its inverse.
Proof. Write $Q$ as $Q = [q_1 \mid \cdots \mid q_m]$. Since the $j$th column of $Q^Q$ is given by $Q^q_j = e_j$, it follows that the columns of $Q$ form an orthonormal set. So $Q$ is invertible from which it follows that
\[Q^* Q = I \quad\implies\quad Q^* = Q^{-1}\]by multiplying by $Q^{-1}$ on the right.
Corollary. $QQ^* = I$.
Theorem. If $Q$ is unitary, then $|Qx| = |x|$.
Proof. $\langle Qx, Qx\rangle = \langle Q^*Qx, x\rangle = \langle x, x\rangle$. Now just take a square root of both sides.
Notice that the above simply states that unitary matrices are isometries. Intuitively, this means that we can think of them as reflections and rotations. The next proposition shows that this is the case.
| Proposition. If $Q$ is unitary, then $ | \det Q | ^2 = 1$. |
Proof. Since $QQ^* = I$, it follows that
\[1 = \det I = \det(QQ^*) = (\det Q)(\det Q^*) = (\det Q)\overline{(\det Q)} = |\det Q|^2.\]Accordingly, $\det Q$ has unit norm and so it acts by rotation/reflection on $\mathbb{F}^m$. This geometric intuition will useful later for motivating the singular value decomposition of a matrix.