Back to Algebra Page

On this page:

• Introduction
• Inner Product Spaces
• Normed Spaces
• Isometries
• Orthogonality
• Gram-Schmidt
• Fourier Expansion
• Riesz Representation Theorem
• Worked Exercises

Introduction

Inner products and notions of orthogonality are ubiquitous throughout mathematics. As such, we will spend time investigating inner product spaces.

Inner Product Spaces

[definition] [deftitle]Definition %counter% (Inner product space)[/deftitle]

Let $V$ be a vector space over $F$ where $F = \RR$ or $F = \CC$. An inner product on $V$ is a mapping $\innerprod{-}{-}:V \to V \to F$ such that

1. (Positive definiteness) For every $v\in V$, $\innerprod{v}{v} \ge 0$ and $\innerprod{v}{v} = 0$ if and only if $v = 0$.
2. (Conjugate symmetry) For all $u,v \in V$, $\innerprod{u}{v} = \overline{\innerprod{v}{u}}$.
3. (Linearity in first argument) For every $u, v, w\in V$ and $r, s\in F$, $\innerprod{ru + sv}{w} = r\innerprod{u}{w} + s\innerprod{v}{w}$.

A vector space $V$ paired with an inner product is called an inner product space. [/definition]

[theorem] [thmtitle]Proposition (Real inner products are $\RR$-bilinear)[/thmtitle]

Let $V$ be a real inner product space. Then the inner product is bilinear. [/theorem]

[proof] Inner products are linear in the first argument so it remains to show that a real inner product is linear in the second argument. Let $u, v, w\in V$ and $a, b \in \RR$. Then

as desired. [/proof]

[theorem] [thmtitle]Proposition (Complex inner products are sesquilinear)[/thmtitle]

Let $V$ be a complex inner product space. Then the inner product is sesquilinear. [/theorem]

[proof] Inner products are linear in the first argument so it remains to show that a complex inner product is conjugate linear in the second argument. Let $u, v, w\in V$ and $a, b \in \CC$. Then

\(\begin{align*} \innerprod{u}{av + bw} &= \overline{\innerprod{av + bw}{u}} \\ &= \overline{a}\overline{\innerprod{v}{u}} + \overline{b}\overline{\innerprod{w}{u}} \\ &= \overline{a}\innerprod{u}{v} + \overline{b}\innerprod{u}{w} \end{align*}\) [/proof]

There are some easy-to-see examples of inner product spaces.

[example] [extitle]Example %counter% (Dot product on $\RR$) [/extitle]

The vector space $\RR^n$ has a canonical inner product space structure via the dot product:

Because this example is so important, we often refer to the dot product as the standard inner product. [/example]

[example] [extitle]Example %counter% (Dot product on $\CC$) [/extitle]

The vector space $\CC^n$ has a canonical inner product space structure via the dot product:

Because this example is so important, we often refer to the dot product as the standard inner product. [/example]

[example] [extitle]Example %counter% ($\ell^2$)[/extitle]

A noteworthy inner product space (Hilbert space, in fact) is $\ell^2(F)$ where $F = \RR$ or $F = \CC$. We define $\ell^2(F)$ to be the vector space of sequences $(z_n)$ such that

The choice of inner product on this space is

Notice that the right side converges since

which is easily seen via the inequality $(|z_n| - |w_n|)^2 \ge 0$. [/example]

[example] [extitle]Example %counter% ($L^2$)[/extitle]

Another important example of an inner product space is $L^2(X, \mathcal{A}, \mu)$ where $L^2(X, \mathcal{A}, \mu)$ is the space of square integrable functions on $X$ with respect to the measure $\mu$ and $\sigma$-algebra $\mathcal{A}$. The inner product structure is given by

[/example]

[theorem] [thmtitle]Lemma %counter%[/thmtitle]

Let $V$ be an inner product space such that

for every $w \in V$. Then $u = v$. [/theorem]

[proof] The equation given is equivalent to stating

for all $w \in V$. In particular, if we choose $w = u - v$, then we obtain

and this occurs if and only if $u - v = 0$. [/proof]

Now, real and complex spaces have vastly different structures. This can be seen from the fact that multiplication by complex numbers allows for both scaling and rotation (in the naive geometric sense). This is also reflected in their inner product structures as well as we will soon see.

[theorem] [thmtitle]Theorem %counter%[/thmtitle]

Let $T$ be a linear operator on the inner product space $V$. If $\innerprod{Tv}{u} = 0$ for every $v, u\in V$, then $T = 0$. [/theorem]

[proof] Equivalently, we can just notice that this is just saying

for every $v, u \in V$ (where the second equality follows by linearity in the first argument). So the lemma implies that $Tv = 0$ for all $v \in V$ which just means that $T = 0$. [/proof]

In the complex inner product space setting, we can making a significantly strong claim because multiplication by $e^{i\theta}$ is rotation by $\theta$ radians. Accordingly, this means it suffices to just look at $\innerprod{Tv}{v} = 0$ instead.

[theorem] [thmtitle]Theorem %counter%[/thmtitle]

Let $T$ be a linear operator on the complex inner product space $V$. If $\innerprod{Tv}{v} = 0$ for every $v\in V$, then $T = 0$. [/theorem]

[proof] Let $r \in \CC$ and write $v = rx + y$ for $x, y\in V$ (note that we can actually generate the entire vector space this way by setting $r = 0$ and $y = v$). Then

Setting $r = 1$ and $r = i$ gives the equations

So it follows that $\innerprod{Tx}{y} = 0$ for all $x, y \in V$. The previous Theorem implies that this means $T = 0$. [/proof]

[example] [extitle]Remark %counter%[/extitle]

This theorem is clearly false if the vector space is a real vector space. Say $V = \RR^2$ and $T$ is the operator that rotates about the origin by $\pi/2$ radians. Clearly, $\innerprod{Tv}{v} = 0$ under the standard inner product but $T \neq 0$. [/example]

Normed Spaces

We recall the definition of a norm and some basic facts about normed spaces.

[definition] [deftitle]Definition %counter% (Normed space)[/deftitle]

Let $V$ be a vector space over $F$. A function $\norm{-}:V \to \RR$ is called a norm provided that:

1. (Positive definiteness) For every $v \in V$, $\norm{v} \ge 0$ and $\norm{x} = 0$ if and only if $v = 0$.
2. (Homogeneity) For every $c\in F$ and $v \in V$, $\norm{cv} = |c|\norm{v}$.
3. (Triangle inequality) For every $v, u\in V$, $\norm{v+u} \le \norm{v} + \norm{u}$.

A vector space $V$ paired with a norm is called a normed space. [/definition]

[example] [extitle]Example %counter% (Euclidean norm on $\RR$)[/extitle]

Recall that the Euclidean norm of a vector in $\RR^n$ is given by

This is the canonical example of a normed space. A similar setup allows us to define Euclidean norm on $\CC^n$. [/example]

[example] [extitle]Example %counter% ($p$-norm on $\RR$)[/extitle]

Recall that the $p$-norm of a vector in $\RR^n$ is given by

In analysis, one spends time showing carefully that this makes $\RR^n$ into a normed space. Note that if $p = 2$, then we just recover the Euclidean norm. A similar setup allows us to define $p$-norm on $\CC^n$. [/example]

[example] [extitle]Example %counter% ($\ell^p$ norm)[/extitle]

Recall that $\ell^p$ is the space of sequences $(z_n)$ of real (or complex) numbers for which

Taking the $p$-th root of the series above gives the $\ell^p$ norm of $(z_n)$. [/example]

[example] [extitle]Example %counter% ($L^p$ norm)[/extitle]

Let $(X, \mathcal{A}, \mu)$ be a measure space. Then

defines a norm on $L^p(X, \mathcal{A}, \mu)$ where $1 \le p < \infty$. [/example]

One particularly important norm is the following: Let $V$ be an inner product space. Then we will define

We get the first two properties of a norm immediately from the properties of the inner product. To see the third axiom, we will need another result to help us out.

[theorem] [thmtitle]Theorem %counter% (Cauchy-Schwarz inequality)[/thmtitle]

Let $V$ be an inner product space. Then

[/theorem]

[proof] Clearly, if $u$ or $v$ is zero, then the result is immediate. Assume that $u, v \neq 0$. Now consider

Set $\overline{r} = \innerprod{v}{u}/\innerprod{v}{v}$ gives

Rearranging the terms gives the desired result. [/proof]

[theorem] [thmtitle]Theorem %counter%[/thmtitle]

The norm induced by a norm is actually a norm. [/theorem]

[proof] We have shown the first two axioms above. For the triangle inequality, note that

where the inequality follows from $\Re z \le |z|$ for any $z \in \CC$ and Cauchy-Schwarz. Taking the square root of both sides gives the desired result and we are done. [/proof]

We conclude this section by mentioning the polarization identities. In short, the induced norm is so strongly connected to its inner product that the inner product can be fully recovered from the norm.

[theorem] [thmtitle]Theorem %counter% (Polarization identity for $\RR$-vector spaces)[/thmtitle]

Let $V$ be a real inner product space. Then

for every $u, v \in V$. [/theorem]

[proof] A result that we will mention, but not prove, is the parallelogram law which states that

To prove this statement, just expand the right side in terms of inner products and the claim follows immediately. In any case, applying the parallelogram law gives

and the claim follows. [/proof]

[theorem] [thmtitle]Theorem %counter% (Polarization identity for $\CC$-vector spaces)[/thmtitle]

Let $V$ be a complex inner product space. Then

for every $u, v \in V$. [/theorem]

Proving this result is pretty similar to the real case, so we omit the proof.

Isometries

[definition] [deftitle]Definition %counter% (Isometry)[/deftitle]

Let $V$ and $W$ be innerproduct spaces and $T \in \Hom(V, W)$. Then we say that $T$ is an isometry provided that

for every $u, v\in V$. If $T$ is, in addition, a bijection, then we call $T$ an isometric isomorphism and we say that $V$ and $W$ are isometrically isomorphic. [/definition]

[theorem] [thmtitle]Theorem %counter% (Isometries are injections)[/thmtitle]

Let $T:V \to W$ be an isometry. Then $T$ is injective. [/theorem]

[proof] Suppose that $v\in \ker T$. Then

Since $\innerprod{v}{v} = 0$ precisely when $v = 0$, the claim follows. [/proof]

[example] [extitle]Example %counter% (Isometry that is not a surjection)[/extitle]

Let $T:\ell^2 \to \ell^2$ be defined by

Clearly $T$ is an isometry but it is most definitely not a surjection. [/example]

An alternative way to characterize isometry is to use the induced norm.

[theorem] [thmtitle]Theorem %counter% (Isometry $\iff$ preserve norm)[/thmtitle]

Let $T:V \to W$ be a linear transformation of inner product spaces. Then $T$ is an isometry if and only if $\norm{Tv} = \norm{v}$ for every $v \in V$. [/theorem]

[proof] ($\implies$) Obvious.

($\impliedby$) Let us consider the complex case. By the polarization identity,

Since the real case can be realized in the same way by removing the “imaginary part”, the claim follows. [/proof]

Orthogonality

We finally have made it to orthogonality! In any case, the inner product induces a geometry on the vector space. As such, notions of “perpendicular” make sense to talk about in an inner product space. Here we go!

[definition] [deftitle]Definition %counter% (Orthogonal)[/deftitle]

Let $V$ be an inner product space. We say that $u, v\in V$ are orthogonal provided that

We denote this relationship by $u \perp v$. Similarly, we say that $X, Y \subseteq V$ are orthogonal provided that

We denote this relationship by $X \perp Y$. [/definition]

Given any subspace of a vector, it is very natural to ask if there is a complement to that subspace. Even of more interest are orthogonal complements.

[definition] [deftitle]Definition %counter% (Orthogonal complement)[/deftitle]

The orthogonal complement of a subset $X \subseteq V$ is the set

[/definition]

[example] [extitle]Example %counter% ($0\oplus 0\oplus \RR \perp \RR^2 \oplus 0$)[/extitle]

A trivial example is that the subspace of $\RR^3$ spanned by $e_1$ and $e_2$ (standard basis vectors) has orthogonal complement $\span (e_3)$. [/example]

A couple of easy results:

[theorem] [thmtitle]Proposition %counter%[/thmtitle]

Let $X$ be a subset of an inner product space $V$. Then $X^\perp$ is a subspace of $V$. [/theorem]

[proof] Obviously $0 \in X^\perp$. Let $u, v \in X^\perp$, $x\in X$, and $a,b\in F$. Then

Thus, $au + bv \in X^\perp$. [/proof]

[theorem] [thmtitle]Proposition %counter%[/thmtitle]

Let $S$ be a subspace of an inner product space $V$. Then $S \cap S^\perp = 0$. [/theorem]

[proof] It is immediate that $0 \in S \cap S^\perp$. Now we show that $0$ is the only element of said space. Let $v \in S \cap S^{\perp}$. Then this means that $v$ is orthogonal to itself, i.e.

This happens if and only if $v = 0$ so the claim follows. [/proof]

If we take the sum of two orthogonal subspaces, they have trivial intersection and so it follows that the sum is actually a direct sum. As well will see, this has very profound consequences.

[definition] [deftitle]Definition %counter% (Orthogonal direct sum)[/deftitle]

Let $V$ be an inner product space. We say that $V$ is an orthogonal direct sum of subspaces $S$ and $T$ if

We notate this as $S \odot T$. More generally,

provided that $V = \bigoplus_{i=1}^n S_i$ and the $S_i$ are pairwise orthogonal. [/definition]

[theorem] [thmtitle]Theorem %counter%[/thmtitle]

Let $V$ be an inner product space. Then $V = S\odot T$ if and only if $V = S \oplus T$ and $T = S^\perp$. [/theorem]

[proof] The $\impliedby$ direction is the definition of an orthogonal direct sum so we only need to show the $\implies$ direction. If $V = S\odot T$, then $T \subseteq S^\perp$. If $v \in S^\perp$, then it can be written

where $s \in S$ and $t\in T$. Both $s$ is orthogonal to both $v$ and $t$ so $s = v - t$ is orthogonal to itself which means $s = 0$, forcing $v \in T$. Thus, $T = S^\perp$. [/proof]

Gram-Schmidt

Given a set of linearly independent vectors $a_1, \ldots, a_n$, a common goal is find a list of pairwise orthogonal vectors $q_1, \ldots, q_n$ such that

for each $k\in[n]$. In the case where the vectors $a_1, \ldots, a_n$ are the columns of an $n \times n$ matrix, we are really computing

which is known as the QR factorization. Another reason for wanting to find the list $q_1, \ldots, q_n$ is because we can construct an orthogonal basis of the vector space which we know to be extremely nice to work with.

[definition] [deftitle]Definition %counter% (Orthogonal set)[/deftitle]

Let $V$ be an inner product space and let

where $I$ is some indexing set. If the $q_i$’s are all pairwise orthogonal, then we say that $X$ is an orthogonal set. If, in addition, each $q_i$ is a unit vector, then $X$ is an orthonormal set. [/definition]

[theorem] [thmtitle]Theorem %counter% (Orthogonal sets are linearly independent)[/thmtitle]

Let $X$ be an orthogonal set of nonzero vectors in the inner product space $V$. Then $X$ is linearly independent. [/theorem]

[proof] Suppose that

where each $q_i \in X$ and $a_i \in F$. Taking the inner product of both sides gives

Now, since $q_j\neq 0$, it follows that $a_j = 0$. Thus, $X$ is linearly independent. [/proof]

[theorem] [thmtitle]Theorem %counter% (Classical Gram-Schmidt Algorithm)[/thmtitle]

Let $V$ be an inner product space and let

be a sequence of vectors in $V$. Define a sequence

by setting $q_1 = b_1$ and

for $k > 1$ where

Then $\mathcal{Q}$ is an orthogonal sequence of vectors in $V$ such that

for every $k > 0$. Furthermore, $q_k = 0$ if and only if $b_k \in \span(b_1, \ldots, b_{k-1})$. [/theorem]

The result is pretty easy to prove by induction and the later claim is by construction so we omit a proof. Our final result in this section is about Hilbert bases.

[definition] [deftitle]Definition %counter% (Hilbert basis)[/deftitle]

Let $V$ be an inner product space. A maximal orthonormal set in an inner product space is called a Hilbert basis for $V$. [/definition]

[theorem] [thmtitle]Theorem %counter% (Hilbert bases always exist)[/thmtitle]

Let $V$ be an inner product space. Then $V$ has a Hilbert basis. [/theorem]

[proof] Let

Clearly this family is nonempty as all singletons of $V$ are in $\mathcal{F}$ (in particular, the trivial subspace is an element). Now suppose that we have an increasing chain $(X_i)$ in $\mathcal{F}$ and consider

If $u, v \in \mathcal{X}$, then they are both in some $X_i$ in the chain. In particular, this means that $\innerprod{u}{v}$ is zero when $u \neq v$ and $1$ when $u = v$. So $\mathcal{X}$ itself is in $\mathcal{F}$. This implies that every chain has an upper bound in $\mathcal{F}$ and Zorn’s lemma implies the existence of a maximal element of $\mathcal{F}$ as desired. [/proof]

Fourier Expansion

Perhaps one of the most useful (in an applications sense) of orthonormal bases is that one can do a Fourier expansion as follows. Say $\mathcal{Q} = {q_1, \ldots, q_n}$ is an orthonormal basis of $V$ and $v\in V$. Then

where each $c_i \in F$. We can determine each $c_i$. Take the inner product of both sides with respect to $q_j$ to obtain

Thus, it follows that

The $c_j$’s are called the Fourier coefficients of $v$ with respect to $\mathcal{Q}$ and the expansion

is called the Fourier expansion of $v$ with respect to $\mathcal{Q}$.

Riesz Representation Theorem

A classical problem of analysis is to understand what kind of objects the elements of the continuous dual space are. This is of importance as linear functionals are ubiquitous throughout mathematics. A special case arises if we have a finite-dimensional vector space in which case the continuous and algebraic duals coincide.

[theorem] [thmtitle]Theorem %counter% (Riesz Representation Theorem)[/thmtitle]

Let $V$ be a finite-dimensional inner product space. The the map

is a conjugate isomorphism (i.e. a conjugate linear bijection). [/theorem]

[proof] (Conjugate linearity) Immediate since the inner product is conjugate linear in the second argument.

(Injectivity) Suppose that $\innerprod{-}{v} = \innerprod{-}{u}$. Then, for all $x \in V$, we have

but this implies that $v = u$.

(Surjectivity) Let $L$ be a linear functional on $V$. Picking out an orthonormal basis $b_1, \ldots, b_n$ for $V$ gives the dual basis $L_1, \ldots, L_n$ with the property that

where $\delta_{ij}$ is the Kronecker delta. Let

and

Then

So setting $v = \conj{d_1}b_1 + \cdots + \conj{d_n}b_n$ yields

and we have surjectivity! [/proof]

Worked Exercises

[example] [extitle]Cooperstein Advanced Linear Algebra Exercise 5.1.8[/extitle]

Let $V$ be an inner product space. Prove that the vectors $v_1, \ldots, v_n$ are linearly independent if and only if

is invertible. [/example]

[proof] Recall that an equivalent formulation of singularity of $A$ is that the columns are not linearly independent. Keeping this in mind, consider the linear combination

The left side can be realized as

So it follows that

for every $j\in [n]$. Multiplying each equation by $\conj a_j$ gives

and adding all the equations together gives

Thus, $\sum_{i=1}^n a_iv_i = 0$. It follows that there is a nontrivial linear dependency between the $v_i$’s so the claim follows. Running through the argument in reverse gives the converse direction. [/proof]

[example] [extitle]Cooperstein Advanced Linear Algebra Exercise 5.2.14 (Modified)[/extitle]

Let $u, v$ be vectors in an inner product space $V$ and assume that $\norm{u + v} = \norm{u} + \norm{v}$. Prove that for all $c, d \in \RR$ that

Note: The original exercise asked the reader to prove that $\norm{cu + dv}^2 = c^2\norm{u}^2 + d^2\norm{v}^2$ which is a false statement. I took the liberty of slightly modifying the original exercise into one that is actually a true statement. [/example]

[proof] Since $\norm{u + v} = \norm{u} + \norm{v}$, it follows that

But as $\innerprod{u+v}{u+v} = \norm{u+v}^2$ and

it follows that $\Re\innerprod{u}{v} = \norm{u}\norm{v}$. So then

as desired. [/proof]

[example] [extitle]Cooperstein Advanced Linear Algebra Exercise 5.1.16[/extitle]

Let $V$ be an inner product space, $x \in V$ a unit vector, and $y \in V$. Prove $\innerprod{y}{x}\innerprod{x}{y} \le \innerprod{y}{y}$. [/example]

[proof] Note that

Furthermore, Cauchy-Schwarz tells us that

Thus, it follows that

as desired. [/proof]

[example] [extitle]Axler Linear Algebra Done Right Exercise 6.A.6[/extitle]

Let $V$ be an inner product space over $F$ (where $F = \RR$ or $F = \CC$). Prove that $\innerprod{u}{v} = 0$ if and only if

for all $a \in F$. [/example]

[proof] ($\implies$) Note that

as desired.

($\impliedby$) Squaring both sides of the inequality and translating to inner products gives

Now we set $a = re^{i\theta}$ where $r > 0$ (notice that this is okay because if we’re in a real inner product space, we can restrict $\theta$ to only allow it to take on values $\pm \pi$). This gives

Dividing through by $r$ and then taking the limit as $r\downarrow 0$ gives

Now suppose we write $\innerprod{u}{v} \eqqcolon x + iy$. If $x > 0$, then setting $\theta = \pi$ gives $0 \le -x$ or that $x \le 0$. So $x$ cannot be positive. Similarly, if $x < 0$, then setting $\theta = 0$ gives $0 \le x$. So $x$ cannot be negative. Thus, $x = 0$. Now if $y$ is anything but $0$, then we can set $\theta = \pm \pi/2$ to show that $0\le y \le 0$ as well. Thus, $\innerprod{u}{v} = 0$ and we are done. [/proof]

[example] [extitle]Axler Linear Algebra Done Right Exercise 6.B.16[/extitle]

Suppose $V$ is finite-dimensional. Suppose $\innerprod{-}{-}_1$ and $\innerprod{-}{-}_2$ are inner products on $V$ with corresponding norms $\norm{-}_1$ and $\norm{-}_2$. Prove that there exists a positive number $c$ such that $\norm{v}_1 \le c\norm{v}_2$ for every $v \in V$. [/example]

[proof] Let $q_1, \ldots, q_n$ be an orthonormal basis of $V$ with respect to $\innerprod{-}{-}_2$. Then if $v = \sum_{i=1}^n a_i q_i$, it follows that

where $M = \max_{i\in [n]} |a_i|$. So then we have

[/proof]