Topics in Linear Algebra

Michael Sever

1 Inner product spaces

We assume given a linear vector space V over a field F. Throughout these notes, unless explicitly stated otherwise, F = ℝ or F = ℂ. Often, but not always, V will be finite dimensional.

A \underline{norm} on V is simply a measure of the size of the elements of V, and is required to satisfy the following: For all u, v ∈ V, α ∈ F

(1.1) ∥u∥ ≥ 0, and ∥u∥ = 0 if and only if u = 0

(1.2) ∥αu∥ = |α| ∥u∥

(1.3) ∥u + v∥ ≤ ∥u∥ + ∥v∥

Since ∥u∥ = ∥u − v + v∥ ≤ ∥u − v∥ + ∥v∥, an equivalent form of the triangle inequality (1.3) is ∥u∥ − ∥v∥ ≤ ∥u − v∥.

Examples: For F = ℝ, V = ℝ^{n}, u = (u_{1} ; u_{2} ; ⋮ ; u_{n}) ∈ V, u_{1}, u_{2}, …, u_{n} ∈ ℝ:

(1.4) the ℓ_{1} norm ∥u∥_{1} = ∑_{i=1}^{n} |u_{i}|

(1.5) the ℓ_{2} norm ∥u∥_{2} = (∑_{i=1}^{n} u_{i}^{2})^{1/2}

(1.6) the ℓ_{∞} norm ∥u∥_{∞} = maximum_{i=1,2,...,n} |u_{i}|

A linear vector space V, equipped with a norm ∥·∥, is said to be \underline{complete} if every Cauchy sequence within the space converges to a limit within the space, i.e.

Let {v_{m}}_{m=1}^{∞} be a sequence of elements v_{m} in V, such that for any ε > 0 there exists N_{ε} such that for all j, k > N_{ε}, ∥v_{j} − v_{k}∥ < ε; then there exists v ∈ V such that ∥v_{m} − v∥ → 0 as m → ∞.

A complete normed linear space is called a \underline{Banach} space.

Example 1: V = ℝ^{n}, with any of the norms (1.4), (1.5), (1.6)

Example 2: The space of continuous functions on the interval [0, 1] with the

norm ∥u∥ = sup_{x∈[0,1]}|u(x)| for u ∈ C[0, 1].

Example 3: The space of continuous functions on the interval (0, 1), given

the norm ∥u∥ = (∫_{0}^{1} u(x)^{2}dx)^{1/2} is not complete, as there are sequences of

continuous functions, convergent with respect to this norm, such that the limit is not continuous.

A special example of a Banach space is an inner product space, or \underline{Hilbert} space. A Hilbert space H is equipped with an inner product, denoted here by (·, ·), a mapping of H × H into F. For any p, q, r, s ∈ H, α, β ∈ F, the inner product satisfies (1.7) (p + q, r + s) = (p, r) + (p, s) + (q, r) + (q, s) (1.8) (αp, q) = α(p, q) (1.9) (p, βq) = β̄(p, q) (1.10) (p, q) = \overline{(q, p)} (1.11) (p, p) ≥ 0, and (p, p) = 0 only for p = 0

In (1.9), (1.10), we have tacitly assumed F = ℂ, and – denotes complex conjugate. For F = ℝ, all such complex conjugate symbols, here and below, are understood omitted.

Furthermore, the norm of a Hilbert space is directly related to the inner product, and is obtained from (1.12) ∥p∥ = (p, p)^{1/2}

Example 1: H = ℝ^{n}, p = (p_{1} ; ⋮ ; p_{n}) , q = (q_{1} ; ⋮ ; q_{n})

(1.13)  (p, q) = ∑_{i=1}^{n} p_{i}q_{i} ,  ∥p∥ = (∑_{i} p_{i}^{2})^{1/2}

Example 2: H = ℂ^{n}, p, q as in example 1,

(1.14)  (p, q) = ∑_{i=1}^{n} p_{i}q̄_{i} , ∥p∥ = (∑_{i} |p_{i}|^{2})^{1/2}

Example 3: H = ℝ^{n}, p, q as above, α_{1}, α_{2}, … α_{n} strictly positive real numbers, we could use a weighted inner product,

(1.15)  (p, q)_{α} = ∑_{i=1}^{n} α_{i}p_{i}q_{i} , ∥p∥ = (∑_{i} α_{i}p_{i}^{2})^{1/2}

From (1.11), it is clear that the norm obtained from (1.12) satisfies (1.1), and (1.2) follows from (1.12) as well, using (1.8), (1.9).

Theorem 1.1. [the \underline{Schwarz} inequality]: For any p, q ∈ H,

(1.16)  |(p, q)| ≤ ∥p∥ ∥q∥

Proof. For p or q = 0, or if (p, q) = 0 this is obvious, so we assume (p, q) ≠ 0. Then for any t ∈ F, from (1.11), using (1.8 - 1.10), (1.12)

0 ≤ (p + tq, p + tq)
= (p, p) + t(q, p) + t̄(p, q) + |t|^{2}(q, q)

= ∥p∥^{2} + 2Re(t(q, p)) + |t|^{2}∥q∥^{2} (1.17)

This has to hold for any t ∈ F. We choose the phase of t so that t(q, p) is a real negative number, and the magnitude |t| to minimize the right side of (1.17). This is accomplished by the choice

(1.18)  t = −\frac{(p, q)}{|(p, q)|} \frac{∥p∥}{∥q∥},

and inserting this value into (1.17) we obtain

0 ≤ ∥p∥^{2} − 2\frac{(p, q)\overline{(p, q)}}{|(p, q)|} \frac{∥p∥}{∥q∥} + ∥p∥^{2}

= 2∥p∥^{2} − 2|(p, q)| ∥p∥/∥q∥ (1.19)

from which (1.16) follows easily.

Lemma 1.2. The norm obtained from (1.12) always satisfies the triangle inequality (1.3).

Proof. Given p, q ∈ H,

∥p + q∥^{2} = (p + q, p + q)
= ∥p∥^{2} + (p, q) + (q, p) + ∥q∥^{2}
= ∥p∥^{2} + 2Re(p, q) + ∥q∥^{2}
≤ ∥p∥^{2} + 2|(p, q)| + ∥q∥^{2}
≤ ∥p∥^{2} + 2∥p∥ ∥q∥ + ∥q∥^{2}

= (∥p∥ + ∥q∥)^{2} (1.20) using the Schwarz inequality in the next to last step.

A \underline{subspace} U within a Banach space V is a linear vector space within V, equipped with the same norm as V, and required to be complete with respect to this norm. In the case that V is also a Hilbert space, U is also a Hilbert space, equipped with the same inner product as V.

A \underline{bounded linear functional} ℓ on a Banach space V is a mapping ℓ : V → F satisfying, for all p, q ∈ V, α, β ∈ F:

(1.21)  ℓ(αp + βq) = αℓ(p) + βℓ(q)

(1.22)  |ℓ(p)| ≤ ∥ℓ∥ ∥p∥

for some nonnegative real constant ∥ℓ∥, independent of p in particular, called the norm of the functional ℓ.

For V of finite or countable dimension, the form of a bounded linear functional follows from (1.21). Let {e_{i}} be a basis for V; then any v ∈ V is

of the form v ≡ ∑_{i} α_{i}e_{i}, α_{i} ∈ F, and ℓ(v) = ℓ(Σα_{i}e_{i}) = ∑_{i} α_{i}ℓ(e_{i}) = ∑_{i} α_{i}ℓ_{i},

with each ℓ_{i} = ℓ(e_{i}).

Theorem 1.3. The set of all bounded linear functionals on a given Banach space V, equipped with this norm, is itself a Banach space. It is called the dual space of V, and denoted typically by V^{∗}.

Proof. For ℓ, b ∈ V^{∗}, α ∈ F, p ∈ V we identify

(1.23)  (ℓ + b)(p) = ℓ(p) + b(p)

(1.24)  (αℓ)(p) = αℓ(p)

and it is obvious that V^{∗} is a linear vector space. From (1.23),

|(ℓ + b)(p)| = |ℓ(p) + b(p)|
≤ |ℓ(p)| + |b(p)|
≤ (∥ℓ∥ + ∥b∥) ∥p∥

so the triangle inequality (1.3) holds for the norm of V^{∗}. It remains to prove completeness; let {ℓ_{n}}_{n=1}^{∞} be any Cauchy sequence within V^{∗}. Then for any p ∈ V, {ℓ_{n}(p)}_{n=1}^{∞} is a Cauchy sequence within F, as given any ε > 0, j, k > N_{ε} (so that ∥ℓ_{j} − ℓ_{k}∥ ≤ ε)

|ℓ_{j}(p) − ℓ_{k}(p)| = |(ℓ_{j} − ℓ_{k})(p)|
≤ ∥ℓ_{j} − ℓ_{k}∥ ∥p∥

≤ ε∥p∥ (1.25)

and ∥p∥ is obviously independent of j, k. Since F is complete, with respect to 1.1 (whether F is ℝ or ℂ), the sequence {ℓ_{n}(p)} converges to a limit, which we identify as ℓ(p). It is easily verified that ℓ : V → F so determined is indeed a bounded linear functional, i.e. an element of V^{∗} and one can take

(1.26)  ∥ℓ∥ = lim_{n→∞} ∥ℓ_{n}∥.

To finish the proof of completeness of V^{∗}, we need to show that ∥ℓ_{n}−ℓ∥ → 0 as n → ∞, that is for any ε > 0 there exists M_{ε} such that |ℓ_{n}(u)−ℓ(u)| ≤ ε for any n ≥ M_{ε} and any u ∈ V with ∥u∥ = 1. We choose M_{ε} = N_{ε/2}, and then k ≥ M_{ε}, depending on ε and the given vector u, such that |ℓ_{k}(u)−ℓ(u)| ≤ ε/2. Then for any n ≥ M_{ε}

|ℓ_{n}(u) − ℓ(u)| ≤ |ℓ_{n}(u) − ℓ_{k}(u)| + |ℓ_{k}(u) − ℓ(u)|
≤ ∥ℓ_{n} − ℓ_{k}∥ ∥u∥ + |ℓ_{k}(u) − ℓ(u)|
≤ ε

In the same manner we can find a Banach space V^{∗∗}, the dual of V^{∗}. Indeed, for any p ∈ V, ℓ ∈ V^{∗}, we can identify p as an element of V^{∗∗} by taking

(1.27)  p(ℓ) = ℓ(p)

so

|p(ℓ)| = |ℓ(p)|

≤ ∥ℓ∥ ∥p∥ (1.28)

so we can use the same norm ∥p∥ for p, regarded as an element of V^{∗∗}, as for p regarded as an element of V. Thus we have shown

Theorem 1.4. V is a subspace of V^{∗∗}.

The following example, however, shows that V^{∗∗} can be bigger than V. Let V be the space of infinite bounded sequence of real numbers, i.e. v = {v_{n}}_{n=1}^{∞}, equipped with the norm

(1.29)  ∥v∥_{V} = maximum_{n=1,2,...} |v_{n}|

and satisfying

(1.30)  v_{n} → 0 as n → ∞.

Then V^{∗} is also a space of infinite bounded sequences of real numbers, ℓ = {ℓ_{n}}_{n=1}^{∞}, ℓ_{n} ∈ ℝ, identifying ℓ(v) = ∑_{n} ℓ_{n}v_{n}; V^{∗} is equipped with the norm

∥ℓ∥_{V^{∗}} = ∑_{n=1}^{∞} |ℓ_{n}|

Similarly, V^{∗∗} is the space of infinite bounded sequences of real numbers; for v ∈ V^{∗∗}, ℓ ∈ V^{∗}, we identify v(ℓ) = ∑_{n} v_{n}ℓ_{n}. Thus V^{∗∗} is equipped with

the same norm ∥·∥_{V} given by (1.29), but the elements of V^{∗∗} are not required to satisfy (1.30). Thus the sequence (1, 1, 1, …) is an element of V^{∗∗} but not of V.

It will be noted that these spaces V, V^{∗}, V^{∗∗} are not Hilbert spaces, because of the specific choice of norm; the norms chosen do not obtain from an inner product. For such infinite dimensional spaces, the specific choice of norm is quite important. (For finite dimensional spaces, it generally is not; the norms given in (1.4 - 1.6), for example, satisfy

(1.31)  ∥u∥_{∞} ≤ ∥u∥_{2} ≤ ∥u∥_{1} ≤ \sqrt{n}∥u∥_{2} ≤ n∥u∥_{∞}

so a Cauchy sequence with respect to any of these norms is a Cauchy sequence with respect to the others as well.)

A Hilbert space, however, is its own dual, whether finite or infinite dimensional. We first prove the following lemma:

Lemma 1.5. Let U be a proper subspace of Hilbert space H. Then there exists a nonzero element x ∈ H, such that

(1.32)  (x, u) = 0 for all u ∈ U, often written x⊥U

Proof. Since U is a proper subspace of H, there exists an element v ∈ H, v ∉ U. Consider

(1.33)  inf_{u∈U} ∥v − u∥^{2} = α

Clearly α ≥ 0. Furthermore α cannot be zero, as if α = 0, there is a sequence {u_{n}}_{n=1}^{∞} in U such that ∥v−u_{n}∥ → 0 as n → ∞. But U is complete, so any such limit v is also within U, contradicting our specification v ∉ U.

But for any sequence {ε_{n}}, ε_{n} ↓ 0 as n → ∞ there will be a sequence {u_{n}} ∈ U such that

(1.34)  ∥v − u_{n}∥^{2} ≤ α + ε_{n}^{2}

And for any fixed n, let t ∈ F, w ∈ U such that ∥w∥ = 1, otherwise arbitrary; then u_{n} + tw ∈ U, and from (1.33) we still have

(1.35)  ∥v − u_{n} − tw∥^{2} ≥ α.

Expanding (1.35), using (1.34)

α ≤ ∥v − u_{n} − tw∥^{2}
= ∥v − u_{n}∥^{2} − 2Re(t̄(v − u_{n}, w)) + |t|^{2}

≤ α + ε_{n}^{2} − 2Re(t̄(v − u_{n}, w)) + |t|^{2}. (1.36)

For any w such that (v − u_{n}, w) ≠ 0, we choose

(1.37)  t = ε_{n}(v − u_{n}, w)/|(v − u_{n}, w)|,

obtaining from (1.36)

(1.38)  0 ≤ ε_{n}^{2} − ε_{n}|(v − u_{n}, w)|

equivalently

(1.39)  |(v − u_{n}, w)| ≤ ε_{n} for all w ∈ U, ∥w∥ = 1.

Fix any ε > 0, and obtain N_{ε} such that ε_{n} < ε/2 for all n > N_{ε}. Then for any m, n > N_{ε}, we have (1.39) and u_{m} ∈ U such that

(1.40)  |(v − u_{m}, w)| ≤ ε_{m}, for any w ∈ U, ∥w∥ = 1.

Combining (1.39) and (1.40),

|(u_{n} − u_{m}, w)| ≤ ε_{n} + ε_{m}

≤ ε; (1.41)

Since u_{n}, u_{m} ∈ U, so is u_{n} − u_{m}, and we can take w = (u_{n} − u_{m})/∥u_{n} − u_{m}∥ in (1.41), obtaining

(1.42)  ∥u_{n} − u_{m}∥ ≤ ε for all n, m > N_{ε}

Thus the sequence {u_{j}}_{j=1}^{∞} is in fact Cauchy, and as U is necessarily complete, u_{n} → z as n → ∞, for some z ∈ U. Then for any w ∈ U, ∥w∥ = 1, and any n

|(v − z, w)| ≤ |(v − u_{n}, w)| + |(u_{n} − z, w)|

≤ ε_{n} + ∥u_{n} − z∥, (1.43)

and the right side of (1.43) clearly approaches zero as n → ∞. Since z ∈ U and v ∉ U, v − z ≠ 0, and x = v − z satisfies (1.32). We note that x⊥U achieves the minimum value α in (1.33).

Theorem 1.6. (The Riesz representation theorem for Hilbert spaces): A bounded linear functional ℓ on a Hilbert space H admits a representation as the inner product with a unique element of H, i.e. for any ℓ ∈ H^{∗} there exists a unique v ∈ H such that

(1.44)  ℓ(u) = (u, v) for all u ∈ H.

Proof. Set U = {u ∈ H|ℓ(u) = 0}. It is easily verified that U is a subspace of H, and a proper subspace of H unless ℓ(u) = 0 for all u ∈ H, in which case (1.44) holds with v = 0. For U a proper subspace of H, there exists x ∈ H, x⊥U, x ≠ 0 by lemma 1.5. Thus ℓ(x) ≠ 0, and setting η = x/ℓ(x) we have ℓ(η) = 1, η⊥U. Now η is unique within H, for if η̃ satisfies ℓ(η̃) = 1, η̃⊥U, then η − η̃⊥U, but ℓ(η − η̃) = 0 so η − η̃ ∈ U and thus η = η̃. Therefore the one-dimensional space K = {αη|α ∈ F } is the perpendicular complement of u in H, i.e. K = {y ∈ H|y⊥U}. Now for any u ∈ H, we can write

(1.45)  u = u_{U} + u_{K}, u_{U} = u − \frac{(u, η)η}{∥η∥^{2}} , u_{K} = \frac{(u, η)η}{∥η∥^{2}}

clearly u_{K} ∈ K and u_{U}⊥η. We claim that u_{U} ∈ U. If not, i.e. if ℓ(u_{U}) ≠ 0, then η̃ = u_{U}/ℓ(u_{U}) is perpendicular to η and satisfies ℓ(η̃) = 1. Thus ℓ(η − η̃) = 0, so η − η̃ ∈ U. As η⊥U, η⊥η − η̃. This is impossible, as η̃⊥η and (η, η) = ||η∥^{2} ≠ 0, thus proving the claim u_{U} ∈ U. Thus

ℓ(u) = ℓ(u_{U} + u_{K})
= ℓ(u_{U}) + ℓ(u_{K})
= 0 + (u, η)ℓ(η)/∥η∥^{2}

= (u, η)/∥η∥^{2} (1.46)

so v = η/∥η∥^{2} satisfies (1.44). And should another such ṽ ∈ H satisfy (1.44) as well, then for all u ∈ H

0 = ℓ(u) − ℓ(u)
= (u, v) − (u, ṽ)
= (u, v − ṽ)

= ∥v − ṽ∥^{2} (1.47)

taking u = v − ṽ.

Lemma 1.7. Let U be a proper subspace of a Hilbert space H, and let U_{⊥} = {v ∈ H, v⊥U}. Then U_{⊥} is also a proper subspace of H, U ∩ U_{⊥} = {0}, and H = U ⊕ U_{⊥}, i.e. for any h ∈ H, there exist unique u ∈ U, v ∈ U_{⊥} such that h = u + v.

Proof. Only the last statement requires proof. Given h ∈ H, we have a bounded linear functional on U determined from ℓ(ω) = (ω, h) for all ω ∈ U. By the Riesz theorem, there exists u ∈ U such that ℓ(ω) = (ω, h) = (ω, u) for all ω ∈ U. Thus h − u is perpendicular to U and so is in U_{⊥}. Suppose uniqueness fails, and h = u + v = ũ + ṽ, with u, ũ ∈ U, v, ṽ ∈ U_{⊥}. Then u − ũ = ṽ − v, but u − ũ ∈ U and ṽ − v ∈ U_{⊥}.

Problems

1. Let V, V^{∗}, V^{∗∗} be the spaces of real sequences in the example following Theorem 1.4. Show that each is a Banach space, i.e. prove completeness. Show that V^{∗} is the dual of V, i.e. that ℓ ∈ V^{∗} if and only if ∥ℓ∥_{V^{∗}} < ∞. Similarly, show v ∈ V^{∗∗} if and only if ∥v∥_{V} < ∞.

2. Let H = ℝ^{4}, and U = span ((\begin{matrix}1 \\ 0 \\ 1 \\ 0 \end{matrix}) , (\begin{matrix}1 \\ −1 \\ −1 \\ 1 \end{matrix}) , (\begin{matrix}1 \\ 1 \\ 3 \\ −1 \end{matrix})) find U_{⊥}.

3. Let H ⊂ C[0, 1], such that any u ∈ H satisfies u(0) = 0, ∫_{0}^{1}(\frac{du}{dx})^{2}dx < ∞,

with (u, v) = ∫_{0}^{1} \frac{du}{dx} \frac{dv}{dx}dx. For each of the following linear functionals on

H, find a suitable bound for ∥ℓ∥, and find z ∈ H such that ℓ(u) = (u, z) for all u ∈ H

a) ℓ_{1}(u) = u(1/2)

b) ℓ_{2}(u) = ∫_{1/2}^{1} u(x)dx

c) ℓ_{3}(u) = ∫_{0}^{1} \frac{u(x)dx}{x}

(More precisely, the functions u ∈ H are differentiable almost every-

where in the interval (0, 1) and satisfy u(z) − u(y) = ∫_{y}^{z} \frac{du}{dx}dx, for any

0 ≤ y < z ≤ 1. You may assume that this space is indeed a Hilbert space.)

4. A bilinear form [ , ] on a Hilbert space H is a mapping of H × H into F, i.e. [p, q] ∈ F for any p, q ∈ H, required to satisfy (1.7), (1.8), (1.9), but not necessarily (1.10) or (1.11).

a) Assume H a real Hilbert space (i.e. F = ℝ) and that [p, p] = 0 for all p ∈ H. Show [p, q] = −[q, p] for all p, q ∈ H, and find a nontrivial example for H = ℝ^{2}.

b) Assume H complex (F = ℂ) and [p, p] = 0 for all p ∈ H. Show that [p, q] = 0 for all p, q ∈ H.

c) Again assume H complex, and assume that α∥p∥^{2} ≤ [p, p] ≤ β∥p∥^{2} for some positive constants α, β. Find a constant C, depending on α, β but independent of p, q ∈ H, such that |[p, q]| ≤ C∥p∥ ∥q∥].

5. Construct an example of a linear functional on a real Hilbert space, satisfying (1.21), which is not bounded, i.e. for any constant C there exist p ∈ H, ∥p∥ = 1, such that ℓ(p) > C.

6. Let V be a Banach space of finite dimension n. Show that V^{∗∗} is isomorphic to V, and find the dimensions of V^{∗} and V^{∗∗}.

7. Let H be the space of problem 3, and let U ⊂ H be a subspace of H, the elements of which satisfy u(1) = 0 as well. Find U_{⊥}.

8. Let U be a subspace of a Hilbert space H, and let P : H → U be the projection of H into U, i.e. for all h ∈ H, v = P h ∈ U and satisfies (u, h) = (u, v) for all u ∈ U.

a) show that P is defined, i.e. that P h is unique

b) show that P is onto U

c) show that ∥P ∥ = 1, i.e. ∥P h∥ ≤ ∥h∥ for all h ∈ H, and that there exists h ∈ H such that ∥P h∥ = ∥h∥

d) show that P is symmetric, i.e. that (g, P h) = (P g, h) for all g, h ∈ H.

9. For H = ℝ^{n}, denote by |u|_{α} = (∑_{i=1}^{n} |u_{i}|^{α})^{1/α} for any 0 < α < ∞.

a) For u ∈ ℝ^{n} fixed, show that |u|_{α} → ∥u∥_{∞}, given by (1.6), as α → ∞.

b) Show that | · |_{α} is \underline{not} a norm for α < 1. A simple counterexample will suffice.

c) For 1 ≤ α < β < ∞, show |u|_{β} ≤ |u|_{α} and find a constant C, depending on α, β, such that |u|_{α} ≤ C|u|_{β}. For which vectors u does equality hold in each case?

10^{∗}. Let [ , ] be a bilinear form on a Hilbert space H, i.e. [p, q] ∈ F for all p, q ∈ H, and [ , ] satisfies (1.7), (1.8), (1.9), and (1.11), but not necessarily (1.10). Furthermore |[p, q]| ≤ ∥p∥ ∥q∥.

Statement: Every bounded linear functional on H can be represented in terms of [ , ], i.e. for any ℓ ∈ H^{∗} there exists a unique u ∈ H such that ℓ(v) = [v, u] for all v ∈ H.

Justification: Uniqueness of u is easy, using (1.11). To show existence, we note that for any fixed u, [v, u] determines a bounded linear functional on H, i.e. b(v) = [v, u] for any v ∈ H. Therefore by the Riesz theorem, for any v ∈ H, b(v) = (v, z) = [v, u] for some unique z ∈ H.

Consider the mapping Q : H → H determined by Qu = z for any u ∈ H. This mapping is linear, using (1.7 - 1.9), and bounded. Therefore it suffices to show that Q is onto H. Suppose not, then QH = {Qu, u ∈ H\} is a \underline{proper} subspace of H, and by lemma 1.5, there exists a nonzero element of H, x⊥QH, i.e. for any u ∈ H, 0 = (x, Qu) = [x, u]. Then choosing u = x gives a contradiction with (1.11).

This argument is incorrect. It is true for finite - dimensional Hilbert spaces, and it is true with the stronger assumption

(∗) [p, p] ≥ α∥p∥^{2} for all p ∈ H, with some constant α > 0. (With the assumption (∗) it is well-known as the Lax-Milgram theorem.)

a) Find the mistake in the argument as stated.

b) Fix the argument, given the assumption (∗).

11. For H of dimension n, let the ℓ_{1}, ℓ_{2} and ℓ_{∞} norms on H be as shown in (1.4), (1.5), (1.6). For A = {A_{ij}}_{i,j=1}^{n} any n × n matrix, we determine the corresponding \underline{induced norms} from

∥A∥_{1} = sup_{h≠0} ∥Ah∥_{1}/∥h∥_{1}, ∥A∥_{2} = sup_{h≠0} ∥Ah∥_{2}/∥h∥_{2},

∥A∥_{∞} = sup_{h≠0} ∥Ah∥_{∞}/∥h∥_{∞}. Sometimes it is more convenient to work with other matrix norms, such as |||A|||_{∞} = maximum_{i,j}|A_{ij}| or

|||A|||_{1} = ∑_{i,j=1}^{n} |A_{ij}|. Show that these norms are all equivalent, i.e. that any one of them majorizes all the others, with factors depending only on n. A convenient way of stating these results is in terms of chains of inequalities of the form (1.31). Show that for any such choice of ∥A∥, there is a positive constant α such that ∥A^{m}∥ ≤ α^{m}∥A∥^{m} for all positive integers m. Here α may depend on n and on the specific choice for ∥A∥, but is independent of A and m. Try to make all of these results “best possible”, i.e. the inequalities can become equalities for a specific choice of the matrix A.

Show that all of these definitions of the norm of a matrix satisfy the triangle inequality 12^{∗}. Let A be a bounded linear operator on an infinite-dimensional Hilbert

space H. Prove the existence of another bounded linear operator A^{†} : H → H, such that for all x, y ∈ H, (Ax, y) = (x, A^{†}y). Hint: Denote by ℓ_{y} : H → F the mapping determined by ℓ_{y}(x) = (Ax, y) for all x ∈ H, any fixed y, and use the Riesz theorem.

13. Let X be a given Hilbert space, and A : X → X a given linear transformation, with A0 = 0 in particular.

(a) Show that if X is finite-dimensional, either there exists nonzero x ∈ X such that Ax = 0, or else A^{−1} : X → X exists, i.e. for every y ∈ X there exists a unique z ∈ X such that Az = y. (This is commonly referred to as the Fredholm alternative.)

(b) Give an example showing that this is not true for infinite-dimensional X.

14. In the proof of theorem 1.3:

(a) Show V^{∗} as defined is a linear vector space

(b) Show that the limit (1.26) exists, and that ∥ℓ∥ so obtained satisfies (1.22)

15. Let ℓ : V → F be a bounded linear functional on V, ρ_{i} ∈ V, i = 1, …,

such that ∑_{i=1}^{∞} ∥ρ_{i}∥ is finite.

With ℓ( ∑_{i=1}^{∞} ρ_{i}) defined as lim_{k→∞} ℓ( ∑_{i=1}^{k} ρ_{i}), show that ℓ( ∑_{i=1}^{∞} ρ_{i}) exists and

is equal to ∑_{i=1}^{∞} ℓ(ρ_{i}).

16. Suppose that |(u, v)| = ∥u∥ ∥v∥ for some u, v ∈ H, u, v ≠ 0. Show that v = αu for some α ∈ F.

17^{∗} Let H = L^{2}(0, 1), i.e. for u ∈ H, u : (0, 1) → ℝ such that ∫_{0}^{1} u(x)^{2}dx <

∞, with the inner product (u, v) = ∫_{0}^{1} u(x)v(x)dx. Let Y : H → H

be determined from Y u(x) = 2u(\frac{x}{2}) − u(1 − x), 0 < x < 1. Find Y^{†} : H → H such that (Y u, v) = (u, Y^{†}v) for all u, v ∈ H. Find a number K such that ∥Y u∥ ≤ K∥u∥ for all u ∈ H.

18^{∗} Let H = L^{2}(0, 1) as in problem 17, and let Z : H → H be determined from Zu(x) = u(x^{2}), 0 < x < 1 for all u ∈ H. Prove that Z is unbounded, that for all (large) M there exists u_{M} such that ∥Zu_{M}∥ ≥ M∥u_{M}∥.

19 Let X ⊂ H be a proper subspace, and P : H → X determined by (u, x) = (P u, x) for all x ∈ X. Show that P is symmetric, i.e. (P u, v) = (u, P v) for all u, v ∈ H.

2 Some basic properties of matrices

For a fixed positive integer n, let H = ℝ^{n} or ℂ^{n} depending on whether F is ℝ or ℂ. A linear operator A : H → H then admits a representation as an n × n matrix, and we shall denote both the operator and the matrix by the same symbol, thus assuming a given basis for H.

Given an n × n matrix A, we have the following definitions:

(2.1) The \underline{range} of A, R(A) = \{η ∈ H|η = Ax for some x ∈ H\}

(2.2) the \underline{kernel} of A, ker(A) = \{x ∈ H|Ax = 0\}

Lemma 2.1. Both R(A) and ker (A) are subspaces of H.

The proof is immediate, since A is a linear operator, but depends on the fact that H is finite-dimensional. Thus ker(A) is complete, that is closed, because A is bounded, ∥Ah∥ ≤ ∥A∥ ∥h∥ for any h ∈ H, for some nonnegative number ∥A∥, the \underline{norm} of A. And R(A) is also complete, as it is simply span {Ae_{i}}_{i=1}^{n}, where {e_{i}}_{i=1}^{n} is a basis for H. For example, taking the e_{i} as unit vectors, (e_{i})_{j} = δ_{ij}, we note that R(A) is just the span of the columns of A, viewed as separate n-dimensional vectors.

Special cases are A nonsingular, i.e. A is 1-1 onto H, so R(A) = H, ker(A) = {0}, and A identically zero, so R(A) = {0}, ker(A) = H. For A singular but not identically zero, both R(A) and ker(A) are proper subspaces of H. Alternatively, R(A) is denoted by AH = {Ah|h ∈ H}.

The relation between R(A) and ker(A) can be described easily in terms of the transpose or adjoint of A, respectively given by

(2.3)  (A^{t})_{ij} = A_{ji} , i, j = 1, …, n
(2.4)  (A^{†})_{ij} = \overline{A_{ji}}, i, j = 1, …, n

For F = ℝ (i.e. A real) A^{t} = A^{†}; for F = ℂ, A^{†} = \overline{A^{t}}. From (2.3) or (2.4) it follows easily that (A^{t})^{t} = (A^{†})^{†} = A.

Theorem 2.2. For any x, y ∈ H,

(2.5)  (A^{†}x, y) = (x, Ay)

Proof. Since (p, q) = ∑_{i=1}^{n} p_{i}q̄_{i},

(x, Ay) = ∑_{i=1}^{n} x_{i}\overline{(Ay)}_{i} = ∑_{i=1}^{n} x_{i}\overline{(∑_{j=1}^{n} A_{ij}y_{j})}

= ∑_{i=1}^{n} ∑_{j=1}^{n} x_{i}\overline{A_{ij}}ȳ_{j} = ∑_{j=1}^{n}(∑_{i=1}^{n} \overline{A_{ij}}x_{i})ȳ_{j}

= ∑_{j=1}^{n}(∑_{i=1}^{n} A_{ji}^{†}x_{i})ȳ_{j} = ∑_{j=1}^{n}(A^{†}x)_{j}ȳ_{j}

= (A^{†}x, y) (2.6)

In the case of F real, A^{†} becomes A^{t}.

Theorem 2.3.

(2.7)  R(A)⊥ ker(A^{†}) and ker(A)⊥R(A^{†}).

Proof. Let y ∈ R(A) and z ∈ ker(A^{†}). Then y = Ax for some x ∈ H, and

(2.8)  (z, y) = (z, Ax) = (A^{†}z, x) = (0, x) = 0

using (2.5). Similarly, for z ∈ ker(A), q ∈ R(A^{†}), q = A^{†}p for some p ∈ H, and

(2.9)  (z, q) = (z, A^{†}p) = (Az, p) = (0, p) = 0.

In general it is not true that R(A) and ker(A) are perpendicular.

These results extend very easily to non-square matrices. Let H_{n} = ℝ^{n} (or ℂ^{n}) and H_{m} = ℝ^{m} (or ℂ^{m}). Then a linear operator A : H_{m} → H_{n} is represented by an n × m matrix. A^{t} and A^{†} are m × n matrices, obtained analogously with (2.3), (2.4). In this case the respective inner products on H_{m} and H_{n} are

(2.10)  (p, x)_{n} = ∑_{i=1}^{n} p_{i}x̄_{i} , (q, y)_{m} = ∑_{i=1}^{m} q_{i}ȳ_{i}

for x, p ∈ H_{m}, y, q ∈ H_{n}. Theorem 2.2 now holds in the form

(2.11)  (A^{†}x, y)_{m} = (x, Ay)_{n}

(same proof as above), and Theorem 2.3 also remains valid, with R(A)⊥ ker(A^{†}) in H_{n} and R(A^{†})⊥ ker(A) in H_{m} (again, same proof as above).

From (2.3) or (2.4), we have (A + B)^{†} = A^{†} + B^{†} whenever A and B are matrices of the same dimensions, and for A, B such that AB is defined,

(2.12)  (AB)^{†} = B^{†}A^{†}

immediately from (2.4) and the definition of matrix multiplication. Alternatively, one can use (2.5), which is a more general definition of an adjoint operator, and note that (x, ABy) = (x, A(By)) = (A^{†}x, By) = (B^{†}(A^{†}x), y) = (B^{†}A^{†}x, y) = ((AB)^{†}x, y) for all x, y ∈ H, thus identifying B^{†}A^{†} with (AB)^{†}. For any nonsingular A, taking B = A^{−1} in (2.12), since I^{†} = I we find

(2.13)  (A^{−1})^{†} = (A^{†})^{−1}.

Next we introduce two classes of n × n matrices of special structure. A \underline{diagonal} matrix D = diag(d_{1}, d_{2}, … d_{n}), d_{i} ∈ F, i = 1, …, n, has zero elements except on its main diagonal, i.e.

(2.15)  D_{ij} = d_{i}δ_{ij} 1 ≤ i, j ≤ n

where here and below, δ is the Kroneker delta,

(2.16)  δ_{ii} = 1, δ_{ij} = 0 for i ≠ j.

We shall also use a class of elementary matrices. The term elementary matrix is used somewhat differently in different texts, and we shall use a restricted class of such matrices here. Herein, an n × n elementary matrix, is of the form E_{kma} where 1 ≤ k, m ≤ n, k ≠ m, and a ∈ F. This matrix has diagonal elements equal to 1, its km element is equal to a, and it is 0 everywhere else,

(2.17)  (E_{kma})_{ij} = δ_{ij} + aδ_{ik}δ_{jm} 1 ≤ i, j ≤ n

From (2.17), it is easily verified that

(2.18)  det(E_{kma}) = 1 and (E_{kma})^{−1} = E_{km(−a)}.

Multiplication of a given n×n matrix by a diagonal or elementary matrix is especially simple. Given an n × n matrix A, let r_{i}, i = 1, …, n, denote the i-th row of A, and let c_{i}, i = 1, …, n denote the i-th column of A. We can think of r_{i} as a row vector of dimension n, and c_{i} as a column vector of dimension n. Furthermore, the matrix A is just the r_{i} assembled rowwise, or the c_{i} assembled columnwise,

(2.19)  A = ( r_{1} ; r_{2} ; ⋮ ; r_{n} ) = (c_{1}c_{2} … c_{n})

or we can use the notation

(2.20)  A_{ij} = (r_{i})_{j} = (c_{j})_{i} , 1 ≤ i, j ≤ n.

Then with D = diag(d_{1}, d_{2}, …, d_{n}) as above, we have

\begin{matrix}(DA)_{ij} & = d_{i}A_{ij} = d_{i}(r_{i})_{j} = (d_{i}r_{i})_{j} \\ (AD)_{ij} & + A_{ij}d_{j} = (c_{j})_{i}d_{j} = (d_{j}c_{j})_{i} 1 ≤ i, j ≤ n \end{matrix} ; (2.21)

left multiplication of A by D multiplies each row r_{i} by the corresponding d_{i}, while right multiplication of A by D multiplies each column c_{j} of A by the corresponding d_{j}.

Next we compute

(E_{kma}A)_{ij} = ∑_{ℓ=1}^{n}(E_{kma})_{iℓ}A_{ℓj}

= A_{ij} + a ∑_{ℓ=1}^{n} δ_{ik}δ_{ℓm}A_{ℓj}

+ A_{ij} + aδ_{ik}A_{mj} = (r_{i})_{j} + aδ_{ik}(r_{m})_{j}

= (r_{i} + aδ_{ik}r_{m})_{j}, 1 ≤ i, j ≤ n; (2.22) for i ≠ k, this is just A_{ij}, but

(2.23)  (E_{kma}A)_{kj} = A_{kj} + aA_{mj}, 1 ≤ j ≤ n.

Left multiplication of A by E_{kma} simply adds a ∈ F times the m-th row of A to the k-th row of A, leaving the other rows unchanged. Similarly

(AE_{kma})_{ij} = ∑_{ℓ} A_{iℓ}(E_{kma})_{ℓj}

= A_{ij} + a ∑_{ℓ} A_{iℓ}δ_{ℓk}δ_{mj}

= A_{ij} + aA_{ik}δ_{mj} = (c_{j})_{i} + aδ_{mj}(c_{k})_{i}

+ (c_{j} + aδ_{mj}c_{k})_{i} , 1 ≤ i, j ≤ n. (2.24)

For j ≠ m, this is just A_{ij}, but

(2.25)  (AE_{kma})_{im} = A_{im} + aA_{ik},  1 ≤ i ≤ n,

so right multiplication of A by E_{kma} adds a times the k-th column of A to the m-th column of A, leaving the other columns unchanged. In the “notation” of (2.19)

(2.26)  DA = ( d_{1}r_{1} ; d_{2}r_{2} ; ⋮ ; d_{n}r_{n} ) , AD = (d_{1}c_{1} d_{2}c_{2} … d_{n}c_{n})

(2.27)  E_{kma}A = ( \begin{matrix}r_{1} \\ r_{k} + ar_{m} \\ r_{n} \end{matrix} ) , AE_{kma} = (c_{1} … c_{m} + ac_{k} … c_{n})

Lemma 2.4. For any n × n matrix A, diagonal matrix D, and elementary matrix E_{kma},

(2.28)  det(AD) = det(DA) = det A det D
(2.29)  det(AE_{kma}) = det(E_{kma}A) = det A

Proof. We use the expressions

(2.30)  det A = ∑_{σ∈P_{n}} N_{σ} Π_{i=1}^{n}A_{iσ(i)}

(2.31)  det A = ∑_{σ=P_{n}} N_{σ} Π_{i=1}^{n}A_{σ(i)i}

where P_{n} is the permutation group of order n, and

(2.32)  N_{σ} = Π_{1≤j<i≤n} \frac{i − j}{σ(i) − σ(j)}.

Then using (2.21) and (2.30)

det(DA) = ∑_{σ∈P_{n}} N_{σ} Π_{i=1}^{n}(DA)_{iσ(i)}

= ∑_{σ∈P_{n}} N_{σ} Π_{i=1}^{n}d_{i}A_{iσ(i)}

= ( ∑_{σ∈P_{n}} N_{σ} Π_{i=1}^{n}A_{iσ(i)})( Π_{i=1}^{n}d_{i})

= (det A)(det D) (2.33)

The other half of (2.28) is proved in the same manner.

To compute det(E_{kma}A), we use (2.22), (2.23),

det(E_{kma}A) = ∑_{σ∈P_{n}} N_{σ} Π_{i=1}^{n}(A_{iσ(i)} + aδ_{ik}A_{mσ(i)})

= ∑_{σ∈P_{n}} N_{σ}( Π_{i=1}_{i≠k}^{n}A_{iσ(i)})(A_{kσ(k)} + aA_{mσ(k)})

= ∑_{σ∈P_{n}} N_{σ}[ Π_{i=1}^{n}A_{iσ(i)} + a( Π_{i=1}_{i≠k}^{n}A_{iσ(i)})A_{mσ(k)}]

= det A + det Ã_{k}^{m} (2.34)

where Ã_{k}^{m} is the matrix A except that the k-th row is replaced by the m-th. Thus the k-th and m-th rows of Ã_{k}^{m} coincide, and det Ã_{k}^{m} = 0, proving one of the statements (2.29). The other is proved similarly.

Next we prove a representation theorem for any n × n matrix A, in terms of diagonal and elementary matrices.

Theorem 2.5. Any n × n matrix A may be written

(2.35)  A = ( Π_{i=1}^{M}E_{k_{i}m_{i}a_{i}})D( Π_{i=M+1}^{N}E_{k_{i}m_{i}a_{i}})

where D is a diagonal n×n matrix, the E_{k_{i}m_{i}a_{i}} are elementary matrices, and N ≤ n^{2}.

Remark. The representation (2.35) is not unique; there are in fact an infinite number of such representations. But from lemma 2.4, (2.29) in particular, it follows that

(2.36)  det A = det D

for any such representation.

Proof. We construct a finite sequence of matrices, A^{(1)}, A^{(2)}, …, A^{(n)}. We take A^{(1)} = A, and identify A^{(n)}, which will be diagonal, as D. For ℓ = 2, …, n, we construct A^{(ℓ)} from A^{(ℓ−1)} by no more than 2(n − ℓ) + 3 left or right multiplications by elementary matrices.

For ℓ > 1, A^{(ℓ)} is partially diagonal, that is

(2.37)  A_{ij}^{(ℓ)} = 0 if i ≠ j and either i < ℓ or j < ℓ.

Given A^{(ℓ)}, we proceed as follows:

First, suppose that both the ℓ-th row and the ℓ-th column of A^{(ℓ)} vanish identically. Then we can simply take A^{(ℓ+1)} = A^{(ℓ)} and (2.37) remains satisfied.

Next, suppose that A_{ℓℓ}^{(ℓ)} ≠ 0. Then take B^{(ℓ)} = A^{(ℓ)} and proceed as below. If A_{ℓℓ}^{(ℓ)} = 0, then either A_{ℓj}^{(ℓ)} ≠ 0 for some j > ℓ or else A_{iℓ}^{(ℓ)} ≠ 0 for some

i > ℓ. Take B^{(ℓ)} = A^{(ℓ)}E_{jℓ1} in the first case and B^{(ℓ)} = E_{ℓi1}A^{(ℓ)} in the second. However determined, B^{(ℓ)} satisfies (2.37) as well and B_{ℓℓ}^{(ℓ)} ≠ 0.

Given B^{(ℓ)}, we make the off-diagonal elements of the ℓ-th row and the ℓ-th column equal to zero by elementary operations, as in ordinary Gaussian elimination. Specifically, we find

(2.38)  A^{(ℓ+1)} = ( Π_{i>ℓ}_{B_{iℓ}^{(ℓ)}≠0}E_{iℓβ_{i}})B^{(ℓ)}( Π_{j>ℓ}_{B_{ℓj}^{(ℓ)}≠0}E_{ℓjγ_{j}})

where β_{i} = −B_{iℓ}^{(ℓ)}/B_{ℓℓ}^{(ℓ)} and γ_{j} = −B_{ℓj}^{(ℓ)}/B_{ℓℓ}^{(ℓ)}. The matrix A^{(ℓ+1)} so determined satisfies (2.37) as required.

By successive application of this algorithm, we arrive at A^{(n)}

(2.39)  A^{(n)} = ( Π_{i=1}^{M}E_{p_{i}q_{i}z_{i}})A( Π_{i=M+1}^{N}E_{p_{i}q_{i}z_{i}});

from (2.37), it follows that A^{(n)} is diagonal, so we can identify A^{(n)} = D. Now multiplying (2.39) successively by the E_{p_{i}q_{i}z_{i}}^{−1}, using (2.18), we obtain (2.35). This completes the proof.

Corollary 1. Right inverses are also left inverses (and vice versa), i.e. if A, B are n × n matrices, AB = I if and only if BA = I.

Proof. Assume AB = I; and using (2.35)

(2.40)  A = Π_{i=1}^{M}E_{i}D Π_{i=M+1}^{N}E_{i},

where here and below, we use the abbreviation E_{i} = E_{k_{i}m_{i}a_{i}} where no ambiguity is possible. The multiplying successively by the E_{i}^{−1} and D^{−1} we find

(2.41)  B = Π_{j=M+1}^{N}E_{N+M+1−j}^{−1}D^{−1} Π_{j=1}^{M}E_{M+1−j}^{−1}

and one easily checks that BA = I. The same argument holds, assuming BA = I initially.

Corollary 2. For any n × n matrices A, B,

(2.42)  det(AB) = (det A)(det B)

This follows immediately from (2.35), (2.28), and (2.29).

We shall also use Theorem 2.5 to simplify a discussion of the rank of a matrix. For any matrix A, the rank of A, denoted here by r(A), is simply dim R(A). We shall denote by γ(A) the dimension of the kernel of A, i.e. dim{x ∈ H|Ax = 0}. These definitions apply whether the matrix A is square or not, but we can simplify the discussion by first assuming square matrices and then extending the results.

Lemma 2.6. Let U ⊂ H, H of dimension n, and Z a nonsingular n × n matrix. Then

(2.43)  dim(ZU) = dim U.

Proof. Obviously dim(ZU) ≤ dim U. But if dim(ZU) < dim U, since U is itself a linear space, there exists u ∈ U, u ≠ 0, such that Zu = 0 contradicting the fact that Z is nonsingular.

Theorem 2.7. For any n × n matrix A, and any representation of A of the form (2.35),

(2.44)  r(A) = the number of nonzero diagonal elements of D;
(2.45)  γ(A) = the number of zero diagonal elements of D.

Furthermore, r(A) is the number of linearly independent rows of A and also the number of linearly independent columns of A.

As a corollary of (2.44), (2.45), we have

(2.46)  r(A) + γ(A) = n

Proof.

r(A) = dim(AH)

= dim( Π_{i=1}^{M}E_{i}D Π_{i=M+1}^{N}E_{i}H)

= dim( Π_{i=1}^{M}E_{i}DH)

= dim(DH)

using lemma 2.6 in the last step. Obviously dim(DH) is the number of nonzero diagonal elements of D.

Next let U = ker A, and

(2.47)  V = Π_{i=M+1}^{N}E_{i}U

From (2.35) it is clear that AU = 0 if and only if DV = 0. Clearly the dimension of V is the number of zero diagonal elements of D. And from lemma 2.6, dim V = dim U = γ(A), so (2.45) holds as well.

Now from (2.22) and (2.24), or perhaps more clearly form (2.27), it follows that left or right multiplication of a matrix by an elementary matrix does not change the number of linearly independent rows or columns. Since D has r(D) = r(A) linearly independent rows and columns, so does A.

Corollary 3. For any n × n matrix A, r(A^{†}) = r(A^{t}) = r(A)

(2.48)  and γ(A^{†}) = γ(A^{t}) = γ(A).

These results extend easily to non-square matrices. Let A be an m × n matrix, i.e. A : H_{n} → H_{m}. Let Ã be a square matrix of dimension maximum (m, n), obtained by adding rows or columns of zeros to A. For n > m, we add n−m rows of zeros to the bottom of A to obtain Ã, which is an n×n matrix, mapping H_{n} to H_{n}. Clearly r(A) = r(Ã), γ(A) = γ(Ã), and r(Ã)+γ(Ã) = n from (2.46) so r(A) + γ(A) = n. For m > n, we add m − n columns of zeros to A to form Ã; Ã is m × m. Again r(A) = r(Ã), but now Ã : H_{m} → H_{m} and γ(Ã) = γ(A) + m − n. Now r(Ã) + γ(Ã) = m, so again (!) r(A) + γ(A) = n. Thus we have proved

Corollary 4. For any m × n matrix A,

(2.49)  r(A) + γ(A) = n.

We note that r(A) ≤ minimum (m, n).

The rank of the product of two matrices is not uniquely determined by the rank of each factor, but satisfies an important inequality.

Theorem 2.8 (Sylvester’s theorem). : Let A be an m × n matrix and B an n × k matrix, so AB is defined and is an m × k matrix. Then

(2.50)  r(A) + r(B) − n ≤ r(AB) ≤ minimum (r(A), r(B))

Proof. A : H_{n} → H_{m}, B : H_{k} → H_{n}, and (AB) : H_{k} → H_{m}.

r(AB) = dim(AB)H_{k}
= dim(A(BH_{k})

= dim(AR(B)) (2.51)

Obviously dim(AR(B)) ≤ dim(AH_{n}) = r(A), since R(B) ⊂ H_{n}. And dim(AR(B)) ≤ dim R(B) = r(B). Using (2.51) this proves the upper bound in (2.50).

To get the lower bound for r(AB), we note that

(2.52)  ker(AB) = {x ∈ H_{k}|Bx ∈ ker(A)}

so that

γ(AB) = dim ker(AB)

(2.53) = dim(R(B) ∩ ker(A)) + dim ker B.

As dim(R(B) ∩ ker(A)) ≤ dim ker(A) = γ(A), (2.53) becomes

(2.54)  γ(AB) ≤ γ(A) + γ(B).

Applying (2.49) to A, B and AB, we find

(2.55)  r(A) + γ(A) = n, r(B) + γ(B) = k, r(AB) + γ(AB) = k;

combining (2.54) and (2.55) gives the lower bound in (2.50).

We continue with a brief discussion of the transformation of vectors and matrices under a change of basis. Let {e_{i}}_{i=1}^{n} be a given basis for V^{n}, a linear vector space of dimension n. Let {e′_{j}}_{j=1}^{n} be an alternative basis, which necessarily can be expressed by

(2.56)  e′_{j} = ∑_{i=1}^{n} T_{ij}e_{i}

with some n × n matrix T thus determined. Indeed, T is obtained by assembling columnwise the vectors e′_{j}; expressed with respect to the basis {e_{i}}. The vectors representing the e′_{j} must be linearly independent, so T is necessarily nonsingular.

Now a vector v ∈ V can be represented with respect to either basis, for example

(2.57)  v = ∑_{i=1}^{n} p_{i}e_{i} = ∑_{j=1}^{n} q_{j}e′_{j};

equivalently, we can say that v is represented with respect to the basis {e_{i}} by p and with respect to {e′_{j}} by q, with

(2.58)  p = (p_{i} ; ⋮ ; p_{n}) , q = (q_{1} ; ⋮ ; q_{n}) .

The relation between p and q, that is the transformation of the vector v under this change of basis, is readily obtained from (2.56), (2.57),

v = ∑_{i=1}^{n} p_{i}e_{i} = ∑_{j=1}^{n} q_{j}e′_{j}

= ∑_{j=1}^{n} q_{j} ∑_{i=1}^{n} T_{ij}e_{i}

= ∑_{i=1}^{n}(∑_{j=1}^{n} T_{ij}q_{j})e_{i} (2.59)

so clearly

(2.60)  T q = p, q = T^{−1}p.

This expresses the transformation of vectors, under a change of basis. Next let A be an n × n matrix. Under this change of basis, p is transformed to T^{−1}p and Ap is necessarily transformed to T^{−1}Ap = (T^{−1}AT)(T^{−1}p), so if A is the representation of a given linear operator on V with respect to the basis {e_{i}}, the representation with respect to the basis {e′_{j}} is T^{−1}AT. This is often called a \underline{similarity transformation} of a matrix.

Finally, let B : V^{n} → V^{m} be a transformation from V^{n} to V^{m}, of dimension m. With respect to the basis {e_{i}}_{i=1}^{n} for V^{n} and {β_{i}}_{i=1}^{m} for V^{m}, B is represented by on m × n matrix, also denoted by B. Now suppose we wish to change basis in (either or) both spaces, via transformations (2.56)

for V^{n} and a similar relation β′_{j} = ∑_{i=1}^{m} Q_{ij}β_{i} for V^{m} with some nonsingular

m × m matrix Q. For any p ∈ V^{n}, Bp ∈ V^{m}; p transforms to T^{−1}p and Bp to Q^{−1}Bp = Q^{−1}BT (T^{−1}p) under these basis changes, so B transforms to Q^{−1}BT. This is the generalization of a similarity transformation to such non-square matrices.

A common example of such a change of basis occurs when one wishes an \underline{orthonormal} basis for a Hilbert space H^{n}, i.e. basis functions \{e′_{j}\}_{j=1}^{n} satisfying

(2.61)  (e′_{j}, e′_{k}) = δ_{jk}, 1 ≤ j, k ≤ n.

Given any basis {e_{i}}_{i=1}^{n}, one can easily compute such an orthonormal basis by the \underline{Gram-Schmidt} procedure,

e′_{1} = e_{1}/∥e_{1}∥
(2.62)  α_{j} = e_{j} − ∑_{i=1}^{j−1}(e_{j}, e′_{i})e′_{i}, e′_{j} = α_{j}/∥α_{j}∥, j = 2, …, n

In (2.62), none of the α_{j} can be zero, for this would contradict linear independence of the {e_{i}}.

A change of basis is often used to transform some given square matrix into an attractive structural form, e.g. diagonal. Not all matrices, however, can be so “diagonalized” by a similarity transformation, as we shall discuss in the following chapter. Furthermore, a real matrix A may have complex eigenvalues, and thus be similar to a complex diagonal matrix via a complex similarity transformation; in some applications, this is quite inconvenient.

We are thus motivated to introduce a broader class of matrices than diagonal, that of block diagonal matrices. An n × n matrix A is \underline{block diagonal}, with “diagonal blocks” b_{1}, …, b_{m} if there exist integers α_{k} and blocks b_{k} (2.63) 0 = α_{0} < α_{1} < α_{2} < ⋯ < α_{m} = n, b_{k} = {α_{k−1} +1, ⋯, α_{k}} k = 1, ⋯, m

such that

(2.64)  A_{ij} = 0 for all i, j, i ∈ b_{k}, j ≠ b_{k} for some k,

i.e. A_{ij} is zero unless i and j are in the same diagonal block.

The extreme examples of block diagonal matrices correspond to diagonal matrices, i.e. m = n, α_{k} = k for all k in (2.63), (2.64), and “full” matrices, with m = 1 in (2.63) and (2.64) vacuous. We shall see in chapter 3 that the Jordan canonical form of a matrix (not similar to a diagonal matrix) is in a convenient block diagonal form, and that real matrices can often be transformed to convenient block diagonal structure by real similarity transformations.

The attractiveness of block diagonal structure of a matrix is that for many purposes, the individual blocks can be treated separately and the analysis considerably simplified. the simplification results essentially from the following lemma:

Lemma 2.9. Suppose a square matrix A is block diagonal. Then so are A^{†}, A^{p} for any positive integer p, and A^{−1} (if A is nonsingular) with the same diagonal blocks as for A.

The proof follows from the basic rule for matrix multiplication, and is left as an exercise. Furthermore, if B has the same block diagonal structure as A, then so do the products AB and BA.

The block structure of matrices, square matrices in particular, is used more generally. Let A be an n × n matrix and blocks b_{k} as in (2.63). Then we may regard A as a matrix or assembly of its blocks [A]_{kℓ}, 1 ≤ k, ℓ ≤ m, where each block [A]_{kℓ} is a matrix of dimension dim b_{k} × dim b_{ℓ}

(2.65)  [A]_{kℓ} = {A_{ij}} i ∈ b_{k}, j ∈ b_{ℓ}.

If two n × n matrices A, B are so partitioned, with the same blocks b_{k}, then matrix multiplication holds in block form

(2.66)  [AB]_{kℓ} = ∑_{j=1}^{m}[A]_{kj}[B]_{jℓ} , 1 ≤ k, ℓ ≤ m.

In this notation, a matrix A is block diagonal if and only if [A]_{kℓ} = 0 for k ≠ ℓ.

Let G_{n} denote the set of nonsingular n × n matrices containing only n nonzero elements. Obviously, such matrices will have only one nonzero element in any one row or any one column, and the determinant expansion of such a matrix will have only one nonvanishing term. Nonsingular diagonal matrices are special examples of elements of G_{n}.

The set G_{n} is obviously not a linear vector space, but it is a (noncommutative) multiplicative group. In particular

Lemma 2.10. Let A, B ∈ G_{n}. Then

(2.67)  AB, BA ∈ G_{n}

(2.68) A^{†} ∈ G_{n}, A^{†}A and AA^{†} are diagonal with strictly positive diagonal elements

(2.69)  A^{−1} ∈ G_{n}.

Proof. (2.67) and (2.68) are verified immediately from the rule for matrix multiplication. In particular, for A ∈ G_{n} and i, j such that A_{ij} ≠ 0, then (AA^{†})_{ii} = (A^{†}A)_{jj} = |A_{ij}|^{2}. To show (2.69), we use A^{−1} = (A^{†}A)^{−1}A^{†}, which is obviously true for any nonsingular matrix A, not just those in G_{n}. However, for A ∈ G_{n}, A^{†}A is diagonal, so (A^{†}A)^{−1} is also diagonal, and in G_{n}, and thus so in the product (A^{†}A)^{−1}A^{†}, using (2.67). Indeed, for A ∈ G_{n} and i, j such that A_{ij} ≠ 0, A_{ji}^{−1} = 1/A_{ij}.

Problems

1. Let p, q, r, s ∈ ℝ^{3}, and let the 3 × 3 matrix A be given by A = pq^{t} + rs^{t}, i.e. A_{ij} = p_{i}q_{j} + r_{i}s_{j} for i, j = 1, 2, 3. Prove that A is singular. Generalize this result to ℝ^{n}.

2. Let p, q, r ∈ ℝ^{n} be given, and x ∈ ℝ^{n} determined from x + p(p, x) − q(q, x) = r. What condition(s) on p, q, r are necessary so that x is uniquely determined? Find x explicitly in this case.

3. Given A : ℝ^{4} → ℝ^{3}, given by A = (\begin{matrix}1 & 0 & 1 & 0 \\ 1 & −1 & −1 & 1 \\ 1 & 1 & 3 & −1 \end{matrix}); find ker(A), R(A),

ker(A^{t}), R(A^{t}).

4. Given A = (\begin{matrix}0 & 1 & −1 \\ 1 & 3 & 1 \\ 0 & 0 & 0 \end{matrix}); find a diagonal matrix D and elementary

matrices E_{1}, …, E_{N} such A = Π_{i=1}^{M}E_{i}D Π_{i=M+1}^{N}E_{i}. Also find a second example of a representation of A in this form.

5. A square matrix A is called symmetric if it is real and A^{t} = A, and, more generally, Hermitian if A = A^{†}. Let H be a complex, finitedimensional Hilbert space (F = ℂ, H = ℂ^{n}) and A an n × n matrix such that (h, Ah) is always real, i.e. (h, Ah) ∈ ℝ for all h ∈ H. Prove that this happens if and only if A is Hermitian.

6. Find an example of a matrix A which does \underline{not} admit a representation

of the form Π_{i=1}^{N}E_{i}D, where the E_{i} are elementary matrices and D is diagonal.

7.^{∗} Show that if a (square) matrix A is nonsingular, then it does admit such “one-sided” representations, i.e. there exist E_{i}, D such that A =

Π_{i=1}^{N}E_{i}D (or A = D Π_{i=1}^{N}E_{i}).

8. Given A = (\begin{matrix}0 & 1 & −1 \\ 1 & 3 & 1 \\ 0 & 0 & 0 \end{matrix}) , B = (\begin{matrix}1 & −2 & −3 \\ 0 & 1 & 2 \\ −2 & 1 & 0 \end{matrix}); find r(AB) and r(BA).

9. True or false: If A, B are n × n matrices, then r(AB) = r(BA). Either prove this or find a counterexample.

10. Let A be an m × n matrix, and B an n × m matrix, with m > n. True or false: AB is always singular. Either prove this or find a counterexample.

11. True or false: If A is an n × n matrix invariant under similarity transformations, i.e. T^{−1}AT = A for all nonsingular n × n T, then A = αI, α ∈ F, I the n×n identity matrix. Prove or find a counterexample.

12^{∗}. Show that the space of n × n matrices, equipped with the norm ∥A∥ = sup_{h∈H^{n}}_{h≠0}∥Ah∥/∥h∥, is a Banach space. You may choose whichever equiva-

lent norm you wish for H^{n}.

13. Prove Gershgorin’s theorem: Let A be an n × n matrix, and for i = 1, ⋯, n, let ρ_{i} be a closed circle in the complex plane, ρ_{i} = {z ∈ ℂ||z − A_{ii}| ≤ ∑_{j≠i} |A_{ij}|}. Show that all of the eigenvalues of A lie in

the region n ; ⋃_{i=1}ρ_{i}.

14. For vectors in H, of dimension n, let ∥ · ∥ denote the ℓ_{2} norm given by (1.5), and let the norm of an n × n matrix be as shown in problem 12. Let σ(A) = maximum_{i}|λ_{i}(A)|, where λ_{i}(A) are the eigenvalues of A_{j}. σ(A) is called the \underline{spectral radius} of A. Prove the following:

(a) σ(A) ≤ ∥A∥

(b) ∥AB∥ ≤ ∥A∥ ∥B∥, for any n × n matrices A, B

(c) ∥A∥ ∥A^{−1}∥ ≥ 1, for any nonsingular A. The product ∥A∥ ∥A^{−1}∥ is sometimes called the \underline{condition number} of A, it is a measure of how close A is to being singular.

15. Prove lemma 2.9.

16. Let H be of dimension 5, spanned by {v, w, x, y, z}, and let A be a linear operator on H satisfying Av = w − y, Aw = v + 2z, Ax = v −z, Ay = w −x−y, Az = −2x. Find a matrix representation for A with respect to each of the following bases for H: (1) {v, w, x, y, z};

(2) {w, v, x, y, z}; (3) {v + w, v − w, x + y, x − y, v − x + z}.

17. Let A : H_{m} → H_{n} be represented by an n×m matrix, where H_{m} = ℂ_{m} and H_{n} = ℂ_{n}, but with inner products

(p, x)_{n} = n ; ∑_{i=1}β_{i}p_{i}x̄_{i}, (q, y)_{m} = m ; ∑_{i=1}γ_{i}q_{i}ȳ_{i}

where the β_{i}, γ_{i} are given positive (real) numbers. Determine a formula for A^{†} such that (2.11) holds for all x ∈ H_{n}, y ∈ H_{m}.

18. Let X = ℝ^{2} with the inner product (p, q)_{x} = 4p_{1}q_{1} + p_{1}q_{2} + p_{2}q_{1} + p_{2}q_{2}

a) Show this is indeed an inner product for X.

b) Given A : X → X, find an expression for A^{†} such that (Ap, q)_{x} = (p, A^{†}q)_{x} for all p, q ∈ X.

19. Given nonzero p, q ∈ ℝ^{n}, and a real n×n matrix B satisfying r·Br > 0 for all nonzero r ∈ ℝ^{n} satisfying r · p = 0. Let A be an (n + 1) × (n + 1) matrix with block structure

A = (\begin{matrix}B & p + Bq \\ p^{†} & p · q \end{matrix})

Prove that A is nonsingular.

20. Prove that r(AA^{†}) = r(A^{†}A) for any matrix A, whether or not a square matrix.

3 Invariant subspaces and Jordan form of a matrix

Throughout this chapter, F = ℂ for definiteness and simplicity.

Given an n × n matrix A : H → H, H of dimension n, it is natural to ask for what values of λ ∈ ℂ is the matrix A − λI singular. Here I is the n × n identity matrix; hereafter we shall write simply A − λ for A − λI, etc.

The matrix A − λ is singular if and only if λ satisfies

(3.1)  P_{A}(λ) = 0

where

(3.2)  P_{A}(λ) = det(A − λ)

is the \underline{characteristic polynomial} of A. Obviously P_{A}(λ) is a polynomial of order exactly n,

(3.3)  P_{A}(λ) = (−λ)^{n} + (trA)(−λ)^{n−1} + ⋯ + det A

where trA = ∑_{i=1}^{n} A_{ii} is called the \underline{trace} of A. Clearly P_{A}(λ) can be factored,

(3.4)  P_{A}(λ) = Π_{i=1}^{k}(λ_{i} − λ)^{m_{i}}

where the λ_{i} ∈ ℂ are the distinct \underline{eigenvalues} of A, satisfying λ_{i} ≠ λ_{j} for all i ≠ j, and the positive integer m_{i} the \underline{multiplicity} of λ_{i}. Since the λ_{i} are the roots of (3.1), we have from (3.3)

(3.5)  Π_{i=1}^{k}λ_{i}^{m_{i}} = det A, ∑_{i=1}^{k} m_{i}λ_{i} = trA, ∑_{i=1}^{k} m_{i} = n

A closely related question is that of the \underline{minimal polynomial} M_{A} of A, the polynomial of lowest degree such that

(3.6)  M_{A}(A) = 0

(The zero in the right side of (3.6) means the ideatically zero n × n matrix, since M_{A}(A) is obviously an n × n matrix.)

The minimal polynomial M_{A} is unique up to normalization (multiplication by a constant) and divides every polynomial Q such that Q(A) = 0.

The polynomial M_{A} is of order no greater than n^{2}, as A has only n^{2} elements; we shall see that in fact it is of order no greater than n. The polynomial M_{A} can also be factored, and we claim

Lemma 3.1.

(3.7)  M_{A}(λ) = Π_{i=1}^{k}(λ_{i} − λ)^{μ_{i}}

where the μ_{i} are positive integers and λ_{i} the eigenvalues of A.

Proof. Suppose M_{A}(A) contained a factor ξ − A, where ξ is not one of the eigenvalues. Then ξ − A is nonsingular, and since all of the factors of M_{A}(A) commute, we can write M_{A}(A) = B(ξ − A), where the matrix B, so determined, is just the product of the remaining factors.

As ξ − A is nonsingular, R(ξ − A) = H, and if M_{A}(A)H = 0, then B(ξ − A)H = BH = 0, so B is identically zero. Thus M_{A} is not the minimal polynomial of A, since B is also a polynomial, of order are less.

Furthermore, every eigenvalue λ_{i} appearing in (3.4) must also appear in (3.7), with some positive μ_{i}. Suppose not, suppose (3.7) contains no factors of (λ_{j} − A). Since A − λ_{j} is singular, there exists a nonzero vector x_{j} such that (A − λ_{j})x_{j} = 0. And

M_{A}(A)x_{j} = Π_{i≠j}(λ_{i} − A)^{μ_{i}}x_{j} = Π_{i≠j}(λ_{i} − λ_{j})^{μ_{i}}x_{j} ≠ 0,

since λ_{i} ≠ λ_{j} for all i ≠ j. This contradicts (3.6), proving our assertion.

Both the characteristic and minimal polynomials of a matrix are invariant under similarity transformations. Let A, B be n × n matrices, with B = T^{−1}AT for some nonsingular T. Then det(B − λI) = det(T^{−1}AT − λI) = det(T^{−1}(A − λI)T) = det(A − λI), so P_{B}(λ) = P_{A}(λ). And 0 = M_{B}(B) = M_{B}(T^{−1}AT) = T^{−1}M_{B}(A)T, since M_{B} is a polynomial, so M_{B}(A) = 0 and M_{A}(B) = 0. Thus if each is the minimal polynomial for its respective matrix, normalized, for example, as in (3.7), they must coincide.

Suppose not, i.e., suppose that M_{A}(λ) ≠ M_{B}(λ). We first assert that degree M_{A}(λ) = degree M_{B}(λ). If, for example, degree M_{A}(λ) < degree M_{B}(λ), then as M_{A}(B) = 0, M_{B} is not the minimal polynomial for B. With M_{A} and M_{B} normalized, as in (3.7), for example, since they are of the same degree it follows that M_{A}(λ) − M_{B}(λ) is a polynomial of degree at most one less than that of M_{A}(λ) or M_{B}(λ). Since M_{A}(A) − M_{B}(A) = M_{A}(B) = M_{B}(B) = 0, it follows that if M_{A}(λ) − M_{B}(λ) is not identically zero, neither M_{A} nor M_{B} is the minimal polynomial for its respective matrix.

Often the eigenvalues of A are distinct, with k = n and all the m_{i} = 1 in (3.4), (3.5). In this case, there are always n linearly independent eigenvectors. Thus A is always similar to a diagonal matrix, and the characteristic and minimal polynomials coincide, i.e. P_{A}(λ) = M_{A}(λ). To justify these statements, assume that the λ_{i}, i = 1, …, n are distinct but that the corresponding eigenvectors {x_{i}} are linearly dependent. Then possibly after a reordering of the λ_{i}, x_{i} there exist α_{2}, …, α_{n} ∈ ℂ such that

x_{1} = ∑_{j=2}^{n} α_{j}x_{j}

so that

Π_{i=2}^{n}(A − λ_{i}I)x_{1} = Π_{i=2}^{n}(A − λ_{i}I) ∑_{j=2}^{n} α_{j}x_{j}

= ∑_{j=2}^{n} α_{j} Π_{i=2}^{n}(λ_{j} − λ_{i})x_{j}

= 0

whereas the left side is

Π_{i=2}^{n}(λ_{1} − λ_{i})x_{1} ≠ 0

by the assumption of distinct eigenvalues.

So the {x_{i}} are linearly independent, and the matrix T, the i-th column of which is x_{i}, i = 1, …, n, is nonsingular. Then one easily verifies that

T^{−1}AT = diag(λ_{1}, …, λ_{n}).

But a simple example shows that this is not always the case: let

B = (\begin{matrix}2 & 0 \\ 0 & 2 \end{matrix}) , G = (\begin{matrix}2 & 1 \\ 0 & 2 \end{matrix});

then P_{B}(λ) = P_{G}(λ) = (2 − λ)^{2}, M_{B}(λ) = 2 − λ but M_{G}(λ) = (2 − λ)^{2}.

Let

(3.8)  U_{ij} = ker(A − λ_{i})^{j}, for i = 1, …, k; j = 1, …, μ_{i}.

The U_{ij} are subspaces of H, and from (3.8) we have

(3.9)  U_{i(j−1)} ⊂ U_{ij}, j > 1, and dim U_{ij} ≥ 1.

We give a proof of the strict inclusion in (3.9). First we show that U_{i(μ_{i}−1)} ⊂ U_{iμ_{i}}. If not, i.e. if these spaces coincide, set

V = Π_{j=1}_{j≠i}^{k}(A − λ_{j}I)^{μ_{j}}H.

Recalling M_{A} given by (3.7), it follows that V ⊆ U_{iμ_{i}} and thus V ⊆ U_{i(μ_{i}−1)} by assumption. But then

(A − λ_{i}I)^{μ_{i}−1}V = (A − λ_{i}I)^{μ_{i}−1} Π_{j=1}_{j≠i}^{k}(A − λ_{j}I)^{μ_{j}}H = 0,

contradicting the fact that M_{A} given by (3.7) is the minimal polynomial.

Thus there exists x ∈ U_{iμ_{i}}, x ∉ U_{i(μ_{i}−1)}, and for any j = 1, …, μ_{i}−2, y_{j} = (A − λ_{i}I)^{j}x satisfies y_{j} ∈ U_{i(μ_{i}−j)} y_{j} ∉ U_{i(μ_{i}−j−1)}.

Theorem 3.2. The U_{ij} are invariant under multiplication by A − ξ, for any ξ ∈ F, i.e. (A − ξ)U_{ij} ⊂ U_{ij}. Furthermore, if ξ ≠ λ_{i} and x ∈ U_{ij}, x ≠ 0, then (A − ξ)x ≠ 0.

Proof. For z ∈ U_{ij}, y = (A − ξ)z satisfies

(3.10)  (A − λ_{i})^{j}y = (A − λ_{i})^{j}(A − ξ)z = (A − ξ)(A − λ_{i})^{j}z = 0

since these factors commute, so y ∈ U_{ij}.

The second statement is proved by contradiction. Suppose not, i.e. suppose there exists x ≠ 0, x ∈ U_{ij} such that (A − ξ)x = 0 with ξ ≠ λ_{i}. Then

0 = (A − λ_{i})^{j}x
= (A − ξ + (ξ − λ_{i}))^{j}x
= (A − ξ)^{j}x + j(A − ξ)^{j−1}(ξ − λ_{i})x + ⋯ + (ξ − λ_{i})^{j}x
= (ξ − λ_{i})^{j}x

which is not zero, as x ≠ 0 and ξ ≠ λ_{i}. This contradiction proves the statement.

Theorem 3.3. The subspaces U_{iμ_{i}} are linearly independent, i.e. if ∑_{i=1}^{k} α_{i}x_{i} =

0 with each x_{i} ∈ U_{iμ_{i}}, x_{i} ≠ 0, α_{i} ∈ F, then all of the α_{i} = 0.

Proof. First we show that any U_{i1} is linearly independent of the other U_{jμ_{j}}. Suppose not, then there exists x ∈ U_{i1}, x ≠ 0, such that

(3.11)  x = ∑_{j≠i} α_{j}x_{j}, x_{j} ∈ U_{jμ_{j}}, α_{j} ∈ F.

Denote by

(3.12)  M_{i} = Π_{j≠i}(λ_{j} − A)^{μ_{j}}

then

(3.13)  M_{i}x_{j} = 0 , x_{j} ∈ U_{jμ_{j}},

for any j ≠ i, using (3.8). But x ∈ U_{i1} satisfies (A − λ_{i})x = 0, and so by application of the second statement of Theorem 3.2, M_{i}x ≠ 0. Thus applying M_{i} to both sides of (3.11) gives a contradiction, proving this result.

Now by induction, for some integer β ≥ 2, suppose that U_{i(β−1)} is linearly independent of the U_{jμ_{j}} but U_{iβ} is not. Then (3.11) holds for some x ∈ U_{iβ}, x ∉ U_{i(β−1)}. Set y = (A − λ_{i})x. Since x ∉ U_{i(β−1)}, y ≠ 0, but as x ∈ U_{iβ}, y ∈ U_{i(β−1)}. Applying (A − λ_{i}) to both sides of (3.11), we find

(3.14)  y = (A − λ_{i})x

= ∑_{j≠i} α_{j}(A − λ_{i})x_{j}.

From Theorem 3.2, (A − λ_{i})x_{j} ∈ U_{jμ_{j}}, so (3.14) now contradicts the induction hypothesis that U_{i(β−1)} is linearly independent of the U_{jμ_{j}}. This completes the proof.

Theorem 3.4.

(3.15)  ⊕_{i=1}^{k}U_{iμ_{i}} = H

Proof. Denote by X = ⊕_{i=1}^{k}U_{iμ_{i}}; clearly X ⊂ H, so it suffices to show that

dim X = n. We use k − 1 applications of Sylvester’s theorem, (2.54) in particular, to obtain, using (3.6), (3.7)

n = γ(M_{A}(A))

= γ( Π_{i=1}^{k}(λ_{i} − A)^{μ_{i}})

≤ ∑_{i=1}^{k} γ((λ_{i} − A)^{μ_{i}})

= ∑_{i=1}^{k} dim U_{iμ_{i}}. (3.16)

From theorem 3.3, the spaces U_{iμ_{i}} are linearly independent, so

(3.17)  dim X = ∑_{i=1}^{k} dim U_{iμ_{i}},

and from (3.16), (3.17), we find dim X ≥ n. Thus dim X = n, completing the proof.

From Theorem 3.4, it follows that if we can find a convenient representation for the matrix A on any such subspace U_{iμ_{i}}, we will have such a representation for A on all of H. Thus our problem is reduced to finding a convenient basis for U_{iμ_{i}}.

First we note that if μ_{i} = 1, then A restricted to U_{iμ_{i}} = U_{i1} is just λ_{i} times the identity matrix, so any basis for U_{i1} may be used. Thus we consider the case μ_{i} > 1.

Denote by V_{iν}, the subspaces determined by V_{i1} = U_{i1},

(3.18)  V_{iν} = U_{i1} ∩ R((A − λ_{i})^{ν−1}), ν = 2, ⋯, μ_{i};

since (A − λ_{i})^{μ_{i}} appears in the minimal polynomial, V_{iμ_{i}} is not trivial (but V_{i(μ_{i}+1)} would be if defined similarly).

Clearly

V_{iμ_{i}} ⊆ V_{i(μ_{i}−1)} ⊆ ⋯ ⊆ V_{i2} ⊆ V_{i1} = U_{i1};

we choose a basis for U_{i1} by first choosing a basis for V_{iμ_{i}}, then adding additional elements to get a basis for V_{i(μ_{i}−1)}, etc. Thus a set of vectors {e_{j1}^{i}}, j = 1, ⋯, dim V_{iν}, is a basis for V_{iν}, for any ν = 1, ⋯, μ_{i}. And given j, for ν = ν(j) determined such that is e_{j1}^{i} ∈ V_{iν} but e_{j1}^{i} ∉ V_{i(ν+1)}, that is

(3.19)  dim V_{i(ν+1)} < j ≤ dim V_{iν} (dim V_{i(μ_{i}+1)} ≡ 0)

it follows from (3.18) that there is a nonzero vector e_{jν}^{i} ∈ U_{iμ_{i}} such that (A − λ_{i})^{ν−1}e_{jν}^{i} = e_{j1}^{i}. We determine a basis \{e_{jθ}^{i}\}_{θ=1}^{ν(j)} for a \underline{Jordan block} Y_{j}^{i}, of dimension ν(j) obtained form (3.19), by

(3.20)  (A − λ_{i})e_{jθ}^{i} = e_{j(θ−1)}^{i}, θ = 2, ⋯, ν(j), (A − λ_{i})e_{j1}^{i} = 0,

alternatively written in the form

A (e_{j1}^{i} ; e_{j2}^{i} ; ⋮ ; e_{jν}^{i}) = λ_{i} (e_{j1}^{i} ; e_{j2}^{i} ; ⋮ ; e_{jν}^{i}) + ( 0 ; e_{j1}^{i} ; ⋮ ; e_{j(ν−1)}^{i})

or in block form, A restricted to Y_{j}^{i} with this basis, assumes the form

(3.21)  A|_{Y_{j}^{i}} = (\begin{matrix}λ_{i} & & 1 & & 0 \\ & ... & & ... \\ & & & & 1 \\ 0 & & & & λ_{i} \end{matrix})

which we call a Jordan block. We shall show that

(3.22)  U_{iμ_{i}} = ⊕_{j=1}^{dim U_{i1}}Y_{j}^{i}

so that {{e_{jθ}^{i}}_{j=1}^{ν(j)}}_{j=1}^{dim U_{i1}} is a basis for U_{iμ_{i}}, and with respect to this basis, A restricted to U_{iμ_{i}} is just an assembly at Jordan blocks, of a form

(3.23)  A|_{U_{iμ_{i}}} = (\begin{matrix}λ_{i} & 1 \\ & & 1 & & & & 0 \\ & & λ_{i} & 0 \\ & & & λ_{i} & 1 \\ & & & & ... & 1 \\ & & & & & λ_{i} & 0 \\ & 0 & & & & & λ_{i} \\ & & & & & & & λ_{i} \end{matrix})

Next we justify (3.22). Clearly the right side of (3.22) is a subspace of U_{iμ_{i}} and contains U_{i1} as a subspace. Thus, suppose it contains U_{i(η−1)} as a subspace but not U_{iη}, for some η, 2 ≤ η ≤ μ_{i}. For arbitrary x ∈ U_{iη}, y = (A − λ_{i})^{η−1}x satisfies y ∈ U_{i1} and y ∈ R((A − λ_{i})^{η−1}), so y ∈ V_{iη} from (3.18).

Then from (3.19), there exist constants τ_{j} such that

y = ∑_{j=1}^{dim V_{iη}} τ_{j}e_{j1}^{i}

and from (3.20), it follows that

x − ∑_{j} τ_{j}e_{jη}^{i} ∈ U_{i(η−1)}

so that ⊕_{j}Y_{j}^{i} also contains U_{iη} as a subspace, proving our claim.

Finally extending to all of H, using theorem 3.4, we choose a basis {{{e_{jθ}^{i}}_{θ=1}^{dim Y_{j}^{i}}}_{j=1}^{dim U_{i1}}}_{i=1}^{k}, and the matrix A is put in (upper) Jordan canonical form. The diagonal elements are the eigenvalues λ_{i}; the off - diagonal elements are equal to zero, except that

(3.24)  if A_{ℓℓ} = A_{(ℓ+1)(ℓ+1)}, then A_{ℓ(ℓ+1)} may be 0 or 1.

From this construction we see that the Jordan form of a matrix A is diagonal, i.e. A is similar to a diagonal matrix, if all of the μ_{i} appearing in (3.7) are equal to 1.

More generally, we see that the dimensions of the Jordan blocks are determined uniquely by the dimensions of the subspaces U_{ij}, which are obviously independent of the specific choice of basis. Thus the Jordan form of a matrix is unique up to possible reordering of the Jordan blocks.

Two n × n matrices A, B are similar to each other, i.e. B = T^{−1}AT for some nonsingular T, if and only if their Jordan forms coincide (up to recordering of the Jordan blocks). For if Q^{−1}BQ = J is in Jordan form, then J = Q^{−1}T^{−1}AT Q = (T Q)^{−1}A(T Q) is also the Jordan form for A. And if B = QJQ^{−1} and A = SJS^{−1}, then B = Q(S^{−1}AS)Q^{−1} = (SQ^{−1})^{−1}A(SQ^{−1}).

Finally, we return to the characteristic polynomial of A. The dimension of the matrix in (3.23) is dim U_{iμ_{i}}, so by inspection

(3.25)  det(A − λ) = Π_{i=1}^{k}(λ_{i} − λ)^{dim U_{iμ_{i}}},

i.e. m_{i} in (3.4) is equal to dim U_{iμ_{i}} = ∑_{j=1}^{dim U_{i1}} ν(j). As each Y_{j}^{i} ⊂ U_{iμ_{i}}, dim Y_{j}^{i} = ν(j), obtained from (3.19), and

(3.26)  μ_{i} = maximum ν(j), j = 1, …, dim U_{i1},

it follows from (3.26) that

(3.27)  μ_{i} ≤ m_{i}, i = 1, …, k.

Thus from (3.4) and (3.7), M_{A}(λ) divides P_{A}(λ) and we have proved

Theorem 3.5 (Cayley-Hamilton theorem). Every n × n matrix satisfies its own characteristic equation, P_{A}(A) = 0.

In the special case that the matrix A is real, its eigenvalues are either real or occur in complex conjugate pairs. if all of the eigenvalues happen to be real, then so is the Jordan form of A, and the similarity transformation matrix T, such that T^{−1}AT is in Jordan form can also be taken real. More generally, at least some of the eigenvalues are complex, and we wish to transform A to some sort of canonical form by a real similarity transformation.

The idea of this construction is very simple; instead of using the e_{jν}^{i} as basis vectors, we use their real and imaginary parts. More specifically, we can place the eigenvalues in any convenient order, so we consider any complex pair in the form

(3.28)  Reλ_{i} = Re(λ_{i+1}) = ξ, Im(λ_{i}) = −Im(λ_{i+1}) = η

where ξ, η are real. The characteristic and minimal polynomials of A, factored into real factors, contain factors of

(3.29)  ((λ − ξ)^{2} + η^{2})^{m_{i}}, ((λ − ξ)^{2} + η^{2})^{μ_{i}}

respectively, and we have, for such values of i,

(3.30)  m_{i} = m_{i+1}, μ_{i} = μ_{i+1}, dim Y_{j}^{i} = dim Y_{j}^{i+1}, j = 1, …, μ_{i}

(3.31)  e_{jν}^{i+1} = \overline{e_{jν}^{i}}, ν = 1, …, ν(j), j = 1, …, μ_{i}.

Instead of working with U_{ij} and U_{i+1,j} separately, we consider a basis for their direct sum

(3.32)  X_{j} = U_{ij} ⊕ U_{i+1,j} = {x ∈ H|[(A − ξ)^{2} + η^{2}]^{j}x = 0}

We rewrite (3.20), taking Re(λ_{i}) = ξ, Im(λ_{i}) = η, Re(e_{jν}^{i}) = p_{jν}^{i}, Im(e_{jν}^{i}) = q_{jν}^{i}, with ξ, η, p_{jν}^{i}, q_{jν}^{i} all real. Taking real and imaginary parts we obtain

Ap_{j1}^{i} = ξp_{j1}^{i} − ηq_{j1}^{i},
Aq_{j1}^{i} = ηp_{j1}^{i} + ξq_{j1}^{i};
Ap_{jν}^{i} = ξp_{jν}^{i} − ηq_{jν}^{i} + p_{j(ν−1)}^{i},
(3.33)  Aq_{jν}^{i} = ηp_{jν}^{i} + ξq_{jν}^{i} + q_{j(ν−1)}^{i}; for ν > 1

instead of (3.20). Taking {p_{jν}^{i}, q_{jν}^{i}}_{ν=1}^{dim Y_{j}^{i}} as a basis for W_{j} = span {p_{jν}^{i}, q_{jν}^{i}}_{ν=1}^{dim Y_{j}^{i}}, from (3.33) we get a real representation for A of the form

(3.34)  A|_{W_{j}} = (\begin{matrix}ξ & η & 1 & 0 & & & 0 \\ −η & ξ & 0 & 1 \\ & & ξ & η & 1 & 0 \\ & & −η & ξ & 0 & 1 \\ & 0 \\ & & & & & & ξ & η \\ & & & & & & −η & ξ \end{matrix})

analogously with (3.21). This is the form of “Jordan block” obtainable by a real similarity transformation. As a special case, if ξ ± iη are simple eigenvalues of A, i.e. μ_{i} = m_{i} = 1, we obtain the “diagonal block”

(\begin{matrix}ξ & η \\ −η & ξ \end{matrix})

in the real canonical form of A.

Problems

1. Find the Jordan canonical form of each of the following

(\begin{matrix}1 & −1 & −1 \\ 2 & 3 & 1 \\ 0 & 0 & 0 \end{matrix}) (\begin{matrix}1 & 2 & −1 \\ 0 & 1 & −1 \\ 0 & 0 & 2 \end{matrix})  (\begin{matrix}−1 & 0 & 2 \\ 0 & −1 & 0 \\ 0 & 0 & −1 \end{matrix})

2. Prove that there is no n × n matrix A, such that T^{−1}AT is diagonal for some nonsingular T, whereas Q^{−1}AQ is in Jordan canonical form, but not diagonal, for some nonsingular Q.

3. Let A be an n × n matrix, all of whose eigenvalues are equal, but such that A is not a scalar multiple of the n × n identity matrix. Prove that A is not similar to any diagonal matrix.

4. Prove that any n × n matrix A can be uniformly approximately by matrices similar to diagonal matrices, i.e. for any ε > 0, there exists A_{ε}, similar to a diagonal matrix, such that |A_{ij} − A_{εij}| < ε for any 1 ≤ i, j ≤ n. Outline: A = T^{−1}JT for some J in Jordan canonical form, but presumably not diagonal. But J can be perturbed (show how) to J̃, such that J̃−J is arbitrarily small, but the eigenvalues of J̃ are distinct. Thus J̃ is similar to a diagonal matrix. Take A_{ε} = T^{−1}J̃T; it remains to show that if J̃ is similar to a diagonal matrix, so is A_{ε}, and that A_{ε} − A is small as described above.

Explain why it is \underline{not} possible to pass to the limit as ε → 0 and thus show that A must be similar to a diagonal matrix.

5. True or false, for A, B n × n matrices:

a) If A, B are both similar to diagonal matrices, then so are AB and BA.

b) If E_{jka} is an n × n elementary matrix, as described in chapter 2, and if A is similar to a diagonal matrix, then so are AE and EA.

c) If AB = BA and A is similar to a diagonal matrix, then so are B and AB.

d) If A is a real matrix, and is similar to a diagonal matrix D, i.e. D = T^{−1}AT, then T can always be taken to be a real matrix.

e) If A is similar to a diagonal matrix and B is itself diagonal, then A + B is similar to a diagonal matrix.

f) If A is a 3 × 3 matrix and M_{A}(λ) = λ^{2}, then r(A) = 1, where r(A) is the rank of A as in chapter 2.

g) If dim U_{i1} = n for an n × n matrix A, with U_{ij} as given in (3.8), then A = λ_{i}I.

h) If two n×n matrixes A, B, have the same characteristic and minimal polynomials, i.e. P_{A}(λ) = P_{B}(λ) and M_{A}(λ) = M_{B}(λ), then they are similar to each other (B = T^{−1}AT for some nonsingular T.)

6. Show that a real 2 × 2 matrix can always be transformed into one of

the forms

(\begin{matrix}ξ & 0 \\ 0 & η \end{matrix}) , (\begin{matrix}ξ & 1 \\ 0 & ξ \end{matrix}) , or (\begin{matrix}ξ & η \\ −η & ξ \end{matrix})

by a real similarity transformation, with ξ, η real. What is the corresponding statement for real 3 × 3 matrices?

7. Let λ_{i} be an eigenvalue of an n × n matrix A. Show that dim ker(A − λ_{i})^{j} = dim ker(A^{†} − λ̄_{i})^{j} for any positive integer j, and thus that the dimensions of the invariant subspaces U_{ij} of A are the same as those of the corresponding subspaces for A^{†}.

8. Use the result of problem 4 to get an independent proof of the Cayley-Hamilton theorem.

9. Suppose that J = T^{−1}AT is in Jordan canonical form, but not diagonal. Describe the structure of the matrices T^{−1}A^{j}T for j a positive integer.

10. Let A be a 4 × 4 matrix such that M_{A}(λ) = (λ − 2)^{2}(λ + 1). What are the possible structures of the Jordan canonical form of A?

11. Let J be the Jordan form of an n × n matrix A. Show that J = D + G, where D is diagonal and G satisfies G^{μ} = 0, where μ =→_{i} maximum μ_{i}, μ_{i} the exponents appearing in the minimal polynomial of A, given by (3.7).

12. Find the characteristic polynomial of each of the following matrices of dimension n, for a, a_{1}, …, a_{n} ∈ ℂ given. In the second example, an induction argument on the dimension n will work.

(\begin{matrix}0 & 1 & & 0 \\ & 0 & & 0 & 1 \\ a_{1} & a_{2} & … & a_{n−1} & a_{n} \end{matrix}) (\begin{matrix}0 & a \\ a & 0 & a & 0 \\ & a & 0 \\ & 0 & & 0 & a \\ & & & a & 0 \end{matrix})

4 Some special classes of square matrices

Let f be a polynomial function of a single variable and A an n×n matrix. If f(A) = 0, meaning that the n × n matrix f(A) vanishes identically, then we easily verify

(4.1)  f(T^{−1}AT) = T^{−1}f(A)T = 0 for any nonsingular T

(4.2)  f̄(A^{†}) = (f(A))^{†} = 0,

where f̄ is the complex conjugate of f, i.e. the polynomial whose coefficients are the complex conjugates of those of f.

Lemma 4.1. A square matrix A is similar to a diagonal matrix if and only if its minimal polynomial contains no repeated factors, i.e. if all of the μ_{i} are equal to 1 in (3.7).

Proof. Choose T such that J = T^{−1}AT is in Jordan form. Then from (4.1), J satisfies M_{A}(J) = 0. If J is diagonal, then no repeated factors are needed in (3.7) to satisfy M_{A}(J) = 0. If J is not diagonal, i.e. a Jordan block for some λ_{i} is not diagonal, then the corresponding block of J − λ_{i} also does not vanish, and thus if M_{A}(λ) contains no additional factors of (λ − λ_{i}), neither does M_{A}(J).

Corollary 1. If A is similar to a diagonal matrix, then so is A^{†}.

Proof. Since M_{A}(A) = 0, M̄_{A}(A^{†}) = 0 from (4.2), it is easily verified that M̄_{A} is the minimal polynomial for A^{†}. Thus if M_{A} has no repeated factors, neither does M̄_{A}.

Remark. The adjoint property is not, in general, invariant under similarity transformations, i.e. in general

(4.3)  (T^{−1}AT)^{†} ≠ T^{−1}A^{†}T

Thus in general, if T^{−1}AT is diagonal, T^{−1}A^{†}T is not.

Corollary 2. Let f be a polynomial with no repeated factors. If f(A) = 0, then A is similar to a diagonal matrix.

Proof. Either f is itself the minimal polynomial for A, or else M_{A}(λ) is obtained from f(λ) by dividing f(λ) by those factors (λ − α) such that α_{i} is not an eigenvalue of A, i.e. such that A − α_{i} is nonsingular. In either case, the minimal polynomial of A contains no respected factors.

An application is the class of \underline{idempotent} matrices, n × n matrices A satisfying

(4.4)  Q^{2} = Q

equivalently f(Q) = Q(Q − 1) = 0, f a polynomial without repeated factors. Here obviously either Q = 0, Q = I, or else (4.4) is actually the minimal polynomial for Q.

Clearly idempotent matrices are similar to diagonal matrices, and have only 0, 1 as possible eigenvalues.

The class of idempotent matrices is invariant under similarity transformations, i.e. if Q is idempotent,

(T^{−1}QT)(T^{−1}QT) = T^{−1}Q^{2}T
= T^{−1}QT

so T^{−1}QT is also idempotent. Consequently, any matrix A which has only 0, 1 as eigenvalues, and which is similar to a diagonal matrix, is idempotent.

An especially important class of idempotent matrices are the projection operators Q : H → K, K ⊂ H a closed subspace satisfying

(4.5)  (h − Qh, y) = 0 for all h ∈ H, y ∈ K.

If we choose h = Qx ∈ K in (4.5), then y = Qx − Q^{2}x ∈ K is a legal choice also, and it follows immediately that Q satisfies (4.4). In addition, for any x, y ∈ H, Qx and Qy are in K, so by repeated use of (4.5), we have (Qx, y) = (Qx, Qy) = (x, Qy) = (x, Q^{†}y) so Q = Q^{†}. Such matrices are called Hermitian, and will be discussed further below.

Two n × n matrices A, B \underline{commute} if AB = BA. This is obviously a very special property, but it occurs in a number of interesting cases.

For example, if B commutes with a diagonal matrix D, then comparing (BD)_{ij} with (DB)_{ij}, 1 ≤ i, j ≤ n, we find B_{ij}(D_{ii} − D_{jj}) = 0 for all 1 ≤ i, j ≤ n, generally a very strong restriction on the off-diagonal elements of B. For example, if the diagonal elements of D are distinct, that is D_{ii} ≠ D_{jj} for all i ≠ j, and if BD = DB, B is also diagonal. Furthermore, commutivity survives similarity transformations; if A, B commute, so do T^{−1}AT and T^{−1}BT.

Theorem 4.2. Let A, B be commuting n × n matrices. Then:

(1) If U_{ij} = ker(A − λ_{i})^{j}, then B : U_{ij} → U_{ij}.

(2) If the eigenvalues of A are distinct, then A, B can be made diagonal simultaneously by a similarity transformation, i.e. these exists a nonsingular T such that T^{−1}AT and T^{−1}BT are both diagonal. Indeed, T^{−1}BT is diagonal for any T such that T^{−1}AT is diagonal.

(3) If A is similar to a diagonal matrix, there exists T such that T^{−1}AT is diagonal and T^{−1}BT is in Jordan canonical form.

Proof. For x ∈ U_{ij}, Bx satisfies

(4.6)  (A − λ_{i})^{j}Bx = B(A − λ_{i})^{j}x = 0,

so Bx ∈ U_{ij}. Now if the eigenvalues of A are distinct, then for λ_{i} any eigenvalue of A, U_{ij} is of dimension 1, so x ∈ U_{ij} implies Bx ∈ U_{ij} implies Bx = αx for some α ∈ F. Thus the eigenvectors of A are also the eigenvectors of B, and there are n linearly independent eigenvectors, since the eigenvalues of A are distinct. Choosing the eigenvectors of A as a basis for H puts A, B simultaneously into diagonal form.

If A is similar to a diagonal matrix, then for any eigenvalue λ_{i}, U_{ij} = U_{i1} for j ≥ 1, but dimU_{i1} can be larger than 1. Now B : U_{i1} → U_{i1}, but A, restricted to the subspace U_{i1}, is just λ_{i} times the identity matrix, so we learn nothing further from the commutivity property. We can choose a basis for U_{i1} which puts B, restricted to U_{i1}, in Jordan canonical form. Such a change of basis will obviously not change the from of A, on this subspace. Proceeding in this manner for each U_{i1} such that dimU_{i1} > 1, we retain A in diagonal form, but can simultaneously put B in Jordan canonical form. Of course, if it is also known that B is similar to a diagonal matrix, then the Jordan form of B is necessarily diagonal, so A and B can be made diagonal simultaneously.

An n × n matrix A is said to be \underline{normal} if it commutes with its adjoint

(4.7)  AA^{†} = A^{†}A.

Immediately from (4.7) we have ∥Ax∥ = ∥A^{†}x∥ for any vector x, as

∥Ax∥^{2} = (Ax, Ax)
(A^{†}Ax, x) = (AA^{†}x, x)

= (A^{†}x, A^{†}x)

= ∥A^{†}x∥^{2}.

Theorem 4.3. A normal matrix A is similar to a diagonal matrix. Furthermore ker(A) = ker(A^{†}), and R(A)⊥ ker(A).

Proof. Suppose not, then from Lemma 4.1 it follows that μ_{i} ≥ 2 in (3.7) for one (or more) of the eigenvalues λ_{i}, so U_{i1} is a proper subspace of U_{i2}. From Lemma 1.5, there exists a nonzero x ∈ U_{i2}, x⊥U_{i1}, i.e. x satisfies

(4.8)  Ax = λ_{i}x + y for some y ≠ 0 in U_{i1}

(4.9)  (x, w) = 0 for all w ∈ U_{i1}.

Now from (4.7) and the first statement of Theorem 4.2,

(4.10)  A^{†} : U_{i1} → U_{i1}.

Thus, from (4.8), using (4.9)

(Ax, y) = λ_{i}(x, y) + ∥y∥^{2}
= ∥y∥^{2}

> 0, (4.11)

but

(Ax, y) = (x, A^{†}y)

= 0

using (4.10) and (4.9), since y ∈ U_{i1}. This contradiction proves the first part of the theorem.

Suppose x ∈ ker(A^{†}). Then

0 = (A^{†}x, A^{†}x)
= (AA^{†}xx) = (A^{†}Axx) = (Ax, Ax)

so x ∈ ker(A), and vice versa. From (2.7) we now have R(A)⊥ ker(A).

Corollary. An n × n matrix A is similar to a diagonal matrix if and only if it is similar to a normal matrix.

Proof. Since diagonal matrices are normal, this part is trivial. Thus assume T^{−1}AT = B, where B is normal. By Theorem 4.3, there exists nonsingular Q such that Q^{−1}BQ = D, D diagonal. Thus

D = Q^{−1}(T^{−1}AT)Q
+ (Q^{−1}T^{−1})A(T Q)
= (T Q)^{−1}A(T Q),

so A is similar to a diagonal matrix.

Two important classes of normal matrices are \underline{unitary} matrices, which are nonsingular and satisfy

(4.12)  A^{−1} = A^{†},

and \underline{Hermitian} matrices, satisfying

(4.13)  A = A^{†}.

For F = ℝ, a nonsingular matrix such that A^{t} = A^{−1} is called \underline{orthogonal}, and a matrix satisfying A = A^{t} is called \underline{symmetric}. The term \underline{self-adjoint} is also sometimes used for Hermitian and/or symmetric matrices, but this term means something stronger in the context of unbounded operators an infinite dimensional Hilbert spaces. The terms symmetric and Hermitian are sometimes used interchangeably when no ambiguity is possible.

From (4.12) it follows immediately that a product of unitary matrices is unitary.

Theorem 4.4. Let r_{i}, i = 1, …, n, and c_{i}, i = 1, …, n denote the rows and columns, respectively, of an n × n matrix A. The matrix A is unitary if and only if

(4.14)  (r_{i}, r_{j}) = (c_{i}, c_{j}) = δ_{ij}, 1 ≤ i, j ≤ n.

Proof. A is unitary if and only if AA^{†} = A^{†}A = I. By inspection

(4.15)  (AA^{†})_{ij} = (r_{i}, r_{j}) and (A^{†}A)_{ij} = \overline{(c_{i}, c_{j})}, 1 ≤ i, j ≤ n.

Theorem 4.5. Inner products of pairs of vectors and lengths of vectors are invariant under unitary transformations. Furthermore, the eigenvalues of a unitary matrix are of magnitude unity.

Proof. Let A be unitary, and x, y any vectors in H. Then

(Ax, Ay) = (A^{†}(Ax), y)
= (A^{†}Ax, y)

= (x, y) (4.16)

using (4.12). Setting x = y, we have ∥Ax∥ = ∥x∥ from (4.16). Conversely, if (4.16) holds for all x, y ∈ H then A satisfies (4.12). And for x any eigenvector of A, corresponding to eigenvalue λ,then ∥x∥ = ∥Ax∥ = ∥λx∥ = |λ|∥x∥ so |λ| = 1.

Hermitian or symmetric matrixes satisfy several special conditions.

Theorem 4.6. A linear operator A on a complex Hilbert space H is Hermitian if and only if

(4.17)  (Az, z) ∈ ℝ for all z ∈ H.

Proof. If H is Hermitian, then for any z ∈ H

(z, Az) = (A^{†}z, z)
= (Az, z)

= \overline{(z, Az)} (4.18)

so (z, Az) ∈ ℝ. Next assume (4.17) holds. For arbitrary p, q ∈ H, set z = p + tq in (4.17), t ∈ ℂ to be chosen below. Then

(4.19) (A(p + tq), p + tq) = (Ap, p) + |t|^{2}(Aq, q) + t(Aq, p) + t̄(Ap, q),

so by hypothesis, the expression

(4.20)  t(Aq, p) + t̄(Ap, q) = t(Aq, p) + t̄\overline{(q, Ap)}

has to be real for any choice of t. By direct computation

Im(t(Aq, p) + t̄\overline{(q, Ap)})
Im(t)Re((Aq, p) − (q, Ap))

+ Re(t)Im((Aq, p) − (q, Ap)) (4.21)

which is zero for any t only if (Aq, p) = (q, Ap). And if (Aq, p) = (q, Ap) for any p, q ∈ H, then A is Hermitian.

Remark. This theorem is true for infinite-dimensional Hilbert spaces as well, with the same proof.

Corollary. The eigenvalues of a Hermitian matrix are real.

Theorem 4.7. The eigenvectors of a Hermitian matrix may be made an orthonormal basis for H.

Proof. Normalize all the eigenvectors x_{i} so that ∥x_{i}∥ = 1. Next let x_{i}, x_{j} be eigenvectors corresponding to different eigenvalues λ_{i}, λ_{j}. Then

(Ax_{i}, x_{j}) = λ_{i}(x_{i}, x_{j})
= (x_{i}, Ax_{j})

= λ_{j}(x_{i}, x_{j}) (4.22)

since λ_{j} is real, so (λ_{i} − λ_{j})(x_{i}, x_{j}) = 0 and (x_{i}, x_{j}) = 0. If there are eigenvalues λ_{i} of multiplicity m_{i} > 1, choose any orthonormal basis for U_{i1}, for example by application of the Gram-Schmidt procedure. All of the eigenvectors are now orthonormal, and as A is similar to a diagonal matrix, they form a basis for H.

Corollary. The matrix T such that T^{−1}AT is diagonal can be taken to be unitary.

Proof. Given the eigenvectors for A, the matrix T is just the set of eigenvectors of A, assembled columnwise. Choose the eigenvectors of A to be orthonormal, and T is unitary by application of Theorem 4.4.

Denote by

(4.23)  ∥A∥ = sup_{x∈H}_{x≠0} \frac{∥Ax∥}{∥x∥}

where ∥ · ∥ is the ℓ_{2} norm given in (1.5), and by

(4.24)  σ(A) = maximum |λ_{i}| {λ_{i}} the eigenvalues of A.

σ(A) is called the \underline{spectral radius} of A; as x in (4.23) can be chosen any eigenvector of A, it follows that

(4.25)  σ(A) ≤ ∥A∥.

Theorem 4.8. If A is Hermitian, then

(4.26)  ∥A∥ = σ(A).

Proof. Let {x_{i}}_{i=1}^{n} be the orthonormal eigenvectors of A, corresponding to eigenvalues λ_{i}. Then any x ∈ H is of the form

(4.27)  x = ∑_{i=1}^{n} α_{i}x_{i}, α_{i} ∈ F;

(4.28)  Ax = ∑_{i=1}^{n} λ_{i}α_{i}x_{i}

∥x∥^{2} = (x, x)

= ∑_{i,j} α_{i}ᾱ_{j}(x_{i}, x_{j})

= ∑_{i} |α_{i}|^{2} (4.29)

since the {x_{i}} are orthonormal, and

∥Ax∥^{2} = (Ax, Ax)

= ∑_{i,j} λ_{i}λ̄_{j}α_{i}ᾱ_{j}(x_{i}, x_{j})

= ∑_{i} |λ_{i}|^{2}|α_{i}|^{2} (4.30)

From (4.29) and (4.30) it now follows that ∥A∥ ≤ σ(A), which together with (4.25) gives (4.26).

A matrix satisfying A^{†} = −A is called \underline{anti-Hermitian} or \underline{skew-Hermitian}; a real matrix such that A^{t} = −A is also called \underline{antisymmetric} or \underline{skew-symmetric}.

Theorem 4.9. A linear operator A on a complex Hilbert space H is anti-Hermitian if and only if (z, Az) is pure imaginary for all z ∈ H. The eigenvalues of an anti-Hermitian matrix are pure imaginary.

Theorem 4.10. The eigenvectors of an anti-Hermitian matrix may be made an orthonormal basis for H.

Theorem 4.11. If A is anti-Hermitian, σ(A) = ∥A∥.

If A is anti-Hermitian, iA is Hermitian, and these results follow from the corresponding results for Hermitian matrices.

Any square matrix can be (trivially) written as the sum of a Hermitian and an anti-Hermitian matrix,

(4.31)  A = \frac{1}{2}(A + A^{†}) + \frac{1}{2}(A − A^{†})

A Hermitian matrix is \underline{positive definite} (respectively \underline{positive semidefinite, negative} \underline{definite, negative semidefinite}) if

(4.32)  (Ax, x) > 0 for all x ∈ H, x ≠ 0.

(respectively ≥ 0, < 0, ≤ 0 in (4.32)). Sometimes these terms are also applied to nonsymmetric matrices A on a real Hilbert space, in which case they refer to the symmetric part of A, as obtained from (4.31). A real matrix A is thus positive definite, for example, if and only if \frac{1}{2}(A + A^{†}) is.

Lemma 4.12. A Hermitian matrix A is positive definite if and only if all of its eigenvalues are positive.

Proof. Obviously positive eigenvalues are necessary to satisfy (4.32). Let {x_{i}}_{i=1}^{n} be orthonormal eigenvectors of A, corresponding to eigenvalues λ_{i}. Then for any x ∈ H, from Theorem 4.7,

x = ∑_{i=1}^{n} α_{i}x_{i}

Ax = ∑_{i=1}^{n} λ_{i}α_{i}x_{i}
(Ax, x) = ∑_{i,j} λ_{i}α_{i}ᾱ_{j}(x_{i}, x_{j})

= ∑_{i} λ_{i}|α_{i}|^{2}; (4.33)

obviously if all of the λ_{i} > 0, then (4.32) will be satisfied.

The following theorem provides a relatively easy method of checking whether a given Hermitian matrix is indeed positive definite. Given an n × n Hermitian matrix A : H^{n} → H^{n}, for m = 1, 2, …, n, denote by

(4.34)  A^{(m)} = {A_{ij}}_{i,j=1}^{m}

i.e. the m×m matrix obtained by truncating the matrix A to its first m rows and columns. For example, A^{(1)} = (A_{11}) and A^{(n)} = A. (These truncated matrices A^{(m)} have various names; the literature is not consistent on this point.)

We think of A^{(m)} as mapping H^{m} to H^{m}, where H^{m}, of dimension m, obviously, is isomorphic to a subspace of H^{n}, the elements of which have zeros in their last n − m components.

Theorem 4.13 (Sylvester’s Theorem). : A Hermitian matrix A is positive definite if and only if

(4.35)  det A^{(m)} > 0, m = 1, …, n

Proof. First suppose that (4.35) fails, i.e. that det A^{(k)} ≤ 0 for some k. From (3.5), det A^{(k)} is the product of the eigenvalues of A^{(k)}, so one or more of the eigenvalues of A^{(k)} is nonpositive. As A^{(k)} is also Hermitian, from lemma 4.12 it follows that A^{(k)} is not positive definite, i.e. there exists nonzero x^{(k)} ∈ H^{k} such that (A^{(k)}x^{(k)}, x^{(k)}) ≤ 0. Obtain x^{(n)} ∈ H^{n} from x^{(k)} by placing zeros in the last n − k components; then (Ax^{(n)}, x^{(n)}) = (A^{(k)}x^{(k)}, x^{(k)}) ≤ 0, so (4.32) fails and A is not positive definite.

Therefore we assume now that (4.35) holds. In this case, we proceed by induction. Obviously the theorem holds for matrices of dimension 1; therefore we assume it is true for matrices of dimension up to and including n − 1, and prove it for matrices of dimension n. In particular, we may assume that A^{(n−1)} is positive definite.

Now as detA > 0, by another application of (3.5) and lemma 4.12, we may infer that if A is not positive definite, there are (at least) two negative real eigenvalues, denoted by λ_{1}, λ_{2}. Let x_{1}, x_{2} be the corresponding eigenvectors, taken orthonormal, using Theorem 4.7.

Given x_{1}, x_{2}, there is a nontrivial linear combination

(4.36)  x = α_{1}x_{1} + α_{2}x_{2}, α_{1}, α_{2} ∈ F, |α_{1}|^{2} + |α_{2}|^{2} > 0

such that the last (n-th) component of x is zero.

Now

(Ax, x) = (α_{1}λ_{1}x_{1} + α_{2}λ_{2}x_{2}, α_{1}x_{1} + α_{2}x_{2})
= λ_{1}|α_{1}|^{2} + λ_{2}|α_{2}|^{2}

< 0, (4.37)

since both λ_{1}, λ_{2} < 0. But let x^{(n−1)} ∈ H^{n−1} be the vector x with that last component, equal to zero, dropped. Then

(Ax, x) = (A^{(n−1)}x^{(n−1)}, x^{(n−1)})

> 0 (4.38)

since A^{(n−1)} is known to be positive definite. This contradiction with (4.37) completes the proof.

As an application of several of the above results, we have the following.

Theorem 4.14. A normal matrix whose eigenvalues are all real (respectively, all pure imaginary or zero) is Hermitian (respectively anti-Hermitian). For N normal, there is a unitary T such that T^{−1}NT is diagonal.

The proof of this uses the following

Lemma 4.15. If T is unitary and A is Hermitian (or anti-Hermitian), then so is T^{−1}AT.

Proof of lemma:

(T^{−1}AT)^{†} = (T^{†}AT)^{†}
= T^{†}A^{†}T
= T^{−1}A^{†}T

= ±T^{−1}AT. (4.39)

for A Hermitian or anti-Hermitian, respectively.

Proof of theorem: Given a normal matrix N, set S = \frac{1}{2}(N + N^{†}), A = \frac{1}{2}(N −N^{†}), which are Hermitian and anti-Hermitian, respectively, and satisfy

(4.40)  N = S + A.

Since N is normal, NN^{†} = N^{†}N, and therefore

(4.41)  SA = AS.

Let T_{0} be a unitary matrix such that

D_{0} = T_{0}^{−1}ST_{0}

is diagonal. Using lemma 4.14,

(4.42)  A_{0} = T_{0}^{−1}AT_{0}

is anti-Hermitian, and from (4.41)

(4.43)  D_{0}A_{0} = A_{0}D_{0}.

From (4.43), the elements of D_{0}, A_{0} satisfy

(4.44)  (D_{0jj} − D_{0kk})A_{0jk} = 0, 1 ≤ j, k ≤ n

so A_{0} is also block diagonal, the blocks {b_{i}} determined by the requirement that the diagonal elements of D_{0} are equal within each block.

If the eigenvalues of D_{0} happen to be distinct, then A_{0} is already diagonal. Otherwise, we find unitary matrices {T_{i}}, one for each diagonal block b_{i} of A_{0}, such that T_{i} is the identity matrix outside of the block b_{i},and

(4.45)  A_{i} = T_{i}^{−1}A_{i−1}T_{i}

is diagonal within the block b_{i}. (The matrix T_{i} is unitary because A_{i−1} is anti-Hermitian within the block b_{i}). As D_{0}, restricted to b_{i}, is a multiple of the identity matrix, it is unchanged,

(4.46)  D_{0} = T_{i}^{−1}D_{0}T_{i}.

By induction, each A_{i−1} is anti-Hermitian and thus so is A_{i}, from (4.45) and the unitary property of T_{i}.

We conclude this process with two diagonal matrices D_{0} and A_{m}, and T obtained from

(4.47)  T = T_{0}T_{1} … T_{m},

such that

(4.48)  T^{−1}ST = D_{0}

(4.49)  T^{−1}AT = A_{m}

As the product of unitary matrices, T is unitary, so D_{0} is Hermitian and A_{m} anti-Hermitian. Therefore {D_{0jj}} are real and {A_{mjj}} are pure imaginary, and since the eigenvalues of D_{0} + A_{m} are the same as those of S + A, using (4.48), (4.49), from (4.40) we see that the D_{0jj}, A_{mjj} are just the real and imaginary parts of the eigenvalues of N.

Suppose that the eigenvalues of N are real (the pure imaginary case is proved exactly the same way). Then A_{mjj} = 0 for all j = 1, …, n, and since A_{m} is diagonal, A_{m} = 0 and A = 0 from (4.49). Then N is Hermitian from (4.40).

Since T is unitary in (4.47), we have also proved the following:

Corollary. The eigenvectors of a normal matrix N can be chosen to be orthonormal, satisfying (x_{i}, x_{j}) = δ_{ij}, 1 ≤ i, j ≤ n; with the eigenvectors so chosen, N admits a representation of the form

(4.50)  N = ∑_{i=1}^{n} λ_{i}x_{i}x_{i}^{†}

where λ_{i} is the corresponding eigenvalue.

In (4.50), the eigenvalues λ_{i} are in general complex, and each product x_{i}x_{i}^{†} is a matrix of rank 1.

These results may be summarized as follows, giving a comparison between normal, Hermitian, and unitary matrices.

Theorem 4.16. A normal matrix A admits a representation of the form

(4.51)  A = T^{−1}DT,

with D diagonal and T unitary. A is Hermitian if and only if D is real. A is anti-Hermitian if and only if D is pure imaginary. A is unitary if and only if the diagonal elements of D are of unit magnitude, i.e. |D_{ii}| = 1 for i = 1, …, n. A is idempotent and Hermitian, i.e. a projection operator onto its range if and only if each of the diagonal elements of D is either 1 or 0. Finally, it follows from (4.51) that σ(A) = ∥A∥ (as determined by (4.23), (4.24)) for any normal matrix A, and not just for Hermitian matrices.

Problems 1. Consider the mixed eigenvalue problem, Bx = λAx, where B, A are

n × n Hermitian matrices, A is nonsingular, λ ∈ F is the eigenvalue and x the eigenvector. Show by example that in general the λ are not real. Prove that if A is positive definite, then λ is real.

2. What can be said about the eigenvalues, eigenvectors, and minimal polynomial of a matrix Q satisfying Q^{2} = Q^{†}?

3. Find an example of a real matrix Q, \underline{not symmetric} which satisfies Q = Q^{2}.

4. Show that a real, antisymmetric matrix of odd dimension is always singular.

5. True or false:

a) If A is Hermitian and positive definite, there exists a Hermitian, positive definite “square root”, i.e. S such that S^{2} = A.

b) There is no square matrix A, of any (finite) dimension, which satisfies A = I + AA^{†}, I the identity matrix.

c) There is no Hermitian matrix, of any dimension, whose minimal polynomial M_{A}(A) = (A^{2} − 1)(A^{3} − A^{2} − 4A + 4).

d) If A is any n × n matrix and m any positive integer, then ∥A^{m}∥ ≤ ∥A∥^{m}, with ∥A∥ defined in (4.23).

e) σ(A + B) ≤ σ(A) + σ(B) for any n × n A, B, with σ defined (4.24).

f) σ(A^{2}) = (σ(A))^{2}

6. For what real values of t is the matrix

(\begin{matrix}1 + t & t & t \\ t & t^{2} & sin t \\ t & −sin t & 1 − t \end{matrix})

positive definite?

7. Let {x_{i}}_{i=1}^{n} be the given eigenvectors of an unknown n × n matrix X, which is known to be similar to a diagonal matrix. A vector b and Xb are also given. Under what conditions on b, Xb, {x_{i}} is the matrix X uniquely determined?

8. Let A, B be two nonsingular, n × n matrices which anticommute, i.e. AB = −BA. Show that n is even.

9^{∗}. Let A be a nonsingular n × n matrix, with only one nonvanishing element in each row (or each column). Show that A is similar to a diagonal matrix. Show that the set of such matrices forms a multiplicative group.

10^{∗}. For A any n×n matrix, let ∥A∥ be as given in (4.23) and σ(A) as given in (4.24). Prove that lim_{m→∞} ∥A^{m}∥^{1/m} = σ(A).

Hint: Consider the action of A^{m} = (A − λ_{i} + λ_{i})^{m} on the subspace U_{ij} given in (3.8) and use Theorem 3.4.

11. Let Q, S be given n × n Hermitian matrices, with Q positive definite. Prove existence of an n × n matrix A satisfying A^{†}Q + QA = S.

12. Let A, B be n × n matrices with B nonsingular, and satisfying AB = BA^{†}. What can be said about the eigenvalues and eigenvectors of A?

13^{∗}. Let A be a given n × n matrix, regarded as a linear map of H → H, H of dimension n. Denote by Z = lim_{ε→0}(A^{†}A + ε)^{−1}A^{†}.

(a) Show that Z determines a (bounded) linear map of H → H, i.e. that for any x ∈ H, Zx is uniquely determined within H. Find Z

explicitly if A = (\begin{matrix}0 & 1 \\ 0 & 0 \end{matrix}).

(b) Show that if A is nonsingular, then Z = A^{−1}.

(c) Show that for any x ∈ H, x−ZAx = P x, where P is the symmetric projection onto ker (A^{†}A).

(d) Show that Z is a pseudoinverse of A in the following sense: ZAx = x, for all x ∈ R(A^{†}A); y − AZy⊥R(A) for all y ∈ R(A).

(e) Show that if A is normal (presumably singular) then AZx = ZAx = x for all x ∈ R(A).

14. Suppose AB = BA and BC = CB, for A, B, C given n × n matrices. Under what condition on B may it be concluded that AC = CA?

15. Suppose AB = BA, A, B n × n matrices, where B is of a special structure as given below. In each case, what may be concluded about the structure of A?

a) B = E_{jma}, an elementary matrix as described in chapter 2

b) B is nonsingular, with only one nonvanishing element in each row or each column.

c) B is nonsingular and block diagonal

d) B_{ij} = 1 for j = i + 1 and 0 otherwise

16^{∗} Given a square matrix A satisfying A^{2} = AA^{†}, show that A is similar to a diagonal matrix.

17^{∗} For any square matrix A, show that ∥A∥^{2} = σ(A^{†}A), with ∥A∥ the induced norm of A given in (4.23).

Hint: Consider ∥A(x + ty)∥ for t ∈ F, ∥x∥ = 1 such that ∥Ax∥ = ∥A∥, and (x, y) = 0.

18. Find the minimal polynomial M_{A}(λ) of a matrix A which is both Hermitian and unitary, but is not a multiple of the identity matrix.

19. Let A be an n × n orthogonal (real, unitary) matrix with n odd. Prove that detA is an eigenvalue of A with odd multiplicity.

20. For I the n × n identity matrix and w ∈ ℝ^{n}, show that the matrix I − 2ww^{t} is orthogonal if ∥w∥ = 1 (Euclidean norm).

21^{∗} Given an n × n real matrix A, show the existence of an orthogonal matrix Q and an upper triangular matrix R (R_{ij} = 0 for i > j) such that A = QR. This does not determine Q, R uniquely - explain.

22. Let A be an n × n matrix with distinct eigenvalues, and B be an n × n matrix commuting with A. Show that B can be written as a polynomial function of A, of degree not exceeding n − 1.

23^{∗} Let A be an n×n matrix whose minimal polynomial is of degree n, and let B be such that AB = BA. Show that B is a polynomial function of A, of degree not exceeding n − 1.

24. Let A, B, C be n × n matrices, satisfying AB = BA and C = AB − B^{3} + 6I. Furthermore C is nonsingular. Show that AC^{−1} = C^{−1}A.

25∗. Let A be a Hermitian n×n matrix, and denote by λ_{i}^{(m)}, i = 1, …, m the eigenvalues of the truncated matrix A^{(m)} as in the proof of Sylverter’s theorem (theorem 4.13). Suppose that {λ_{i}^{(k)}, i = 1, …, k} ⊂ {λ_{i}^{(m)}, i = 1, …, m} for all 1 ≤ k < m ≤ n. Prove that A is diagonal.

26. Given an n × n matrix A which (simultaneously) satisfies Q_{1}(A) = Q_{2}(A) = Q_{3}(A) = 0, where Q_{1}(λ) = λ^{2}(λ − 1)(λ + 2)(λ − 3)^{2}, Q_{2}(λ) = (λ − 1)^{2}(λ + 3)(λ − 3), Q_{3}(λ) = λ(λ − 1)(λ + 2)^{3}(λ − 3)(λ + 3)^{2}

a) Prove that A^{−1} exists and is similar to a diagonal matrix.

b) If, in addition, it is known that A is Hermitian, show that ∥A^{−1}∥ ≤ 1. Give an example with ∥A^{−1}∥ < 1.

c) Give an example of non-Hermitian A such that ∥A^{−1}∥ > 1.

5 Matrices depending on a parameter

Very often one encounters matrices whose elements are not simply constants, but depend on one or more variables. Here we discuss the variation of various properties of such a matrix as these variables change in value.

For the present discussion, it will suffice to consider matrices depending on a single real variable t, for t ∈ L, L some open interval in ℝ. Furthermore, only square matrices will be discussed, so throughout this chapter all matrices are understood as of dimension n × n, for some n fixed. Thus a matrix A(t) = {A_{ij}(t)_{i,j=1}^{n}}, t ∈ L, may be considered as determining a mapping (also denoted by A) of L into the space of n × n matrices over F, denoted here by M. The space M is equipped with any one of the matrix norms as appearing in chapter 1 or 2 above.

Then for A such a matrix-valued function on L, we have the obvious definitions of continuity and differentiability.

Definition 5.1. A : L → M is continuous at t_{0} ∈ L if for all ε > 0, there exists a δ > 0 such that

(5.1)  ∥A(t_{0} + t) − A(t_{0})∥ < ε for all |t| < δ

Definition 5.2. A : L → M is differentiable at t_{0} ∈ L, with derivative A′(t_{0}) = \frac{dA}{dt}(t_{0}) ∈ M, if there exists such a matrix A′(t_{0}) ∈ M satisfying

(5.2)  ∥A(t_{0} + t) − A(t_{0}) − tA′(t_{0})∥ ≤ o(|t|)

(The notation o(|t|) is generic, for any function of t such that o(|t|)/t → 0 as t → 0.)

From these definitions, we have an obvious statement

Lemma 5.3. A : L → M is continuous (resp. differentiable) at t_{0} if and only if each of its components A_{ij} is a continuous (resp. differentiable) function of t at t_{0}.

Further, if A is differentiable at t_{0}, then

(5.3)  A′(t_{0})_{ij} = \frac{d}{dt}A_{ij}(t_{0}) i, j = 1, …, n,

and if A, B are both matrices differentiable at t_{0}, then so is the product AB, and

(5.4)  (AB)′(t_{0}) = A′(t_{0})B(t_{0}) + A(t_{0})B′(t_{0})

Proof. The norm of a matrix is small if and only if the magnitude of each element is, so (5.3) follows immediately from (5.2). Then (5.4) follows from (5.3) and the ordinary expression for the product of two matrices.

As det A, now a mapping of L → F, is just a sum of products of elements of A, we have immediately

Lemma 5.4. If A is continuous (resp. differentiable) at t_{0}, then so is det A.

There is a representation for \frac{d}{dt}(det A), which is occasionally used. For k = 1, …, n denote by A_{(k)}, A^{(k)} the matrices with elements given by

\begin{matrix}A_{(k)ij} = A_{ij}, i ≠ k \\ A_{(k)kj} = \frac{d}{dt}A_{kj} \end{matrix} ; (5.5)

\begin{matrix}A_{ij}^{(k)} = A_{ij}, j ≠ k \\ A_{ik}^{(k)} = \frac{d}{dt}A_{ik}, \end{matrix} ; (5.6)

i.e. one takes the t-derivative of the k-th row (resp. k-th column) of A to get A_{(k)} (resp. A^{(k)}).

Lemma 5.5. If A is differentiable at t_{0}, then

(5.7)  \frac{d(det A)}{dt}(t_{0}) = ∑_{k=1}^{n} det A_{(k)}(t_{0}) = ∑_{k=1}^{n} det A^{(k)}(t_{0})

Proof. Differentiate (2.30) and (2.31) with respect to t, using the “product rule”, and collect the differentiated elements rowwise or columnwise. Then using (5.5) or (5.6), one gets (5.7).

If A is continuous at t_{0}, and if A(t_{0}) is nonsingular, then A^{−1}(t_{0}) exists, and indeed from Lemma (5.4) it follows that A^{−1}(t) exists for t in some neighborhood of t_{0}. And from an application of Cramer’s rule, for nonsingular A(t), A^{−1}(t) is just a sum of products of elements of A(t) divided by detA(t). Thus we obtain

Lemma 5.6. If A is continuous (resp. differentiable) at t_{0}, and if A(t_{0}) is nonsingular, then A^{−1} is also continuous (resp. differentiable) at t_{0}. If A^{−1} is differentiable at t_{0}, then

(5.8)  (A^{−1})′(t_{0}) ≡ \frac{d}{dt}A^{−1}(t_{0}) = −A^{−1}(t_{0})A′(t_{0})A^{−1}(t_{0})

Proof. We need only obtain (5.8). Differentiating A(t)A^{−1}(t) = I with respect to t and using (5.4), this is immediate.

Unlike (5.7), the result (5.8) is often used.

The behavior of the eigenvalues and eigenvectors of a matrix A(t) is more interesting. We first illustrate the situation by several examples.

Example 1: A(t) = (\begin{matrix}1 & t \\ 0 & 1 \end{matrix}) · A(t) is similar to a diagonal matrix only

for t = 0.

Example 2: A(t) = (\begin{matrix}1 & t^{2} \\ t & 1 \end{matrix}) · A(t) is similar to a diagonal matrix for all

values of t, but the eigenvalues are 1 ± t^{3/2}, real for t ≥ 0 and complex for t < 0. Although A is a smooth function of t, the eigenvalues are only once continuously differentiable in a neighborhood of t = 0. The corresponding

eigenvectors are \binom{± t^{1/2}}{1}, and are not differentiable at t = 0.

Example 3: A(t) = (\begin{matrix}1 + t & 1 \\ 0 & 1 \end{matrix}). Now A is similar to a diagonal matrix

for all t ≠ 0. The eigenvalues are 1 and 1 + t, with corresponding eigenvectors \binom{1}{−t} and \binom{1}{0}. As t → 0, the two eigenvectors coincide, i.e. linear independence is lost.

Example 4: A(t) = (\begin{matrix}1 & 1 \\ t & 1 \end{matrix}). Again A is similar to a diagonal matrix

for all t ≠ 0. The eigenvalues are 1 ± t^{1/2}, with corresponding eigenvectors

\binom{1}{± t^{1/2}}. A is a smooth function of t, but the eigenvalues and eigenvectors

change from real to complex, and are not differentiable, at t = 0.

These examples share a common feature - the eigenvalues become equal at t = 0. When the eigenvalues of a matrix are distinct, both the eigenvalues and eigenvectors are as smooth, with respect to t, as the matrix itself.

Theorem 5.7. If A is continuous at t_{0}, then so are its eigenvalues.

Proof. P_{A(t)}(λ) is a polynomial of order n, whose leading coefficient is (−1)^{n}, i.e. not zero uniformly in t, and the other coefficients of which are continuous functions at t at t = t_{0}. Therefore so are the roots of P_{A(t)}(λ), which are the eigenvalues of A(t).

Corollary. If the eigenvalues of A(t_{0}) are distinct, and if A is continuous at t_{0}, then the eigenvalues of A are distinct for t is an open neighborhood of t_{0}.

Thus we consider such a matrix A with distinct eigenvalues, λ_{i} ≠ λ_{j} for i ≠ j. Thus m_{i} = μ_{i} = 1 for all values of i and k = n in (3.4), (3.7). Denote by {x_{i}}_{i=1}^{n} the corresponding eigenvectors,

(5.9)  Ax_{i} = λ_{i}x_{i}, i = 1, …, n

which are linearly independent, using theorem 3.3. We normalize the eigenvectors so that ∥x_{i}∥ = 1.

For each t, t = 1, …, n, x_{i}(t) ∈ H, and A(t) = H → H, H of dimension n. We regard H as a Hilbert space, equipped with the inner product as given in (1.13) or (1.14).

Now as λ̄_{i} ≠ λ̄_{j} for i ≠ j, the eigenvalues of A^{†} are also distinct, and thus there is a full set of eigenvectors for A^{†}, also linearly independent,

(5.10)  A^{†}y_{i} = λ̄_{i}y_{i}, i = 1, …, n.

Taking the adjoint of (5.10), we get

(5.11)  y_{i}^{†}A = λ_{i}y_{i}^{†},

i.e. the y_{i}^{†}, viewed as row vectors, are thus the left eigenvectors for A.

Lemma 5.8. For i ≠ j, (x_{i}, y_{j}) = 0. Furthermore (x_{i}, y_{i}) ≠ 0 for i = 1, …, n.

Proof. For all i ≠ j,

(Ax_{i}, y_{j}) = λ_{i}(x_{i}, y_{j})
= (x_{i}, A^{†}y_{j})
= λ_{j}(x_{i}, y_{j})

so as λ_{i} ≠ λ_{j}, (x_{i}, y_{j}) = 0. As the {x_{i}} span H, since y_{i} ∈ H is not zero, it cannot be perpendicular to x_{i}. Typically the y_{i} are normalized by the condition

(5.12)  (x_{i}, y_{j}) = δ_{ij}, i, j = 1, …, n

Theorem 5.9. Assume that the eigenvalues of A(t_{0}) are distinct. If A is continuous at t_{0}, so are the eigenvalues and eigenvectors. If A is differentiable at t_{0}, then so are the eigenvalues and eigenvectors, and their t-derivatives are given explicitly by

(5.13)  λ′_{i}(t_{0}) = (A′(t_{0})x_{i}, y_{i})

(5.14)  x′_{i}(t_{0}) = ∑_{j≠i} \frac{(A′(t_{0})x_{i}, y_{j})}{λ_{i} − λ_{j}}x_{j} + α_{i}x_{i}, i = 1, …, n,

assuming the x_{i}, y_{j} satisfy (5.12). The constants α_{i} are arbitrary, determined by the normalization condition on the {x_{i}}.

In (5.13), (5.14) we have abbreviated x_{i}(t_{0}) by x_{i}, etc., and as above, primes indicate t-derivatives.

Proof. For A continuous at t_{0}, we have already seen that the eigenvalues {λ_{i}} are continuous at t_{0}, so it suffices to check the eigenvectors {x_{i}}. For ε > 0 small, satisfying in particular

(5.15)  ε < \frac{1}{3}minimum_{i≠j}|λ_{i}(t_{0}) − λ_{j}(t_{0})|

we use the continuity of A and the {λ_{i}}, of which there are only a finite number, to obtain δ > 0 such that

(5.16)  ∥A(t) − A(t_{0})∥ < ε,
(5.17)  |λ_{i}(t) − λ_{i}(t_{0})| < ε, i = 1, …, n, for all t ∈ (t_{0} − δ, t_{0} + δ).

From (5.17) and (5.15) an application of the triangle inequality gives

|λ_{i}(t) − λ_{j}(t)| > \frac{1}{3}|λ_{i}(t_{0}) − λ_{j}(t_{0})|

(5.18) ≥ constant > 0

for all i ≠ j, t ∈ (t_{0} −δ, t_{0} +δ), i.e. the eigenvalues remain distinct, uniformly with respect to t in this interval.

Now fix t arbitrarily within the interval (t_{0}−δ, t_{0}+δ); since the {x_{i}(t_{0})}_{i=1}^{n} form a basis for H, we can make an expansion

(5.19)  x_{i}(t) = ∑_{j=1}^{n} α_{j}x_{j}(t_{0}).

Our claim that x_{i} is continuous at t_{0} will now follow if we can show that the |α_{j}|/|α_{i}| are small for j ≠ i. For this we use (5.9), (5.16) and (5.17)

A(t)x_{i}(t) = λ_{i}(t)x_{i}(t)
= (λ_{i}(t_{0}) + O(ε))x_{i}(t)

= (λ_{i}(t_{0}) + O(ε)(α_{i}x_{i}(t_{0}) + ∑_{j≠i} α_{j}x_{j}(t_{0})) (5.20)

A(t)x_{i}(t) = A(t)(α_{i}x_{i}(t_{0}) + ∑_{j≠i} α_{j}x_{j}(t_{0}))

= (A(t_{0}) + O(ε))(α_{i}x_{i}(t_{0}) + ∑_{j≠i} α_{j}x_{j}(t_{0}))

= λ_{i}(t_{0})α_{i}x_{i}(t_{0}) + ∑_{j≠i} α_{j}λ_{j}(t_{0})x_{j}(t_{0})

+ O(ε))(α_{i}x_{i}(t_{0}) + ∑_{j≠i} α_{j}x_{j}(t_{0})) (5.21)

Here and below, we employ the common generic notation O(ε) for quantities satisfying |O(ε)| ≤ constant ε for ε > 0 small.

Comparing (5.20) and (5.21) we find

∑_{j≠i} α_{j}(λ_{i}(t_{0}) − λ_{j}(t_{0}) + O(ε))x_{j}(t_{0}) = O(ε)α_{i}x_{i}(t_{0})

= O(ε), (5.22)

and for ε > 0 small, j ≠ i, λ_{i}(t_{0}) − λ_{j}(t_{0}) + O(ε) is bounded away from zero, and so (5.22) implies |α_{j}/α_{i}| = O(ε) for all j ≠ i. This proves the first part of Theorem 5.9.

Next suppose that A is differentiable at t_{0}. Then there exists a matrix A′(t_{0}), such that for |t − t_{0}| small,

(5.23)  ∥A(t) − A(t_{0}) − (t − t_{0})A′(t_{0})∥ ≤ o(|t − t_{0}|).

Our theorem asserts the existence of λ′_{i}(t_{0}) ∈ F, x′_{i}(t_{0}) ∈ H, i = 1, …, n given by (5.13), (5.14), respectively, such that

(5.24)  |λ_{i}(t) − λ_{i}(t_{0}) − (t − t_{0})λ′_{i}(t_{0})| ≤ o(|t − t_{0}|)

(5.25)  ∥x_{i}(t) − x_{i}(t_{0}) − (t − t_{0})x′_{i}(t_{0})∥ ≤ o(|t − t_{0}|)

We make the expansions

(5.26)  λ_{i}(t) = λ_{i}(t_{0}) + (t − t_{0})γ_{i} + θ_{i}(t)

(5.27)  x_{i}(t) = x_{i}(t_{0}) + (t − t_{0}) ∑_{j=1}^{n} α_{j}x_{j}(t_{0}) + η_{i}(t)

in which γ_{i} and the α_{j} are to be chosen below and θ_{i}, η_{i} thus determined. Our objective, of course, is to choose γ_{i} and the α_{i} such that θ_{i}(t) and η_{i}(t) are o(|t − t_{0}|), and thus γ_{i} is identified with λ′_{i}(t_{0}) and ∑ α_{j}x_{j}(t_{0}) with x′_{i}(t_{0}) using (5.24), (5.25).

With λ_{i}(t), x_{i}(t) given by (5.26), (5.27), and A(t) satisfying (5.23), (5.9) still has to be satisfied at each value of t,

A(t)x_{i}(t) = λ_{i}(t)x_{i}(t)

= (λ_{i}(t_{0}) + (t − t_{0})γ_{i} + θ_{i}(t))(x_{i}(t_{0}) + (t − t_{0}) ∑_{j} α_{j}x_{j}(t_{0}) + η_{i}(t))

= λ_{i}(t_{0})x_{i}(t_{0}) + (t − t_{0})[γ_{i}x_{i}(t_{0}) + λ_{i}(t_{0}) ∑_{j} α_{j}x_{j}(t_{0})]

+ θ_{i}(t)x_{i}(t) + λ_{i}(t)η_{i}(t) + O(t − t_{0})^{2} (5.28)

using (5.26), (5.27). Using (5.23) and (5.27),

A(t)x_{i}(t) = (A(t_{0}) + (t − t_{0})A′(t_{0}) + o(|t − t_{0}|))

(x_{i}(t_{0}) + (t − t_{0}) ∑_{j} α_{j}x_{j}(t_{0}) + η_{i}(t))

= A(t_{0})x_{i}(t_{0}) + (t − t_{0})[A′(t_{0})x_{i}(t_{0})

+ A(t_{0}) ∑_{j} α_{j}x_{j}(t_{0})] + A(t)η_{i}(t) + o(|t − t_{0}|) (5.29)

Comparing (5.28) and (5.29), we separate the linear terms in (t−t_{0}) from those of “higher order”, and require each to be satisfied separately. Thus we choose γ_{i} and {α_{j}} so that

(5.30)  A′(t_{0})x_{i}(t_{0}) + ∑_{j=1}^{n} α_{j}(λ_{j}(t_{0}) − λ_{i}(t_{0}))x_{j}(t_{0}) = γ_{i}x_{i}(t_{0})

and then θ_{i} and η_{i} must satisfy

(5.31)  (A(t) − λ_{i}(t))η_{i}(t) − θ_{i}(t)x_{i}(t) = o(|t − t_{0}|).

Taking the inner product of (5.30) with y_{i}(t_{0}) and with y_{k}(t_{0}) for some k ≠ i, we find, using (5.12)

(5.32)  γ_{i} = (A′(t_{0})x_{i}(t_{0}), y_{i}(t_{0}))

(5.33)  α_{k} = (A′(t_{0})x_{i}(t_{0}), y_{k}(t_{0}))/(λ_{i}(t_{0}) − λ_{k}(t_{0})), k ≠ i;

(5.13) now follows from (5.32), (5.24) and (5.26), and (5.14) from (5.33), (5.25) and (5.27), but we still have to show that θ_{i}(t) and η_{i}(t) are o(|t − t_{0}|).

Denote by ζ the right side of (5.31). Taking the inner product of (5.31) with y_{i}(t) and y_{j}(t), j ≠ i, using (5.12) again, we find

(5.34)  θ_{i}(t) = (−ζ, y_{i}(t))

(5.35)  η_{i}(t) = ∑_{j≠i}(ζ, y_{j}(t))x_{j}(t)/(λ_{j}(t) − λ_{i}(t))

Now using (5.18), |ζ| = o(|t − t_{0}|), and ∥x_{j}(t)∥ = 1, we find from (5.34), (5.35)

(5.36)  |θ_{i}(t)| + ∥η_{i}(t)∥ ≤ o(|t − t_{0}|) →_{j} maximum∥y_{j}(t)∥,

so our proof is reduced to showing that the ∥y_{j}(t)∥ are uniformly bounded for |t − t_{0}| < δ. This is not completely trivial, as the y_{j}(t) are normalized by (5.12), with each ∥x_{i}(t)∥ = 1.

We apply the first part of the theorem to A^{†}(t). If A is continuous, at t_{0}, so is A^{†}, and as the eigenvalues λ_{i}(t_{0}) of A(t_{0}) are distinct, so are the λ̄_{i}(t_{0}), the eigenvalues of A^{†}(t_{0}). Therefore by the first part of the theorem, the normalized eigenvectors of A^{†}, y_{j}/∥y_{j}∥, j = 1, …, n are continuous at t_{0}. If x_{i} and y_{i}/∥y_{j}∥ are continuous at t_{0}, then so is their product, (x_{i}, y_{i})/∥y_{i}∥ = 1/∥y_{i}∥. Thus the ∥y_{i}∥ are continuous at t_{0}, and since there are only a finite number of them, taking δ smaller if necessary, they are uniformly bounded with respect to t in (t_{0} − δ, t_{0} + δ).

This completes the proof.

When two or more of the eigenvalues of A(t_{0}) are equal, differentiability of the eigenvalues or eigenvectors cannot be expected in general. Examples given above illustrate this. Furthermore, we find factors of λ_{i}(t_{0}) − λ_{j}(t_{0}) in the denominator of (5.14) and (5.35), and numerous appearance of y_{j}(t_{0}) normalized by (5.12). If x_{i}(t) − x_{j}(t) → 0 as t → t_{0} with ∥x_{i}(t)∥ = ∥x_{j}(t)∥ = 1, then ∥y_{j}(t)∥ → ∞ as t → t_{0}.

Nevertheless, we saw in Theorem 5.7 that if A is continuous at t_{0}, then so are its eigenvalues, and it is obviously natural to ask if there is not some corresponding condition on the eigenvectors. Indeed, in the simple examples given above one will note that when A(t_{0}) is not similar to a diagonal matrix but A(t) is, for t ≠ t_{0}, the two eigenvectors coalesce as t → t_{0}, and the limit is indeed the one eigenvector of A(t_{0}). This is not accidental.

The subspaces U_{ij} discussed in chapter 3 satisfy a continuity condition, even though their dimensions may change abruptly as the matrix A(t) varies.

Theorem 5.10. Let U_{ij}(t_{0}) = ker(A(t_{0})−λ_{i}(t_{0}))^{j}, and suppose that A(t), λ(t), x(t) satisfy

(5.37)  |λ(t) − λ_{i}(t_{0})| ≤ ε
(5.38)  ∥A(t) − A(t_{0})∥ ≤ ε
(5.39)  ∥x(t)∥ = 1
(5.40)  x(t) ∈ ker(A(t) − λ(t))^{j}, i.e.(A(t) − λ(t))^{j}x(t) = 0

Then there exists x_{0} ∈ U_{ij}(t_{0}) and a constant c, independent of ε, such that

(5.41)  ∥x(t) − x_{0}∥ ≤ cε.

As an example, suppose that A is continuous at t_{0}, and that λ_{i}(t_{0}) is of multiplicity m_{i}(t_{0}) > 1. Suppose further that m_{i}(t_{0}) = μ_{i}(t_{0}) in (3.4), (3.7), so there is only one Jordan block for λ_{i}(t_{0}) and U_{i1}(t_{0}) is of dimension 1, spanned by x_{0}, with ∥x_{0}∥ = 1. For |t − t_{0}| ≠ 0 but small, there will be m_{i}(t_{0}) eigenvalues of A(t) close to λ_{i}(t_{0}) by Theorem 5.7. Suppose these eigenvalues are all simple (distinct), i.e. there are m_{i} linearly independent corresponding eigenvectors x(t), normalized by ∥x(t)∥ = 1. Then as t → t_{0}, all of these eigenvectors coalesce, and their limit as t → t_{0} is x_{0} (up to multiplication by a phase factor). This is the phenomenon we observed in example 3 and 4 above.

Proof. Denote by

(5.42)  W = ⊕_{ℓ=1}_{ℓ≠i}^{k(t_{0})}U_{ℓμ_{ℓ}(t_{0})}(t_{0}),

(5.43)  Z = {z ∈ U_{iμ_{i}(t_{0})}^{(t_{0})}|z⊥U_{ij}(t_{0})};

in (5.42), (5.43), k(t_{0}) is the k appearing in (3.4), (3.7), etc. for the matrix A(t_{0}), μ_{ℓ}(t_{0}) the corresponding μ_{ℓ} in (3.7), and

(5.44)  U_{ℓμ_{ℓ}(t_{0})}(t_{0}) = ker(A(t_{0}) − λ_{ℓ}(t_{0}))^{μ_{ℓ}(t_{0})}

as discussed in chapter 3. (For j = μ_{i}(t_{0}) in the statement of Theorem 5.12, the space Z is not included.) Now from Theorem 3.3, noting that U_{iμ_{i}(t_{0})}(t_{0}) = U_{ij}(t_{0}) ⊕ Z and U_{ij}(t_{0}) ∩ Z = {0}, the spaces W, U_{ij}(t_{0}) and Z are linearly independent, that is there are no nontrivial linear combinations of elements from these spaces equal to zero. All of these spaces are closed, finite dimensional subspaces of H. From Theorem 3.4, these spaces together span H,

(5.45)  H = W ⊕ U_{ij}(t_{0}) ⊕ Z,

and therefore we have

(5.46)  x(t) = w + x_{0} + z, w ∈ W, x_{0} ∈ U_{ij}(t_{0}), z ∈ Z.

Using x(t) given by (5.46) in (5.40) with the estimates (5.37), (5.38), we obtain

(A(t) − λ(t))^{j}x(t) = 0
= (A(t_{0}) − λ_{i}(t_{0}) + O(ε))^{j}x(t)
= (A(t_{0}) − λ_{i}(t_{0}))^{j}(w + x_{0} + z) + O(ε)∥x(t)∥

= (A(t_{0}) − λ_{i}(t_{0}))^{j}(w + z) + O(ε) (5.47)

using (5.39) and x_{0} ∈ U_{ij}(t_{0}) is the last step.

We claim that there exists a positive constant τ such that for any q ∈ W ⊕ Z

(5.48)  ∥(A(t_{0}) − λ_{i}(t_{0}))^{j}q∥ ≥ τ∥q∥;

putting q = w + z in (5.47) and using (5.46), will obtain (5.41) immediately. Thus it suffices to prove the claim. Suppose (5.38) fails, then there is an infinite sequence {q_{s}}_{s=1}^{∞}, such that each q_{s} ∈ W ⊕ Z, ∥q_{s}∥ = 1, and

(5.49)  ∥(A(t) − λ_{i}(t_{0}))^{j}q_{s}∥ → 0 as s → ∞.

Since W ⊕ Z is finite dimensional, taking a subsequence of {q_{s}} is necessary, we may assume that {q_{s}} converges to a limit q_{0}, which will satisfy ∥q_{0}∥ = 1, q_{0} ∈ W ⊕ Z, as W ⊕ Z is closed, and by continuity

(5.50)  ∥(A(t_{0}) − λ_{i}(t_{0})^{j}q_{0}∥ = 0

But from (5.50), q_{0} ∈ U_{ij}(t_{0}), which contradicts the linear independence of W, U_{ij}(t_{0}) and Z, thus proving the claim and the theorem.

Problems

1. Suppose the derivative of a matrix A were defined by (5.23). Show that A′(t_{0}) thus determined, if it exists, is unique.

2. Suppose ∥x_{j}(t)∥, ∥x_{i}(t)∥ = 1 and ∥x_{i}(t) − x_{j}(t)∥ → 0 as t → t_{0}. Let the y_{j} be normalized by (5.12). Show that ∥y_{i}(t)∥ → ∞ as t → t_{0}.

3. Find explicit expressions for the constants α_{i} in (5.14), assuming the α_{i} real and that the normalization of the x_{i} is via ∥x_{i}∥ = (x_{i}, x_{i})^{1/2} = 1. Explain why it is no loss of generality to assume the α_{i} real.

4. Suppose that the eigenvalues of A(t_{0}) are distinct and that all the eigenvalues λ_{i}(t) and eigenvectors x_{i}(t) are continuous at t_{0}. Show that A is continuous at t_{0}.

5. True or false: Let x_{i}(t), x_{j}(t) be normalized eigenvectors of A(t), corresponding to eigenvalues λ_{i}(t), λ_{j}(t). For t ≠ t_{0}, λ_{i}, λ_{j} are simple eigenvalues. Furthermore, A is continuous at t_{0}. As t → t_{0}, x_{i}(t)−x_{j}(t) → 0. Then, necessarily, λ_{i}(t) − λ_{j}(t) → 0 as t → t_{0}, and A(t_{0}) is \underline{not} similar to a diagonal matrix. Either prove this result or find a counterexample.

6. Given a unitary matrix U, find a matrix A(t), depending smoothly on t and unitary for all t ∈ [0, 1], such that A(0) = I and A(1) = U. Under what conditions on U can A(t) be made real for all t ∈ [0, 1]?

6 Elementary functions of matrices

Throughout this chapter, matrices are understood as of dimension n × n for arbitrary fixed n unless stated otherwise. For definiteness, we take F = ℂ; the particular results for real matrices will be stated explicitly.

Elementary functions of scalar variables satisfy numerous identities and differential equations. Here we wish to define elementary functions of matrices, in such a way that many, if not all of these properties are preserved.

Our point of departure is to note that elementary functions often admit representations as infinite power series

(6.1)  f(z) = ∑_{j=0}^{∞} a_{j}z^{j}

for example, converging in some region |z| < ρ. Then the series (6.1) converges absolutely in this region, and so do sums and products of such series, the series obtained by term by term differentiation or integration, etc. Thus many formal computations with such series are in fact justified, and numerous properties of the function f(z) obtained in this manner.

We can do the same for elementary functions of matrices, determined analogously from

(6.2)  f(A) = ∑_{j=0}^{∞} a_{j}A^{j},

where A is some matrix, a_{j} ∈ F, and A^{0} = I by convention, provided the series (6.2) also converges. This question, however, has a simple answer.

Theorem 6.1. If the series (6.1) converges for |z| < ρ, then the series (6.2) converges absolutely for A such that

(6.3)  σ(A) < ρ.

Here σ(A) is the spectral radius of A, given in (4.24). We shall prove this by a method which provides a procedure for computing f(A) explicitly.

Proof. We have A = T^{−1}JT, T nonsingular, J in Jordan canonical form. For any partial sum

f_{m}(A) = ∑_{j=0}^{m} a_{j}A^{j}

= ∑_{j=0}^{m} a_{j}(T^{−1}JT)^{j}

= ∑_{j=0}^{m} a_{j}T^{−1}J^{j}T

= T^{−1}f_{m}(J)T, (6.4)

so if the partial sums f_{m}(J) converge absolutely, so do the f_{m}(A), and in addition we shall have established

(6.5)  f(A) = T^{−1}f(J)T

for the infinite series as well.

Thus it suffices to consider the f_{m}(J), which is facilitated by the simple structure of the Jordan form J. In particular, J, is block diagonal, with each block of dimension no greater than μ, μ the largest of the exponents μ_{i} appearing in the minimal polynomial for A, given by (3.7). In any event, μ ≤ n. Now since J is block diagonal, so are all the J^{j} and thus the f_{m}(J), with the same block structure, by application of lemma 2.9. Thus it suffices to prove the convergence of f_{m}(J), restricting attention to any single Jordan block.

The matrix J, restricted to a single block, of dimension k, say, is of a very special structure

(6.6)  J = λI_{k} + L

where I_{k} is the k × k identity matrix, L a k × k matrix satisfying

(6.7)  L_{ij} = 1 for j = i + 1, L_{ij} = 0 otherwise

and λ is one of the eigenvalues of J, or of A. In particular

(6.8)  |λ| ≤ σ(A).

The matrix L obviously commutes with I_{k}; furthermore, the powers of L are easily seen to satisfy

(6.9)  (L^{m})_{ij} = 1 if j = i + m, 0 otherwise,

and since L is of dimension k, with k not exceeding μ,

(6.10)  L^{μ} = 0.

As the series (6.1) converges for |z| < ρ, it follows that

(6.11)  |a_{j}| |z|^{j} ≤ K(|z|) |z| < ρ

where the bound K depends on |z| (possibly blowing up as |z| → ρ) but is independent of j.

Thus for any N > M, the absolute sum of the terms in f_{N}(J) − f_{M}(J) is

∑_{j=M+1}^{N}|a_{j}| ∥ (λI_{k} + L)^{j}∥

= ∑_{j=M+1}^{N} |a_{j}| ∥ ∑_{m=0}^{j} \binom{j}{m}λ^{j−m}L^{m}∥

= ∑_{j=M+1}^{N} |a_{j}| ∥ ∑_{m=0}^{μ−1} \binom{j}{m}λ^{j−m}L^{m}∥

≤ ∑_{m=0}^{μ−1} ∥L^{m}∥ ∑_{j=M+1}^{N} |a_{j}|\binom{j}{m}|λ|^{j−m}

≤ ∑_{m=0}^{μ−1} ∥L^{m}∥\frac{1}{σ(A)^{m}m!} ∑_{j=M+1}^{N} |a_{j}|j^{m}σ(A)^{j} (6.12)

using (6.10), (6.8), and \binom{j}{m} = j!/(m!(j − m)!).

In (6.12), choose M initially sufficiently large that

(6.13)  σ(A)j^{(μ−1)/j} ≤ (2σ(A) + ρ)/3 for j > M,

using σ(A) < ρ. Then the second sum in (6.12) is estimated

∑_{j=M+1}^{N}|a_{j}|(\frac{2σ(A) + ρ}{3})^{j}

≤ ∑_{j=M+1}^{N} |a_{j}|(\frac{σ(A) + 2ρ}{3})^{j}(\frac{2σ(A) + ρ}{σ(A) + 2ρ})^{j}

≤ K(\frac{σ(A) + 2ρ}{3}) ∑_{j=M+1}^{∞} (\frac{2σ(A) + ρ}{σ(A) + 2ρ})^{j}

= K(\frac{σ(A) + 2ρ}{3})(\frac{σ(A) + 2ρ}{ρ − σ(A)})(\frac{2σ(A) + ρ}{σ(A) + 2ρ})^{M+1} (6.14)

Again using σ(A) < ρ, the right side of (6.14) is made arbitrarily small by choosing M sufficiently large. This completes the proof.

Proceeding as in obtaining (6.12), we see that the corresponding block of f(J) is given explicitly by

(6.15)  ∑_{m=0}^{μ−1} L^{m} ∑_{j=m}^{∞} a_{j}λ^{j−m}\binom{j}{m} = ∑_{m=0}^{μ−1} \frac{f^{(m)}(λ)}{m!}L^{m}

which reduces to f(λ) in the case of μ = 1, i.e. this block of J in fact diagonal.

Some implications of this result are worth noting. If A is a real matrix, if f(z) is real for z real and if σ(A) < ρ, then f(A), defined by (6.2), is a real matrix - the eigenvalues of A do not have to be real.

Entire functions of matrices, like e^{A} and sin A are defined for any matrix, and satisfy relations such as

(6.16)  sin^{2} A + cos^{2} A = I, \frac{d}{dt}(e^{tA}) = Ae^{tA}

as these are readily obtained from the power series expansions. But for two matrices, A, B in general

(6.17)  e^{A+B} ≠ e^{A}e^{B}, sin(A + B) ≠ sin A cos B + cos A sin B

because the proof of such results for scalars, based on the power series expansions for the respective functions, depends on interchanging products BA and AB. Thus equality applies in relations such as (6.17) if A, B commute.

The regularity of a function f(A) so determined is described by the following lemma:

Lemma 6.2. Suppose the series (6.1) converges for |z| < ρ, and that σ(A) < ρ. Then for any matrix B, f(A + tB) is defined and is an infinitely differentiable function of the real parameter t in some neighborhood of t = 0. Furthermore,

(6.18)  If AB = BA, \frac{d}{dt}f(A + tB)|_{t=0} = f′(A)B.

Proof. As σ(A) < ρ, σ(A + tB) < ρ for |t| sufficiently small, and thus formal differentiation any number of times of the power series for f(A + tB) is justified. If A, B commute, the power series for A + tB, |t| small, may be reassembled in the form f(A) + tf′(A)B + O(t^{2}), and (6.18) follows.

In view of (6.10), the right side of (6.15) is just the formal Taylor series for f(λI + L). The generalization of this is the following lemma:

Lemma 6.3. Suppose A, B commute, and σ(A + tB) < ρ for all 0 ≤ t ≤ 1, where the series (6.1) converges for |z| < ρ. Then for any integer m

(6.19) f(A + B) = ∑_{j=0}^{m} \frac{f^{(j)}(A)}{j!}B^{j} + \frac{1}{m!} ∫_{0}^{1}(1 − t)^{m}f^{(m+1)}(A + tB)dtB^{m+1}.

Proof. For any t ∈ [0, 1], σ(A + tB) < ρ and A + tB commutes with B, so

(6.20)  \frac{d}{dt}f(A + tB) = f′(A + tB)B;

integrating (6.20) we obtain

(6.21)  f(A + B) − f(A) = ∫_{0}^{1} f′(A + tB)dtB.

The higher derivatives of f(A + tB) are defined by their power series, which is known to converge. Therefore we simply apply m partial integrations to (6.21) to obtain (6.19).

Other functions of matrices are defined as the inverse functions of functions defined by convergent power series. In this manner, one is often able to define such a function in a larger region than would be possible by a power series. Thus the formal power series for ℓn(I + A), for example, converges for σ(A) < 1, but we shall see that in fact any nonsingular matrix has a logarithm.

Theorem 6.4. Suppose the series (6.1) converges for |z| < ρ. Given a matrix Z suppose that for each eigenvalue λ_{i} of Z there exists η_{i}, such that

(6.22)  |η_{i}| < ρ

(6.23)  f(η_{i}) = λ_{i}

and

(6.24)  f′(η_{i}) ≠ 0.

Then f^{−1}(Z) is defined, i.e. there exists a matrix X such that f(X) = Z, with σ(X) < ρ.

Proof. Put Z = T^{−1}JT, J in Jordan form. If we can solve

(6.25)  f(Y) = J

for a matrix Y such that

σ(Y) < ρ,

then X = T^{−1}Y T satisfies f(X) = Z,

f(T^{−1}Y T) = T^{−1}f(Y)T
= T^{−1}JT

= Z. (6.26)

Of course we seek Y in the same block diagonal structure as that of J. For a block of dimension k, J is of the form (6.6), with L given by (6.7) and λ one of the eigenvalues of Z. By hypothesis, therefore, there exists η satisfying

(6.27)  |η| < ρ

(6.28)  f(η) = λ

(6.29)  f′(η) ≠ 0.

We seek the corresponding block of the matrix Y in the form

(6.30)  ηI_{k} + ∑_{j=1}^{k−1} ζ_{j}L^{j}

noting that L^{k} = 0, and the lower powers of L obtained from (6.9). The ζ_{j} are scalars to be determined.

We wish to apply (6.19) to (6.30), identifying A in (6.19) with ηI_{k}, B with ∑ ζ_{j}L^{j}, and m in (6.19) with k − 1. Obviously A, B commute, and the remainder term in (6.19) vanishes, because of the factor L^{k}. From the structure of L and (6.30), B is upper triangular with zero diagonal elements, and thus for any t, σ(A + tB) = η < ρ. Therefore in this case

f(A + B) = f(A) + f′(A)B + ⋯ + \frac{1}{(k − 1)!}f^{(k−1)}(A)B^{k−1}

= f(η)I_{k} + ∑_{m=1}^{k−1} \frac{1}{m!}f^{(m)}(ηI_{k})(∑_{j=1}^{k−1} ζ_{j}L^{j})^{m}

= f(η)I_{k} + ∑_{j=1}^{k−1} L^{j}(f′(η)ζ_{j} + g_{j}(f′(η), …, f^{(j)}(η), ζ_{1} …, ζ_{j−1})) (6.31)

where the functions g_{j} are obtained by expanding the sum in (6.31) and collecting the powers of L. In particular, g_{j} includes all the quadratic and higher order terms in the ζ_{i}, and therefore g_{j} depends on the ζ_{i} only for i < j, and g_{1} vanishes identically.

We wish to solve f(A+B) = λI_{k} +L; using (6.31) and comparing powers of L, using (6.28) it will suffice that

(6.32)  f′(η)ζ_{1} = 1, f′(η)ζ_{j} + g_{j} = 0, j = 2, …, k − 1

which is easily solved recursively using (6.29). Independently of the ζ_{j}, we will have σ(A + B) = |η| < ρ, so the matrix Y determined in the manner indeed satisfies (6.25).

The matrix f^{−1}(Z) so obtained will be real if Z is a real matrix with real eigenvalues, the η_{i} can be taken real valued, and f(z) is real for real z. Thus lnZ is real, for example, for a real matrix Z with positive (real) eigenvalues, and a real sin^{−1} Z exists for real Z with real eigenvalues and σ(Z) < 1.

1. Evaluate explicitly: exp (\begin{matrix}0 & 1 \\ 1 & 0 \end{matrix}) , ℓn (\begin{matrix}0 & 1 \\ 1 & 0 \end{matrix}) , sin (\begin{matrix}π/2 & 1 \\ 0 & π/2 \end{matrix}) , tan^{−1} (\begin{matrix}1 & 1 \\ 0 & 1 \end{matrix}) .

2. Show that e^{A} is nonsingular for any matrix A.

3. Find an example of nonsingular A, B such that ℓn(AB) ≠ ℓnA + ℓnB

4. Suppose that Z satisfies the conditions of Theorem 6.4, so that X = f^{−1}(Z) is defined. Show that f′(X) is nonsingular; and that XZ = ZX.

5. Show directly that sin^{−1} (\begin{matrix}1 & 1 \\ 0 & 1 \end{matrix}) does not exist. Hint: Any matrix

X commutes with sin X, so the possible structure of such a matrix is limited.

6. Find an example of a nonsingular 2 × 2 matrix A such that A(sin A) = −(sin A)A

7. Show that if sin A = 0, then A is similar to a diagonal matrix, each of whose diagonal elements is an integer multiple of π.

8. Evaluate ln (\begin{matrix}2 & 1 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & 1 \end{matrix})

9. Classify the matrices A ∈ M_{2×2}(ℂ) such that there is no Z ∈ M_{2×2}(ℂ) satisfying Z^{2} = A.

10 Find ρ and the function K satisfying (6.11) for the case a_{j} = j, j = 0, 1, 2, . . ..

11 Prove that A is anti-Hermitian if and only if e^{A} is unitary.

12^{∗} For A, B ∈ M_{n×n}(ℂ) suppose that e^{tA}B = Be^{tA} for t ∈ ℝ. Prove that AB = BA. In this true if cos(tA)B = B cos(tA)?

Appendix A. Roots of polynomials

Lemma A.1. A factorization of a polynomial is unique up to rearrangement of the roots.

Proof. Suppose that for all z ∈ ℂ

(A.1)  H(z) = ∏_{i=1}^{k}(z − r_{i})^{μ_{i}} = ∏_{i=1}^{ℓ}(z − s_{i})^{ν_{i}}

where r_{i}, s_{i} ∈ ℂ, k, ℓ, μ_{i}, ν_{i} are positive integers, and

(A.2)  r_{i} ≠ r_{j}, s_{i} ≠ s_{j} for all i ≠ j.

The zeros of H are independent of such a factorization, so the sets {r_{1}, …, r_{k}}, {s_{1}, …, s_{ℓ}} coincide. From (A.2) it follows that ℓ = k and s_{i} = r_{i}, i = 1, …, k possibly after rearranging the s_{i}. So (A.1) becomes

(A.3)  H(z) = ∏_{i=1}^{k}(z − r_{i})^{ν_{i}} = ∏_{i=1}^{k}(z − r_{i})^{ν_{i}}.

Similarly, the zeros of the z-derivative of H are independent of the specific factorization. For any j = 1, …, k, from the first part of (A.3)

(A.4)  \frac{d^{m}H(r_{j})}{dz^{m}} = 0, m = 0, …, μ_{j} − 1

(A.5)  \frac{d^{μ_{j}}H(r_{j})}{dz^{μ_{j}}} = μ_{j}!_{k ; ∏_{i≠j} ; i=1}(r_{j} − r_{i})^{μ_{i}} ≠ 0

From the second part of (A.3)

(A.6)  \frac{d^{m}H(r_{j})}{dz^{m}} = 0, m = 0, …, ν_{j} − 1; \frac{d^{ν_{j}}(r_{j})}{dz^{ν_{j}}} ≠ 0.

Comparing (A.4), (A.5), (A.6), it follows that ν_{j} = μ_{j} for all j, completing the proof.

For a = (a_{1} ; ⋮ ; a_{n}) , r = (r_{1} ; ⋮ ; r_{n}) ∈ ℂ^{n}, let f : ℂ^{n} → ℂ^{n} be determined by

f(r) = a, where for all z ∈ ℂ

(A.7) P_{a}(z) = z^{n}+a_{1}z^{n−1}+⋯+a_{n−1}z+a_{n} = (z−r_{1})(z−r_{2}) … (z−r_{n}).

Clearly f is a smooth mapping, a polynomial mapping in particular. But f is not one-to-one, as f(r) is unchanged by the interchange of any r_{i} with r_{j}, j ≠ i.

Lemma A.2. For every K > 0

(A.8)  |f(r)| ≤ K implies |r| ≤ n maximum (1, K).

Proof. The components of r are just the zeros of P_{a}(z) given by (4.7) where a = f(r) so from (A.8)

(A.9)  |a| ≤ K.

Without loss of generality assume K ≥ 1. Then for |z| > nK, for any j = 1, …, n,

|a_{j}z^{n−j}| ≤ |a||z|^{n−1},
≤ K|z|^{n−1}

< |z|^{n}/n. (A.10)

From (A.7) and (A.10) it follows from the triangle inequality that P_{a}(z) does not vanish, proving (A.8).

Denote by

(A.11)  D = {r ∈ ℂ^{n}|r_{i} ≠ r_{j} for all i ≠ j}

and

(A.12)  S = f(D)

Lemma A.3. S is an open subset of ℂ^{n}.

Proof. Given a ∈ S, there exists r ∈ D such that f(r) = a, equivalently from (A.7)

(A.13)  P_{a}(z) = 0, for z ∈ {r_{i}, …, r_{n}}.

From the continuity of f, for |ã − a| sufficiently small, |Pã(z)| can be made arbitrarily small for z in a sufficiently small neighborhood of any of of the r_{j}. Furthermore, for any j = 1, …, n fixed

\frac{dP_{a}(r_{j})}{dz} = _{n ; ∏_{i=1} ; i≠j}(r_{j} − r_{i})

≠ 0, (A.14)

since r ∈ D, using (A.11). So by the implicit function theorem, there is a zero r̃_{j} of Pã approaching r_{j} as ã approaches a, for each j. In particular, the r̃_{j} are distinct for |ã − a| sufficiently small.

The next lemma shows that there exists a right inverse of f, defined on the closure of S.

Lemma A.4. Suppose that a^{m} ∈ S, m = 1, 2, …, and that a^{m} → a ∈ ℂ^{n} as m → ∞. Then there exists r ∈ ℂ^{n} such that f(r) = a.

Proof. As a^{m} ∈ S, there exists r^{m} ∈ D such that f(r^{m}) = a^{m}; by lemma A2, as the a^{m} are bounded uniformly with respect to m, so are the r^{m}. By the Bolzano-Weissstrauss theorem, a subsequence of the r^{m} converges to r as r → ∞. Now f(r) = a from the continuity of f.

Theorem A.5. In the complex field, any polynomial of degree n can be factored into n linear terms.

Proof. We have established the existence of a right inverse of f defined on the closure of S, so it suffices to show that the open set S is dense in ℂ^{n}.

Denote by

(A.15)  E = {r ∈ ℂ^{n}|r_{n−1} = r_{n}}

and

(A.16)  Q = f(E)

From lemma A1, although E is not all of the boundary of D, Q is all of the boundary of S. Therefore either S is dense in K^{n} or else Q partitions ℂ^{n} into nonintersecting regions.

There are not enough points in Q to partition ℂ^{n}; it is only here that the assumption of the complex field is used. At any point a ∈ Q, denote by T_{a} the tangent space to Q at a. As E is isomorphic to ℝ^{2n−2}, and f is a smooth map, it follows that T_{a} is defined for all a ∈ Q and that

(A.17)  dim T_{a} ≤ 2n − 2.

However, if Q or a subset thereof partitions ℂ^{n}, isomorphic to ℝ^{2n}, we need T_{a} of dimension 2n − 1. So indeed S is dense in ℂ^{n} and the theorem is proved.

Theorem A.6. In the complex field, the roots of a polynomial of finite degree depend continuously on the coefficients of the polynomial.

Proof. Suppose not, then there exist r, a = f(r) ∈ ℂ^{n}, and a sequence {(r^{m}, a^{m})}_{m=1}^{∞} satisfying

(A.18)  a^{m} = f(r^{m}), m = 1, …

(A.19)  a^{m} → a as m → ∞

but

(A.20)  |r^{m} − r| ≥ α > 0

after any rearrangement of the components of each of the r^{m}.

By lemma A2, as the |a^{m}| are bounded uniformly with respect to m, so are the |r^{m}|, and a subsequence converges to some r̃ ∈ ℂ^{n}. From (A.18), (A.19) and the continuity of f, f(r̃) = a. Now from lemma A1, r̃ = r, possibly after a rearrangement of the components contradicting (A.20) and thus proving the theorem.

Appendix B: Sequences of functions and power series

Let L ⊆ ℝ be an interval, and let f_{n}, n = 1, …, be an infinite sequence of functions from L into ℝ.

Definition. The sequence \{f_{n}\}_{n=1}^{∞} converges \underline{pointwise} in L if for all ε > 0 and all x ∈ L, there exists N_{ε,x} such that

(B.1)  |f_{n}(x) − f_{m}(x)| ≤ ε for all m, n ≥ N_{ε,x}

Definition. The sequence \{f_{n}\}_{n=1}^{∞} converges \underline{uniformly} in L if for all ε > 0 there exists N_{ε} such that

(B.2)  |f_{n}(x) − f_{m}(x)| ≤ ε for all m, n ≥ N_{ε} and x ∈ L.

Throughout we denote by f the limit function of such convergent sequence.

Theorem B.1. Assume that the sequence {f_{n}}_{n=1}^{∞} converges uniformly in L. If each of the f_{n} is continuous at a point x ∈ L, then the limit function is continuous at x. If each of the f_{n} is uniformly continuous in L, then the limit is uniformly continuous in L.

Proof. We prove only the second statement here, as the arguments are very similar. Given γ > 0, choose ε = γ/3, find N_{ε} from (B.2), and fix n ≥ N_{ε}. Then

(B.3)  |f(x) − f_{n}(x)| ≤ γ/3 for all x ∈ L.

Now find δ, depending on γ and on n, such that

(B.4)  |f_{n}(x) − f_{n}(y)| ≤ γ/3 for all |x − y| ≤ δ, x, y ∈ L

using the fact that f_{n} is uniformly continuous in L. Then using (B.3) and (B.4), for any x, y ∈ L, |x − y| ≤ δ,

|f(x) − f(y)| ≤ |f(x) − f_{n}(x)| + |f_{n}(x) − f_{n}(y)| + |f_{n}(y) − f(y)|
≤ γ.

Uniform convergence is needed because we have to choose n without knowing x, y. A simple example where the theorem fails, given only pointwise convergence is given by L = (−1, 1),

f_{n}(x) = \begin{cases}e^{nx}, x < 0 \\ 1, x ≥ 0 \end{cases}

where {f_{n}}_{n=1}^{∞} converges pointwise to the (discontinuous) Heavyside function.

Theorem B.2. Let (a, b) ⊆ L be a bounded interval, and assume that the sequence {f_{n}}_{n=1}^{∞} converges uniformly in the interval (a, b). Then

(B.5)  ∫_{a}^{b} f_{n}(x)dx → ∫_{a}^{b} f(x)dx as n → ∞.

Proof. Given any γ > 0 set ε = γ/(b − a) and fix n ≥ N_{ε} from (B.2). (We need b − a finite so ε > 0.) Then

| ∫_{a}^{b} f_{n}(x)dx − ∫_{a}^{b} f(x)dx| ≤ ∫_{a}^{b} |f_{n}(x) − f(x)dx

∫_{a}^{b} εdx

= (b − a)ε = γ.

The example a = 0, b = 1, f_{n}(x) = ne^{−nx} shows that uniform convergence is needed. The sequence {f_{n}}_{n=1}^{∞} converges pointwise to the zero function on (0, 1), but

∫_{0}^{1} f_{n}(x)dx = 1 − e^{−n}

converges to 1 as n → ∞.

The example a = 0, b = ∞,

f_{n}(x) = \begin{cases}1/n, & 0 < x < n \\ 0, & x ≥ n \end{cases}

shows that a finite interval is needed. The sequence {f_{n}}_{n=1}^{∞} converges uniformly to zero, but for all n

∫_{0}^{∞}f_{n}(x)dx = 1.

Theorem B.3. Assume that each of the f_{n} is continuously differentiable in L, that the sequence {f_{n}}_{n=1}^{∞} converges pointwise to f in L, and that the sequence {f′_{n}}_{n=1}^{∞} converges uniformly to g in L. Then f is differentiable in L and

(B.6)  f′(x) = g(x) for all x ∈ L

Proof. For any fixed x, y ∈ L and any n,

(B.7)  f_{n}(y) − f_{n}(x) = ∫_{x}^{y} f′_{n}(t)dt.

As n → ∞, the left side of (B.7) converges to f(y) − f(x), whereas by

theorem B.2 the right side of (B.7) converges to ∫_{x}^{y}(g(t)dt. So for any

x, y ∈ L

(B.8)  f(y) − f(x) = ∫_{x}^{y} g(t)dt

Now from Theorem B.1, g is continuous in L, so from (B.8) f is differentiable in L and satisfies (B.6).

The application of these results needed in chapter 6 is that where the f_{n} are given as a power series in x

(B.9)  f_{n}(x) = ∑_{j=0}^{n} a_{j}x^{j}

where the a_{j} ∈ ℂ (or ℝ) are given coefficients. The formal limit

(B.10)  f(x) = ∑_{j=0}^{∞} a_{j}x^{j}

obviously exists for x = 0; indeed, we shall show it exists for |x| < ρ, ρ the radius of convergence of the series (B.10), given explicitly by

(B.11)  ρ = 1/ lim sup_{j} |a_{j}|^{1/j}.

As determined from (B.11), ρ may be zero (as in the case a_{j} = j!), a finite positive number (as in the case a_{0} = 0, a_{j} = (−1)^{j+1}/j, corresponding to the Taylor series for log (1 + x)), or infinite (as in the case a_{j} = 1/j!, corresponding to the Taylor series for e^{x}). For any |x| < ρ, from (B.11)

lim sup_{j} |a_{j}x^{j}| = lim sup_{j} |a_{j}|ρ^{j}(\frac{|x|}{ρ})^{j}
= 0,

so there is a number K, depending on |x|, such that

|a_{j}x^{j}| ≤ K for all j,

and in particular (6.11) holds. For |x| > ρ, from (B.11)

lim sup_{j} |a_{j}x^{j}| = ∞;

the limit may or may not exist for x = ±ρ, depending on the a_{j}.

Theorem B.4. Let L ⊂ (−ρ, ρ) be any closed interval. Then f given by (B.10) exists and is an infinitely differentable function in L. Furthermore

f_{n} → f (B.12)

f_{n}^{(ℓ)} → f^{(ℓ)}, ℓ = 1, 2.... (B.13)

both uniformly in L as n → ∞, and for any a, b ∈ L

(B.14)  ∫_{a}^{b} f_{n}(x) → ∫_{a}^{b} f(x)dx as n → ∞

Proof. It suffices to prove that for any arbitrary fixed ℓ = 0, 1, 2, ... that the sequence {f_{n}^{(ℓ)}} converges uniformly in L. Then (B.12) follows from theorem B.1, setting ℓ = 0, (B.13) follows from theorem B.3, and (B.14) from theorem B.2.

With L fixed, denote by

(B.15)  x̄ = maximum_{x∈L}|x|;

clearly

(B.16)  x̄ < ρ.

Given ε > 0, we seek N_{ε} such that (B.2) holds for the sequence {f_{n}^{(ℓ)}}_{n=1}^{∞}. With the f_{n} obtained from (B.9), this becomes

(B.17)  | ∑_{j=m+1}^{n} a_{j} \frac{j!}{(j − ℓ)!}x^{j−ℓ}| ≤ ε

for all m, n ≥ N_{ε} and all x ∈ L.

In fact we shall prove a statement stronger than (B.17), that for N_{ε} sufficiently large, depending on x̄, ρ, ℓ and the function K satisfying (6.11),

(B.18)  ∑_{j=N_{ε}}^{∞} |a_{j}|\frac{j!}{(j − ℓ)!}x̄^{j−ℓ} ≤ ε.

The left side of (B.18) is equal to

\begin{aligned}\frac{1}{(\frac{x̄+ρ}{2})^{ℓ}} ∑_{j=N_{ε}}^{∞} & |a_{j}|(\frac{x̄ + ρ}{2})^{j} \frac{j!}{(j − ℓ)!}(\frac{2x̄}{x̄ + ρ})^{j−ℓ} \\ & ≤ \frac{K(\frac{x̄+ρ}{2})}{(\frac{x̄+ρ}{2})ℓ} ∑_{j=N_{ε}}^{∞} \frac{j!}{(j − ℓ)!}θ^{j−ℓ} \end{aligned} ; (B.19)

using (6.11) with |x| = (x̄ + ρ)/2 < ρ from (B.16) and denoting by

θ = \frac{2x̄}{x̄ + ρ} < 1

also from (B.16). The right side of (B.19) is equal to

\frac{K(\frac{x̄+ρ}{2})}{(\frac{x̄+ρ}{2})^{ℓ}} ∑_{k=ℓ}^{∞} \frac{(k + N_{ε} − ℓ)!}{(k + N_{ε} − 2ℓ)!}θ^{k+N_{ε}−2ℓ}

< \frac{K(\frac{x̄+ρ}{2})}{(\frac{x̄+ρ}{2})^{ℓ}}(N_{ε} − ℓ)^{ℓ}θ^{N_{ε}−ℓ} ∑_{k=ℓ}^{∞} \frac{k!}{(k − ℓ)!}θ^{k−ℓ}

= \frac{K(\frac{x̄+ρ}{2})}{(\frac{x̄+ρ}{2})^{ℓ}}(N_{ε} − ℓ)^{ℓ}θ^{N_{ε}−ℓ} \frac{ℓ!}{(1 − θ)^{ℓ+1}} (B.20)

using θ < 1. Now as N_{ε} → ∞ with ℓ, θ < 1 fixed, the function (N_{ε} − ℓ)^{ℓ}θ^{N_{ε}−ℓ} approaches zero, so taking N_{ε} sufficiently large the right side of (B.20) is made smaller than the given ε.