2.5 Kernel and Image

1Reading¶

Material related to this page, as well as additional exercises, can be found in ALA Ch. 2.5 and LAA 4.2.

2Learning Objectives¶

By the end of this page, you should know:

a kernel (null space) and image (column space) of a matrix
the realtionship between the null space and column space to solutions of linear systems
to apply the superposition principle for solving linear systems with different right-hand vectors

3Introduction¶

In applications of linear algebra, subspaces of $\mathbb{R}^n$ (and general vector spaces $V$ ) typically arise from either

(a) set of all solutions to a system of linear equations of the form $A \vv x = \vv 0$ , called a homogeneous linear system, or (b) as the span of certain specified vectors.

(a) is known as the null space description of a subspace, while (b) is known as the column space or image description of a subspace.

We will see that these are intimately related to systems of linear equations.

4Null Space of a Matrix¶

If we think of the function $f(\vv x) = A \vv x$ that maps $\vv x \mapsto A \vv x$ , then Null $(A)$ is the subset of $\mathbb{R}^n$ that $f(\vv x)$ maps to $\vv 0$ .

Example 1

Consider the following system of homogeneous equations

\begin{align*} x_1 - 3x_2 - 2x_3 &= 0, \\ -5x_1 + 9x_2 + 2x_3 &= 0, \end{align*}

(2)

or in matrix form $A \vv x = \vv 0$ , where $A = \bm1 & -3 & -2 \\ -5 & 9 & 1 \em$ . Recall that the set of $\vv x$ satisfying $A \vv x = \vv 0$ is the solution set of (2). Our goal is to relate this solution set to the matrix $A$ which will give us a geometric interpretation to the solution of the algebraic system.

Is $\vv u = \bm 5 \\ 3 \\ -2 \em$ in Null $(A)$ ?

Evaluating $A \vv u = \bm1 & -3 & -2 \\ -5 & 9 & 1 \em\bm 5 \\ 3 \\ -2 \em = \bm 0 \\ 0\em \Rightarrow A \vv u = \vv 0 \Rightarrow \vv u \in$ Null $(A)$ .

Now, you may be wondering why are we calling Null $(A)$ the null space? That’s because Null $(A)$ is a subspace!

We test it as follows. Suppose that $\vv u, \vv v \in$ Null $(A)$ and $c, d \in \mathbb{R}$ . We need to check if $c \vv u + d \vv v \in$ Null $(A)$ , i.e., is it true that $A (c \vv u + d \vv v) = \vv 0?$

\begin{align*} A(c \vv u + d \vv v) &= c A \vv u + d A \vv v, \ (\textrm{linearity of matrix multiplication}) \\ &= c \vv 0 + d \vv 0, \ (\vv u, \vv v \in \textrm{Null}(A). \ \textrm{so} \ A \vv u = A \vv v = \vv 0) \\ &= \vv 0 \end{align*}

(3)

From (3), Null $(A)$ is a vector space! If $A \in \mathbb{R}^{m \times n}$ , then Null $(A)$ is a subspace of $\mathbb{R}^n$ (where $\vv x$ lived). This property leads to the following incredibly important superposition principle for solutions to homogeneous linear systems.

5Describing the Null Space¶

There is no obvious relationship between the entries of $A$ and Null $(A)$ . Rather, it is defined implicitly via the condition that $\vv x \in$ Null $(A)$ if and only if $A \vv x =\vv 0$ . However, if we compute the general solution to $A \vv x = 0$ , this will give us an explicit description of Null $(A)$ . This can be accomplished via Gaussian Elimination.

Example 2

Let us find a basis for Null $(A)$ , where $A = \bm -3 & 6 & -1 & 1 & -7 \\ 1 & -2 & 2 & 3 & -1 \\ 2 & -4 & 5 & 8 & -4 \em$ . We reduce $\bm A & | & \vv 0\em$ to row echelon form in order ot write the basic variables in terms of the free variables:

\begin{align*} \bm 1 & -2 & 0 & -1 & 3 & - \\ 0 & 0 & 1 & 2 & -2 & 0 \\ 0 & 0 & 0 & 0 & 0 & 0 \em \Leftrightarrow & x_1 - 2x_2 - x_4 + 3x_5 = 0, \\ & x_3 + 2x_4 - 2x_5 = 0, \\ &0 = 0. \end{align*}

(4)

From (4), the general solution is $x_1 = 2x_2 + x_4 - 3x_5, \ x_3 =-2x_4 + 2x_5$ . The free variables are $x_2, x_4, x_4$ and the basic variables are $x_1, x_3$ , since, the pivots are at $(1, 1), (2, 3)$ . We can decompose the general solution as

\bm x_1 \\ x_2 \\ x_3 \\ x_4 \\ x_5\em = \bm 2x_2 + x_4 - 3x_5 \\ x_2 \\ -2x_4 + 2x_5 \\ x_4 \\ x_5\em = x_2 \bm 2 \\ 1 \\ 0 \\ 0 \\ 0\em + x_4 \bm 1 \\ 0 \\ -2 \\ 1 \\ 0\em + x_5 \bm -3 \\ 0 \\ 2 \\ 0 \\ 1\em = x_2 \vv u_1 + x_4 \vv u_2 + x_5 \vv u_3, \\ \textrm{where}, \vv u_1 = \bm 2 \\ 1 \\ 0 \\ 0 \\ 0\em, \ \vv u_2 = \bm 1 \\ 0 \\ -2 \\ 1 \\ 0\em, \ \vv u_3 = \bm -3 \\ 0 \\ 2 \\ 0 \\ 1\em.

(5)

From (5), every linear combination of $\vv u_1, \vv u_2, \vv u_3$ is in Null $(A)$ . Also, $\vv u_1, \vv u_2, \vv u_3$ are linearly independent (think when does (5) become zero and why so?), hence, $\vv u_1, \vv u_2, \vv u_3$ form a basis for Null $(A)$ .

We conclude that Null $(A) \subset \mathbb{R}^5$ is a subspace of dimension 3.

6The Column Space of $A$ ¶

We have seen previously that we can write the matrix-vector product $A \vv x$ as the linear combination

A \vv x = x_1 \vv a_1 + x \vv a_2 + \ldots + x_n \vv a_n,

(6)

of the columns $\vv a_, \vv a_2, \ldots , \vv a_n$ of $A = \bm \vv a_1 & \vv a_2 & \ldots & \vv a_n\em$ weighted by the elements $x_i$ of $\vv x$ . By letting the coefficients $x_1, x_2, \ldots, x_n$ vary, we can descsribe the subspace spanned by the columns of $A$ , aptly named the column space of $A$ .

Example 3

Find a matrix $A$ so that the set

W = \left\{ \bm 6a - b \\ a + b \\ -7a \em : a, b \in \mathbb{R}\right\}

(8)

is equal to Col $(A)$ . To do so, we first write $W$ as a set of linear combinations

\begin{align*} W &= \left\{ a \bm 6 \\ 1 \\ -7\em + b \bm -1 \\ 1 \\ 0\em : a, b \in \mathbb{R}\right\} \\ &= \textrm{span}\left\{\bm 6 \\ 1 \\ -7\em, \bm -1 \\ 1 \\ 0\em\right\} \end{align*}

(9)

Now, we set the vectors in (9) as the columns of $A: A = \bm 6 & -1 \\ 1 & 1 \\ -7 & 0\em$ . It then follows that ${\textrm{Col}(A) = \textrm{span}\left\{\bm 6 \\ 1 \\ -7\em, \bm -1 \\ 1 \\ 0\em\right\} = W}$ .

7The Complete Solution to $A \vv x = \vv b$ ¶

With an understanding of Null $(A)$ and Col $(A)$ , we can completely characterize the solution set to $A \vv x = \vv b$ .

Theorem 2

The linear system $A \vv x = \vv b$ has at least one solution $\vv x^*$ if and only if $\vv b \in$ Col $(A)$ . If this occurs, then $\vv x$ is a solution to $A \vv x = \vv b$ if and only if

\vv x = \vv x^* + \vv n,

(10)

where $\vv n \in$ Null $(A)$ is an element of the null space of $A$ .

Proof 1 (Proof of Theorem 2)

The first part of the theorem was already discussed here. Suppose, both $\vv x$ and $\vv x^*$ are solutions so that $A \vv x = A \vv x^* = \vv b$ , then their difference $\vv n = \vv x - \vv x^*$ satisfies

A \vv n = A (\vv x - \vv x^*) = A \vv x - A \vv x^* = \vv b - \vv b = \vv 0

(11)

so that $\vv n \in$ Null $(A)$ . This means that $\vv x = \vv x^* + (\vv x - \vv x^*) = \vv x^* + \vv n$ .

Exercise 1

Find all solutions to the linear system

\begin{align*} x_1 + 2x_2 + x_3 - 2x_4 &= 9\\ -2x_1 - 3x_2 - x_3 + x_4 &= -7\\ x_1 + x_2 - x_3 - x_4 &= 2 \end{align*}

(12)

Solution to Exercise 1

First, let’s form the augmented matrix:

\begin{align*} \left[ \begin{array}{cccc|c} 1 & 2 & 1 & -2 & 9\\ -2 & -3 & -1 & 1 & -7\\ 1 & 1 & -1 & -1 & 2 \end{array} \right] \end{align*}

(13)

Putting the coefficient matrix in row echelon form, we get the equivalent system

\begin{align*} \left[ \begin{array}{cccc|c} 1 & 2 & 1 & -2 & 9\\ 0 & 1 & 1 & -3 & 11\\ 0 & 0 & -1 & -2 & 4 \end{array} \right] \end{align*}

(14)

Putting the coefficient matrix in reduced row echelon form (i.e., eliminating above the pivots, then scaling the pivots to equal 1), we get the equivalent system

\begin{align*} \left[ \begin{array}{cccc|c} 1 & 0 & 0 & 6 & -17\\ 0 & 1 & 0 & -5 & 15\\ 0 & 0 & 1 & 2 & -4 \end{array} \right] \end{align*}

(15)

This is equivalent to the linear system

\begin{align*} x_1 + 6x_4 &= -17\\ x_2 - 5x_4 &= 15\\ x_3 + 2x_4 &= -4 \end{align*}

(16)

whose solutions are given by:

\begin{align*} \bm x_1 \\ x_2 \\ x_3 \\ x_4 \em &= \bm -17 - 6x_4 \\ 15 + 5x_4 \\ -4 - 2x_4 \\ x_4 \em \\ &= \underbrace{\bm -17 \\ 15 \\ -4 \\ 0 \em}_{\text{particular solution}} + \underbrace{x_4 \bm -6 \\ 5 \\ -2 \\1 \em}_{\text{null space}}\tag{for all $x_4 \in \mathbb R$} \end{align*}

(17)

We can specialize Theorem 3 or square matrices, which allows us to characterize if $A$ is invertible via either its null space or column space.

8The Superposition Principle¶

We already showed that for homogeneous systems $A \vv x = \vv 0$ , superposition lets us generate new solutions by combining known solutions. For inhomogeneous systems $A \vv x = \vv b$ , superposition lets us combine solutions for different RHS( $\vv b$ vectors).

Suppose we have solutions $\vv x_1^*$ and $\vv x_2^*$ to $A \vv x = \vv b_1$ and $A \vv x = \vv b_2$ , respectively. Can I quickly build a solution to $A \vv x = c_1 \vv b_1 + c_2 \vv b_2$ for some $c_1, c_2 \in \mathbb{R}$ ?

The answer is superposition! Let’s try $\vv x^* = c_1 \vv x_1^* + c_2 \vv x_2^*$ :

\begin{align*} A \vv x^* &= A (c_1 \vv x_1^* + c_2 \vv x_2^*) = c_1 (A \vv x_1^*) + c_2 (A \vv x_2^*) \\ &= c_1 \vv b_1 + c_2 \vv b_2. \end{align*}

(18)

It worked! From (18), $\vv x^* = c_1 \vv x_1^* + c_2 \vv x_2^*$ is a solution to $A \vv x = c_1 \vv b_1 + c_2 \vv b_2$ . This is again the power of linear superposition at play.

Example 4

The system

\bm 4 & 1 \\ 1 & 4 \em \bm x_1 \\ x_2 \em = \bm f_1 \\ f_2 \em

(19)

models the mechanical response of a pair of masses connected by springs subject to external forcing.

The solution $\vv x = \bm x_1 \\ x_2 \em$ is the displacement of the masses and the right-hand side $\vv f = \bm f_1 \\ f_2 \em$ are the appplied forces.

For $\vv f = \vv e_1 = \bm 1 \\ 0 \em$ , $\vv x_1^* = \bm \frac{4}{15} \\ -\frac{1}{15}\em$ ; and $\vv f = \vv e_2 = \bm 0 \\ 1 \em$ , $\vv x_2^* = \bm -\frac{1}{15} \\ \frac{4}{15}\em$ .

Hence, we can write the general solution for $\vv f = \bm f_1 \\ f_2 \em = f_1 \vv e_1 + f_2 \vv e_2$ as $\vv x^* = f_1 \vv x_1^* + f_2 \vv x_2^*$ !

8.1The General form¶

The above idea caneasily be extended to several RHSs (solutions to more than two $\vv b$ vectors).

If $\vv x_1^*, \vv x_2^*, \ldots, \vv x_k^*$ are solutions to $A \vv x = \vv b_1, A \vv x = \vv b_2, \ldots, A \vv x = \vv b_k$ , then, for any choice of $c_1, c_2, \ldots, c_k \in \mathbb{R}$ , a particular solution to

A \vv x = c_1 \vv b_1 + c_2 \vv b_2 + \ldots + c_k \vv b_k

(20)

is given by $\vv x^* = c_1 \vv x_1^* + c_2 \vv x_2^* + \ldots + c_k \vv x_k^*$ . The general solution to (20) is then

\vv x = \vv x^* + \vv n = c_1 \vv x_1^* + c_2 \vv x_2^* + \ldots + c_k \vv x_k^* + \vv n,

(21)

where $\vv n \in$ Null $(A)$ .

Important

If we know the particular solutions $\vv x_1^*, \vv x_2^*, \ldots, \vv x_m^*$ to $A \vv x = \vv e_i$ for $i = 1, 2, \ldots, m$ , where $\vv e_1, \ldots, \vv e_m$ are the standard basis vectors of $\mathbb{R}^m$ , then, we can construct a particular solution $\vv x^*$ to $A \vv x = \vv b$ by first writing

\vv b = b_1 \vv e_1 + b_2 \vv e_2 +\ldots + b_m \vv e_m

(22)

to conclude that $\vv x^* = b_1 \vv x_1^* + b_2 \vv x_2^* +\ldots + b_m \vv x_m^*$ is a solution to $A \vv x = \vv b$ .

This is conceptually useful because it tells us how the elements $b_i$ of $\vv b$ affect our solution $\vv x^*$ .