1.7 Matrix Inverses - ESE 2030 📏

1Reading¶

Material related to this page, as well as additional exercises, can be found in ALA Ch. 1.5, LAA Ch 2.2. These notes are mostly based on ALA Ch 1.5.

2Learning Objectives¶

By the end of this page, you should know:

what is the inverse of a matrix
computing inverse for 2x2 matrices
important properties of matrix inverse: existence, product
Gauss Jordan elimination to compute inverses

3Basic Definition¶

The inverse of a matrix is analogous to the reciprocal $a^{−1} =\frac{1}{a}$ of a nonzero scalar $a \neq 0$ . We already encountered the inverses of matrices corresponding to elementary row operations. In this section, we will study inverses of general square matrices. We begin with the formal deﬁnition.

The inverse of a matrix is typically more useful in theory than it is in practice. In fact, a commandment of numerical linear algebra is “thou shalt not invert a matrix” because it tends to cause numerical issues like the ones we described in the last section. Because of this, we will not spend too much time on computing matrix inverses: it is actually very rare that you will ever need to do this outsie of a linear algebra class!

4Formula for 2x2 matrices¶

We want to find the inverse of the matrix $A$ , that is denoted by $X$

A = \begin{bmatrix} a & b \\ c & d \end{bmatrix},\ X = \begin{bmatrix} x & y \\ z & w \end{bmatrix} \Rightarrow AX = I = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}

(2)

The above matrix equation produces a set of four linear equations in the unknowns $(x, y, z, w)$ :

AX = \begin{bmatrix} a & b \\ c & d \end{bmatrix}\begin{bmatrix} x & y \\ z & w \end{bmatrix} = \begin{bmatrix} ax + bz & ay + bw \\ cx + dz & cy + dw \end{bmatrix} = \begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix} = I,

(3)

which holds if an only if $(x,y,z,w)$ satisfy the linear system:

\begin{align*} ax + bz &= 1\\ ay + bw &= 0\\ cx + dz &= 0\\ cy + dw & =1. \end{align*}

(4)

Solving by Gaussian Elimination, we find

x = \frac{d}{ad-bc}, \ y = \frac{-b}{ad-bc}, \ z = \frac{-c}{ad-bc}, \ w = \frac{a}{ad-bc} \Rightarrow X = \frac{1}{ad-bc}\begin{bmatrix} d & -b \\ -c & a \end{bmatrix},

(5)

provided that the common denominator $ad-bc \neq 0$ . You can verify that $XA = I$ also holds, which lets us conclude that $X=A^{-1}$ is the inverse of $A$ .

4.1Python Break!¶

NumPy has a built in function for computing matrix inverses, np.linalg.inv. Let’s compare the output of that function to a matrix inverse computed with our formula (5). Notice that because of numerical errors, $AA^{-1} \approx I$ , but that the approximation error is very small (on the order of 1e-16).

import numpy as np

def my_inv(A):
    a = A[0,0]
    b = A[0,1]
    c = A[1,0]
    d = A[1,1]
    denominator = a*d - b*c
    X = 1/denominator * np.array([[d, -b], [-c, a]])
    return X

A = np.array([[1,2],[-3,5]])
Ainv = my_inv(A)
Ainv_np = np.linalg.inv(A)

print(f'A=\n {A}, \n Ainv =\n {Ainv}, \n Ainv_np = \n {Ainv_np}')
print(f'AA^{-1} = \n{A @ Ainv}')

A=
 [[ 1  2]
 [-3  5]], 
 Ainv =
 [[ 0.45454545 -0.18181818]
 [ 0.27272727  0.09090909]], 
 Ainv_np = 
 [[ 0.45454545 -0.18181818]
 [ 0.27272727  0.09090909]]
AA^-1 = 
[[ 1.00000000e+00  0.00000000e+00]
 [-2.22044605e-16  1.00000000e+00]]

5Some Useful Properties¶

One way to understand the matrix inverse is to take a dynamic view of matrix multiplication. If we think of the matrix $A$ as defining a function $f(\vv x)$ that maps $\vv x$ to a new vector $f(\vv x) = A\vv x$ , then we can intuitively think of the matrix inverse as “undoing” this action, just as we did for elementary operation matrices and their inverses.

For example, the elementary operation of adding twice the first row to the third row is given by

E = \begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ 2 & 0 & 1 \end{bmatrix},

(7)

while the inverse operation is given by

L = \begin{bmatrix}1 & 0 & 0 \\ 0 & 1 & 0 \\ -2 & 0 & 1 \end{bmatrix},

(8)

and you can verify that $L = E^{-1}$ . You can also verify similarly that for permutation matrices with exactly one interchange that

P = \begin{bmatrix} 0 & 1 & 0 \\ 1 & 0 & 0 \\ 0 & 0 & 1 \end{bmatrix} = P^{-1},

(9)

i.e., that $P$ is its own inverse! Our observation above gives us some easy intuition for understanding why this is true, but you can also check this directly by using the formula (5).

The above statement will be proved later, but for now think about the scalar analogy. The equation $ax=b$ has a unique solution $x = a^{-1}b$ if and only if $a \neq 0$ . Similarly, $A \textbf{x} = \textbf{b}$ has a unique solution $\textbf{x} = A^{-1}\textbf{b}$ if and only if $A^{-1}$ exists.

Properties

The inverse of a square matrix, if it exists, is unique.
If $A$ is invertible, so is $A^{-1}$ and $\left(A^{-1}\right)^{-1} = A$ .
If $A$ and $B$ are invertible matrices of the same size, then their product $AB$ is also invertible, and
$(AB)^{-1} = B^{-1}A^{-1} \\ \textbf{order is reversed!}$
(10)
Again, we can intuit why this is true by thinking about the transformations $\vv x \mapsto B \vv x$ and $\vv y \mapsto A \vv y$ : we have to undo the transformation of $\vv x$ computed by first transforming $\vv y = B \vv x$ , and then $\vv z = A \vv y = AB \vv x$ in the right order:
$\textbf{x} \xrightarrow{B} B \textbf{x} \xrightarrow{A} AB\textbf{x} \xrightarrow{A^{-1}} A^{-1}AB\textbf{x} = B\textbf{x} \xrightarrow{B^{-1}} B^{-1}B\textbf{x} = \textbf{x}$
(11)

Exercise 1 (Matrix inverses)

For each of the following inveritble matrices, find their inverse.

a. $\bm 1&2\\3&4 \em$

b. $\bm 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & -3 \em$

c. $I_n$ (the identity matrix in $n$ dimensions)

d. $\bm 1&2&0\\0&1&2\\0&0&1 \em$

Solution to Exercise 1

a. Using the formula for inverses of $2 \times 2$ matrices,

\begin{align*} \bm 1&2\\3&4 \em^{-1} = \frac{1}{1\cdot 4 - 2 \cdot 3}\bm 4 & -2 \\ -3 & 1\em = \bm -2 & 1 \\ \frac 3 2 & -\frac 1 2\em \end{align*}

(12)

b. We can confirm that

\begin{align*} \bm 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & -3 \em\bm 1 & 0 & 0 \\ 0 & \frac 1 2 & 0 \\ 0 & 0 & -\frac 1 3 \em = \bm 1&0&0\\0&1&0\\0&0&1 \em = \bm 1 & 0 & 0 \\ 0 & \frac 1 2 & 0 \\ 0 & 0 & -\frac 1 3 \em\bm 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & -3 \em \end{align*}

(13)

which means that

\begin{align*} \bm 1 & 0 & 0 \\ 0 & 2 & 0 \\ 0 & 0 & -3 \em^{-1} = \bm 1 & 0 & 0 \\ 0 & \frac 1 2 & 0 \\ 0 & 0 & -\frac 1 3 \em \end{align*}

(14)

c. Remember that $I_n \times I_n = I_n$ , which means $I_n^{-1} = I_n$ .

Exercise 2 (Inverse of a product of (invertible) matrices)

Given matrices

\begin{align*} A = \bm 1&2\\3&4 \em, \quad B = \bm 1&1\\2&3 \em \end{align*}

(15)

Find $A^{-1}$ , $B^{-1}$ , and $(AB)^{-1}$ . Check that $(AB)^{-1} = B^{-1}A^{-1}$ .

Solution to Exercise 2

Multiplying out $AB$ , we get that

\begin{align*} AB = \bm 5&7\\11&15 \em \end{align*}

(16)

Using the formula for inverse of $2\times 2$ matrices, we get that

\begin{align*} A^{-1} = \bm -2&1\\ \frac 3 2 & -\frac 1 2 \em,\quad B^{-1} = \bm 3 & -1\\ -2 & 1 \em, \quad (AB)^{-1} = \bm -\frac {15} 2 & \frac 7 2 \\ \frac{ 11} 2 & -\frac 5 2 \em \end{align*}

(17)

Finally, we check that $B^{-1}A^{-1} = (AB)^{-1}$ :

\begin{align*} B^{-1}A^{-1} = \bm 3 & -1\\ -2 & 1 \em\bm -2&1\\ \frac 3 2 & -\frac 1 2 \em = \bm -\frac {15} 2 & \frac 7 2 \\ \frac{ 11} 2 & -\frac 5 2 \em = (AB)^{-1} \end{align*}

(18)

Looks good!

5.1Python Break!¶

# Inverse of the product
A = np.array([[1, 2, 3], 
              [4, -5, 6], 
              [7, 8, 9]])

B = np.array([[1, 2, 0],
              [1, 3, 0],
              [1, -3, 1]])

A_inv = np.linalg.inv(A)
B_inv = np.linalg.inv(B)

AB_inv = np.linalg.inv(A@B)
print("\nInverse of product: \n", AB_inv, "\nProduct of inverses:\n", B_inv @ A_inv)


Inverse of product: 
 [[-2.425       0.35        0.575     ]
 [ 0.825      -0.15       -0.175     ]
 [ 5.45833333 -0.75       -1.20833333]] 
Product of inverses:
 [[-2.425       0.35        0.575     ]
 [ 0.825      -0.15       -0.175     ]
 [ 5.45833333 -0.75       -1.20833333]]

6Gauss-Jordan Elimination¶

Gauss-Jordan Elimination (GJE) is the principal algorithm for computing inverses of a nonsingular matrix.

Here is some good news: we already have all of the tools needed to solve $AX = I$ for the unknown matrix $X$ . Our starting point is to recognize that the matrix equation $AX = I$ is really $n$ linear systems of the form $A\vv x_i = \vv e_i$ in parallel, where the $\vv x_i$ and $\vv e_i$ are the columns of the matrix $X$ and the identify matrix $I$ , respevtively.

We define the $n \times 1$ unit vectors $\textbf{e}_i$ :

\textbf{e}_1 = \begin{bmatrix} 1 \\ 0 \\ 0 \\ \vdots \\ 0 \end{bmatrix}, \ \textbf{e}_2 = \begin{bmatrix} 0 \\ 1 \\ 0 \\ \vdots \\ 0 \end{bmatrix}, \cdots, \ \textbf{e}_n = \begin{bmatrix} 0 \\ 0 \\ \vdots \\ 0 \\ 1 \end{bmatrix}

(19)

as the vectors with exactly one entry of 1 in the $i^{th}$ position and zeros elsewhere. The vectors $\textbf{e}_i$ are the columns of the identity matrix $I_n$ :

I_n = \begin{bmatrix} \textbf{e}_1 & \textbf{e}_2 & \cdots & \textbf{e}_n \end{bmatrix}

(20)

Hence, the right inverse equation can be written as

\begin{align*} AX = I \iff A \begin{bmatrix} \textbf{x}_1 & \textbf{x}_2 & \cdots & \textbf{x}_n \end{bmatrix} = \begin{bmatrix} A \textbf{x}_1 & A \textbf{x}_2 & \cdots & A \textbf{x}_n \end{bmatrix} = \begin{bmatrix} \textbf{e}_1 & \textbf{e}_2 & \cdots & \textbf{e}_n \end{bmatrix} \iff A\textbf{x}_1 = \textbf{e}_1, A\textbf{x}_2 = \textbf{e}_2, \cdots, A\textbf{x}_n = \textbf{e}_n. \end{align*}

(21)

The above defines a set of $n$ systems of linear equations. A key feature here is that all $n$ linear systems have the same coefficient matrix $A$ . We can take advantage of that to build one large augmented matrix $M$ that stacks all $n$ right hand sides on the right of the coefficient matrix $A$ :

M = \left[ \begin{array}{c|ccc} A & \textbf{e}_1 & \textbf{e}_2 & \cdots & \textbf{e}_n \end{array}\right] = \left[ \begin{array}{c|c} A & I \end{array}\right].

(22)

We can then apply our row operations (scaling and adding, swapping) to (22) to reduce $A$ to upper triangular form

M = \left[ \begin{array}{c|c} A & I \end{array}\right] \to N = \left[ \begin{array}{c|c} U & C \end{array}\right],

(23)

which is equivalent to reducing the original $n$ linear systems to

U\textbf{x}_1 = \textbf{c}_1, U\textbf{x}_2 = \textbf{c}_2, \cdots, U\textbf{x}_n = \textbf{c}_n,

(24)

which we could then solve via back substituion for the columns $\vv x_i$ of the matrix inverse $X$ .

For example, consider the following $A$ matrix, corresponding augmented matrix $M$ , and corresponding upper triangular form

A = \begin{bmatrix}0 & 2 & 1 \\ 2 & 6 & 1 \\ 1 & 1 & 4 \end{bmatrix},\ M = \left[ \begin{array}{ccc|ccc} 0 & 2 & 1 & 1 & 0 & 0 \\ 2 & 6 & 1 & 0 & 1 & 0 \\ 1 & 1 & 4 & 0 & 0 & 1 \end{array}\right] \to N = \left[ \begin{array}{ccc|ccc} 2 & 6 & 1 & 0 & 1 & 0 \\ 0 & 2 & 1 & 1 & 0 & 0 \\ 0 & 0 & \frac{9}{2} & 1 & -\frac{1}{2} & 1 \end{array}\right].

(25)

Although we could stop here, it’s worth highlighting that a more common version of GJE continues to apply row operations to fully reduce the augmented matrix to the form $\left[ \begin{array}{c|c} I & X \end{array}\right]$ so that $X$ is the inverse of $A$ .

We first note that both $U$ and $I$ have zeros below the diagonal, so this is a good start! However, in our current form $\left[ \begin{array}{c|c} U & C \end{array}\right]$ , the diagonal pivots of $U$ are not 1. We need another row operation!

The scaling operation on $N$ in (25) reduces the augmented matrix to

N = \left[ \begin{array}{ccc|ccc} 2 & 6 & 1 & 0 & 1 & 0 \\ 0 & 2 & 1 & 1 & 0 & 0 \\ 0 & 0 & \frac{9}{2} & 1 & -\frac{1}{2} & 1 \end{array}\right] \to\left[ \begin{array}{c|c} V & B \end{array}\right] = \left[ \begin{array}{ccc|ccc} 1 & 3 & \frac{1}{2} & 0 & \frac{1}{2} & 0 \\ 0 & 1 & \frac{1}{2} & \frac{1}{2} & 0 & 0 \\ 0 & 0 & 1 & \frac{2}{9} & -\frac{1}{9} & \frac{2}{9} \end{array}\right],

(26)

where we divide each row by its corresponding pivot. Now, to make $V$ identity, we perform use the same idea as we did in Gaussian Elimination, but to zero out entries above the pivot. In this case, we start with the $(3,3)$ pivot to zero out the $(2, 3)$ and $(1,3)$ entries, and then use the $(2,2)$ pivot to zero out the $(2,1)$ entry.

\left[ \begin{array}{c|c} V & B \end{array}\right] = \left[ \begin{array}{ccc|ccc} 1 & 3 & \frac{1}{2} & 0 & \frac{1}{2} & 0 \\ 0 & 1 & \frac{1}{2} & \frac{1}{2} & 0 & 0 \\ 0 & 0 & 1 & \frac{2}{9} & -\frac{1}{9} & \frac{2}{9} \end{array}\right] \to \left[ \begin{array}{ccc|ccc} 1 & 0 & 0 & -\frac{23}{18} & \frac{7}{18} & \frac{2}{9} \\ 0 & 1 & 0 & \frac{7}{18} & \frac{1}{18} & -\frac{1}{9} \\ 0 & 0 & 1 & \frac{2}{9} & -\frac{1}{9} & \frac{2}{9} \end{array}\right]

(27)

Finally, the right-hand matrix in (27) is the inverse of $A$ :

A^{-1} = \left[ \begin{array}{ccc} -\frac{23}{18} & \frac{7}{18} & \frac{2}{9} \\ \frac{7}{18} & \frac{1}{18} & -\frac{1}{9} \\ \frac{2}{9} & -\frac{1}{9} & \frac{2}{9} \end{array}\right]

(28)

Exercise 3 (Gauss-Jordan Elimination)

Given the matrix

\begin{align*} A = \bm 1&1&1 \\ 1&2&4 \\ 1&3&9 \em \end{align*}

(30)

Find $A^{-1}$ using Gauss-Jordan elimination.

Solution to Exercise 3

Autmenting $A$ to the identity, we get

\begin{align*} \left[ \begin{array}{ccc|ccc} 1 & 1 & 1 & 1 & 0 & 0\\ 1 & 2 & 4 & 0 & 1 & 0\\ 1 & 3 & 9 & 0 & 0 & 1 \end{array}\right] \end{align*}

(31)

Using Row 1 to eliminate Column 1, we get

\begin{align*} \left[ \begin{array}{ccc|ccc} 1 & 1 & 1 & 1 & 0 & 0\\ 0 & 1 & 3 & -1 & 1 & 0\\ 0 & 2 & 8 & -1 & 0 & 1 \end{array}\right] \end{align*}

(32)

Using Row 2 to eliminate Column 2, we get

\begin{align*} \left[ \begin{array}{ccc|ccc} 1 & 0 & -2 & 2 & -1 & 0\\ 0 & 1 & 3 & -1 & 1 & 0\\ 0 & 0 & 2 & 1 & -2 & 1 \end{array}\right] \end{align*}

(33)

Using Row 3 to eliminate Column 3, we get

\begin{align*} \left[ \begin{array}{ccc|ccc} 1 & 0 & 0 & 3 & -3 & 1\\ 0 & 1 & 0 & -2.5 & 4 & -1.5\\ 0 & 0 & 2 & 1 & -2 & 1 \end{array}\right] \end{align*}

(34)

Scaling Row 3 so it’s entry is one,

\begin{align*} \left[ \begin{array}{ccc|ccc} 1 & 0 & 0 & 3 & -3 & 1\\ 0 & 1 & 0 & -2.5 & 4 & -1.5\\ 0 & 0 & 1 & 0.5 & -1 & 0.5 \end{array}\right] \end{align*}

(35)

Hence we have found that

\begin{align*} A^{-1} = \bm 3 & -3 & 1\\ -2.5 & 4 & -1.5\\0.5 & -1 & 0.5 \em \end{align*}

(36)