3.1 Inner Products and Induced Norms

0.1Reading¶

Material related to this page, as well as additional exercises, can be found in ALA Ch. 3.1.

0.2Learning Objectives¶

By the end of this page, you should know:

an inner product on a vector space
a norm induced by an inner product
some important examples of inner products

0.3Introduction¶

In the last two lectures, we generalized how we add and scale vectors in $\mathbb{R}^2$ and $\mathbb{R}^3$ to general Euclidean spaces $\mathbb{R}^n$ , and more general vector spaces $V$ . In this lecture, we bring over other key concepts from $\mathbb{R}^2$ and $\mathbb{R}^3$ to vector spaces, namely the ideas of angle, length, and distance.

The notions of angle, length, and distance in general vector spaces play a foundational role in modern applications of engineering, economics, and AI. By the end of the next few lectures, you will be equipped with both conceptual and computational tools that will allow you to solve some really interesting problems with immediate real-world application!

1The Dot Product and the Euclidean Norm¶

We start with a familiar example of an inner-product for vectors in $\mathbb{R}^n$ , the dot product:

The Pythagorean theorem extends to $n-$ dimensional space and tells us that $\vv v \cdot \vv v$ is the square of the length of $\vv v$ . We use this observation to define the Euclidean norm (or length):

In $\mathbb{R}^2$ , the formula for Euclidean norm looks a lot like the familiar Pythagorean theorem! This generalizes our idea of length from $\mathbb{R}^2$ and $\mathbb{R}^3$ to $\mathbb{R}^n$ .

Python break!¶

In NumPy, the dot product of a vector v1 with another vector v2 is given by np.dot(v1, v2) as illustrated below.

# Dot product

import numpy as np

v1 = np.array([1, 2, 3])
v2 = np.array([4, 5, 6])
dot_prod = np.dot(v1, v2)
print("Dot product <v1, v2> is: ", dot_prod)

Dot product <v1, v2> is:  32

1.1Inner Products¶

The Eucledian norm $\|\vv v\|$ of a vector $\vv v$ has some intuitive properties. For example, if $\vv v \neq 0$ , then $\|\vv v\| > 0$ (all nonzero vectors have positive length), and $\|\vv v\| = 0$ if and only if $\vv v = 0$ (only the zero vector has zero length).

These properties, and those of the dot product, inspire the following abstract definition for more general inner-products:

Definition 3 (Inner Products)

An inner product on the real vector space $V$ is a pairing that takes two vectors $\vv v, \vv w \in V$ and produces a real number $\langle \vv v, \vv w\rangle \in \mathbb{R}$ . The inner product is required to satisfy the following three axioms for all $\vv u, \vv v, \vv w \in V$ , and scalars $c, d \in \mathbb{R}$ .

Bilinearity

\begin{align*} \langle c \vv u + d \vv v, \vv w \rangle &= c \langle \vv u, \vv w \rangle + d \langle \vv v, \vv w \rangle, \\ \langle \vv u, c \vv v + d \vv w \rangle &= c \langle \vv u, \vv v \rangle + d \langle \vv u, \vv w \rangle. \end{align*}

(5)

Symmetry

\begin{align*} \langle \vv v , \vv w \rangle &= \langle \vv w , \vv v \rangle. \end{align*}

(6)

Positivity

\langle \vv v , \vv v \rangle > 0 \ \textrm{whenever} \ \vv v \neq \vv 0, \ \textrm{while} \ \langle \vv 0, \vv 0 \rangle = 0.

(7)

As we will see soon, an inner product allows us to define notions of angle, length, and distance in a vector space. This added structure is very useful, so when a vector space is equipped with an inner product, we call it an inner product space.

You can (and should!) check that the dot product satisfies Definition 3.

Example 1 (Weighted dot product)

Let’s define a different inner product on $\mathbb{R}^2$ . Instead of the typical dot product, let’s consider a weighted dot product:

\begin{align*} \langle \vv v, \vv w \rangle = 2v_1w_1 + 5v_2w_2 \end{align*}

(9)

for $\vv v = \bm v_1 \\ v_2 \em,\vv w = \bm w_1 \\ w_2\em$ . Let’s verify that this is indeed an inner product by checking that it satisfies the 3 axioms:

Bilinearity. For any scalars $c, d$ and vectors $\vv u, \vv v, \vv w$ ,

\begin{align*} \langle c \vv u + d \vv v, \vv w\rangle &= 2(c u_1 + d v_1)w_1 + 5(c u_2 + d v_2) w_2\\ &= c(2 u_1 w_1 + 5 u_2w_2) + d(2 v_1w_1 + 5 v_2 w_2)\\ &= c \langle \vv u, \vv w \rangle + d\langle \vv v, \vv w \rangle \end{align*}

(10)

Hence $\langle \cdot, \cdot \rangle$ is linear in its first argument when the second one is fixed. Try to prove that $\langle \cdot, \cdot \rangle$ is linear in its second argument when the first one is fixed (the proof should look very similar!). This will show bilinearity.

Symmetry.

\begin{align*} \langle \vv u, \vv v\rangle &= 2 u_1 v_1 + 5 u_2v_2 \\ &= 2 v_1 u_1 + 5 v_2u_2\\ &= \langle \vv v, \vv u\rangle \end{align*}

(11)

This shows symmetry.

Positivity.

\begin{align*} \langle \vv v, \vv v\rangle &= 2 v_1^2 + 5 v_2^2 \end{align*}

(12)

If $\vv v = \vv 0$ , then clearly this expression is zero because all terms are zero. On the other hand, if either component of $\vv v$ is nonzero, then this expression is strictly positive. This shows positivity.

Therefore, the weighted dot product above is indeed an inner product.

The weighted norm induced by the weighted dot product on $\mathbb{R}^2$ is then

\|\vv v\| = \sqrt{ \langle \vv v, \vv v\rangle} = \sqrt{2 v_1^2 + 5 v_2^2}

(13)

which assigns more weights to the second coordinate relative to the first.

We can generalize Example 1 to $\mathbb{R}^n$ and arbitrary positive weights. Let $c_1, ..., c_n > 0$ we positive numbers. Then the corresponding weighted inner product and weighted norm on $\mathbb{R}^n$ is defined to be

\begin{align*} \langle \vv v, \vv w\rangle = c_1v_1w_1 + ... + c_nv_nw_n \\ \|\vv v\| = \sqrt{ \langle \vv v, \vv v\rangle} = \sqrt{c_1v_1^2 + c_2v_2^2 + ... + c_nv_n^2} \end{align*}

(14)

The numbers $c_i > 0$ are called the weights.

Example 2 (An inner product over a continuous function space)

We saw before that we can define vector spaces where vectors are doubly infinite sequences or even functions! We will not work with such functions spaces in the rest of the class, but you should know what we can define inner products on these vector spaces too!

For example, recall we saw that $C^0[0, 1]$ , the space of all continuous functions defined over the interval $[0, 1]$ , is a vector space. A commonly used inner product on this space is

\begin{align*} \langle f, g\rangle = \int_{0}^{1}{f(t)g(t) \:dt} \end{align*}

(15)

for any two functions $f, g \in C^0[0, 1]$ . One can verify that this inner product satisfies this definition; we won’t do that here but it isn’t hard (just remember the properties of integration)!

For some intuition, let’s consider the sampled function versions $\vv f, \vv g \in \mathbb{R}^{T+1}$ , where remember, we define

\begin{align*} \vv f = \bm f(0) \\ f(\tau) \\ \vdots \\ f(T\tau) \em = \bm f_1 \\ f_2 \\ \vdots \\ f_{T+1} \em, \quad \vv g = \bm g(0) \\ g(\tau) \\ \vdots \\ g(T\tau) \em = \bm g_1 \\ g_2 \\ \vdots \\ g_{T+1} \em \end{align*}

(16)

for τ a sampling time chosen so that $T\tau = 1$ , These two vectors live in $\mathbb{R}^{T+1}$ , and let’s consider a weighted inner product with all weights $c_i = \tau$ . Then the inner product between $\vv f$ and $\vv g$ is

\begin{align*} \langle \vv f, \vv g\rangle &= \tau f_1g_1 + \tau f_1g_1 + ... + \tau f_{T+1}g_{T+1}\\ &= \sum_{t = 0}^{T}{\tau f(i\tau) g(i\tau)} \end{align*}

(17)

This should remind your of how the Riemann integral for $h(t) = f(t)g(t)$ is defined. Indeed, if we let $\tau \to 0$ , we recover the integral inner product defined above. Since our inner product $\langle f, g\rangle$ is an inner product for any τ, it shouldn’t be too surprising that the ntegral inner product (Int) does too.

Int then defines the $L_2$ norm of a function:

\|\vv f\| = \sqrt{\int_0^1 f(t)^2 dt}

(18)

which generalizes the notion of length to functions. These ideas might seem very abstract, but they are immensely practical and lie at the heart of modern applications of Fourier analysis, differential equations, and numerical analysis.

Python break!¶

Let’s use the np.inner function in NumPy to compute the inner product between vectors and notice how it is different from the np.dot function. We also compute weighted inner products. Then, we use np.linalg.norm to directly compute the Eucledian norm of a vector and compare it with the induced norm of the corresponding inner product (dot product).

# Inner product

import numpy as np

v1_c = v1.reshape((-1,1)) # Notice the shape of the vectors
v2_c = v2.reshape((-1,1))
print("The vectors are v1: \n", v1_c, "\n and v2: \n", v2_c)
inner_prod = np.inner(v1_c, v2_c) # What happens if you use np.dot?  
print("Inner product <v1, v2> is: \n", inner_prod)

# weighted inner product

weights = np.array([2, 5, 3])
inner_weighted = np.inner(v1, weights*v2)
print("Weighted inner product <v1, v2> is: \n", inner_weighted)

The vectors are v1: 
 [[1]
 [2]
 [3]] 
 and v2: 
 [[4]
 [5]
 [6]]
Inner product <v1, v2> is: 
 [[ 4  5  6]
 [ 8 10 12]
 [12 15 18]]
Weighted inner product <v1, v2> is: 
 112

# Norms

print("Norm of v1: ", np.linalg.norm(v1))
print("Induced norm of v1: ", np.sqrt(np.dot(v1, v1)))

Norm of v1:  3.7416573867739413
Induced norm of v1:  3.7416573867739413