Essence Of Linear Algebra (3Blue1Brown Inspired)

To locate a point on a map, we can count steps east and north. We could also count steps along two diagonal directions. The destination stays the same, but the numbers change. How do we translate between these descriptions—and can choosing different directions make a mathematical operation easier to understand?

We will follow one arrow through both descriptions, build the matrix that translates between them, and then examine an operation that stretches the plane. Its special directions will give us a coordinate system in which that stretching becomes especially simple.

An original illustrated companion to Grant Sanderson’s 3Blue1Brown videos on change of basis and eigenvectors and eigenvalues. This tutorial focuses on those two topics; the linked series develops the wider subject.

One arrow, two descriptions

Start at the origin and move three units right, then one unit up. Draw an arrow from the origin to that destination; call it v. The two numbers (3, 1) tell us how many of each step to take.

The rightward unit step and upward unit step are our basis vectors, e₁ and e₂. A basis is a set of vectors that lets us describe every vector in the space in exactly one way by scaling and adding them. Here, scaling a step means changing its length; a negative multiplier reverses its direction.

e_1=\begin{pmatrix}1\\0\end{pmatrix},\qquad e_2=\begin{pmatrix}0\\1\end{pmatrix}

Now use a step one unit right and one unit up, b₁ = (1, 1), and a step one unit right and one unit down, b₂ = (1, −1). Two b₁ steps reach (2, 2); adding one b₂ step reaches (3, 1). Follow the colored paths:

Horizontal and vertical steps

3 right steps + 1 up step. Coordinates: (3, 1).

Diagonal steps

2 up-right steps + 1 down-right step. Coordinates: (2, 1). The gold arrow stays fixed.

Both paths reach the same gold arrow. Multiplying vectors by numbers and adding them is called a linear combination. The numbers in that combination are the vector’s coordinates in the chosen basis:

v=3e_1+1e_2=2b_1+1b_2

We will write x for the column of standard coordinates, and c for the column of diagonal coordinates. The basis tells us what each entry multiplies:

Standard coordinates · multiply e₁, e₂

x=\begin{pmatrix}3\\1\end{pmatrix}

Diagonal coordinates · multiply b₁, b₂

c=\begin{pmatrix}2\\1\end{pmatrix}

The diagonal vectors are themselves (1, 0) and (0, 1) in their own basis: one copy of the first step, or one copy of the second. Their standard coordinates tell us where those steps lead on the original grid. This is the information we need to translate.

Put the new steps into the columns

Suppose we receive the diagonal coordinates c = (2, 1). To find the standard coordinates, take two copies of b₁ and one copy of b₂, using their standard entries:

2\begin{pmatrix}1\\1\end{pmatrix}+1\begin{pmatrix}1\\-1\end{pmatrix}=\begin{pmatrix}2+1\\2-1\end{pmatrix}=\begin{pmatrix}3\\1\end{pmatrix}

We need to perform this same operation for any pair of diagonal coordinates. Put b₁ in the first column of an array and b₂ in the second. Multiplying each column by the corresponding input coordinate, then adding, is matrix–vector multiplication. Call this matrix A:

A=\left[\underbrace{\begin{matrix}1\\1\end{matrix}}_{b_1}\;\underbrace{\begin{matrix}1\\-1\end{matrix}}_{b_2}\right],\qquad Ac=x

A takes diagonal coordinates in and gives standard coordinates out. Its columns are the new basis vectors written in the standard basis. Try translating other coordinates with those same columns:

First diagonal coordinate c₁: 2Second diagonal coordinate c₂: 1

Cyan contributes c₁b₁; blue contributes c₂b₂. Their sum is the gold vector.

\begin{pmatrix}1&1\\1&-1\end{pmatrix}\begin{pmatrix}2\\1\end{pmatrix}=\begin{pmatrix}3\\1\end{pmatrix}

Input: diagonal coordinates.
Output: horizontal/vertical coordinates.

Choose another vector by changing its diagonal coordinates. The basis and translation matrix stay fixed.

To translate in the other direction, we know x and need c. Write the standard coordinates as (x₁, x₂). The first row says x₁ = c₁ + c₂; the second says x₂ = c₁ − c₂. Add the equations to isolate c₁; subtract them to isolate c₂:

\begin{aligned}x_1+x_2&=2c_1&\quad c_1&=\frac{x_1+x_2}{2}\\x_1-x_2&=2c_2&c_2&=\frac{x_1-x_2}{2}\end{aligned}

These two rules form the matrix that undoes A: its inverse, written A⁻¹. Using it on our original arrow recovers the two diagonal coordinates:

\begin{gathered}A^{-1}=\frac12\begin{pmatrix}1&1\\1&-1\end{pmatrix}\\c=A^{-1}x=\frac12\begin{pmatrix}3+1\\3-1\end{pmatrix}=\begin{pmatrix}2\\1\end{pmatrix}\end{gathered}

A translates the diagonal description into the standard description; A⁻¹ translates back. Describing a fixed vector in a different basis is a change of basis. We can do this because the two chosen steps are independent: neither is a multiple of the other, so together they give unique coordinates throughout the plane.

Describe an operation by what it does to two steps

We can now translate descriptions of a fixed arrow. Our next goal is to describe an operation that changes the arrow itself. Suppose the rightward unit step is sent to (2, 1), and the upward unit step is sent to (1, 2).

Require the operation to preserve scaling and addition: transforming three copies of a vector gives three copies of its transformed vector, and transforming a sum gives the sum of the transformed vectors. Such an operation is a linear transformation. Once we know where the two basis vectors go, this rule tells us where every vector goes.

Our vector is 3e₁ + e₂. Its image must therefore be three copies of the image of e₁, plus one copy of the image of e₂. Package those two images as the columns of a matrix M:

\begin{gathered}M=\begin{pmatrix}2&1\\1&2\end{pmatrix}\\Mx=3\begin{pmatrix}2\\1\end{pmatrix}+1\begin{pmatrix}1\\2\end{pmatrix}=\begin{pmatrix}7\\5\end{pmatrix}\end{gathered}

Move from the input to its image: 0%

At 100%, M sends:

Cyan: e₁ = (1, 0) → (2, 1).
Blue: e₂ = (0, 1) → (1, 2).
Gold: v = (3, 1) → (7, 5).

The faint square grid is the fixed standard coordinate system. The colored grid follows the transformed steps.

The slider traces straight paths between the original and transformed positions. Only its 100% endpoint is the transformation M.

Here M takes standard coordinates of the input and returns standard coordinates of its image. The gold arrow changes from (3, 1) to (7, 5). The coordinate language stays fixed.

Why can A also look like it moves the grid?

The same numerical multiplication can describe a translation of coordinates or a transformation of vectors. The interpretation depends on what the input and output numbers refer to.

Read (2, 1) as diagonal coordinates: it already describes our gold arrow. A gives (3, 1), its standard description. Now read (2, 1) as standard coordinates: it describes a different arrow, two units right and one up. Applying A as a transformation moves that arrow to (3, 1).

In the second interpretation, A sends e₁ to b₁ and e₂ to b₂, so a moving-grid animation takes the standard grid into the diagonal grid. In the first interpretation, those same columns translate diagonal coordinates into standard coordinates. Labeling the coordinate language at each end keeps the two uses clear.

M mixes the standard coordinates: each output entry uses both input entries. Could another choice of basis expose a simpler description of this operation?

Find the directions that only get scaled

For M to act simply along a chosen direction, a vector pointing that way must come out along the same line through the origin. Then one number can describe the effect: how much that vector was scaled. Keep M fixed and turn the input arrow to look for these directions:

Direction of the input arrow: 0°

Gold: input (1, 0).
Pink ring: output (2, 1).

The output has left the dashed input line. Scaling the input alone cannot reach it.

Use the two direction buttons to find the lines where M acts by scaling alone.

M stays fixed. This control changes which direction we examine; the input arrow always has length one.

The up-right and down-right directions have exactly this property. Check the diagonal steps we already know by multiplying them by M:

Along b₁ · three times as long

\begin{aligned}Mb_1&=\begin{pmatrix}2+1\\1+2\end{pmatrix}\\&=\begin{pmatrix}3\\3\end{pmatrix}=3b_1\end{aligned}

Along b₂ · unchanged

\begin{aligned}Mb_2&=\begin{pmatrix}2-1\\1-2\end{pmatrix}\\&=\begin{pmatrix}1\\-1\end{pmatrix}=1b_2\end{aligned}

A nonzero vector that a transformation scales this way is an eigenvector. Its scale factor is the corresponding eigenvalue, usually written λ. For a candidate vector w, the condition is:

Mw=\lambda w,\qquad w\ne0

For this M, b₁ has eigenvalue 3 and b₂ has eigenvalue 1. Every nonzero multiple of either vector has the same eigenvalue. In other transformations an eigenvalue can be negative, reversing the arrow along its line, or zero, sending it to the origin.

Find those directions algebraically

The picture suggested two directions. To find all the possibilities directly from M, write the unknown vector as w = (p, q). The eigenvector condition asks for a common factor λ in both entries:

\begin{aligned}2p+q&=\lambda p\\p+2q&=\lambda q\end{aligned}

Move the right sides to the left. The first equation gives q = (λ − 2)p. Substitute it into the second:

\begin{gathered}(2-\lambda)p+q=0\\p+(2-\lambda)q=0\\\bigl[1-(\lambda-2)^2\bigr]p=0\end{gathered}

If p were zero, the first equation would also make q zero. Since an eigenvector must be nonzero, the bracket must vanish. Thus λ − 2 = ±1, giving λ = 3 or λ = 1. Substituting back gives q = p or q = −p: the two diagonal lines.

There is a geometric way to package the same calculation. Subtract λ times the original vector from M’s output; we want the remainder to be zero. The matrix I with ones on its diagonal and zeros elsewhere leaves a vector unchanged, so it is called the identity matrix. Our condition becomes:

\begin{gathered}(M-\lambda I)w=0\\M-\lambda I=\begin{pmatrix}2-\lambda&1\\1&2-\lambda\end{pmatrix}\end{gathered}

For this modified matrix to send a nonzero vector to zero, its two columns must be dependent: their contributions can cancel. Here is what that looks like at λ = 3:

For λ = 3, the columns of M − λI point along one line. Their sum is zero: the input b₁ = (1, 1) disappears under this modified matrix.

The area enclosed by the two output columns then vanishes. A matrix’s signed area scale in two dimensions is its determinant. For a two-by-two matrix, multiply the top-left and bottom-right entries, then subtract the product of the other two entries. Requiring zero area gives the same two values:

\begin{aligned}\det(M-\lambda I)&=(2-\lambda)^2-1\\&=(\lambda-3)(\lambda-1)=0\end{aligned}

The determinant locates the possible scale factors. Solving (M − λI)w = 0 for each one finds the vectors with that factor.

We now have a reason to choose the diagonal basis: each of its vectors changes independently by a single multiplier. The next step is to express the whole transformation in those coordinates.

Translate in, transform, translate back

Suppose our input is given in diagonal coordinates and we want the transformed output in diagonal coordinates too. M currently accepts standard coordinates. We already have both translations needed to use it:

1 · Translate with A

Diagonal input → standard input.

\begin{pmatrix}2\\1\end{pmatrix}\longmapsto\begin{pmatrix}3\\1\end{pmatrix}

2 · Transform with M

Standard input → standard output.

\begin{pmatrix}3\\1\end{pmatrix}\longmapsto\begin{pmatrix}7\\5\end{pmatrix}

3 · Translate with A⁻¹

Standard output → diagonal output.

\begin{pmatrix}7\\5\end{pmatrix}\longmapsto\begin{pmatrix}6\\1\end{pmatrix}

Write c′ for the diagonal coordinates after the transformation. Applying the three operations in this order gives c′ = A⁻¹(M(Ac)). In a matrix product the rightmost matrix acts first. The combined matrix is M expressed in the diagonal basis.

Its first column describes the image of b₁ in that basis: three copies of b₁ and zero copies of b₂, giving (3, 0). Its second column describes the image of b₂: (0, 1). A matrix with entries only on its main diagonal is a diagonal matrix. Call this one D:

\underbrace{D}_{\text{M in the new basis}}=A^{-1}MA=\begin{pmatrix}3&0\\0&1\end{pmatrix}

c'=Dc=\begin{pmatrix}3&0\\0&1\end{pmatrix}\begin{pmatrix}2\\1\end{pmatrix}=\begin{pmatrix}6\\1\end{pmatrix}

The original vector was 2b₁ + b₂. Its image is 6b₁ + b₂, which has standard coordinates (7, 5). We can now read the operation as “triple the first diagonal coordinate; keep the second.” A basis made of eigenvectors is an eigenbasis; expressing the transformation in it is diagonalization.

What this buys us, and when it is possible

Apply M twice: the b₁ contribution is multiplied by 3 twice, while the b₂ contribution stays fixed. After n applications, the diagonal coordinates are (3ⁿc₁, c₂). To express the answer in standard coordinates, translate back with A:

M^n=A\begin{pmatrix}3^n&0\\0&1\end{pmatrix}A^{-1}

In two dimensions, this requires two independent eigenvectors. A 90° rotation has no real eigenvector: every nonzero real arrow leaves its original line. A shear can have just one line of eigenvectors, which also cannot provide a basis for the plane. Our example has the two independent directions required.

We have used the inverse to translate back. A particularly convenient case arises when the new basis vectors are perpendicular and each has length one. That is the form used in PCA.

Make each coordinate a signed distance

Our diagonal step (1, 1) has length √(1² + 1²) = √2. One diagonal coordinate unit therefore travels √2 ordinary distance units. To make a coordinate directly measure distance along its axis, shorten each diagonal step to length one by dividing by √2.

\begin{gathered}u_1=\frac{b_1}{\sqrt2},\qquad u_2=\frac{b_2}{\sqrt2}\\Q=[u_1\;u_2]\end{gathered}

The shorter arrows below point along the same two directions. Each now travels one ordinary distance unit:

The dashed extensions reach b₁ and b₂. The solid arrows u₁ and u₂ stop at the unit circle. The same vector needs more of these shorter steps.

The directions stay perpendicular. Perpendicular unit vectors form an orthonormal basis. Call our vector’s coordinates in this normalized basis s. Because each step is shorter, we need √2 times as many steps along each direction:

\begin{gathered}v=2b_1+b_2=(2\sqrt2)u_1+(\sqrt2)u_2\\s=\begin{pmatrix}2\sqrt2\\\sqrt2\end{pmatrix},\qquad x=Qs\end{gathered}

To recover s from x, multiply matching entries of x and u₁ and add them. This operation, the dot product, reads the signed distance along the unit direction u₁. It keeps the u₁ contribution and removes the perpendicular u₂ contribution. Do the same with u₂ for the second coordinate.

Turning each column uᵢ into a row lets row-by-column multiplication perform that dot product. Swapping rows and columns is the transpose, written Qᵀ:

Columns build the vector

Q=\left[\;u_1\;u_2\;\right],\qquad x=Qs

Rows read its coordinates

Q^\top=\begin{pmatrix}u_1^\top\\u_2^\top\end{pmatrix},\qquad s=Q^\top x

\begin{aligned}s_1&=\begin{pmatrix}1/\sqrt2&1/\sqrt2\end{pmatrix}\begin{pmatrix}3\\1\end{pmatrix}\\&=\frac4{\sqrt2}=2\sqrt2\\s_2&=\begin{pmatrix}1/\sqrt2&-1/\sqrt2\end{pmatrix}\begin{pmatrix}3\\1\end{pmatrix}\\&=\frac2{\sqrt2}=\sqrt2\end{aligned}

Why the transpose is the inverse here

Multiplying Qᵀ by Q pairs each row with each column. A unit vector dotted with itself gives 1; perpendicular vectors give 0. The product is the identity matrix, so Qᵀ undoes Q:

\begin{aligned}Q^\top Q&=\begin{pmatrix}u_1^\top u_1&u_1^\top u_2\\u_2^\top u_1&u_2^\top u_2\end{pmatrix}\\&=\begin{pmatrix}1&0\\0&1\end{pmatrix}=I\end{aligned}

Thus Q⁻¹ = Qᵀ. For our earlier, unnormalized A, each column has squared length 2, so AᵀA = 2I. The inverse needs the extra factor 1/2 we derived earlier. In general, translation back uses the inverse; orthonormal columns make the transpose perform that operation.

Shortening an eigenvector leaves its eigenvalue unchanged: M(b₁/√2) = 3(b₁/√2). Consequently, the normalized basis gives the same simple description of M:

Q^\top M Q=D,\qquad M=Q D Q^\top

Reading the second equation from right to left: translate into the eigenbasis, scale its coordinates, then translate back. For a real symmetric matrix—one whose entries match across the diagonal—an orthonormal eigenbasis always exists. Expressing it as QDQᵀ is its spectral decomposition.

In principal component analysis, the matrix being studied is the covariance matrix, which summarizes how the measurements vary together. It is real symmetric. Its unit eigenvectors supply Q; their eigenvalues give the variance along those directions. Each centered observation receives coordinates through Qᵀ, and Q rebuilds it from those coordinates.

Symbols and sources

v: The running geometric vector, three units right and one up.
e₁, e₂; b₁, b₂: Standard unit steps; diagonal steps (1, 1) and (1, −1), written in standard coordinates.
x; c; s: Coordinates in the standard, diagonal, and normalized diagonal bases, respectively.
x₁, x₂; c₁, c₂; s₁, s₂: The two entries of each coordinate column.
A; A⁻¹: Diagonal-to-standard translation; its inverse.
M; D; c′: The transformation in standard coordinates; the same transformation in its eigenbasis; transformed diagonal coordinates.
w = (p, q); λ: A candidate nonzero eigenvector in standard coordinates; its eigenvalue.
I; det: Identity matrix; determinant (signed area scale in two dimensions).
u₁, u₂; Q; Qᵀ: Normalized diagonal vectors; matrix with those vectors as columns; its transpose.
n: Number of repeated applications of a transformation.

Conceptual inspiration: Grant Sanderson / 3Blue1Brown, Essence of Linear Algebra: Change of basis and Eigenvectors and eigenvalues. The explanations, running example and interactive diagrams on this page were created for this tutorial.