Linear Algebra 1: Basics#

Vectors, what are they?#

Vectors as Arrays of Numbers#

We will define vector as an ordered collection of numbers which describes states of physical systems.

  • \(a = (-2, 8)\): A 2D vector.

  • \(b = (1.34, 4.23, 5.98)\): A 3D vector.

  • \(c = (1, -2, 4i, 3 + 2i)\): A 4D vector with complex components.

  • \(f = (1, 2, 3, 4, 5, 6, \ldots, \infty)\): An infinite-dimensional vector with integer components.

  • Note: Vectors can belong to real or complex spaces, depending on their components.

Vectors Defined with Respect to a Basis#

  • In classical physics, vectors are often visualized as arrows in space, with both magnitude and direction. A vector is typically defined with respect to a basis, where unit vectors (\(\vec{e_i}\)) span the space. For instance, in 3D Euclidean space, the standard basis is:

    \[e_1 = (1, 0, 0), \,\, e_2 = (0, 1, 0), \,\, e_3 = (0, 0, 1)\]

  • Here is an example of vector \(v\) in terms of unit vectors of euclidean space.

    \[v = (2,3,4) = 2e_1+3e_2+4e_3\]

  • In a different coordinate system or basis, the same vector will have different components, but its intrinsic properties remain unchanged. This flexibility in representation is a fundamental concept in quantum mechanics and linear algebra.

Row vectors vs column vector#

  • We will defined row and column vectors which are related to one another by operation of transpose. They describe the same physical state or coordinate of a system.

\[\begin{split} \begin{pmatrix} a_1 \\ a_2 \end{pmatrix} = (a_1, a_2)^{T} \end{split}\]

  • However row vectors live with their own universe of row vectors and column vectors in their own universe of column vectors.

  • Later we will see that to take scalar or dot product row vector needs to be hit with column and vice versa.

Example: Vector expressed in two different basis#

Let’s take a 2D vector \(\mathbf{v}\) in the standard Cartesian unit basis:

\[\begin{split} \mathbf{v} = \begin{pmatrix} 2 \\ 3 \end{pmatrix} \end{split}\]
\[\begin{split} \mathbf{v} = 2 \mathbf{e_1} + 3 \mathbf{e_2} = 2\begin{pmatrix} 1 \\ 0 \end{pmatrix}+3\begin{pmatrix} 0 \\ 1 \end{pmatrix} \end{split}\]

Now, suppose we switch to a new basis, where each new basis vector is a scalar multiple of the standard basis vectors. Let’s define the new basis vectors as:

\[\begin{split} \mathbf{b_1} = 2 \mathbf{e_1} = \begin{pmatrix} 2 \\ 0 \end{pmatrix}, \quad \mathbf{b_2} = 2 \mathbf{e_2} = \begin{pmatrix} 0 \\ 2 \end{pmatrix} \end{split}\]

in the new basis, the vector \(\mathbf{v}\) is represented as:

\[ \mathbf{v}_{\text{new}} = 1 \mathbf{b_1} + 3/2 \mathbf{b_2} \]
\[\begin{split} \mathbf{v}_{\text{new}} = \begin{pmatrix} 1 \\ \frac{3}{2} \end{pmatrix} \end{split}\]

  • This shows that when moving to a new basis the components of the vector change accordingly, but the vector itself remains the same geometrically!

Vector Notation#

Depending on the context, we may emphasize the basis or omit it when the basis is implied:

  • In terms of a coordinate basis:

    \[\vec{a} = 2\vec{e_i} + 3\vec{e_j}\]

  • As an ordered tuple of components:

    \[a = (2, 3)\]

  • Using Dirac notation (common in quantum mechanics):

    \[\mid a \rangle = 2 \mid e_i \rangle + 3 \mid e_j \rangle\]

  • Vectors can represent a wide range of phenomena, such as the position of a particle in space, the population of countries, or temperature variations in different regions of a forest.

Vector Operations#

What defines vectors in a mathematical framework are the operations performed on them. Let’s illustrate these operations using a simple 2D vectors \(a = (3, 2)\) and \(b=(1,1)\) as an example.

1. Addition or Subtraction#

  • Component-wise addition:

    \[\begin{split}a + b = \begin{pmatrix} 2\\ 3 \end{pmatrix} + \begin{pmatrix} 1\\ 1 \end{pmatrix} = \begin{pmatrix} 2+1\\ 3+1 \end{pmatrix} = \begin{pmatrix} 3\\ 4 \end{pmatrix}\end{split}\]

  • In Dirac notation:

    \[|a \rangle + |b \rangle = (2 + 1)\mid e_1 \rangle + (3 + 1)\mid e_2 \rangle\]

2. Multiplication by a Scalar#

Multiplying a vector by a scalar \(\beta=10\) scales the vector’s magnitude without changing its direction.

  • Component-wise scalar multiplication:

    \[\begin{split}10\cdot a = 10 \begin{pmatrix} 2\\ 3 \end{pmatrix} = \begin{pmatrix} 20\\ 30 \end{pmatrix}\end{split}\]

  • In Dirac notation:

    \[10\mid a \rangle = 20 \mid e_1 \rangle + 30 \mid e_2 \rangle\]

System of Linear Equations as Matrix Operations#

A system of linear equations can be interpreted as a matrix operating on a vector to produce another vector. For example, consider the system:

\[\begin{split} \begin{aligned} a_{11}x_1 + a_{12}x_2 &= b_1 \\ a_{21}x_1 + a_{22}x_2 &= b_2 \end{aligned} \end{split}\]

This system can be written compactly in matrix form as:

A system of linear equation as matrix vector product

\[ A \mathbf{x} = \mathbf{b} \]

  • \(A\) is a matrix containing the coefficients of the system,

  • \(\mathbf{x}\) is the vector of unknowns, and

  • \(\mathbf{b}\) is the vector of constants (right-hand side).

For a 2D system, this becomes:

\[\begin{split} \begin{pmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{pmatrix} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = \begin{pmatrix} b_1 \\ b_2 \end{pmatrix} \end{split}\]

Matrix vector Multiplication#

Row by column product rule

Matrix multiplication operates by taking the dot product of the rows of the matrix \(A\) with the vector \(\mathbf{x}\). For the above example:

\[\begin{split}A \mathbf{x} = \begin{pmatrix} a_{11}x_1 + a_{12}x_2 \\ a_{21}x_1 + a_{22}x_2 \end{pmatrix} \end{split}\]

This shows how the system of equations is equivalent to multiplying the matrix by the vector.

Lower level definition of Matrix Multiplication

In general, if \(A\) is an \(m \times n\) matrix and \(\mathbf{x}\) is a vector of dimension \(n\), the matrix-vector multiplication \(A \mathbf{x}\) is defined as:

\[ b_i = (A \mathbf{x})_i = \sum_{j=1}^{n} a_{ij} x_j \]

Where \(a_{ij}\) are the elements of matrix \(A\), and \(x_j\) are the components of vector \(\mathbf{x}\).

Solving linear equation via matrix Inversion

\[{\bf x} = A^{-1}{\bf b}\]
Hide code cell source 
import numpy as np
# Define the coefficients matrix A and the right-hand side vector b
print('Solving Ax=b via matrix inversion')

A = np.array([[2, 3],
              [1, -2]])

b = np.array([7, 1])

print('A', A)
print('b', b)

# Calculate the inverse of matrix A
A_inv = np.linalg.inv(A)

# Solve for the unknown vector x using matrix inversion: x = A_inv * b
x = np.dot(A_inv, b)

print("Solution using matrix inversion:")
print("x =", x)

Solving Ax=b via matrix inversion
A [[ 2  3]
 [ 1 -2]]
b [7 1]
Solution using matrix inversion:
x = [2.42857143 0.71428571]

Visualizing Action of a Matrix on a Vector#

These transformations illustrate different ways of modifying vectors and geometric shapes through scaling, shearing, and identity operations. We use a 2D vector and matrices as an example.

  1. Shearing and Scaling. \(\begin{bmatrix} 2 & 1.5 \\ 0.5 & 0.75 \end{bmatrix}\)

    • This matrix scales the \(x\)-direction by 2 and the \(y\)-direction by 0.75. Additionally, it applies both horizontal and vertical shear, distorting the object by shifting the \(x\)-coordinate in proportion to the \(y\)-coordinate and the \(y\)-coordinate in proportion to the \(x\)-coordinate. This results in a combination of scaling and shearing, altering both the size and shape of objects.

  2. Stretch. \(\begin{bmatrix} 2 & 0 \\ 0 & 1.5 \end{bmatrix}\)

    • This matrix stretches the \(x\)-direction by 2 and compresses \(y\)-direction by 1.5. It changes the size of the object, expanding it in the \(x\)-direction \(y\)-direction, while keeping its shape intact without any angular distortion.

  3. Stretch/compress. \(\begin{bmatrix} 3 & 0 \\ 0 & 0.75 \end{bmatrix}\)

    • This matrix stretches the object by a factor of 3 along the \(x\)-axis and compresses it by a factor of 0.75 along the \(y\)-axis. This operation changes the proportions of the object without distorting angles, stretching it non-uniformly.

  4. Identity (No Change). \(\begin{bmatrix} 1 & 0 \\ 0 & 1 \end{bmatrix}\)

    • The identity matrix leaves the object unchanged. Applying this transformation results in the same object without any alteration to its size, shape, or orientation.

Hide code cell source 
import numpy as np
import matplotlib.pyplot as plt

# Define the original vector
vector = np.array([2, 3])

# Define the transformation matrices
shear_scale_matrix = np.array([[2, 1], [0, 0.5]])
scaling_matrix = np.array([[2, 0], [0, 1.5]])
stretching_matrix = np.array([[3, 0], [0, 0.75]])
identity_matrix = np.eye(2)

# Apply the transformations
transformed_vector = shear_scale_matrix @ vector
scaled_vector = scaling_matrix @ vector
stretched_vector = stretching_matrix @ vector
identity_vector = identity_matrix @ vector

# Create subplots for the four transformations
fig, axs = plt.subplots(2, 2, figsize=(12, 12))

# Plot 1: Shearing and Scaling Transformation
axs[0, 0].axhline(0, color='black', linewidth=0.5)
axs[0, 0].axvline(0, color='black', linewidth=0.5)
axs[0, 0].quiver(0, 0, vector[0], vector[1], angles='xy', scale_units='xy', scale=1, color='blue', label='Original Vector')
axs[0, 0].quiver(0, 0, transformed_vector[0], transformed_vector[1], angles='xy', scale_units='xy', scale=1, color='red', label='Sheared & Scaled Vector')
axs[0, 0].set_xlim(-6, 6)
axs[0, 0].set_ylim(-6, 6)
axs[0, 0].set_title('Shearing and Scaling Transformation')
axs[0, 0].legend()
axs[0, 0].grid(True)
axs[0, 0].set_aspect('equal', adjustable='box')

# Plot 2: Scaling Transformation
axs[0, 1].axhline(0, color='black', linewidth=0.5)
axs[0, 1].axvline(0, color='black', linewidth=0.5)
axs[0, 1].quiver(0, 0, vector[0], vector[1], angles='xy', scale_units='xy', scale=1, color='blue', label='Original Vector')
axs[0, 1].quiver(0, 0, scaled_vector[0], scaled_vector[1], angles='xy', scale_units='xy', scale=1, color='green', label='Scaled Vector')
axs[0, 1].set_xlim(-6, 6)
axs[0, 1].set_ylim(-6, 6)
axs[0, 1].set_title('Scaling Transformation')
axs[0, 1].legend()
axs[0, 1].grid(True)
axs[0, 1].set_aspect('equal', adjustable='box')

# Plot 3: Stretching Transformation
axs[1, 0].axhline(0, color='black', linewidth=0.5)
axs[1, 0].axvline(0, color='black', linewidth=0.5)
axs[1, 0].quiver(0, 0, vector[0], vector[1], angles='xy', scale_units='xy', scale=1, color='blue', label='Original Vector')
axs[1, 0].quiver(0, 0, stretched_vector[0], stretched_vector[1], angles='xy', scale_units='xy', scale=1, color='purple', label='Stretched Vector')
axs[1, 0].set_xlim(-6, 6)
axs[1, 0].set_ylim(-6, 6)
axs[1, 0].set_title('Stretching Transformation')
axs[1, 0].legend()
axs[1, 0].grid(True)
axs[1, 0].set_aspect('equal', adjustable='box')

# Plot 4: Identity Transformation
axs[1, 1].axhline(0, color='black', linewidth=0.5)
axs[1, 1].axvline(0, color='black', linewidth=0.5)
axs[1, 1].quiver(0, 0, vector[0], vector[1], angles='xy', scale_units='xy', scale=1, color='blue', label='Original Vector')
axs[1, 1].quiver(0, 0, identity_vector[0], identity_vector[1], angles='xy', scale_units='xy', scale=1, color='orange', label='Identity Vector (No Change)')
axs[1, 1].set_xlim(-6, 6)
axs[1, 1].set_ylim(-6, 6)
axs[1, 1].set_title('Identity Transformation (No Change)')
axs[1, 1].legend()
axs[1, 1].grid(True)
axs[1, 1].set_aspect('equal', adjustable='box')

# Display the plots
plt.show()
../_images/a5a48c39ecaa48e5e4a31fc4dbc2962394f898b008a088eb4915f67b02377139.png

Visualize action of a matrix on several vectors#

  • Lets pick four vectors and act our matrix on all of them to see how the relationship between their distances and angles changes

\[(0,0), (0.5,0.5), (0.5, 1.5), (0,1), (0,0)\]

  • We connect the points defined by this vectors to aid our visualization

Hide code cell source 
import matplotlib.pyplot as plt
import numpy as np

coords = np.array([[0,0],[0.5,0.5], [0.5,1.5], [0,1], [0,0]] )
coords = coords.transpose()

x = coords[0,:]
y = coords[1,:]

# Create the figure and axes objects
fig, ax = plt.subplots()

# Plot the points.  x and y are original vectors, x_LT1 and y_LT1 are images
ax.plot(x,y,'ro')

ax.axis([-2,2,-1,2])
# Connect the points by lines
ax.plot(x,y,'r',ls="--", label='x')
ax.grid(True)
../_images/d2e58afc22ebe1fc40dcfc5eb5fead471de7f065af26c0a02fda03768da1382b.png

Stretch matrix A#

\[\begin{split} \begin{equation} A = \left[ \begin{array}{cc} 2 & 0 \\ 0 & 1 \end{array}\right] \end{equation} \end{split}\]
Hide code cell source 
import matplotlib.pyplot as plt
import numpy as np

A = np.array([[2,0],[0,1]]) 

A_coords = A@coords

###
x_LT1 = A_coords[0,:]
y_LT1 = A_coords[1,:]

# Create the figure and axes objects
fig, ax = plt.subplots()

# Plot the points.  x and y are original vectors, x_LT1 and y_LT1 are images
ax.plot(x,y,'ro')
ax.plot(x_LT1,y_LT1,'bo')

# Connect the points by lines
ax.plot(x,y,'r',ls="--", label='x')
ax.plot(x_LT1,y_LT1,'b', label='Ax')

# Edit some settings 
ax.axvline(x=0,color="k",ls=":")
ax.axhline(y=0,color="k",ls=":")
ax.grid(True)
ax.axis([-2,2,-1,2])
ax.set_aspect('equal')
plt.legend()

<matplotlib.legend.Legend at 0x7f90a3f752e0>
../_images/2f36d0e9a4eb1ab817034add91cef3c3a394b19337cc10816b6c9cd6a4ac6dc7.png

Rotation matrix R#

To rotate vectors in the plane, we choose an angle \(\theta\) and write down the matrix that represents the rotation counterclockwise by an angle \(\theta\). Basic trigonometry can be used to calculate the columns in this case.

(25)#\[\begin{equation} R = \left[ \begin{array}{cc} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{array}\right] \end{equation}\]
Hide code cell source 
import matplotlib.pyplot as plt
import numpy as np

theta = np.pi/6
R = np.array([[  np.cos(theta), -np.sin(theta)],
                [np.sin(theta),  np.cos(theta)]])
R_coords = R@coords

x_LT3 = R_coords[0,:]
y_LT3 = R_coords[1,:]

# Create the figure and axes objects
fig, ax = plt.subplots()

# Plot the points.  x and y are original vectors, x_LT1 and y_LT1 are images
ax.plot(x,y,'ro')
ax.plot(x_LT3,y_LT3,'bo')

# Connect the points by lines
ax.plot(x,y,'r',ls="--")
ax.plot(x_LT3,y_LT3,'b')

# Edit some settings 
ax.axvline(x=0,color="k",ls=":")
ax.axhline(y=0,color="k",ls=":")
ax.grid(True)
ax.axis([-2,2,-1,2])
ax.set_aspect('equal')
ax.set_title("Rotation");
../_images/b6e9bef139d3a4d2c0650b726b46b5442cf0693bb46298861a61eb62962e7c3a.png

Shear Matrix, S#

In the study of mechanics, shear forces occur when one force acts on some part of a body while a second force acts on another part of the body but in the opposite direction. To visualize this, imagine a deck of playing cards resting on the table, and then while resting your hand on top of the deck, sliding your hand parallel to the table.

\[\begin{split} \begin{equation} S= \left[ \begin{array}{cc} 1 & 2 \\ 0 & 1 \end{array}\right] \end{equation} \end{split}\]
Hide code cell source 
import matplotlib.pyplot as plt
import numpy as np

S = np.array([[1,2],[0,1]])
S_coords = S@coords

x_LT4 = S_coords[0,:]
y_LT4 = S_coords[1,:]


# Create the figure and axes objects
fig, ax = plt.subplots()

# Plot the points.  x and y are original vectors, x_LT1 and y_LT1 are images
ax.plot(x,y,'ro')
ax.plot(x_LT4,y_LT4,'bo')

# Connect the points by lines
ax.plot(x,y,'r',ls="--")
ax.plot(x_LT4,y_LT4,'b')

# Edit some settings 
ax.axvline(x=0,color="k",ls=":")
ax.axhline(y=0,color="k",ls=":")
ax.grid(True)
ax.axis([-2,4,-1,2])
ax.set_aspect('equal')
ax.set_title("Shear");
../_images/8d750fde86da32b8efccc8b3182a38144eb9345d30edb4badaa2e87f33e5641b.png