Resources Introduction What is Linear Algebra? Linear Algebra studies vectorial spaces and its representation as matrices. You could associate it with linear equations.
Ecosystem https://discuss.boardinfinity.com/t/what-are-tensors/19073 Story Related work FAQ Vector A V vector is a tuple of a set of numbers in n n n n n n R n \mathbb{R^n} R n v = < a 1 , a 2 , a 3 , . . . , a n > v=<a_1,a_2,a_3,...,a_n> v =< a 1 , a 2 , a 3 , ... , a n >
You can represent vectors geometrically, however we warning you about the limitations that you must keep in mind such as the dimensionality, magnitude, direction, relative scale, complex numbers.
https://www.desmos.com/calculator/d43yjwzrhl
You operate vectors with special sum and multiplication rules.
Let v , w v,w v , w R n \mathbb{R^n} R n v + w = ( v 1 + w 1 , v 2 + w 2 , . . . , v n + w n ) v+w=(v_1+w_1,v_2+w_2,...,v_n+w_n) v + w = ( v 1 + w 1 , v 2 + w 2 , ... , v n + w n )
https://web.ma.utexas.edu/users/ysulyma/matrix/
https://www.desmos.com/calculator/sit26yznye
Chapter 1.2.
Chapter 1.3
Chapter 2.1
Matrix A matrix is a rectangular array (arrange a group of things in a particular way ) of numbers or "entries".
The idea of Elimination Elimination (It's a numerical method):
Ax=b ->
[(entry to eliminate*pivot) / (pivot)] - entry to eliminate ->
keep going to find all n pivots and the upper triangular U->
Ux=c -> Back substitution or breakdown
(U ) Upper triangle allows back substitution.
Rules:
pivot = first nonzero.
The multiplier is l = (entry to eliminate) / (pivot) .
if pivot == 0, then you exchange. When breakdown (pivot == 0) is permanent, Ax=b has no solution or infinitely many.
Note 1:
We want to reduce, hence We subtract:
[ax -by = c] Row 1
[dx - ey = d] Row 2
Numerical Variables
c - d = (ax-by) - (dx-ey)
Note 2:
new row = Row 2 - Row 1*(entry/pivot)
= (dx-ey=d) - (ax-by=c) * (d/a)
is the same operation
Row 1 * d By d(ax-by) = dc
Row 2 * c By a(dx-ey) = ad
new row = d(ax-by) - a(dx-ey) = dc - ad
Elimination Using Matrices Vandermonde matrix is a matrix with the terms of a geometric progression in each row.
Example: [1 1 1; 1 2 3; 1 4 9].
Rules for Matrix Operations Graph or network has n nodes Inverse matrices Sparse matrix
In numerical analysis and scientific computing , a sparse matrix or sparse array is a matrix in which most of the elements are zero. By contrast, if most of the elements are nonzero, then the matrix is considered dense . The number of zero-valued elements divided by the total number of elements (e.g., m × n for an m × n matrix) is called the sparsity of the matrix (which is equal to 1 minus the density of the matrix). Using those definitions, a matrix will be sparse when its sparsity is greater than 0.5.
Hermitian, symmetric, triangular, tridiagonal, or bidiagonal
Elimination = Factorization: A=LU ( E 32 E 31 E 21 ) A = U becomes A = ( E 21 − 1 E 31 − 1 E 32 − 1 ) U = L U (E_{32}E_{31}E_{21})A=U \text{ becomes }A=(E_{21}^{-1}E^{-1}_{31}E^{-1}_{32})U=LU ( E 32 E 31 E 21 ) A = U becomes A = ( E 21 − 1 E 31 − 1 E 32 − 1 ) U = LU L is a lower triangular product of elimination inverse and it contains the numbers l i j l_{ij} l ij
The key reason why A=LU: Row n of U = (Row n of A) − l 31 (Row 1 of U) − l 32 (Row 2 of U) − . . . \text{ Row } n \text{ of U }=\text{(Row } n \text{ of A)}- l_{31}\text{(Row } 1 \text{ of U)}- l_{32}\text{(Row } 2 \text{ of U)}-... Row n of U = (Row n of A) − l 31 (Row 1 of U) − l 32 (Row 2 of U) − ... Row n of A = Row n of U + l 31 (Row 1 of U) + l 32 (Row 2 of U) − . . . \text{Row } n \text{ of A}=\text{ Row } n \text{ of U }+ l_{31}\text{(Row } 1 \text{ of U)}+ l_{32}\text{(Row } 2 \text{ of U)}-... Row n of A = Row n of U + l 31 (Row 1 of U) + l 32 (Row 2 of U) − ...
E − 1 E^{-1} E − 1 E E E l i j l_{ij} l ij − l i j -l_{ij} − l ij
LDU. Lower triangular L times diagonal D times upper triangular U.A = L D U = L [ d 1 d 2 . . . d n ] [ 1 u 12 / d 1 u 13 / d 1 . . . 1 u 23 / d 2 . . . . . . . . . 1 ] A=LDU=L\begin{bmatrix} d_1 & & \\ & d_2 & \\ & & .\\ & & & .\\ & & & &.\\ & & & &&d_n\\ \end{bmatrix} \begin{bmatrix} 1 & u_{12}/d_1 & u_{13}/d_1 &.&.&.\\ & 1 & u_{23}/d_{2}&.&.&.\\ & & .&&&.\\ & & & . &&.\\ & & & &.&.\\ & & & & &1\\ \end{bmatrix} A = L D U = L d 1 d 2 . . . d n 1 u 12 / d 1 1 u 13 / d 1 u 23 / d 2 . . . . . . . . . . . . 1
One square system. System: A x = b \text{System: } Ax=b System: A x = b O ( n 3 ) Factor: A = L U O(n^3)\text{ Factor: }A=LU \text{ } O ( n 3 ) Factor: A = LU O ( n 2 ) Solve: L c = b , U x = c O(n^2) \text{ Solve: }Lc=b \text{, } Ux=c O ( n 2 ) Solve: L c = b , Ux = c Gauss elimination with no row exchanges \text{ Gauss elimination with no row exchanges} Gauss elimination with no row exchanges
A band matrix B has w nonzero diagonals below and above its main diagonal.
Factor: O ( n 3 ) O(n^3) O ( n 3 ) O ( n w 2 ) O(nw^2) O ( n w 2 ) O ( n 2 ) O(n^2) O ( n 2 ) O ( 2 n w ) O(2nw) O ( 2 n w )
Most matrices in practice are sparce (many zero entries). In that case A=LU is much faster.
n=1 0 3 10^3 1 0 3
n=1 0 4 10^4 1 0 4 1 0 3 10^3 1 0 3
n=1 0 5 10^5 1 0 5 1 0 6 10^6 1 0 6
This is too expensive without a supercomputer, but that these matrices are full. Most matrices in practice are sparse (many zero entries). In that case, A=LU is much faster.
Transposes and Permutations Unprofesional definition ( A T ) i j = A j i (The columns A T are the rows of A) \text{Unprofesional definition }(A^T)_{ij} = A_{ji} \text{ (The columns } A^T \text{ are the rows of A)} Unprofesional definition ( A T ) ij = A ji (The columns A T are the rows of A) Profesional definition. ( A x ) T y = x T ( A T y ) \text{Profesional definition. } (Ax)^Ty = x^T(A^Ty) Profesional definition. ( A x ) T y = x T ( A T y ) Scalar ( k A ) T = k A T \text{Scalar } (kA)^T=kA^T Scalar ( k A ) T = k A T Sum ( A + B ) T = A T + B T \text{Sum } (A+B)^T=A^T+B^T Sum ( A + B ) T = A T + B T Product ( A B ) T = B T A T \text{Product } (AB)^T=B^TA^T Product ( A B ) T = B T A T Proof If B is a vector x, such that( A x ) T = x T A T (Ax)^T=x^TA^T ( A x ) T = x T A T Ax combines the columns of A \text{Ax combines the columns of A } Ax combines the columns of A A x = a 1 x 1 + a 2 x 2 + . . . a n x n = b Ax=a_1x_1+a_2x_2+...a_nx_n=b A x = a 1 x 1 + a 2 x 2 + ... a n x n = b We have ( A x ) T = ( a 1 x 1 + a 2 x 2 + . . . a n x n ) T = b T \text{We have }(Ax)^T=(a_1x_1+a_2x_2+...a_nx_n)^T=b^T We have ( A x ) T = ( a 1 x 1 + a 2 x 2 + ... a n x n ) T = b T ( A x ) T = ( a 1 x 1 ) T + ( a 2 x 2 ) T + . . . + ( a n x n ) T (Ax)^T=(a_1x_1)^T+(a_2x_2)^T+...+(a_nx_n)^T ( A x ) T = ( a 1 x 1 ) T + ( a 2 x 2 ) T + ... + ( a n x n ) T ( A x ) T = a 1 T x 1 + a 2 T x 2 + . . . + a n T x n = [ x 1 x 2 . . . x n ] [ a 1 T a 2 T . . . a n T ] (Ax)^T=a_1^Tx_1+a_2^Tx_2+...+a_n^Tx_n=[x_1 x_2 ...x_n] \begin{bmatrix} & a^T_1 &\\ & a^T_2 &\\ & . &\\ & . &\\ & . &\\ & a^T_n &\\ \end{bmatrix} ( A x ) T = a 1 T x 1 + a 2 T x 2 + ... + a n T x n = [ x 1 x 2 ... x n ] a 1 T a 2 T . . . a n T thus ( A x ) T combines the rows of A T \text{thus }(Ax)^T \text{ combines the rows of } A^T thus ( A x ) T combines the rows of A T ( A x ) T = [ x 1 x 2 . . . x n ] [ a 1 T a 2 T . . . a n T ] = x T A T (Ax)^T=[x_1 x_2 ...x_n] \begin{bmatrix} & a^T_1 &\\ & a^T_2 &\\ & . &\\ & . &\\ & . &\\ & a^T_n &\\ \end{bmatrix}=x^TA^T ( A x ) T = [ x 1 x 2 ... x n ] a 1 T a 2 T . . . a n T = x T A T So ( x T A T ) combines the rows of A T \text{So }(x^TA^T) \text{ combines the rows of } A^T So ( x T A T ) combines the rows of A T In A they are columns, A T in they are rows. \text{In A they are columns, } A^T \text{ in they are rows.} In A they are columns, A T in they are rows. ( [ column of A x ] T ) = ( row of x T A T ) ([\text{column of }Ax]^T)=(\text{row of }x^TA^T) ([ column of A x ] T ) = ( row of x T A T ) ( A x ) T = x T A T (Ax)^T=x^TA^T ( A x ) T = x T A T
If B is [ x 1 x 2 . . . ] [x_1 x_2 ...] [ x 1 x 2 ... ] ( A B ) T = [ A x 1 A x 2 . . . ] T = [ A x 1 A x 2 . . . ] = [ x 1 T A T x 2 T A T . . . ] = B T A T (AB)^T=\begin{bmatrix} & & \\ Ax_1 & Ax_2 & ... \\ & & \\ \end{bmatrix}^T=\begin{bmatrix} &Ax_1 & \\ & Ax_2 & \\ & . & \\&.&\\&.&\\ \end{bmatrix}=\begin{bmatrix} &x_1^TA^T & \\ & x_2^TA^T & \\ & . & \\&.&\\&.&\\ \end{bmatrix}=B^TA^T ( A B ) T = A x 1 A x 2 ... T = A x 1 A x 2 . . . = x 1 T A T x 2 T A T . . . = B T A T
Inverse ( A − 1 ) T = ( A T ) − 1 \text{Inverse } (A^{-1})^T=(A^T)^{-1} Inverse ( A − 1 ) T = ( A T ) − 1 Proof A T ( A T ) − 1 = I A^T (A^{T})^{-1}=I A T ( A T ) − 1 = I ( A A − 1 ) T = I T = I (AA^{-1})^T=I^T=I ( A A − 1 ) T = I T = I ( A − 1 ) T A T = I (A^{-1})^TA^T=I ( A − 1 ) T A T = I ( A − 1 ) T = ( A T ) − 1 (A^{-1})^T=(A^T)^{-1} ( A − 1 ) T = ( A T ) − 1 The meaning of Inner Products T is inside. The dot product or inner product is x T y = x ⋅ y = n u m b e r = < x ∣ y > x^Ty=x·y=number=<x|y> x T y = x ⋅ y = n u mb er =< x ∣ y >
T is outside. The rank one product or outer product is x y T = ∣ x > < y ∣ = m a t r i x xy^T=|x><y|=matrix x y T = ∣ x >< y ∣ = ma t r i x
Symmetric Matrices Definition. S T = S \text{Definition. }S^T=S Definition. S T = S These are the most important matrices of all.S = L D U = L D L T S=LDU=LDL^T S = L D U = L D L T
Permutation matrices Definition. A permutation matrix P has the rows of the identity I in any order
Single 1 in every row and every column. There are n! permutation matrices of order n.
And P − 1 = P T P^{-1}=P^T P − 1 = P T
If A is invertible then a permutation P will reorder its rows for
Worked examples A T A = 0 A^TA=0 A T A = 0 A ≠ 0 A\ne 0 A = 0 d i a g ( A T A ) = ( ( c o l i of A ) T ( c o l i of A ) ) i i ≥ 0 A T A = 0 d i a g ( A T A ) = 0 ( ( c o l i of A ) T ( c o l i of A ) ) i i = 0 ( c o l i of A ) = 0 A = 0 diag(A^TA)=((col\text{ }i\text{ of }A)^T(col\text{ }i\text{ of }A))_{ii}\geq0 \\ \text{ }A^TA=0\\ diag(A^TA)=0\\ ((col\text{ }i\text{ of }A)^T(col\text{ }i\text{ of }A))_{ii}=0\\ (col\text{ }i\text{ of }A)=0\\ A=0 d ia g ( A T A ) = (( co l i of A ) T ( co l i of A ) ) ii ≥ 0 A T A = 0 d ia g ( A T A ) = 0 (( co l i of A ) T ( co l i of A ) ) ii = 0 ( co l i of A ) = 0 A = 0 Find P 3 x 3 3 = I P^3_{3x3}=I P 3 x 3 3 = I ( 123 ) , ( 231 ) , ( 312 ) , ( 123 ) P [ 1 2 3 ] = [ 2 3 1 ] P [ 2 3 1 ] = [ 3 1 2 ] P [ 3 1 2 ] = [ 1 2 3 ] P = [ 1 0 0 0 0 1 0 1 0 ] [ 0 1 0 1 0 0 0 0 1 ] = [ 0 1 0 0 0 1 1 0 0 ] Thus, P 3 = I (123),(231),(312),(123)\\ P\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}=\begin{bmatrix} 2 \\ 3 \\ 1 \end{bmatrix}\\ P\begin{bmatrix} 2 \\ 3 \\ 1 \end{bmatrix}=\begin{bmatrix} 3 \\ 1 \\ 2 \end{bmatrix}\\ P\begin{bmatrix} 3 \\ 1 \\ 2 \end{bmatrix}=\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}\\ P=\begin{bmatrix} 1 & 0 & 0\\ 0 & 0 & 1\\ 0 & 1 & 0 \end{bmatrix}\begin{bmatrix} 0 & 1 & 0\\ 1 & 0 & 0\\ 0 & 0 & 1 \end{bmatrix}=\begin{bmatrix} 0 & 1 & 0\\ 0 & 0 & 1\\ 1 & 0 & 0 \end{bmatrix}\\ \text{Thus, }P^3=I ( 123 ) , ( 231 ) , ( 312 ) , ( 123 ) P 1 2 3 = 2 3 1 P 2 3 1 = 3 1 2 P 3 1 2 = 1 2 3 P = 1 0 0 0 0 1 0 1 0 0 1 0 1 0 0 0 0 1 = 0 0 1 1 0 0 0 1 0 Thus, P 3 = I 3. Find P 4 x 4 4 ≠ I P^4_{4x4}\neq I P 4 x 4 4 = I ( 1234 ) , ( 1342 ) , ( 1423 ) , ( 1234 ) , ( 1342 ) P = [ 1 P 3 ] Thus, P 4 x 4 4 ≠ I (1234),(1342),(1423),(1234),(1342)\\ P=\begin{bmatrix} 1 & &&\\ & & P^3&\\ & & & \end{bmatrix}\\ \text{Thus, }P^4_{4x4}\neq I ( 1234 ) , ( 1342 ) , ( 1423 ) , ( 1234 ) , ( 1342 ) P = 1 P 3 Thus, P 4 x 4 4 = I
4. Prove that the identity matrix cannot be te product of three row exchanges (or five). It can be the product of two exchanges (or four).
We have a permutation matrix E p E_p E p I I I
Key properties: E p = ( E p ) T = ( E p ) − 1 E_p=(E_p)^T=(E_p)^{-1} E p = ( E p ) T = ( E p ) − 1 E P 2 n = I , E p 2 n + 1 = E p E_P^{2n}=I,E_p^{2n+1}=E_p E P 2 n = I , E p 2 n + 1 = E p
Vector Spaces This the highest level of understating about matrix calculations. DEFINITION. The standard n-dimensional space R n R^n R n v v v n n n The "vectors" in S can be matrices or functions of x. M (2 by 2 matrices) and F (functions) and Z (zero vector alone) are vector spaces. M \text{ (2 by 2 matrices) and } F \text{ (functions) and } Z \text{ (zero vector alone) are vector spaces.} M (2 by 2 matrices) and F (functions) and Z (zero vector alone) are vector spaces.
Properties x + y = y + x x + ( y + z ) = ( x + y ) + z x + 0 = x x + ( − x ) = 0 1 × x = x ( c 1 c 2 ) x = c 1 ( c 2 x ) c ( x + y ) = c x + c y ( c 1 + c 2 ) = c 1 x + c 2 x x+y=y+x\\ x+(y+z)=(x+y)+z\\ x+0=x\\ x+(-x)=0\\ 1\times x=x\\ (c_1c_2)x=c_1(c_2x)\\ c(x+y)=cx+cy\\ (c_1+c_2)=c_1x+c_2x x + y = y + x x + ( y + z ) = ( x + y ) + z x + 0 = x x + ( − x ) = 0 1 × x = x ( c 1 c 2 ) x = c 1 ( c 2 x ) c ( x + y ) = c x + cy ( c 1 + c 2 ) = c 1 x + c 2 x Max-plus vector space An interesting “max-plus” vector space comes from the real numbers R combined with −∞. Change addition to give x + y = max(x, y) and change multiplication to
Subspace DEFINITION. A subspace of a vector space is a set of vectors (incluiding 0) that two requirements: If v and w are vectors in the subspace and c is any scalar, then
v + w v+w v + w
c v cv c v
An operator H H H
H ( f 1 + f 2 ) = H ( f 1 ) + H ( f 2 ) H(f_1+f_2)=H(f_1)+H(f_2) H ( f 1 + f 2 ) = H ( f 1 ) + H ( f 2 )
H ( α f 1 ) = α H ( f 1 ) H(\alpha f_1)=\alpha H (f_1) H ( α f 1 ) = α H ( f 1 )
Worked examples Is m a x max ma x
Is the sum a linear operator?
Is the square root a linear operator?
Is the division a linear operator?
x x x
The smallest subspace The linear span (also called the linear hull or just span) of a set S of vectors in a vector space is the smallest linear subspace that contains the set.
Column space C(A) is a combination of r (number of pivots) columns.
Nullspace N(A) is a combination of n-r special solutions.
Eigenvectors and eigenvalues Eigenfunctions Least algebras https://textbooks.math.gatech.edu/ila/least-squares.html
Linear Regression https://twitter.com/leonpalafox/status/1676246463389618176
Simple problems https://www.algebra.com/algebra/homework/word/age/Really-intricate-age-word-problem.lesson
A father is 27 years older than his son ten years ago he was twice as old as his son how old is the father?
Model.
f a t h e r − 27 = s o n father-27=son f a t h er − 27 = so n
0.5 ( f a t h e r − 10 ) = s o n − 10 0.5(father-10)=son-10 0.5 ( f a t h er − 10 ) = so n − 10
So,
f a t h e r = 64 father=64 f a t h er = 64
Four years ago, a mother was 4 times as old as her son. In 4 years’ time, the sum of their ages will be 56 years. Find the age of the mother when the son was born.
Model.
m-4=4(s-4)
m+4+s+4=56
The answer is m-s.
Next steps Matrix Methods in Data Analysis, Signal Processing, and Machine Learning
References 
