MATH 414 Lecture 4

From Notes
Jump to navigation Jump to search

« previous | Wednesday, January 22, 2014 | next »


Least Squares

Given time t, we can measure samples of a function y(t) as points yk for k=1,2,….

Suppose we want to find a linear regression y=at+b for parameters a and b that minimizes the following sum of squares:

∑k=0n−1(yk−(ak+b))2

Case

Let us define y→=at→+b1→, where t→=⟨0,1,2,…,n−1⟩ and 1→=⟨1,1,…,1⟩.

Then ∑k=0n−1(yn−(ak+b))=(y→D−y→)T(y→D−y→)=‖y→D−y→‖2, where y→D is our vector of samples.


Let W:=Span{1→,t→} be a vector space of all possible regressions for y. For some unit vector w^∈W, we can find a vector tw^ such that tw^=y→−y→0.

Therefore we can find a minimum value for t in ‖y→D−(y→0+tw^)‖2:

‖y→D−(y→0+tw^)‖=⟨y→D−(y→0+tw^),y→D−(y→0+tw^)⟩=…=t2−2t⟨yD−y0,w^⟩

We find the minimum of this function at t=0, so we find that ⟨y→D−y→0,w^⟩=0 and furthermore ⟨y→D−y→0,y→⟩=0 for all y→∈W.


Punchline

Theorem. Let V be an inner product space and let W⊂V be a subspace of V. Also, let v→ be any vector in V. Then, w→0 minimizes ‖v→−w→‖ over all w→∈W if and only if v→−w→0⊥W.

Proof. Follows mutatis mutandis [1] from the special case illustrated above.

quod erat demonstrandum


Finding w→0

How do we find w→0 (or in the special case y→0)?

w→0 is the orthogonal projection of v→ onto the vector space W.

Let {e^1,e^2,…,e^n} be an orthonormal basis for W. Then

projW(v→)=∑k=1n⟨v→,e^k⟩e^k


Footnotes

  1. ↑ mutatis mutandis = "change for change"