Hello MIMO Control Friends,

PURPOSE OF NOTE
The purpose of this note is to precisely define the so-called
least square error problem and to discuss its solution. An illustrative
example
is given.
_____________________________________________________________________________

PROBLEM SETUP
Consider the linear system of equations Ax = b where b does not lie in the
column space of A; i.e. b is not a linear combination of the columns of A
and hence a solution does not exist.

_____________________________________________________________________________

LEAST SQUARE ERROR PROBLEM
Because a solution does not exist, we seek to find that x which minimizes
the distance between Ax and b; i.e. which minimizes the following expression:

      || b - Ax || = Norm ( b - Ax )

ORTHOGONALITY CONDITION: TOWARD A SOLUTION
>From Pythagorean Orthogoanlity ideas, it foillows that we need to
project b onto the column space of A and then solve for x. Given this, it
follows from Pythagorean Orthogoanlity ideas that a solution to this
problem is given by any vector
x which satisfies the following orthogonality condition:

         b - Ax is orthogonal to column space of A

Draw a picture for yourselves!

_____________________________________________________________________________

SOLUTION TO LEAST SQUARES PROBLEM
Given the above, it follows that x is a solution if and only if it
satisfies the following orthogonmality condition:

    b - Ax is orthogonal to Aw for all vectors w

or
    w^H A^H ( b - Ax ) = 0  for all vectors w

or
    w^H  ( A^H b - A^H Ax ) = 0  for all vectors w

Since this must hold for any vector w, it must hold for w = A^H b - A^H Ax.
This implies that

    w^H w = 0 for w = A^H b - A^H Ax.

This, , however, implies that

    w = A^H b - A^H Ax = 0

_____________________________________________________________________________

NORMAL EQUATIONS: THE SOLUTION
It thus follows that x is a solution to the least square error problem 
if and only if x satisfies the following linear system of equations:

       A^H Ax = A^H b  

This linear system of equations is referred to in the literature as the
Normal Equations.

_____________________________________________________________________________

SPECIAL CASE: A HAS FULL COLUMN RANK
Suppose that A has full column rank. In such a case the
matrix (A^H A) is inverible and the matrix A possesses
a left inverse given by:

       A^{-L} = (A^H A)^{-1} A^H

When A has full column rank, it therefore follows that the 
least square error solution is given by

       x = A^{-L} b = (A^H A)^{-1} A^H b

_____________________________________________________________________________

PROJECTION MATRIX
With this solution in hand, it is useful to make an additional observation.
Note, for example, that

       Ax = AA^{-L} b = A (A^H A)^{-1} A^H b

Given this, it is natural to interpret the matrix

        P = A (A^H A)^{-1} A^H 

as a matrix which projects any vector b onto the column space of A.

_____________________________________________________________________________

ILLUSTRATIVE EXAMPLE: A LEAST SQUARES PROBLEM
Consider the linear system Ax = b where A = [ 1   1 ]^T and b = [ 1   0 ]^T.
This system does not possess a solution since b does not lie in the column
space of A.
Given this, we seek a least square error solution. From the above
discussion, the least square error solution is given by:

      x = A^{-L} b = (A^H A)^{-1} A^H b = ([1  1][1  1]^T)^{-1} [1  1] [1
0]^T 
                                        = 0.5

which is consistent with our intuition. Right?  

_____________________________________________________________________________


Hope that the above is helpful.
Any feedback on this note would be appreciated.
AAR