Hello MIMO Control Friends,

PURPOSE OF NOTE
The purpose of this note is to develop an understanding for the so-called 
minimum norm problem and its solution as it relates to a linear system of
equations Ax = b.

PROBLEM SETUP
Consider the system of linear equations Ax = b.
Suppose that b lies in the column space of A; i.e. b is a linear combination
of the columns of A and hence a solution x to Ax = b exists.

____________________________________________________________________________

In general, while a solution exists - in the above case - it need not be
unique.
The following simple example illustrates this point as well as other
features which are exhibited in the general case.

EXAMPLE: LINEAR SYSTEM WITH NONUNIQUE SOLUTION
The linear system

      Ax = | 1  1 | x = 1 = b

has a general solution

               x = xr + xn = | 0.5 |  + k |  1 |
                             | 0.5 |      | -1 |
where 
              xr = | 0.5 |  
                   | 0.5 |   

              xn = k |  1 |
                     | -1 |

        k is an arbitrary constant

        Axr = b   i.e. xr is a particular solution
 
        Axn = 0   i.e. xn lies in the right null space of A

        xr lies in the row space of A; i.e. it is a linear combination
                                            of the rows of A
        xr is orthogonal to xn

It should be noted that since xr lies in the row space of A, it is
orthogonal to the right null space of A. Given this, the last property
follows; i.e. xr is orthogonal to xn.

While this example is very specific, it illustrates a few important
features which are exhibited in the general case.

____________________________________________________________________________

GENERAL SOLUTION FOR THE GENERAL CASE
Every solution to the system of linear equations

      Ax = b

may be written in the genaral form

      x = xr + xn

where

      Axr = b  i.e. xr is a particular solution

      Axn = 0  i.e. xn lies in the right null space of A

      xr lies in the row space of A;  i.e. xr is a linear combination
                                           of the rows of A

      xr is orthogonal to xn 

Note: The rows of A are orthogonal to the right null space of A.
      Think about this..this should make sense.

MINIMUM NORM SOLUTION
Since xr lies in row space of A - and is thus orthogonal to the 
right null space of A - it follows that xr is orthogonal to xn.

Given this, it follows from the Pythagorean Theorem that

       norm(x) = SQRT(norm(xr)^2 + norm(xn)^2)

Since this is greater than or equal to norm(xr), it follows from the
Pythagorean Theorem (or Orthogonal Projection Theorem) that the minimum
norm solution to Ax = b is given by any solution xr, satisfying Axr = b,
which lies in the row space of A.

Such a solution may be found by solving the following system of equations:

        xr = A^H z   (This guarantees that xr lies in row space of A)

        A A^H z = b  (This is guaranteed to have a solution z since Ax = b
has a solution)


SPECIAL CASE: A HAS FULL ROW RANK
If A has full row rank, then the matrix AA^H is invertible, A has a right
inverse, the minimum norm solution xr is unique and is given by

        xr = A^{-R} b = A^H (A A^H)^{-1} b

where  
        A^{-R} = A^H (A A^H)^{-1} 

is a right inverse of A.

____________________________________________________________________________


EXAMPLE: MINIMUM NORM SOLUTION
Given the above discussion, the minimum norm solution to the linear system
of equations 

             Ax = | 1  1 | x = 1 = b

is given by

             xr =  [1 1]^H ([1 1][1 1]^H )^{-1} 1 = [0.5  0.5]^T

as suggested by the discussion in the previous example.

____________________________________________________________________________

EXAMPLE: SETTING UP AND SOLVING A MINIMUM NORM PROBLEM (MINIMUM DISTANCE TO
A LINE)
Consider the problem of finding the closest point on the line 

             x2 = 1 - x1 

to the origin x1 = x2 = 0.

(a) Set up this problem up as a minimum norm problem; i.e. find A, b where
we are interested in the minimum norm solution to Ax = b.

(b) Solve the problem which you have posed in (a) for the minimum norm
solution.

____________________________________________________________________________

Hope that the above is helpful.
Please keep asking questions!
AAR