Lucas Kanade isn't going so well.

This week I got a huge database of LUMIS images. Unfortunately, they don't work very well with Lucas Kanade. I think this has to do with the brightness consistency constraints (even after histogram matching). I think its related to finding large number of local minimum when the brightness constancy constraint is changed even a bit. 

Image Subtraction (ghostly parts and blurring are bad)

*The box is a subset of the image being used for matching. 

However, affine transformations do capture the transformation pretty well, even when we aren't taking images of a flat calibration grid:

Image Subtraction after histogram equalization and an affine transformation derived from hand chosen points

The sum of square differences increased 3 fold, but they're still acceptable.