Overflow handling in azspec

[mggr]

[Issue began 5/June/2008, split out from #133 18/June.]

While fixing the FODIS processing (#133), Bill noticed that the overflow value he was looking for was 4096 but should actually have been 4095. In fixing this, he has unearthed a large number of overflows in captured Eagle and Hawk imagery.

In Hawk, which shifts the whole frame out of the CCD at once, these overflows will only affect the data at the bandpixel that has overflowed. The only fix required is to mark the bad pixels and to provide info to the flight crew to allow them to set camera values appropriately.

In Eagle, the CCD shifts the image out a line at a time, which means light continues to accumulate during the shifting procedure. This additional light is removed by "frame smear correction", where a proportion of the light is removed from each pixel, accumulating as the shifts progress through the frame (begins at the red end of the spectrum and proceeds to the blue end). However, if a pixel overflows, it is no longer possible to estimate the proportion of light to remove from the next pixel (as it will be unmeasured due to pixel saturation).

[mggr]

Bill's first fix to the FODIS processing (14/May, finally tested on 2/June) showed the overflow problems:

[mggr]

Try two:

I have been having thoughts about this wrt Ethiopia line 6 and
obviously other sites with eagle overflow. After discussion with
Andrew during which we reviewed our original philosophy in the
area of bad pixel processing, I  decided to implement and supply
the following FSM corr.:

1. as at present all pixels in a raw data frame are checked for
overflow based upon the particular instrument. Overflowed pixels
are set to a flag value so they can be easily recognised in the
rest of the radiometric calibration program. On output to a BIL
or HDF file overflowed pixels are set to either full scale or
zero - user option.

2. [for eagle only] during FSM-c each sensor column is corrected
by starting at the row nearest the output gate (blue for eagle)
and subtracting from each row except the first a value which is
the sum of each row before the current one scaled by the shift
time  proportion of the integration period.

In Jukka's original alg. when an overflowed pixel occurred it is
first just corrected as though it was valid and for the following
row just ignored and the scaled fraction used in that row
correction. If they were then good rows they will essentially not
be corrected sufficiently as the actual value of overflowed rows
would be lower than they should be.

On output for a column like this there would be first correct
bands, then "corrected" overflowed bands and then  inadequately
corrected good bands.

In azspec.118  all overflowed bands are set to zero.

To provide a better solution within what Andrew and I decided
years back about bad data I have implemented a version which uses
the scaled overflowed value (4095 for eagle) in the correction
calculation (like Jukka) but will set each output overflowed
band/pixel to the flag value. band/pixels from after an
overflowed set of bands will be corrected using this sum of
scaled values.

This will at least mean that a) no overflowed pixels are output
and b) at least a reasonable attempt has been made to correct
later bands.


Please find attached azspec.1110  with options:

-sFJ    which will use Jukka's original method, so there will be
no zerod pixels in any band
            by default ie no -sFJ  the above method is
implemented

-sFZ    overflowed output pixels will be set to zero and not the
default of FFFF

[mggr]

azspec 1.1.10 was sent on 9/June. Bill's suggested approach seems like the best fix to the problem (short of avoiding overflows entirely).

Tested and it seems to be munching Hawk at the moment - going to send Bill a more complete set of sample datasets so he can look into it.

On the spectral graphs (scroll right), note that the shape of the azspec hawk curve appears to be similar to the uncalibrated raw data. The band blackouts are in the correct place though.

As Bill says, you can clearly see that Caligeo is making up the numbers in the 1500-1800nm range. The curve there cannot have come from the flat (maxed) raw data and must just be the calibration coefficients and any smear correction.

[mggr]

Comparison of the newest azspec

[mggr]

Comparison of the newest azspec

[mggr]

A couple of comparisons of Caligeo vs old azspec vs new azspec (1.1.10)

Eagle (scroll right!)

Hawk (scroll right!)

[mggr]

After some diagnosis, Bill sent azspec.1111, which appears to have fixed the spikes.

FODIS is also fine (note caution on how to view FODIS with ENVI in #133).