Rescaling dark frames causes calibration errors

[benj]

Processing data in azspec and using the -dIc option to rescale dark frames seems to cause the data to be incorrectly radiometrically calibrated.

I've attached three screenshots. dIc_ex1.png shows spectral profiles for the rescaled flightline from both azspec (top) and CaliGeo (bottom). You can see that the azspec spectrum goes up at both ends, where the CaliGeo spectrum doesn't. The azspec spectrum also has values that are consistently about 600 higher than CaliGeo (discounting the edges of the spectrum, which as mentioned are much higher).

dIc_ex2.png shows spectral profiles from the second flightline, which did not require the dark frames to be rescaled. Azspec and CaliGeo are in very close agreement for this line.

odd_one_out.png is just for illustration - it shows all of the Eagle lines from that flight (as processed through azspec) overlaid. The first flightline (with the rescaled dark frames) is clearly visible as different - much whiter than the other lines.

This may affect any datasets from 2006, and some VOCALS data.

Related tickets: #218 #250

[benj]

Spectral profiles from Eagle line with rescaled dark frames (GB04-06 line 1)

[benj]

Spectral profiles from Eagle line with separate but not rescaled dark frames (GB04-06 line 2)

[benj]

Have reported this issue to Bill for investigation.

[benj]

VOCALS data affected by this issue are as follows (thanks Jane):

eagle:   line 04
         line 05

hawk:    line 04
         line 05


Vocals day 04

hawk:    line 05
         line 06
         line 07
         line 08

Vocals day 05

hawk:    line 02
         line 03
         line 04
         line 05
         line 06
         line 07
         line 08
         line 09
         line 10
         line 11


Vocals day 06

eagle:   line 03

hawk:    line 03

[benj]

Top lines in the previous comment (without a day number) are for day 3 - copy/paste error on my part.

[benj]

Image for illustration showing difference of rescaled line (far left, whitish)

[benj]

Not doing anything about this, going to produce a program to rescale dark frames as appropriate external to azspec (see #250). Closing this as wontfix unless there turns out to be some problem with the other approach.

[benj]

Image showing synthetic dataset hawk with and without rescaling

[benj]

-dIc option does actually appear to be working. It goes wrong for Eagle but that's because Eagle darkframe values don't appear to vary with integration time (see comments in #250), so its use for Eagle is inappropriate. Seems to work fine for Hawk.

Image shows synthetic data processed with -dIc and with dark frames for correct integration time. Values don't match exactly, but that's due to a combination of the fact that the dark frames were produced using an average value (and dark values for Hawk vary with band) and also the rescaling also necessarily uses only an average scaling value. Values are close enough to say it's behaving correctly given the limitations inherent in the method.

Going to reopen ticket and reclose as invalid rather than wontfix.

[mggr]

Added convenience links for later memory assistance ;)