Dark frames missing from some data files

[benj]

Some flights appear to be missing dark frames (should have been collected automatically). This affects the following flights so far:

VOC_03-2008 (#197), Eagle and Hawk, all lines
VOC_04-2008 (#205), Eagle and Hawk, all lines
VOC_05-2008 (#206), Eagle and Hawk, all lines
VOC_06-2008 (#209), Eagle and Hawk, all lines

CEH07_01-2007 (#53), Eagle, file 8
CEH07_05-2007 day 73 (#59), Eagle and Hawk line 1 (not processed at PML)
GB07_05-2007 day 116a (#54), Eagle, files 1-3

WM06_11-2006 day 142 (No ticket), Eagle, all lines
HY05_07-2006 day 194 (No ticket), Eagle and Hawk, all lines
GB06_05-2006 days 195A & C (No ticket), Eagle and Hawk, all lines

Related tickets: #249 #250

[benj]

Phil Goy says as far as he knows the sensors were set to collect dark data automatically - he's going to talk to Specim to ask if they know why it might not have worked.

[mggr]

CEH07_01-2007 (Buckingham) is a known issue - AKW, the processor, was notified of the loss of the dark lines (in this case, it was due to the data being destroyed on the disk - see ticket #53 for details). Adding strikethrough in description to indicate we shouldn't worry about this one.

[benj]

Looks like this should be recoverable - the effects of substituting dark frames from different flights are small-ish (ie within 5%, and mostly within 1-2%).

Note it's a lot of work though - to make it work at all, you have to pick dark frames from a flight that used the same spatial/spectral binning, the same frame rate and the same integration time (also ideally the same altitude). The integration times in particular change per flight line. Any flightlines for which dark frames cannot be obtained that match all three will not be recoverable - running without dark frame correction results in a 100-500% (or even more) error, and failing to match all three items just results in mostly underflowed data).

Eagle:

Hawk:

[benj]

Bill has pointed out that azspec has a couple of options that are relevant here:

-dS copies dark frames to the external ENVI file pair: dark.raw and dark.raw.hdr from a raw file that has dark frames at the end

-dIc s applies a correction to the dark frame average if the raw and dark have different integration periods; dark is scaled by s * raw / dark

The latter in particular should mean that it isn't necessary to find new dark frames for each flightline, since it lets us adjust for the integration time and the binning and frame rates are mostly constant within a flight day.

[benj]

Having done those calculations for a few more data points (initially I only did one... thought the effects would be consistent and they aren't), the effect is a bit bigger than I thought. Basically, the error is mostly within 5% (except for one area on the Hawk where it's within 10%), but for some pixels (particularly for Hawk) it's up to 30% or so. Graphs above have been updated.

[benj]

List of which Vocals flights have which combinations of flight parameters

[benj]

Looks like we need to extract dark frames from fourteen other flights to do substitution (see attached file darkframe_matches.txt). This still leaves 21 flightlines that will need to have substitution done by rescaling integration times.

Suggested substitutions (from darkframe_matches.txt):
Hawk:
Day 2 line 1: Day 3 line 1; Day 4 lines 14,15; Day 6 lines 5,9-12
Day 7 line 2: Day 4 line 3
Day 7 line 5: Day 4 lines 1,2,4,9-13; Day 5 line 1; Day 6 lines 6-8
Day 10 line 2: Day 5 lines 12-14; Day 6 lines 1,2,4
Day 11 line 4: Day 3 lines 2,3,6-14
No match: Day 3 lines 4,5; Day 4 lines 5-8; Day 5 lines 2-11; Day 6 line 3

Eagle:
Day 7 line 2: Day 4 line 3
Day 7 line 5: Day 4 lines 9-11; Day 6 line 8
Day 7 line 8: Day 4 line 1
Day 8 line 1: Day 4 lines 12,13; Day 6 lines 6,7
Day 8 line 5: Day 4 lines 14,15; Day 6 line 5,9-12
Day 9 line 1: Day 3 lines 2,3
Day 9 line 2: Day 3 lines 6-14
Day 10 line 2: Day 6 lines 1,2,4
Day 11 line 1: Day 4 lines 5-8
No match: Day 3 lines 1,4,5; Day 6 line 3

Unfortunately Eagle day 4 lines 2 and 3, and the whole of Eagle day 5 (14 lines) do not have alternative Vocals flights that can be used for rescaling (there's nothing else that matches their binning and framerate settings).

[benj]

Suggested source days for dark frame substitution for flights missing autodark

[benj]

Having run my script for finding matching dark frames on our other (non-vocals) datasets...

1. Only three datasets match the two day 4 lines - Difarsa (climate possibly even less appropriate than UK? Though possibly not, I guess), one Latera line (2007 data, calibration difference), and Buckingham (we really shouldn't use that one, and it's 2007 data anyway). Difarsa's likely to be the best choice but that doesn't make it a good one (and it'll have to be rescaled even so).

2. The 14 day 5 lines use spatial binning of 2 for some reason. No other 2008 datasets use spatial binning 2, and the only ones that do from 2007 are Silver Flowe and Inverclyde, neither of which use the same framerate as the Vocals data. In other words, we have to either work out a (valid) way to rescale dark frames to account for either different spatial binning or different framerate, or we have no viable dark frames at all for the Eagle data for day 5, however large an error we're willing to accept (though it might be feasible to collect some on a future flight).

[benj]

Percentage effect of substituting dark frames from a different flight for Eagle

[benj]

Percentage effects of substituting dark frames from a different flight for Hawk

[jatt]

For Eagle day 3 line 1 there is no alternative Vocals flight that can be used for rescaling (there's nothing else that matches the binning and framerate settings).

[mggr]

Per #250, Eagle dark files are more or less always the same regardless of settings. Other dark files have been created as needed. These should be pulled from ~jatt and put into the correct VOCALS project, then this ticket can be closed.

Picking up the GB07/05 project as having missing dark frames turns out to have been a mistake - the stored .hdr files were zero length, hence it was detected as having no dark frames. Actually the data does have dark frames. See #54 for more info.

[mggr]

Belay that close until W06/11, HY05/07 (no data?) and GB06/05 have been checked..

[mggr]

Added convenience links for later memory assistance ;)

[benj]

I'm going to add something to the unpacking script to check for missing dark frames - it's pretty trivial to check for.