Opened 12 years ago

Closed 12 years ago

Last modified 12 years ago

#110 closed bug (fixed)

Azspec processed Hawk data has spectral spikes

Reported by: benj Owned by: mggr
Priority: immediate Milestone:
Component: az* programs Keywords:
Cc: mggr Other processors:

Description (last modified by benj)

Azspec-processed level 1b hawk data does not appear to be correct - it contains large anomalous spikes in some bands, seemingly correlated to very low data values. Eg from Nigg Bay line 3:

Hawk spectrum from azspec-processed level 1b file over water (8px average)

Compared to Caligeo spectrum from same pixel:

Hawk spectrum from caligeo-processed level 1b file over water (8px average), same place as azspec example

Could be underflowing, but would expect a much higher value for the spikes (~65k) in this case. Might still be underflowing and then changed by maths in azspec though.

Note extended from ticket #106, but separate to Hawk noise issue

Attachments (10)

hawk_lev1_spec_az.png (4.0 KB) - added by benj 12 years ago.
Hawk spectrum from azspec-processed level 1b file over water (8px average)
hawk_lev1_spec_cali.png (3.9 KB) - added by benj 12 years ago.
Hawk spectrum from caligeo-processed level 1b file over water (8px average), same place as azspec example
hawkland_az_cali.png (211.5 KB) - added by benj 12 years ago.
Hawk level 1b spectra over land from azspec and Caligeo
hawkrawmatch.jpg (30.5 KB) - added by benj 12 years ago.
Hawk raw data over water. White line is single pixel, red line is 8px average
hawkazmatch1px.jpg (24.9 KB) - added by benj 12 years ago.
Hawk level 1b spectrum processed by azspec over water - single pixel
spike_vs_raw-vertprofile.jpg (411.6 KB) - added by mggr 12 years ago.
Spikes vs raw, vertical profile
new_azspec.jpg (196.6 KB) - added by mggr 12 years ago.
azoldnewwater.jpg (81.7 KB) - added by benj 12 years ago.
Old and new azspec processed data over water
aznewvscaliwater.jpg (57.7 KB) - added by benj 12 years ago.
New azspec and Caligeo processed data over water
aznewvscaliland.jpg (70.9 KB) - added by benj 12 years ago.
New azspec and caligeo over land

Download all attachments as: .zip

Change History (21)

Changed 12 years ago by benj

Hawk spectrum from azspec-processed level 1b file over water (8px average)

Changed 12 years ago by benj

Hawk spectrum from caligeo-processed level 1b file over water (8px average), same place as azspec example

comment:1 Changed 12 years ago by benj

  • Description modified (diff)

Changed 12 years ago by benj

Hawk level 1b spectra over land from azspec and Caligeo

comment:2 Changed 12 years ago by benj

Azspec output over land (ie values largely not close to 0) is much closer to Caligeo output, but still not the same. Note below magnitude differences, exaggerated absorption feature at ~1600nm, slightly different shape of leftmost peak and lower starting low-end value (note may be due to spectral shifting, see #107).

Hawk level 1b spectra over land from azspec and Caligeo

Changed 12 years ago by benj

Hawk raw data over water. White line is single pixel, red line is 8px average

Changed 12 years ago by benj

Hawk level 1b spectrum processed by azspec over water - single pixel

comment:3 Changed 12 years ago by benj

Checked relationship between raw data bad pixels and spikes in azspec-processed data. Of the two graphs below (top is raw data, white line being single pixel spectrum and red being 8-pixel average, bottom being azspec-processed level 1b single pixel for the same pixel), it can be seen that the azspec processed data has spikes that do not correspond to bad pixels even within the 8-pixel range (red line).

Raw data:

Hawk raw data over water. White line is single pixel, red line is 8px average

Azspec-processed:

Hawk level 1b spectrum processed by azspec over water - single pixel

comment:4 Changed 12 years ago by mggr

This problem has been passed to Bill for investigation (first formal contact 6/March), investigation begun on 10th. Follow up contact on 11th and 12th.

Changed 12 years ago by mggr

Spikes vs raw, vertical profile

comment:5 Changed 12 years ago by mggr

Bill responded yesterday with some initial thoughts, particularly on bad pixels. I did a little more investigation on the exact locations of some of these spikes.

The following image has the following important points to note:

  • very bright speckles in the azspec image are the hawk "spikes"
    • note these are not exactly in lines, as with the obvious bad pixels
  • the two plots on the right are vertical profiles (image-Y coord is the plot x axis) for a single line where speckles manifest, top plot is the raw data and bottom is the azspec calibrated output (spikes are obvious)
  • note that the spikes correspond with raw values that are below the measured dark value (actually align with the pixel before-in-raw/after-in-processing the low value, but this may be something to do with the processing algorithm or the way I measured the Y offset between raw and azspec data)

Minor other comments for reproducing:

  • azspec inserts blank padding lines for missing frames, hence there is an offset of approximately 18 in Y coordinates (measured by hand in the region of interest by finding a small bright pixel present in raw and azspec and computing the different in coords)
  • band 166, line 30 in azspec, line 290 (320-30, due to the flip) was the one examined
  • dark value was measured by hand (couldn't get azspec to report it with -da). Used all the dark frames but the first one, which was obviously non-dark and averaged them to get 5519.77777 (~5520).
    actual values averaged:
    (5520+5536+5531+5516+5509+5512+5509+5519+5520+5513+5539+5522+5523+5509+5521+5522+5520+5519+5533+5517+5520+5528+5529+5521+5512+5518+5522+5510+5519+5512+5504+5530+5520+5514+5523+5520)/36
    


An interpretation is that the bright spikes occur when a pixel goes below the dark value.

Spikes vs raw, vertical profile

comment:6 Changed 12 years ago by mggr

  • Owner changed from benj to mggr
  • Status changed from new to assigned

Bill also found the underflow bug above and has supplied an updated azspec to test. The bad pixel issue is still outstanding (note other ticket for this)

Changed 12 years ago by mggr

comment:7 Changed 12 years ago by mggr

Visual examination indicates spikes are fixed.

(left to right: old-azspec, caligeo, new-azspec)


Changed 12 years ago by benj

Old and new azspec processed data over water

Changed 12 years ago by benj

New azspec and Caligeo processed data over water

Changed 12 years ago by benj

New azspec and caligeo over land

comment:8 Changed 12 years ago by benj

  • Resolution set to fixed
  • Status changed from assigned to closed

New Azspec from Bill seems to have fixed the spike problem - Azspec over water no longer shows spikes (see below), old and new azspec match exactly except for the spikes over both land and water. Marking bug fixed.

Old and new azspec processed data over water

comment:9 Changed 12 years ago by mggr

Work order number was 200803-1

comment:10 Changed 12 years ago by mggr

  • Type changed from task to bug

comment:11 Changed 12 years ago by mggr

Work orders renumbered to make it obvious when one is missing. This work order is now wo2008001.

Note: See TracTickets for help on using tickets.