FODIS processing appears incorrect

[mggr]

With the recent acquisitions of the first FODIS data, we attempted to process some using azspec and appear to have a problem. By comparison with the solar spectrum (​http://fsf.nerc.ac.uk/img/g37_grph.gif) and a Caligeo processing of the same line, there is a clear discrepancy. Image below shows a worst case selection of the azspec FODIS output.

relevant part of processing log:

azspec  -- ver: 1.1.7 Apr  3 2008   (C) Azimuth Systems UK 2005, 2007

Raw file header: eagle/VNIRsecloudtest-1.hdr
Raw file       : eagle/VNIRsecloudtest-1.raw
Dark frames at end of image file, 152 frames starting at: 3932
Image lines corrected from: 4083  to 3931
Using Tr multiplier from spectral binning[2]:  1.00  fsmscale is now: 0.000100  Tr: 0.002000  integ: 20.0
Optics from basic details and binning port fov: 16.129  full fov: 34.688 corr oa: 512.00 scpixels: 953
image file frame count  start: 60092  end: 64047  total lines: 3955 file lines: 3931

Dark frames at end of raw file: 152  darks to be used: 50

Dark frames used:  49.0   scale: 0.02041
Radcal file header: calibration/eagle/SN100022_130207_XB.hdr
Radcal file       : calibration/eagle/SN100022_130207_XB.cal
Config file       : calibration/eagle/SN100022_130207_XB.cfg

Using HDF file: lev1/eJJJ011b_gpt0.0.hdf
HDFStart  fov: 34.7 fovp: 16.1 scpixels: 953

New BIL  bands: 252 pix: 953 lines: 3955
   HDF image data: CAimage  bands: 252 pixels: 953 lines: 3955
   FODIS (ILS) bands: 252 pixels: 1 lines: 3955
2 frame(s) missing from: 60094  to 60097 line: 2 cline: 3
2 frame(s) missing from: 60098  to 60101 line: 6 cline: 5
2 frame(s) missing from: 60102  to 60105 line: 10 cline: 7
1 frame(s) missing from: 60112  to 60114 line: 20 cline: 15
1 frame(s) missing from: 60115  to 60117 line: 23 cline: 17
3 frame(s) missing from: 60118  to 60122 line: 26 cline: 19
2 frame(s) missing from: 60126  to 60129 line: 34 cline: 24
1 frame(s) missing from: 60130  to 60132 line: 38 cline: 26
2 frame(s) missing from: 60133  to 60136 line: 41 cline: 28
2 frame(s) missing from: 60138  to 60141 line: 46 cline: 31
2 frame(s) missing from: 60142  to 60145 line: 50 cline: 33
2 frame(s) missing from: 60146  to 60149 line: 54 cline: 35
1 frame(s) missing from: 60157  to 60159 line: 65 cline: 44
2 frame(s) missing from: 60964  to 60967 line: 872 cline: 850
**FrameProcess - HDF write error for ILS line: 3955
SpecNavProc gotsync: 0  synclater: 1

Nav syncstart from image file header of: 49107.62500 gps secs of day
sync entries set for later use
    start frame: 0 end: 3954
    start frame time: 49107.624900  frame time increment: 0.033333 secs

writing new NAV vgroup

Processing summary...

  data from   : AISA HS Instrument, SW ver 2.800
  sensor ID   : 100022
  lens FOV    : 34.69 degs
  raw file    : eagle/VNIRsecloudtest-1.raw
  BIL lines   : 3931  available
  acq date    : 31/01/2008  day: 31
  rec times   : start: 13:38:27.62  end: 13:40:45.19
  binning     : 2  spectral  1 spatial
  integration : 20.00  msecs
  frame rate  : 30.00  fps
  radiance sc : 1000.000
  HDF file    : lev1/eJJJ011b_gpt0.0.hdf
  Image data  :  pix: 72 to 1024  saved to HDF-SDS item: CAimage
  FODIS (ILS) :  pix: 28 to 32 saved to HDF-SDS item: CAils
  pixels      : 1024
  bands       :  252  re-ordered blue to red
  image lines : 3955  frameIDs at line (0): 60092 and line (3954): 64047

Processing applied...

  frame shift smear correction, line shift scale: 0.00010
  dark frame subtraction, lines averaged: 50  dark frames from raw file
  radiometric calibration using file: calibration/eagle/SN100022_130207_XB.cal
  ** total pixels overflowed after calibration: 556
  FODIS averaged
  ** no FODIS ratio applied

Processing times  : calibration: 64.2 total: 237.6  secs

halting after azspec (no need to do the rest)

Sent a copy of the Caligeo and azspec processed files, the raw data and the script to Bill, with a brief explanatory note.

[mggr]

Incidentally, note the level of noise in the FODIS region, presumably from second order diffraction effects?

Enhanced image showing stray light:

[mggr]

Note: not sure if straylight correction was on or off in the CaliGeo processing.

[benj]

FODIS output(*pi) vs modelled solar spectrum and irradiance at plane

[benj]

Model output file

[benj]

Gnuplot script for generating graph output

[benj]

Compared FODIS spectrum to modelled spectrum from SMARTS. FODIS spectrum is slightly low but generally a good match - possibly the sensor input is just dirty? Note FODIS output below has been multiplied by pi - model output is in W/m², CaliGeo fodis output is in W m^-2 sr^-1 nm^-1, multiplied by pi to get rid of the sr^-1.

[mggr]

Passed to Bill as work order 200805-1.

[mggr]

Bill responded with some fixes (azspec, qcdisplay and azexhdf) on 14/May.

I finally re-ran the test line and extracted the FODIS values with azexhdf, then compared with the caligeo ones in ENVI - they look good :)

Plotting the FODIS data direct from the HDF with ENVI does show weird spiky results, but looking at the data values in ENVI or plotting/examining the HDF with other tools (HDFLook) looks fine. Freelook just crashes when you try to look at it too, so I guess the plotting errors are an ENVI bug..

There are some differences in the spectrum, consistent with the differences between azspec and caligeo in the main image - see #119.

Looks like the FODIS data is now consistent with normal imagery though, so calling this fixed.

[mggr]

comparison of fixed-azspec with caligeo, with bright values (no cloud?)

[mggr]

comparison of fixed-azspec with caligeo, with dark values (under cloud?)

[mggr]

fixed azspec & caligeo vs solar spectrum

[mggr]

For completeness, fixed az & cali (bright values) vs solar spectrum:

[mggr]

comparison of fixed-azspec with caligeo, with bright values (no cloud?)

comparison of fixed-azspec with caligeo, with dark values (under cloud?)

[mggr]

Work orders renumbered to make it obvious when one is missing. This work order is now wo2008003.

[mggr]

Oops, azspec.118 works for FODIS, but has some effects in the main image. Areas that were previously bright whites (but not overflowed) are now black.

[mggr]

Moving overflow issue to #168, retaining this ticket purely for the FODIS fix.

[mggr]

FODIS looks fixedin 1.1.8 and 1.1.11, reclosing.

[mggr]

CAUTION

Note: to get sensible FODIS output, do NOT browse the hdf directly with ENVI. For whatever reason, that comes out as nonsense. You must export the FODIS data with:
azexhdf -Be fodis.bil -d CAils eagle0.hdf

This output file can be viewed in ENVI.