Opened 11 years ago

Closed 11 years ago

Last modified 11 years ago

#133 closed bug (fixed)

FODIS processing appears incorrect

Reported by: mggr Owned by: mggr
Priority: immediate Milestone:
Component: az* programs Keywords:
Cc: Other processors:

Description

With the recent acquisitions of the first FODIS data, we attempted to process some using azspec and appear to have a problem. By comparison with the solar spectrum (http://fsf.nerc.ac.uk/img/g37_grph.gif) and a Caligeo processing of the same line, there is a clear discrepancy. Image below shows a worst case selection of the azspec FODIS output.


relevant part of processing log:

azspec  -- ver: 1.1.7 Apr  3 2008   (C) Azimuth Systems UK 2005, 2007

Raw file header: eagle/VNIRsecloudtest-1.hdr
Raw file       : eagle/VNIRsecloudtest-1.raw
Dark frames at end of image file, 152 frames starting at: 3932
Image lines corrected from: 4083  to 3931
Using Tr multiplier from spectral binning[2]:  1.00  fsmscale is now: 0.000100  Tr: 0.002000  integ: 20.0
Optics from basic details and binning port fov: 16.129  full fov: 34.688 corr oa: 512.00 scpixels: 953
image file frame count  start: 60092  end: 64047  total lines: 3955 file lines: 3931

Dark frames at end of raw file: 152  darks to be used: 50

Dark frames used:  49.0   scale: 0.02041
Radcal file header: calibration/eagle/SN100022_130207_XB.hdr
Radcal file       : calibration/eagle/SN100022_130207_XB.cal
Config file       : calibration/eagle/SN100022_130207_XB.cfg

Using HDF file: lev1/eJJJ011b_gpt0.0.hdf
HDFStart  fov: 34.7 fovp: 16.1 scpixels: 953

New BIL  bands: 252 pix: 953 lines: 3955
   HDF image data: CAimage  bands: 252 pixels: 953 lines: 3955
   FODIS (ILS) bands: 252 pixels: 1 lines: 3955
2 frame(s) missing from: 60094  to 60097 line: 2 cline: 3
2 frame(s) missing from: 60098  to 60101 line: 6 cline: 5
2 frame(s) missing from: 60102  to 60105 line: 10 cline: 7
1 frame(s) missing from: 60112  to 60114 line: 20 cline: 15
1 frame(s) missing from: 60115  to 60117 line: 23 cline: 17
3 frame(s) missing from: 60118  to 60122 line: 26 cline: 19
2 frame(s) missing from: 60126  to 60129 line: 34 cline: 24
1 frame(s) missing from: 60130  to 60132 line: 38 cline: 26
2 frame(s) missing from: 60133  to 60136 line: 41 cline: 28
2 frame(s) missing from: 60138  to 60141 line: 46 cline: 31
2 frame(s) missing from: 60142  to 60145 line: 50 cline: 33
2 frame(s) missing from: 60146  to 60149 line: 54 cline: 35
1 frame(s) missing from: 60157  to 60159 line: 65 cline: 44
2 frame(s) missing from: 60964  to 60967 line: 872 cline: 850
**FrameProcess - HDF write error for ILS line: 3955
SpecNavProc gotsync: 0  synclater: 1

Nav syncstart from image file header of: 49107.62500 gps secs of day
sync entries set for later use
    start frame: 0 end: 3954
    start frame time: 49107.624900  frame time increment: 0.033333 secs

writing new NAV vgroup

Processing summary...

  data from   : AISA HS Instrument, SW ver 2.800
  sensor ID   : 100022
  lens FOV    : 34.69 degs
  raw file    : eagle/VNIRsecloudtest-1.raw
  BIL lines   : 3931  available
  acq date    : 31/01/2008  day: 31
  rec times   : start: 13:38:27.62  end: 13:40:45.19
  binning     : 2  spectral  1 spatial
  integration : 20.00  msecs
  frame rate  : 30.00  fps
  radiance sc : 1000.000
  HDF file    : lev1/eJJJ011b_gpt0.0.hdf
  Image data  :  pix: 72 to 1024  saved to HDF-SDS item: CAimage
  FODIS (ILS) :  pix: 28 to 32 saved to HDF-SDS item: CAils
  pixels      : 1024
  bands       :  252  re-ordered blue to red
  image lines : 3955  frameIDs at line (0): 60092 and line (3954): 64047

Processing applied...

  frame shift smear correction, line shift scale: 0.00010
  dark frame subtraction, lines averaged: 50  dark frames from raw file
  radiometric calibration using file: calibration/eagle/SN100022_130207_XB.cal
  ** total pixels overflowed after calibration: 556
  FODIS averaged
  ** no FODIS ratio applied

Processing times  : calibration: 64.2 total: 237.6  secs

halting after azspec (no need to do the rest)

Sent a copy of the Caligeo and azspec processed files, the raw data and the script to Bill, with a brief explanatory note.

Attachments (9)

eagle_fodis-cropped.jpg (99.6 KB) - added by mggr 11 years ago.
eagle_fodis_straylight.jpg (100.4 KB) - added by mggr 11 years ago.
fodis.png (7.9 KB) - added by benj 11 years ago.
FODIS output(*pi) vs modelled solar spectrum and irradiance at plane
fodis_test.ext.txt (44.4 KB) - added by benj 11 years ago.
Model output file
plotsmartsvsfodis.gpl (667 bytes) - added by benj 11 years ago.
Gnuplot script for generating graph output
light_comparison.png (5.0 KB) - added by mggr 11 years ago.
comparison of fixed-azspec with caligeo, with bright values (no cloud?)
dark_comparison.png (5.2 KB) - added by mggr 11 years ago.
comparison of fixed-azspec with caligeo, with dark values (under cloud?)
vssolar.png (9.3 KB) - added by mggr 11 years ago.
fixed azspec & caligeo vs solar spectrum
azspec_117_118_diff.jpg (265.9 KB) - added by mggr 11 years ago.

Download all attachments as: .zip

Change History (23)

Changed 11 years ago by mggr

Changed 11 years ago by mggr

comment:1 Changed 11 years ago by mggr

  • Status changed from new to assigned

Incidentally, note the level of noise in the FODIS region, presumably from second order diffraction effects?

Enhanced image showing stray light:


comment:2 Changed 11 years ago by mggr

Note: not sure if straylight correction was on or off in the CaliGeo processing.

Changed 11 years ago by benj

FODIS output(*pi) vs modelled solar spectrum and irradiance at plane

Changed 11 years ago by benj

Model output file

Changed 11 years ago by benj

Gnuplot script for generating graph output

comment:3 Changed 11 years ago by benj

Compared FODIS spectrum to modelled spectrum from SMARTS. FODIS spectrum is slightly low but generally a good match - possibly the sensor input is just dirty? Note FODIS output below has been multiplied by pi - model output is in W/m2, CaliGeo fodis output is in W m-2 sr-1 nm-1, multiplied by pi to get rid of the sr-1.

FODIS output(*pi) vs modelled solar spectrum and irradiance at plane

comment:4 Changed 11 years ago by mggr

Passed to Bill as work order 200805-1.

comment:5 Changed 11 years ago by mggr

Bill responded with some fixes (azspec, qcdisplay and azexhdf) on 14/May.

I finally re-ran the test line and extracted the FODIS values with azexhdf, then compared with the caligeo ones in ENVI - they look good :)

Plotting the FODIS data direct from the HDF with ENVI does show weird spiky results, but looking at the data values in ENVI or plotting/examining the HDF with other tools (HDFLook) looks fine. Freelook just crashes when you try to look at it too, so I guess the plotting errors are an ENVI bug..

There are some differences in the spectrum, consistent with the differences between azspec and caligeo in the main image - see #119.

Looks like the FODIS data is now consistent with normal imagery though, so calling this fixed.

Changed 11 years ago by mggr

comparison of fixed-azspec with caligeo, with bright values (no cloud?)

Changed 11 years ago by mggr

comparison of fixed-azspec with caligeo, with dark values (under cloud?)

Changed 11 years ago by mggr

fixed azspec & caligeo vs solar spectrum

comment:6 Changed 11 years ago by mggr

For completeness, fixed az & cali (bright values) vs solar spectrum:

fixed azspec & caligeo vs solar spectrum

comment:7 Changed 11 years ago by mggr

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:8 Changed 11 years ago by mggr

comparison of fixed-azspec with caligeo, with bright values (no cloud?)

comparison of fixed-azspec with caligeo, with bright values (no cloud?)

comparison of fixed-azspec with caligeo, with dark values (under cloud?)

comparison of fixed-azspec with caligeo, with dark values (under cloud?)

comment:9 Changed 11 years ago by mggr

Work orders renumbered to make it obvious when one is missing. This work order is now wo2008003.

comment:10 Changed 11 years ago by mggr

  • Resolution fixed deleted
  • Status changed from closed to reopened

Oops, azspec.118 works for FODIS, but has some effects in the main image. Areas that were previously bright whites (but not overflowed) are now black.

Changed 11 years ago by mggr

comment:11 Changed 11 years ago by mggr


comment:12 Changed 11 years ago by mggr

Moving overflow issue to #168, retaining this ticket purely for the FODIS fix.

comment:13 Changed 11 years ago by mggr

  • Resolution set to fixed
  • Status changed from reopened to closed

FODIS looks fixedin 1.1.8 and 1.1.11, reclosing.

comment:14 Changed 11 years ago by mggr

CAUTION

Note: to get sensible FODIS output, do NOT browse the hdf directly with ENVI. For whatever reason, that comes out as nonsense. You must export the FODIS data with:
azexhdf -Be fodis.bil -d CAils eagle0.hdf

This output file can be viewed in ENVI.

Note: See TracTickets for help on using tickets.