Changes between Version 2 and Version 3 of Procedures/hyperspecprocessing


Ignore:
Timestamp:
Feb 13, 2012, 10:59:00 AM (13 years ago)
Author:
mark1
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Procedures/hyperspecprocessing

    v2 v3  
    1010
    1111When ARSF-DAN receive hyperspectral raw data, it enters the processing chain which consists of the following stages:
    12  * Initial checks and re-formatting project structure <link>
    13     * [include unpacking etc and Digital Elevation Model (DEM) generation]
    14  * Aircraft position and navigation post processing <link>
    15  * APL Hyperspectral chain <link>
    16  * Delivery creation
    17  * Delivery checking <link>
    18  * Dispatch Dispatch <link>
     12 * [wiki:Procedures/NewDataArrival Initial checks and re-formatting project structure]
     13    * This includes [wiki:Help/DEM_scripts Digital Elevation Model (DEM) generation]
     14 * [wiki:Procedures/ProcessingChainInstructions/NavigationProcessing Aircraft position and navigation post processing]
     15 * [#HyperAPLChain APL Hyperspectral chain]
     16 * [wiki:Procedures/EagleHawkProcessing#Makingadelivery Delivery creation]
     17 * [wiki:Procedures/DeliveryChecking Delivery checking]
     18 * [wiki:Procedures/EndUserDelivery Dispatch Dispatch]
    1919
    20 After the data has been dispatched to the PI, the project directories will be tidied up and the data archived at the NERC Earth Observation Data Centre (NEODC) <link>. The data will then be available for use by other parties after an initial embargo period of 1 year after dispatch.
     20After the data has been dispatched to the PI, the project directories will be tidied up and the data archived at the [http://neodc.nerc.ac.uk/ NERC Earth Observation Data Centre (NEODC)]. The data will then be available for use by other parties after an initial embargo period of 1 year after dispatch.
    2121
    2222-------------------------------------------------------------
    23 == Hyperspectral processing using the Airborne Processing Library ==
     23== Hyperspectral processing using the Airborne Processing Library == #HyperAPLChain
    2424
    2525
    … …  
    3131The first stage of processing is to apply a radiometric calibration to the raw data and generate a mask file. This can be summarised as follows:
    3232
    33  * Normalise the data.
     33'''Normalise the data. '''
    3434 
    35 The captured dark frames are used to generate a per CCD pixel (i.e. per sample per band) average value which can be used as an effective "0" value. That is, it gives the value of the pixel when no light is shining on the CCD. This is therefore a measure of the noise of the system at time of capture. The raw data is then normalised to this by subtracting the corresponding average dark value. If the value would drop below 0 after subtraction it is set to equal 0.
     35 The captured dark frames are used to generate a per CCD pixel (i.e. per sample per band) average value which can be used as an effective "0" value. That is, it gives the value of the pixel when no light is shining on the CCD. This is therefore a measure of the noise of the system at time of capture. The raw data is then normalised to this by subtracting the corresponding average dark value. If the value would drop below 0 after subtraction it is set to equal 0.
    3636
    37 * Smear correct Eagle data
     37'''Smear correct Eagle data'''
    3838
    39 The Eagle sensor use a CCD blah blah blah <insert smear text>. The Eagle data therefore needs to be corrected for any light that is coming from the other bands as the data is extracted. The formula used for this is:
    40 Iic = Ii - f*Sum(Ij)
    41 where Iic is the smear corrected image band i, Ii is the image band i, f is the frame smear correction scalar, Ij is the image band j.
     39 The Eagle uses a CCD that shifts data out line by line at the end of a frame. While this readout process is quick, additional light still falls onto the detector during the readout period. The Eagle data therefore needs to be corrected for any light that is coming from the other bands as the data is extracted. The formula used for this is:
    4240
    43  * Apply gains
     41 I,,ic,, = I,,i,, - f*Sum,,j<i,,(I,,j,,)
    4442
    45 At the start and end of the flying season the Eagle and Hawk instruments are "factory" calibrated. Since 2011 this has been done in-house by ARSF and FSF. One of the products of the instrument calibration is the gain multipliers. This is a per sample list of scalars that convert the sensor captured data value into a "real world" at-sensor radiance value. The image data is multiplied by the corresponding gain value to give the at-sensor radiance value.
     43 where I,,ic,, is the smear corrected image band i, I,,i,, is the image band i, f is the frame smear correction scalar, I,,j,, is the image band j.
    4644
    47  * Calibrate FODIS if Eagle data
     45'''Apply gains'''
    4846
    49 The FODIS is blah blah blah <insert FODIS text>. This is calibrated in the same way as the other raw data. When calibrated, the FODIS pixels for the same scan line are averaged together to further reduce random noise.
     47 At the start and end of the flying season the Eagle and Hawk instruments are "factory" calibrated. Since 2011 this has been done in-house by ARSF and FSF. One of the products of the instrument calibration is the gain multipliers. This is a per sample list of scalars that convert the sensor captured data value into a "real world" at-sensor radiance value. The image data is multiplied by the corresponding gain value to give the at-sensor radiance value.
    5048
    51  * Insert Missing Scans
     49'''Calibrate FODIS if Eagle data'''
    5250
    53 Occaisionaly the sensor "drops" a scan line. This can be identified by examining the frame counter through the raw file and observing where it increases by more than 1 between consequetive scans. If a missing scan is identified in the raw image, a dummy scan line of 0s is output to the calibrated image.
     51 The FODIS is blah blah blah <insert FODIS text>. This is calibrated in the same way as the other raw data. When calibrated, the FODIS pixels for the same scan line are averaged together to further reduce random noise.
    5452
    55  * Flag Pixels over/under flown, bad, missing
     53'''Insert Missing Scans'''
    5654
    57 A mask file will be created at the same time that the radiometric calibration is applied. This is a file of the same dimensions as the raw data file that contains the status of each calibrated pixel. The values in the mask file are as follows:
    58 0 - good data
    59 1 - underflown data - the raw value after normalisation is less than 0.
    60 2 - overflown data - the raw value is equal to the maximum (4095 for Eagle, 16383 for Hawk).
    61 4 - bad pixel - usually refers to a Hawk CCD pixel that is considered untrustworthy. Is also used for first 2 pixels of band 1.
    62 8 - Smear affected - a longer wavelength Eagle band has overflown causing the smear correction for this pixel to be incorrect by an unknown quantity.
    63 16 - Dropped scan - a missing scan has been detected and inserted here.
     55 Occaisionaly the sensor "drops" a scan line. This can be identified by examining the frame counter through the raw file and observing where it increases by more than 1 between consequetive scans. If a missing scan is identified in the raw image, a dummy scan line of 0s is output to the calibrated image.
    6456
    65  * Flipping data and writing
     57'''Flag Pixels over/under flown, bad, missing'''
    6658
    67 The calibrated data, FODIS and mask files are stored in ENVI BIL format. Prior to writing out, the image data and mask data is reordered. The Eagle data is "flipped" spectrally such that the first band is the lowest wavelength and the last band is the highest (i.e. such that wavelengths are ordered blue to red). The Hawk data is "flipped" spatially such that each scan line is reveresed (i.e. pixel 1 becomes pixel 320). This is done because the Eagle and Hawk are mounted backwards to each other and flipping allows targets to be more easily compared between the two sensors data.
     59 A mask file will be created at the same time that the radiometric calibration is applied. This is a file of the same dimensions as the raw data file that contains the status of each calibrated pixel. The values in the mask file are as follows:
     60 0 - good data
     61 1 - underflown data - the raw value after normalisation is less than 0.
     62 2 - overflown data - the raw value is equal to the maximum (4095 for Eagle, 16383 for Hawk).
     63 4 - bad pixel - usually refers to a Hawk CCD pixel that is considered untrustworthy. Is also used for first 2 pixels of band 1.
     64 8 - Smear affected - a longer wavelength Eagle band has overflown causing the smear correction for this pixel to be incorrect by an unknown quantity.
     65 16 - Dropped scan - a missing scan has been detected and inserted here.
     66
     67'''Flipping data and writing'''
     68
     69 The calibrated data, FODIS and mask files are stored in ENVI BIL format. Prior to writing out, the image data and mask data is reordered. The Eagle data is "flipped" spectrally such that the first band is the lowest wavelength and the last band is the highest (i.e. such that wavelengths are ordered blue to red). The Hawk data is "flipped" spatially such that each scan line is reveresed (i.e. pixel 1 becomes pixel 320). This is done because the Eagle and Hawk are mounted backwards to each other and flipping allows targets to be more easily compared between the two sensors data.
    6870
    6971