Changes between Version 8 and Version 9 of Procedures/OwlProcessing


Ignore:
Timestamp:
Aug 27, 2015, 10:54:58 AM (9 years ago)
Author:
lah
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/OwlProcessing

    v8 v9  
    11= Owl Processing Guide =
    22
    3 The Owl thermal hyperspectral sensor is new to the arsf and does not yet have a complete processing chain.  As this is developed this wiki page will be updated. Details on the Owl sensor are available [https://arsf-dan.nerc.ac.uk/trac/wiki/Sensors/Owl here] and information on Owl testing is available [https://rsg.pml.ac.uk/intranet/trac/wiki/Projects/ARSF-DAN/Users/lah/OwlNotes here].
     3The Owl thermal hyperspectral sensor is new to the arsf and so new problems may occur during processing.  Please add any new issues to the bottom of this wiki. Details on the Owl sensor are available [https://arsf-dan.nerc.ac.uk/trac/wiki/Sensors/Owl here] and information on Owl testing is available [https://rsg.pml.ac.uk/intranet/trac/wiki/Projects/ARSF-DAN/Users/lah/OwlNotes here].
    44
    5 == Rename Files ==
     5== File Naming ==
    66
    7 Our scripts require that data be named according to the standard convention e.g. OWL219b-14-1, where 219 is the julian day, b the sortie, 14 the year and 1 the flightline. If this is not the case, you must rename both the files and the directories. renameOwl.py can be run with -p specifying the project directory. You will need to set write permissions for all the raw directories and files to do this; taking a backup is strongly recommended.
     7Our scripts require that data be named according to the standard convention e.g. OWL219b-14-1, where 219 is the julian day, b the sortie, 14 the year and 1 the flightline. The files should have been renamed during unpacking, but if this is not the case, run proj_tidy.sh -e -p <project directory> to generate the move commands. Check that every move command is present before executing, as partial renaming cannot be completed in a second run. Taking a backup of the data files is recommended before renaming.
     8
     9== Dark Frames ==
     10
     11The radiometric calibration will fail if the dark frames are saved in a separate file to the data (Ops sometimes do this). Check the capture directory for any extra files labelled like they might be dark frames and append them to the end of the data file. Remember to add the autodarkstartline key to the header file. Alternatively use the script stitchOwl.py.
    812
    913== Radiometric Calibration ==
     
    1923All output files are written to the flight line subdirectory of /processing/owl/flightlines/level1b and logs written to /processing/owl/logfiles. The output files consist of the data (*_proc.bil) and if not specified as inputs, the calibration (*_calibration.rad) and blinker files (*blinkers.dat), each with their own header file. During processing a symlink appears in the output flight line folder to prevent simultaneous processing over the same line.  You may need to remove this if processing is aborted before it is automatically removed.
    2024
    21 If it results in producing weird files like `tp3f35cf3d_7182_4f17_8294_425fa4dc2953.raw` and the logs say stuff like `No autodark found` then the files need to be appended with dark frames.
     25Files named as `tp3f35cf3d_7182_4f17_8294_425fa4dc2953.raw` are temporary files and will be removed on completion. If they remain check the logs for errors. `No autodark found` means the files need to be appended with dark frames.
    2226
    2327=== Blinking Pixels/Vertical Stripes ===
     
    3337|| Level 1b Owl data (2014 238) calibrated with different values of pixstabi (default 0.011, 0.005, 0.003, 0.0025, 0.002) ||
    3438
     39== Dropped Frames ==
     40
     41If there are dropped frames they should be recorded in a .log file in the capture directory. These dropped frames need to be inserted into the level 1b data file as empty lines before geocorrection to map the flight line correctly.
     42
    3543== Geocorrection ==
    3644
    3745Create a dem in exactly the same way as for fenix.  This should be saved in the processing/hyperspectral/dem directory, not the owl directories, so only 1 copy is required.  e.g. from the project directory type create_apl_dem.py --aster.
    3846
    39 Geocorrection is conducted in the same way as for fenix data, using APL. Generate a config file using generate_apl_config.py -o processing/owl/o2014219b.cfg (make sure it begins with an '''o''')to keep processing separate from fenix. Currently you need to add the boresight values to the config file (owl_boresight = 0.12 -0.32 0.32). You may want to delete or set to false the fenix lines if only processing owl data.  Make sure the project directory structure is complete (same as fenix). Submit to the grid using specim_qsub.py <config_file> and calculate the SCTs. Once you have done this you can generate the mapped data (102 bands) by setting owl_bandlist = ALL and setting the SCT value for each line.
     47Geocorrection is conducted in the same way as for fenix data, using APL. Generate a config file using generate_apl_config.py -o processing/owl/o2014219b.cfg (make sure it begins with an '''o''') to keep processing separate from fenix. Currently you need to add the boresight values to the config file ( e.g. owl_boresight = 0.12 -0.32 0.32). You may want to delete or set to false the fenix lines if only processing owl data.  Make sure the project directory structure is complete (same as fenix). Submit to the grid using specim_qsub.py <config_file> and calculate the SCTs. Once you have done this you can generate the mapped data (102 bands) by setting owl_bandlist = ALL and setting the SCT value for each line.
     48
     49Early owl files have corrupt nav files, so if you get this message in the log or 'Requested sync time index is out of bounds in GetSyncDelay()' set the config file to use_nav = false.
     50
     51If there appears to be stripping across the mapped files, bright and dark lines, or sections of a flight line don't match up the sensor has probably dropped frames. These can be checked in the .log file in the capture directory. These empty lines need to be inserted into the level 1b data file before geocorrection to map the flight line correctly.
    4052
    4153== Making a delivery ==