Changes between Version 11 and Version 12 of Processing/atcor


Ignore:
Timestamp:
May 2, 2018 9:18:02 AM (16 months ago)
Author:
dac
Comment:

Added instructions for running atmospheric correction on NERC-ARF system.

Legend:

Unmodified
Added
Removed
Modified
  • Processing/atcor

    v11 v12  
    55== NERC-ARF Specific Instructions ==
    66
    7 For running on the NERC-ARF systems there are a number of utilities to make batch processing easier.
     7For running on the NERC-ARF systems there are a number of conventions and scripts used for processing.
    88
    9 **Work in progress - more documentation will be added as it is written **
     9=== Directory Structure ===
    1010
    11 === Generating the .inn file ===
     11**Processing**
     12{{{
     13processing/hyperspectral/atmospheric_correction/<flightline>
     14}}}
     15ATCOR prefers to have all files for a flightline in a single directory
    1216
    13 To generate the .inn file for ATCOR use the `gen_inn_atcor.py` script.
     17**Delivery**
    1418
    15 * `--calfile` needs to point to the .cal file ATCOR users, this wil be `~arsf/usr/opt/atcor_4/sensor/SENSOR/*.cal` where `SENSOR` is the name used for the sensor definition in ATCOR (e.g., f17_b2).
    16 *  `inputbil` needs to point to the bil file in the delivery folder directory.
    17 * `--outputdir` should be the directory used or ATCOR processing.
     19The following directories are included in addition to the ones included with a standard hyperspectral delivery:
     20{{{
     21flightlines/level2_ac  (the level1 corrected *atm.bil)
     22flightlines/mappped_ac  (the level3 corrected *atm.bil)
     23mosaic/                (if a level3 corrected mosaic has been generated - multiple subdirs if required)
     24doc/                   (standard documentation on procedure - not flight specific)
     25screenshots/
     26           fJJJll3b_atm.jpg
     27AtmosCorr_readme-DATE.pdf   (description of all AC work and parameters, software versions, algorithms, include a table with what
     28                        has been processed, e.g.  | Topography   | YES   |
     29                                                  | BRDF         | YES   |
     30                                                  | Cloud shadow | No    |
     31}}}
    1832
    19 If microtops data has been provided pass this in using `--microtops` to read using the https://github.com/robintw/PyMicrotops library and set a constant visibility based on this. Note if the microtops data has a number of header lines these need to be removed before PyMicrotops can read the file, there should just be one header line with the column names.
     33=== Running ATCOR ===
    2034
    21 After generating the .inn file it should be renamed to use the same name as the image (without  the .bsq extension). For example if the file being processed in ATCOR is 'f168043b_masked_resampled_utm31n.bsq' name 'f168043b_masked_resampled_utm31n.inn'.
     35Running ATCOR comprises generating the .inn parameter file and then running ATCOR through IDL. For files which have entire bands masked out (e.g., Fenix data from late 2014 - 2015) these bands need to be removed before running. The script `run_atm_correction_atcor.py` will run the entire process on a file which has been masked using `aplmask` using the NERC-ARF default parameters.
    2236
    23 Currently ATCOR doesn't always read all the parameters correctly open the .inn file (plain text) and update the values in the ATCOR GUI if needed.
     37{{{
     38run_atm_correction_atcor.py -d <delivery_directory> <masked_bil_file>
     39}}}
    2440
    25 === Batch running .inn files ===
     41The directory the masked bil file is in will be used for all ATCOR files so it is recommended a separate directory is used for each line.
    2642
    27 Once .inn files have been generated and checked using the ATCOR GUI it is possible to run in batch using the script `batch_run_atcor_inn.py`. To do so ssh onto 'rsg-atcor' and run:
     43To run multiple files a separate queue has been set up for ATCOR. To submit ATCOR jobs to the grid a command similar the below can be used:
    2844{{{
    29 batch_run_atcor_inn.py /path/to/files/to/process
     45qsub -q atcor.q -P arsfdan -m n -b y -N acor_test \
     46         -o /users/rsg/dac/scratch_network/atcor_runs/2017_166/f166071b/  \
     47         -e /users/rsg/dac/scratch_network/atcor_runs/2017_166/f166071b/  \
     48         /users/rsg/dac/scratch_network/git_repos/arsf_atcor_scripts/scripts/run_atm_correction_atcor.py \
     49         -d /users/rsg/arsf/arsf_data/2017/flight_data/spain/RG17_26-2017_166_Albacete/delivery/RG17_26-166-hyperspectral-20170823 \
     50         /users/rsg/dac/scratch_network/atcor_runs/2017_166/f166071b/f166071b_masked.bil
    3051}}}
    31 This will recursively search for .inn files and run them through ATCOR.
     52Note only a single job will run at once.
     53
     54**Non-standard parameters**
     55
     56For more control over the parameters passed to ATCOR, for example to include microtops data to define visibility, the `gen_inn_atcor.py` script can be used.
     57The .inn files can be opened in the ATCOR GUI and run through that or the `batch_run_atcor_inn.py` can be used.