Changes between Version 11 and Version 12 of Processing/atcor

Timestamp:

May 2, 2018, 9:18:02 AM (6 years ago)

Author:

dac

Comment:

Added instructions for running atmospheric correction on NERC-ARF system.

Processing/atcor

@@ -5 +5 @@
 == NERC-ARF Specific Instructions ==
 
-For running on the NERC-ARF systems there are a number of utilities to make batch processing easier. 
+For running on the NERC-ARF systems there are a number of conventions and scripts used for processing.
 
-**Work in progress - more documentation will be added as it is written **
+=== Directory Structure ===
 
-=== Generating the .inn file ===
+**Processing**
+{{{
+processing/hyperspectral/atmospheric_correction/<flightline>
+}}}
+ATCOR prefers to have all files for a flightline in a single directory
 
-To generate the .inn file for ATCOR use the `gen_inn_atcor.py` script.
+**Delivery**
 
-* `--calfile` needs to point to the .cal file ATCOR users, this wil be `~arsf/usr/opt/atcor_4/sensor/SENSOR/*.cal` where `SENSOR` is the name used for the sensor definition in ATCOR (e.g., f17_b2).
-*  `inputbil` needs to point to the bil file in the delivery folder directory.
-* `--outputdir` should be the directory used or ATCOR processing.
+The following directories are included in addition to the ones included with a standard hyperspectral delivery:
+{{{
+flightlines/level2_ac  (the level1 corrected *atm.bil)
+flightlines/mappped_ac  (the level3 corrected *atm.bil)
+mosaic/                (if a level3 corrected mosaic has been generated - multiple subdirs if required)
+doc/                   (standard documentation on procedure - not flight specific)
+screenshots/
+           fJJJll3b_atm.jpg
+AtmosCorr_readme-DATE.pdf   (description of all AC work and parameters, software versions, algorithms, include a table with what
+                        has been processed, e.g.  | Topography   | YES   |
+                                                  | BRDF         | YES   |
+                                                  | Cloud shadow | No    |
+}}}
 
-If microtops data has been provided pass this in using `--microtops` to read using the https://github.com/robintw/PyMicrotops library and set a constant visibility based on this. Note if the microtops data has a number of header lines these need to be removed before PyMicrotops can read the file, there should just be one header line with the column names.
+=== Running ATCOR ===
 
-After generating the .inn file it should be renamed to use the same name as the image (without  the .bsq extension). For example if the file being processed in ATCOR is 'f168043b_masked_resampled_utm31n.bsq' name 'f168043b_masked_resampled_utm31n.inn'.
+Running ATCOR comprises generating the .inn parameter file and then running ATCOR through IDL. For files which have entire bands masked out (e.g., Fenix data from late 2014 - 2015) these bands need to be removed before running. The script `run_atm_correction_atcor.py` will run the entire process on a file which has been masked using `aplmask` using the NERC-ARF default parameters.
 
-Currently ATCOR doesn't always read all the parameters correctly open the .inn file (plain text) and update the values in the ATCOR GUI if needed.
+{{{
+run_atm_correction_atcor.py -d <delivery_directory> <masked_bil_file>
+}}}
 
-=== Batch running .inn files ===
+The directory the masked bil file is in will be used for all ATCOR files so it is recommended a separate directory is used for each line.
 
-Once .inn files have been generated and checked using the ATCOR GUI it is possible to run in batch using the script `batch_run_atcor_inn.py`. To do so ssh onto 'rsg-atcor' and run:
+To run multiple files a separate queue has been set up for ATCOR. To submit ATCOR jobs to the grid a command similar the below can be used:
 {{{
-batch_run_atcor_inn.py /path/to/files/to/process
+qsub -q atcor.q -P arsfdan -m n -b y -N acor_test \
+         -o /users/rsg/dac/scratch_network/atcor_runs/2017_166/f166071b/  \
+         -e /users/rsg/dac/scratch_network/atcor_runs/2017_166/f166071b/  \
+         /users/rsg/dac/scratch_network/git_repos/arsf_atcor_scripts/scripts/run_atm_correction_atcor.py \
+         -d /users/rsg/arsf/arsf_data/2017/flight_data/spain/RG17_26-2017_166_Albacete/delivery/RG17_26-166-hyperspectral-20170823 \
+         /users/rsg/dac/scratch_network/atcor_runs/2017_166/f166071b/f166071b_masked.bil
 }}}
-This will recursively search for .inn files and run them through ATCOR.
+Note only a single job will run at once.
+
+**Non-standard parameters**
+
+For more control over the parameters passed to ATCOR, for example to include microtops data to define visibility, the `gen_inn_atcor.py` script can be used.
+The .inn files can be opened in the ATCOR GUI and run through that or the `batch_run_atcor_inn.py` can be used.