Changes between Version 38 and Version 39 of Procedures/OwlProcessing

Timestamp:

Jan 29, 2024, 2:52:14 PM (3 months ago)

Author:

wja

Comment:

--

Procedures/OwlProcessing

@@ -135 +135 @@
 Create a dem in exactly the same way as for fenix.  This should be saved in the processing/hyperspectral/dem directory, not the owl directories, so only 1 copy is required.  e.g. from the project directory type create_apl_dem.py --aster.
 
-Geocorrection is conducted in the same way as for fenix data, using APL. However, if lines need to be split an extra step is required - see "Splitting files" section below. Generate a config file using generate_apl_config.py -o processing/owl/o2014219b.cfg (make sure it begins with an '''o''') to keep processing separate from fenix. You may need to add the boresight values to the config file ( e.g. owl_boresight = 0.12 -0.32 0.32). You may also want to delete or set to false the fenix lines if only processing owl data.  Make sure the project directory structure is complete (same as fenix). Submit to the grid using specim_qsub.py <config_file> and calculate the SCTs. Once you have done this you can generate the mapped data (102 bands) by setting owl_bandlist = ALL and setting the SCT value for each line.
+Geocorrection is conducted in the same way as for fenix data, using APL. However, if lines need to be split an extra step is required - see "Splitting files" section below. Generate a config file using generate_apl_config.py -o processing/owl/o2014219b.cfg (make sure it begins with an '''o''') to keep processing separate from fenix. You may need to add the boresight values to the config file ( e.g. owl_boresight = 0.12 -0.32 0.32). You may also want to delete or set to false the fenix lines if only processing owl data.  Make sure the project directory structure is complete (same as fenix). Submit to the grid using specim_slurm.py <config_file> (more info in the "Processing on the Slurm Grid" section below) and calculate the SCTs. Once you have done this you can generate the mapped data (102 bands) by setting owl_bandlist = ALL and setting the SCT value for each line.
 
 Early owl files have corrupt nav files, so if you get this message in the log or 'Requested sync time index is out of bounds in GetSyncDelay()' set the config file to use_nav = false.
@@ -142 +142 @@
 
 generate_apl_config will check the processing/owl/stitched directory for modified raw files to use before defaulting to the original raw directory, so make sure this directory is present only if you intend to use it.
+
+== Processing on the Slurm Grid ==
+
+`specim_slurm.py` will generate a `sbatch` file (be default into the logfiles directory). This sbatch file is used to submit jobs to Slurm. The sbatch file will run locally if you run it like a bash script. If more than one flightline needs processing (i.e more than one line has `process_line = True` in the APL config, then the sbatch file is configured to submit an array job (one job containing multiple tasks).
+
+To interact with slurm you need the slurm client installed and configured. It is easier to just ssh to the host `rsg-slurm-login-1`. 
+
+You can submit jobs using `sbatch [PATH TO SBATCH SCRIPT].
+
+Monitor jobs using `squeue --me` to view all your own jobs. A specific job can be monitored with `squeue -j [JOB ID]`.
+
+By default, array jobs will display as a single job, with additional taks only displaying if they are processing rather than queing, the `--array` flag will expand this.
+
+Remove a job with `scancel [JOB ID]`
 
 === Splitting files ===