Changes between Version 104 and Version 105 of Procedures/EagleHawkProcessing

Timestamp:

Jan 30, 2024, 3:47:55 PM (10 months ago)

Author:

wja

Comment:

--

Procedures/EagleHawkProcessing

@@ -40 +40 @@
  * pixel size using [https://nerc-arf-dan.pml.ac.uk/pixelsize/pixelsize.html Pixel size (and swath width) calculator] but please round to nearest 0.5m.
  * project metadata details such as pilot, co-pilot, etc
- * mailto email address (the email address to send grid job emails to if you intend to use the `-m` option of `specim_qsub.py`)
  * change the logdir location to processing/hyperspectral/logfiles or processing/owl/logfiles
 
@@ -90 +89 @@
 === Submitting processing to gridnodes ===
 
-To submit jobs to the grid, from the top level directory use: {{{specim_qsub.py <config_file>}}}
+To submit jobs to the grid, from the top level directory use: {{{specim_slurm.py -s <sensor> -c <config_file>}}}
 
-The actual script which does the processing of each job is: process_specim_apl_line.py
+Add the `--final` flag to actually submit the jobs.
 
-Once submitted, you can keep an eye on your jobs using qmon.
+`specim_slurm.py` will generate a sbatch file (be default into the logfiles directory). This sbatch file is used to submit jobs to Slurm. The sbatch file will run locally if you run it like a bash script. If more than one flightline needs processing (i.e more than one line has `process_line = True` in the APL config, then the sbatch file is configured to submit an array job (one job containing multiple tasks).
 
-The number of APL jobs which can run at once is limited using a complex consumable (apl_throttle) as the I/O usage can be high, particularly for mapping all bands. To increase / decrease the number of jobs run (as rsgcode):
-{{{
-qconf -mattr exechost complex_values apl_throttle=5 global
-}}}
-Replacing '5' with the maximum number of APL jobs allowed to run at once. For more details see [https://rsg.pml.ac.uk/intranet/trac/wiki/Processing/Grid_Engine/Maintenance#Throttlingyourself throttling yourself]
+To interact with slurm you need the slurm client installed and configured. It is easier to just ssh to the host rsg-slurm-login-1.
+
+You can submit jobs using `sbatch [PATH TO SBATCH SCRIPT]`.
+
+Monitor jobs using squeue --me to view all your own jobs. A specific job can be monitored with `squeue -j [JOB ID]`.
+
+By default, array jobs will display as a single job, with additional taks only displaying if they are processing rather than queing, the `--array` flag will expand this.
+
+Remove a job with `scancel [JOB ID] `