Hyperspectral Data Processing Guide

This guide describes how to process hyperspectral data using the apl suite. This should be used for 2011 data onwards. If you are processing earlier data, see here for instructions for processing with the az suite.

The apl suite consists of the following programs: aplcal, aplmask, aplnav, aplcorr, apltran, aplmap.

Before starting, make sure the navigation is processed and all raw data is present and correct.

Projects are located ~arsf/arsf_data/<year>/flight_data/<area>/<project>. Processing files and deliveries will be generated under <project>/processing. You should be logged in as the airborne user when doing processing. Check here for the project lay-out and file name standards.

DEM

To return sensible results for all but very flat areas, you will need a dem. You can create a DEM from ASTER data by running the following from within the project directory.

create_apl_dem.py --aster

It is also possible to create from NextMap (UK only, non-free) and SRTM (+/- 60 degrees north) using the --nextmap and --srtm flags.

If an output file is not specified it will put in /processing/hyperspectral/dem. If the script is not run from within the project directory the path to the project needs to be specified so the navigation data can be found.

Note: If you create a nextmap DEM then this can not be included with the delivery as there are restrictions on it's distribution. Generate a separate ASTER DEM to be included with the delivery.

Creating config file

This file will be used to automatically generate the commands necessary to process your hyperspectral lines.

If no config file exists (in <proj_dir>/processing/hyperspectral) then, in top project directory, run:

generate_apl_config.py

This should generate a config file based on the raw data and applanix files, and output it to the processing/hyperspectral directory.
Go through carefully and check everything is correct. Most notably:

• project_code
• dem and dem_origin
• transform_projection is correct for the data
• pixel size using ​Pixel size (and swath width) calculator but please round to nearest 0.5m.
• project metadata details such as pilot, co-pilot, etc
• change the logdir location to processing/hyperspectral/logfiles or processing/owl/logfiles

The boresight and lever arm are pulled automatically from the tables located at: /users/rsg/arsf/usr/share/leverarm_values.csv /users/rsg/arsf/usr/share/boresight_values.csv

Check with someone that these are appropriate for your flight if you're not sure.

For processing data collected in 2023: Please refer to the following information regarding whether the sensor needs flipping: ​https://nerc-arf-dan.pml.ac.uk/trac/wiki/Procedures/installation_summary_2023

Flipping (in the case of the instrument or IMU being installed backwards), can be done during aplcal. To apply this to every line in the config use:

cal_args_default = -FLIPSAMPLES

Or you can add cal_args_extra = -FLIPSAMPLES in the sections of individual lines.

Some flights (including 2023) include data from sensors of which the swath edges are obstructed by the aircraft. This requires a mask to be added to cal_args_default (for all lines in config) or cal_args_extra (for individual lines).

E.g.

cal_args_extra = -qcfailures /users/rsg/arsf/calibration/2021/fenix1k/f1k_swathedge_mask.txt

You can combine multiple arguments.

E.g

cal_args_extra = -FLIPSAMPLES -qcfailures /users/rsg/arsf/calibration/2021/fenix1k/f1k_swathedge_mask.txt

Correct Header Files

Many 2014 flights have incorrect wavelengths in the header files, which will cause an error when submitting to the grid if not corrected. Run replace_wavscale.py with the correct text file (fenix_201405.txt or fenix_201405_vnirbinning4) to produce new header files. Original headers will be saved in a sub-directory. You will need to run it as arsf from /hyperspectral/fenix (don't forget to change permissions again):

replace_wavscale.py -w ~arsf/calibration/2015/fenix/fenix_201509.txt *.hdr

For flights from the 2014 Malaysia campaign point to the 2015 fenix calibration file (fenix_201504.txt)

Fenix1k data from 2021 also needs this run on it with the following command (replacing the TXT file with the appropriate VNIR binning):

replace_wavscale.py -w /users/rsg/arsf/calibration/2021/fenix1k/wlcal2b_vnir_1b_swir.txt hyperspectral/fenix/F*/capture/*hdr

Submitting processing to gridnodes

To submit jobs to the grid, from the top level directory use: specim_slurm.py -s <sensor> -c <config_file>

Add the --final flag to actually submit the jobs.

specim_slurm.py will generate a sbatch file (be default into the logfiles directory). This sbatch file is used to submit jobs to Slurm. The sbatch file will run locally if you run it like a bash script. If more than one flightline needs processing (i.e more than one line has process_line = True in the APL config, then the sbatch file is configured to submit an array job (one job containing multiple tasks).

To interact with slurm you need the slurm client installed and configured. It is easier to just ssh to the host rsg-slurm-login-1.

You can submit jobs using sbatch [PATH TO SBATCH SCRIPT].

Monitor jobs using squeue --me to view all your own jobs. A specific job can be monitored with squeue -j [JOB ID].

By default, array jobs will display as a single job, with additional taks only displaying if they are processing rather than queing, the --array flag will expand this.

Remove a job with scancel [JOB ID]

Individual processing stages

You shouldn't have to worry about this unless something goes wrong. However something often does! Detailed explanations of each step is explained here

Problems

If you have any problems, check the files created in logs e.g.

EUFAR10-03_2010-196_eagle_-2.o293411

The last part of the name is the grid node job number.
Check these for errors (look for stars). Common problems are listed here, along with possible solutions.

SCTs

The script will have produced 21 iterations of each flightline, with a range of sct values. SCT is a timing offset which affects the position and geometry of the image. Currently they range from -0.1 to 0.1 seconds. A BIL file will have been produced for each version, which will be put in <project>/processing/hyperspectral/flightlines/georeferencing/mapped. You will need to go through these using ​tuiview and find the BIL file that looks correct, and note down the sct value. You usually determine the correct image by the amount of wobble in the image. Lines with an incorrect offset will cause kinks in straight lines such as roads where the plane trajectory wobbles. Selecting the image with the straight road is usually what is required. You can use .sync file created together with the config file to store correct SCT values.

Once you have the correct values, write them on the ticket for the relevant flight. If you wrote the values to the .sync file, the script 'sync_to_wiki_table.py' can help with this.

sync_to_wiki_table.py < 2012259.sync

Once the output is entered into the wiki, it should look like this:

Flightline	Eagle SCT	Hawk SCT
1	-0.05	-0.07
2	-0.05	-0.06
3	-0.02	-0.03
4	-0.02	-0.03
5	-0.03	-0.04
6	-0.02	-0.03
7	-0.06	-0.07
8	-0.04	-0.05
9	-0.02	-0.07
10	-0.01	-0.08
11	-0.02	-0.03
12	-0.02	-0.05
13	-0.05	-0.03
14	-0.02	-0.07
15	-0.02	-0.03
16	-0.05	-0.05
17	-0.05	-0.05
18	-0.02	-0.06
19	-0.02	-0.06

Creating final files

Before you do this, check if the PI requires the flight lines to be split. If so, see the page on splitting flight lines.

The stage that creates the geolocated bil files that you use to find SCTs deletes the original level 1 files after it's finished and will also only process three bands. You therefore need to use the config one more time to generate the full set of files for each flightline, and to generate mapped files with all bands.

To do this:

1. Delete the old mapped files or move them to another directory. This will make it easier to find the files required for delivery.
2. Change the sctstart and sctend values so they are both the correct figure or use mergeCFSync.py to put the values from your sync file into the config file
3. In the global section set slow_mode = true.
4. Change the eagle, hawk and fenix 'bandlist' in the config file to 'ALL'. This will map all bands of the data.

Running this with specim_qsub.py will once again submit your lines to the gridnodes and you should soon have all the files you require to make a delivery.

OS vectors

If the project is in the UK, you will need to check the positional accuracy of the images against the OS line overlay. These should have been ordered before hand and are located in ~arsf/vectors.

Making a delivery

See the page on delivery creation

Making the Readme

To make the readme first generate the readme config file using

generate_readme_config.py -d <delivery directory> -r hyper -c <config_file>

The readme config file will be saved as hyp_genreadme-airbone.cfg in the processing directory, do not delete this file as it is required for delivery checking. Check all the information in the readme config file is correct, if not change it.

Then create a readme tex file using: create_latex_hyperspectral_apl_readme.py -f <readme_config_file>

Finally run latex <readme_tex_file> to create the PDF readme. Using fedora 21, will need to run pdflatex <readme_tex_file> instead of the latest command. This readme should be placed in the main delivery folder.

This page details the old manual way for creating the delivery and Readme.