Eagle/Hawk Processing Guide

This guide describes how to process hyperspectral data using the apl suite. This should be used for 2011 data onwards. If you are processing earlier data, see here for instructions for processing with the az suite.

The apl suite consists of the following programs: aplcal, aplnav, apltran, aplmap.

Before starting, make sure the navigation is processed and all raw data is present and correct.

DEM

To return sensible results for all but very flat areas, you will need a dem. One should have already been completed in the unpacking stage. If not, it will need to be created. For the UK, use NextMap. Otherwise use ASTER. If you can't use ASTER for some reason, then you can also create one from our own LiDAR using make_lidardem_or_intensity.sh.

Creating config file

This file will be used to automatically generate the commands necessary to process your hyperspectral lines.

If no config file exists (in <proj_dir>/processing/hyperspectral) then, in top project directory, run:

generate_apl_runscripts.py -s s -n <numlines> -j <jday> -y <year>

This should generate a config file based on the raw data and applanix files, and output it to the processing/hyperspectral directory. The file may need editing, the main parts to check:

• project_code
• dem and dem_origin
• transform_projection is correct for the data

• If using SBETs from IPAS to process the hyperspectral make sure to use these lever arm values (referenced from pav80 not GPS antenna. They should be automatically selected according to the year but best to check):

Eagle : 0.415 -0.014 -0.129

Hawk : 0.585 -0.014 -0.129

And use these boresight values (PRH):

Eagle : -0.322 0.175 0.38

Hawk : -0.345 0.29 0.35

Submitting processing to gridnodes

To submit jobs to the grid, from the top level directory use: specim_qsub.py <config_file>

The actual script which does the processing of each job is: process_specim_apl_line.py

Individual processing stages:

You shouldn't have to worry about this unless something goes wrong. However something often does! Detailed explanation of each step is explained here

Problems

Sync

If you get something in the log file like:

End of POSATT file with NO Specim sync found no sync within 5.00 secs of raw file header time: 49915.66

then there is no sync info in the nav file for that line and and the range of possible sct values will increase (up to a few seconds). You will need to include 'has_sync = false' in the config file line entry and input a wider range of scts, e.g.

[hawk_-11] ... ... ... has_sync = false sctend = -1 sctincrement = -0.1 sctstart = 1

[hawk_-12] ...

A less likely reason for the above output is that the raw header file in question contains invalid GPS times (either start time or end time) and therefore also results in not lying within the times of the .nav file. If both GPS Starting Position and GPS Start Time are missing (accordingly for End Position/End Time) then the best guess is to replace the invalid time/position with the raw header file from the other sensor (eagle/hawk), making sure that it is the correct corresponding flightline. If only one of the two are missing, time or position, then the navigation files should be used to find the missing data, this is easier done when the position is missing (open .gpb file from applanix/extract) but still possible when the time is missing (.gpb again, but will have to track position).

Turns

If you get something in the log file like:

flight line may have a turn in it heading spread over approximately: 120 degs

run terminated due to turn

then there is a bend in the line that is too sharp. You need to add a -bend flag to the azcorr arguments for the line entry in the config file, e.g.

[eagle_-7] ... ... ... azgcorr_args_extra = -bend

[eagle_-8] ...

Once processed, it is best to exclude the lines which are causing the problems by including a -l flag in azspec followed by the line numbers delineating the part of the line you want to keep, then reprocessing. E.g. if the bend is at the start of the line:

[eagle_-7] ... ... ... azgcorr_args_extra = -bend azspec_args_extra = -l 50 2000

will exclude the first 50 lines. Make sure the second value takes into account dark frames: the number of lines given in the header file will include dark frames and if you want to process to the end of the line you should therefore specify the line number before the figure given in 'autodarkstartline' (also in the header file).

Making a delivery

Use the make_hyper_delivery.py script to make the delivery directory. Run it from within the main project directory. By default it runs in dry run mode.

Use --final if happy with what it says it will do. Use -m <config> to generate screenshots and mosaics

To make the readme file use the script: create_latex_hyperspectral_apl_readme.py