= Eagle/Hawk Processing Guide =

This guide describes how to process hyperspectral data using the apl suite. This should be used for 2011 data onwards. If you are processing earlier data, see [wiki:Procedures/AZEagleHawkProcessing here] for instructions for processing with the az suite.

The apl suite consists of the following programs: '''aplcal''', '''aplmask''', '''aplnav''', '''aplcorr''', '''apltran''', '''aplmap'''.

Before starting, make sure the [wiki:Procedures/ProcessingChainInstructions/NavigationProcessing navigation is processed] and all raw data is present and correct.

Projects are located ~arsf/arsf_data/<year>/flight_data/<area>/<project>. Processing files and deliveries will be generated under <project>/processing. You should be logged in as the airborne user when doing processing. Check [wiki:Processing/FilenameConventions here] for the project lay-out and file name standards.

=== DEM ===

To return sensible results for all but very flat areas, you will need a dem. You can create a DEM by running asterdem.sh or nextmapdem.sh and output this in the hyperspectral DEM directory; otherwise use the liDAR aster DEM from the liDAR processing.

Note: If you create a nextmap DEM then this can not be included with the delivery as there are restrictions on it's distribution. Generate a separate ASTER DEM to be included with the delivery.

=== Creating config file ===

This file will be used to automatically generate the commands necessary to process your hyperspectral lines.

If no config file exists (in <proj_dir>/processing/hyperspectral) then, in top project directory, run:

`generate_apl_config.py`

This should generate a config file based on the raw data and applanix files, and output it to the processing/hyperspectral directory.[[BR]]
Go through carefully and check everything is correct. Most notably:

 * project_code
 * dem and dem_origin
 * transform_projection is correct for the data
 * pixel size using [http://arsf-dan.nerc.ac.uk/pixelsize/pixelsize.html Pixel size (and swath width) calculator]
 * project metadata details such as pilot, co-pilot, etc

The boresight and lever arm are pulled automatically from the tables located at:
/users/rsg/arsf/usr/share/leverarm_values.csv
/users/rsg/arsf/usr/share/boresight_values.csv

Check with someone that these are appropriate for your flight if you're not sure.

=== Correct Header Files ===

Many 2014 flights have incorrect wavelengths in the header files, which will cause an error when submitting to the grid if not corrected. Run replace_wavscales.py with the correct text file (fenix_201405.txt or fenix_201405_vnirbinning4) to produce new header files.  Original headers will be saved in a sub-directory.

/users/rsg/arsf/arsf_data/2014/misc/2014_cal_feb_gloucester/software/libarsfcal/replace_wavscales.py
-w ~arsf/calibration/2014/fenix/fenix_201405.txt <header_file(s)_to_fix>

=== Submitting processing to gridnodes ===

To submit jobs to the grid, from the top level directory use: specim_qsub.py <config_file>

The actual script which does the processing of each job is: process_specim_apl_line.py

Once submitted, you can keep an eye on your jobs using qmon.

=== Individual processing stages ===

You shouldn't have to worry about this unless something goes wrong. However something often does! Detailed explanations of each step is explained [wiki:Procedures/AplSuiteDetails here]

=== Problems ===

If you have any problems, check the files created in logs e.g.

EUFAR10-03_2010-196_eagle_-2.o293411

The last part of the name is the grid node job number. [[BR]]
Check these for errors (look for stars). Common problems are listed [wiki:Processing/problemsHS here], along with possible solutions.

=== SCTs ===

The script will have produced 21 iterations of each flightline, with a range of sct values. SCT is a timing offset which affects the position and geometry of the image. Currently they range from -0.1 to 0.1 seconds. A tiff will have been produced for each version, which will be put in <project>/processing/hyperspectral/flightlines/georeferencing/mapped. You will need to go through these using gtviewer and find the image that looks correct, and note down the sct value. You usually determine the correct image by the amount of wobble in the image. Lines with an incorrect offset will cause kinks in straight lines such as roads where the plane trajectory wobbles. Selecting the image with the straight road is usually what is required. You can use .sync file created together with the config file to store correct SCT values.

Once you have the correct values, write them on the ticket for the relevant flight. If you wrote the values to the .sync file, the script 'sync_to_wiki_table.py' can help with this.

{{{
sync_to_wiki_table.py < 2012259.sync
}}}

Once the output is entered into the wiki, it should look like this:
|| Flightline || Eagle SCT || Hawk SCT ||
|| 1 || -0.05 || -0.07 ||
|| 2 || -0.05 || -0.06 ||
|| 3 || -0.02 || -0.03 ||
|| 4 || -0.02 || -0.03 ||
|| 5 || -0.03 || -0.04 ||
|| 6 || -0.02 || -0.03 ||
|| 7 || -0.06 || -0.07 ||
|| 8 || -0.04 || -0.05 ||
|| 9 || -0.02 || -0.07 ||
|| 10 || -0.01 || -0.08 ||
|| 11 || -0.02 || -0.03 ||
|| 12 || -0.02 || -0.05 ||
|| 13 || -0.05 || -0.03 ||
|| 14 || -0.02 || -0.07 ||
|| 15 || -0.02 || -0.03 ||
|| 16 || -0.05 || -0.05 ||
|| 17 || -0.05 || -0.05 ||
|| 18 || -0.02 || -0.06 ||
|| 19 || -0.02 || -0.06 ||


=== Creating final files ===

Before you do this, check if the PI requires the flight lines to be split. If so, see the page on [wiki:Processing/Splitting#FenixSplitting splitting flight lines].

The stage that creates the geolocated tiff's that you use to find SCTs deletes the original level 1 files after it's finished. You therefore need to use the config one more time to generate the full set of files for each flightline, using the correct SCT value. To do this, change the sctstart and sctend values so they are both the correct figure or use `mergeCFSync.py` to put the values from your sync file into the config file. Then in the global section set `slow_mode = true`.

Also, to generate mapped files for the delivery, change the eagle, hawk and fenix 'bandlist' in the config files to 'ALL'. This will map all bands of the data. 

Running this with specim_qsub.py will once again submit your lines to the gridnode and you should soon have all the files you require to make a delivery. Before doing it make sure to delete the old files or move them to another directory.

Once you have the final files run ('''if you will be creating a delivery with make_hyper_delivery.py this step is not required here''')

`aplxml.py --meta_type=p --config_file=<config_file> --output processing/hyperspectral/flightlines/project_information/<xmlfilename>` 

in the main project directory to get the xml project information - you must specify an output filename as airborne has no write permissions in the main project directory. 

=== OS vectors ===

If the project is in the UK, you will need to check the positional accuracy of the images against the OS line overlay. These should have been ordered before hand and are located in ~arsf/vectors.

=== Making a delivery ===

See the page on [wiki:Procedures/DeliveryCreation delivery creation]

==== Making the Readme ====

To make the readme first generate the readme config file using

`generate_readme_config.py -d <delivery directory> -r hyper -c <config_file>`

The readme config file will be saved as hyp_genreadme-airbone.cfg in the processing directory, do not delete this file as it is required for delivery checking. Check all the information in the readme config file is correct, if not change it.  

Then create a readme tex file using: `create_latex_hyperspectral_apl_readme.py -f <readme_config_file>`

Finally run `latex <readme_tex_file>` to create the PDF readme. This readme should be placed in the main delivery folder.  


[wiki:Procedures/DeliveryCreation/Hyperspectral This] page details the old manual way for creating the delivery and Readme.