= APL User Guide - Level 1 to Level 3 processing =

A guide to processing ARSF delivered level 1 hyperspectral data to a mapped image. Please note that this guide is available as a pdf document from [wiki:Downloads here].

-----------------------------------------
== Summary of the Mapping Procedure ==

The hyperspectral data delivered by the Natural Environment Research Council (NERC) Airborne 
Research Survey Facility (ARSF) will almost certainly be in level-1 format. This is data
which has had radiometric calibration routines applied and navigation data appended. To get
level-3 geocorrected data you need to perform a few extra steps. These steps can be summarised
as:

* '''Creating an Input Geometry (IGM) file.''' This is a file which relates each level-1 pixel to a Geographic Latitude and Longitude position.

* '''Reprojecting.''' Since most people do not want to map their data in Geographic Latitude and Longitude, the IGM file must be reprojected into a suitable projection.

* '''Gridding.''' Once the IGM file has been reprojected into the desired projection system the level-1 data can be gridded into a regular map grid.

This document will help you proceed to generate level-3 products from your ARSF data, using
the available software. Please ensure you have the most up to date version of this document,
which will be available from the ARSF Data Analysis Node website, along with the most up to
date versions of the mapping software.
----------------------------------

== Creating an IGM file ==
To create an IGM file from your level-1 data you need to use the aplcorr package. This can be
downloaded from the ARSF-DAN website downloads page [wiki:Downloads here].

aplcorr is designed to take in the calibrated level-1 binary BIL (Band Interleaved by Line) file and create a new binary 3 band BIL file. The bands are defined as:
 * Longitude
 * Latitude
 * Height

This new BIL file will have the same row and column dimensions as the original level-1 data BIL file, meaning that for each pixel of the level-1 data there is a corresponding longitude, latitude, height triple. Once you have this file you will be ready to proceed to the next step of
the geocorrection procedure. 

To create the most accurate IGM possible it is essential to use a Digital Elevation Model (DEM). This will allow the height of the imaged ground scene to be taken into account. It is possible to create an IGM file without a DEM in aplcorr ; in this case it will use the WGS-84 ellipsoid model as a height reference. This will result in errors in the geolocation, the magnitude of which will depend on the elevation difference between the ellipsoid surface and the true ground surface.

Currently aplcorr DEMs have to be in the WGS-84 datum Geographic Latitude/Longitude projection. If your DEM is in another coordinate system it will have to be reprojected prior to using it in the aplcorr program. This can be done using a GIS program such as GRASS. Alternatively the scripts available to download from here can be used.

=== Digital Elevation Model format for aplcorr ===
This section describes the format of the Digital Elevation Models that aplcorr accepts. Currently there is only a single format and projection that is accepted but in future releases this may change.

'''Projections'''

 * '''Geographic Longitude/Latitude''' in the '''WGS-84''' datum. Data values should be in decimal degrees, positive to North and East with zero at the equator and Greenwich Meridian.

'''File Formats'''
 * '''Band Interleaved by Line (BIL)''' 1 band file with data type 8-bit, 16/32-bit unsigned integer, 16/32-bit signed integer or 32/64-bit floating point. This file format is associated with the ENVI image processing package.

 * '''Band Sequential (BSQ)''' 1 band file with data type 8-bit, 16/32-bit unsigned integer, 16/32-bit signed integer or 32/64-bit floating point. This file format is associated with the ENVI image processing package.

=== Reprojecting a DEM into Geographic Longitude/Latitude ===

Most Digital Elevation Models are not going to be projected in latitude/longitude and so will require transformation into this projection. This can be done using a GIS package. There are also scripts available from ARSF-DAN (see the downloads page) that make use of the open source GRASS GIS to reproject DEM data. A description on the use of these scripts is given further below. A suitable DEM, produced from freely available sources such as ASTER or SRTM data, should also be included in your data delivery.

If you do not have access to a GIS then the open source GDAL utilities can be used to reproject a DEM and convert the file format. The gdalwarp program can warp a DEM from one projection to another but currently only supports horizontal reprojection, i.e. the elevation data values are not adjusted for vertical datum shifts. So this can be used for DEMs which are already in the WGS-84 vertical datum, for example, a DEM in WGS-84 UTM Eastings and Northings.

The gdal_translate utility can be used to convert between different file formats. For a full list of supported file formats and help on the GDAL package see the [http://www.gdal.org GDAL] website. For example, to convert a GeoTiff DEM called dem.tif into an ENVI style DEM for aplcorr named dem.bsq:
{{{gdal_translate -of ENVI dem.tif dem.bsq}}}

If you do not have a DEM covering the area of your data then you can download ASTER elevation data and construct a DEM for your flight line coverage. More information about the dataset is available at http://www.ersdac.or.jp/GDEM/E/2.html, the data itself can be downloaded from http://asterweb.jpl.nasa.gov/gdem-wist.asp.

It should be noted that aplcorr will not accept a DEM that contains NULL data in the required region. If it finds a NULL data cell then it will exit at that point.

=== Converting DEM data using the ARSF scripts ===

This section describes how to use the ARSF DEM transformation scripts to convert a DEM that is in a GDAL supported format or GRASS ASCII raster format into an aplcorr format. The procedure is slightly different for data in UK Ordnance Survey National Grid projection than for other projections and so both methods are described. For non-UK re-projections, no extra files are needed assuming that the projection and datum transformations are implemented ‘built-in’ by PROJ4. PROJ4 is an open source library of coordinate transformations. More information about these libraries can be found on the web at http://trac.osgeo.org/proj/. If grid shift files are required to do the transformation for your DEM then a different approach may be needed.

Note that the scripts will apply an interpolation routine to remove any NULL values before outputting the new DEM. This could be unsuitable for DEMs which contain large regions of NULLs (such as a large body of water) or very large DEMs. In these cases a better approach would be to patch the DEM with another suitable surface to remove the NULLs prior to conversion.

[wiki:Processing/Scripts/demconversionscript Please follow this link to see the guide for using these scripts]

=== Example aplcorr commands ===

A few example commands to create an IGM file follow. To see example commands specific to your dataset please refer to the read-me document provided with your ARSF data delivery. 

'''Process an Eagle line using a DEM'''

{{{aplcorr -navfile line1_navigation.bil -lev1file line1_calibrated.bil -vvfile eagle_fov_fullccd_vectors.bil -dem myarea_dem.bil -igmfile output.igm}}}

'''Process an Eagle line using the WGS-84 ellipsoid surface'''

{{{aplcorr -navfile line1_navigation.bil -lev1file line1_calibrated.bil -vvfile eagle_fov_fullccd_vectors.bil -igmfile output.igm}}}

'''Process a Hawk line using the WGS-84 ellipsoid surface with additional height of 50m'''

{{{aplcorr -navfile line1_navigation.bil -lev1file line1_calibrated.bil -vvfile hawk_fov_fullccd_vectors.bil -heightoffset 50 -igmfile output.igm}}}

'''Process a Hawk line and output parameters for use in external atmospheric algorithms'''

{{{aplcorr -navfile line1_navigation.bil -lev1file line1_calibrated.bil -vvfile hawk_fov_fullccd_vectors.bil -dem myarea_dem.bil -igmfile output.igm -atmosfile line1_atmospheric_parameters.bil}}}

--------------------------------------

== Reprojecting the IGM Geolocation Information ==

Often it will be the case where you do not want to create a map in Geographic Longitude/Latitude. In these cases you will require the IGM file created from the previous stage to be re-projected into a more usable projection. This should be easily performed using a GIS system - the IGM file is a 3-band BIL file (ENVI format) which most GIS systems can handle. A package is provided called apltran for users who do not wish to use a GIS to transform the IGM. This is really just a wrapper program for the open source [http://trac.osgeo.org/proj/ PROJ.4] library.

=== Using apltran ===

The usage of apltran is fairly straight forward and should be no problem if you are familiar with using the PROJ.4 libraries. If you have not used PROJ.4 before then this section could be of interest. PROJ.4 uses strings to describe the projections that it is transforming to and from. apltran has some projection strings built into it to make it easier to use.

To reproject to the Ordnance Survey National Grid (OSTN02) you need to download a grid shift file from:

[http://www.ordnancesurvey.co.uk/oswebsite/gps/osnetfreeservices/furtherinfo/ostn02_ntv2.html]

This file needs to be unzipped and the one of interest for apltran is the one with .gsb extension.

=== Example apltran commands ===

'''Transform to OS National Grid'''

If you want to map your data to Ordnance Survey National Grid coordinates then you need to use a command line like:

{{{apltran -igm line01.igm -output line01 OSNG.igm -outproj osng gridfile.gsb}}}

where the gridfile.gsb is the NTv2 format file that performs the OSTN02 grid shift.

'''Transform to UTM''' 

If you want to map your data into the UTM North zone 30 projection system (on the WGS84
ellipsoid) then you would use a command like:

{{{apltran -igm line01.igm -output line01 UTM30.igm -outproj utm wgs84N 30}}}