Version 9 (modified by mark1, 13 years ago) (diff)

--

APL User Guide - Level 1 to Level 3 processing

A guide to processing ARSF delivered level 1 hyperspectral data to a mapped image. Please note that this guide is available as a pdf document from here.

A quick start guide for command line usage can be found here: Quick start guide

Or for the graphical user interface here.

Summary of the Mapping Procedure

The hyperspectral data delivered by the Natural Environment Research Council (NERC) Airborne Research Survey Facility (ARSF) will almost certainly be in level-1 format. This is data which has had radiometric calibration routines applied and navigation data appended. To get level-3 geocorrected data you need to perform a few extra steps. These steps can be summarised as:

  • Creating an Input Geometry (IGM) file. This is a file which relates each level-1 pixel to a Geographic Latitude and Longitude position.
  • Reprojecting. Since most people do not want to map their data in Geographic Latitude and Longitude, the IGM file must be reprojected into a suitable projection.
  • Gridding. Once the IGM file has been reprojected into the desired projection system the level-1 data can be gridded into a regular map grid.

This document will help you proceed to generate level-3 products from your ARSF data, using the available software. Please ensure you have the most up to date version of this document, which will be available from the ARSF Data Analysis Node website, along with the most up to date versions of the mapping software.

Creating an IGM file

To create an IGM file from your level-1 data you need to use the aplcorr package. This can be downloaded from the ARSF-DAN website downloads page here.

aplcorr is designed to take in the calibrated level-1 binary BIL (Band Interleaved by Line) file and create a new binary 3 band BIL file. The bands are defined as:

  • Longitude
  • Latitude
  • Height

This new BIL file will have the same row and column dimensions as the original level-1 data BIL file, meaning that for each pixel of the level-1 data there is a corresponding longitude, latitude, height triple. Once you have this file you will be ready to proceed to the next step of the geocorrection procedure.

To create the most accurate IGM possible it is essential to use a Digital Elevation Model (DEM). This will allow the height of the imaged ground scene to be taken into account. It is possible to create an IGM file without a DEM in aplcorr ; in this case it will use the WGS-84 ellipsoid model as a height reference. This will result in errors in the geolocation, the magnitude of which will depend on the elevation difference between the ellipsoid surface and the true ground surface.

Currently aplcorr DEMs have to be in the WGS-84 datum Geographic Latitude/Longitude projection. If your DEM is in another coordinate system it will have to be reprojected prior to using it in the aplcorr program. This can be done using a GIS program such as GRASS. Alternatively the scripts available to download from here can be used.

Digital Elevation Model format for aplcorr

This section describes the format of the Digital Elevation Models that aplcorr accepts. Currently there is only a single format and projection that is accepted but in future releases this may change.

Projections

  • Geographic Longitude/Latitude in the WGS-84 datum. Data values should be in decimal degrees, positive to North and East with zero at the equator and Greenwich Meridian.

File Formats

  • Band Interleaved by Line (BIL) 1 band file with data type 8-bit, 16/32-bit unsigned integer, 16/32-bit signed integer or 32/64-bit floating point. This file format is associated with the ENVI image processing package.
  • Band Sequential (BSQ) 1 band file with data type 8-bit, 16/32-bit unsigned integer, 16/32-bit signed integer or 32/64-bit floating point. This file format is associated with the ENVI image processing package.

Reprojecting a DEM into Geographic Longitude/Latitude

Most Digital Elevation Models are not going to be projected in latitude/longitude and so will require transformation into this projection. This can be done using a GIS package. There are also scripts available from ARSF-DAN (see the downloads page) that make use of the open source GRASS GIS to reproject DEM data. A description on the use of these scripts is given further below. A suitable DEM, produced from freely available sources such as ASTER or SRTM data, should also be included in your data delivery.

If you do not have access to a GIS then the open source GDAL utilities can be used to reproject a DEM and convert the file format. The gdalwarp program can warp a DEM from one projection to another but currently only supports horizontal reprojection, i.e. the elevation data values are not adjusted for vertical datum shifts. So this can be used for DEMs which are already in the WGS-84 vertical datum, for example, a DEM in WGS-84 UTM Eastings and Northings.

The gdal_translate utility can be used to convert between different file formats. For a full list of supported file formats and help on the GDAL package see the ​GDAL website. For example, to convert a GeoTiff DEM called dem.tif into an ENVI style DEM for aplcorr named dem.bsq: gdal_translate -of ENVI dem.tif dem.bsq

If you do not have a DEM covering the area of your data then you can download ASTER elevation data and construct a DEM for your flight line coverage. More information about the dataset is available at ​http://www.ersdac.or.jp/GDEM/E/2.html, the data itself can be downloaded from ​http://asterweb.jpl.nasa.gov/gdem-wist.asp.

It should be noted that aplcorr will not accept a DEM that contains NULL data in the required region. If it finds a NULL data cell then it will exit at that point.

Converting DEM data using the ARSF scripts

This section describes how to use the ARSF DEM transformation scripts to convert a DEM that is in a GDAL supported format or GRASS ASCII raster format into an aplcorr format. The procedure is slightly different for data in UK Ordnance Survey National Grid projection than for other projections and so both methods are described. For non-UK re-projections, no extra files are needed assuming that the projection and datum transformations are implemented ‘built-in’ by PROJ4. PROJ4 is an open source library of coordinate transformations. More information about these libraries can be found on the web at ​http://trac.osgeo.org/proj/. If grid shift files are required to do the transformation for your DEM then a different approach may be needed.

Note that the scripts will apply an interpolation routine to remove any NULL values before outputting the new DEM. This could be unsuitable for DEMs which contain large regions of NULLs (such as a large body of water) or very large DEMs. In these cases a better approach would be to patch the DEM with another suitable surface to remove the NULLs prior to conversion.

Please follow this link to see the guide for using these scripts?

Example aplcorr commands

A few example commands to create an IGM file follow. To see example commands specific to your dataset please refer to the read-me document provided with your ARSF data delivery.

Process an Eagle line using a DEM

aplcorr -navfile line1_navigation.bil -lev1file line1_calibrated.bil -vvfile eagle_fov_fullccd_vectors.bil -dem myarea_dem.bil -igmfile output.igm

Process an Eagle line using the WGS-84 ellipsoid surface

aplcorr -navfile line1_navigation.bil -lev1file line1_calibrated.bil -vvfile eagle_fov_fullccd_vectors.bil -igmfile output.igm

Process a Hawk line using the WGS-84 ellipsoid surface with additional height of 50m

aplcorr -navfile line1_navigation.bil -lev1file line1_calibrated.bil -vvfile hawk_fov_fullccd_vectors.bil -heightoffset 50 -igmfile output.igm

Process a Hawk line and output parameters for use in external atmospheric algorithms

aplcorr -navfile line1_navigation.bil -lev1file line1_calibrated.bil -vvfile hawk_fov_fullccd_vectors.bil -dem myarea_dem.bil -igmfile output.igm -atmosfile line1_atmospheric_parameters.bil

Reprojecting the IGM Geolocation Information

Often it will be the case where you do not want to create a map in Geographic Longitude/Latitude. In these cases you will require the IGM file created from the previous stage to be re-projected into a more usable projection. This should be easily performed using a GIS system - the IGM file is a 3-band BIL file (ENVI format) which most GIS systems can handle. A package is provided called apltran for users who do not wish to use a GIS to transform the IGM. This is really just a wrapper program for the open source ​PROJ.4 library.

Using apltran

The usage of apltran is fairly straight forward and should be no problem if you are familiar with using the PROJ.4 libraries. If you have not used PROJ.4 before then this section could be of interest. PROJ.4 uses strings to describe the projections that it is transforming to and from. apltran has some projection strings built into it to make it easier to use.

To reproject to the Ordnance Survey National Grid (OSTN02) you need to download a grid shift file from:

​http://www.ordnancesurvey.co.uk/oswebsite/gps/osnetfreeservices/furtherinfo/ostn02_ntv2.html

This file needs to be unzipped and the one of interest for apltran is the one with .gsb extension.

Example apltran commands

Transform to OS National Grid

If you want to map your data to Ordnance Survey National Grid coordinates then you need to use a command line like:

apltran -igm line01.igm -output line01 OSNG.igm -outproj osng gridfile.gsb

where the gridfile.gsb is the NTv2 format file that performs the OSTN02 grid shift.

Transform to UTM

If you want to map your data into the UTM North zone 30 projection system (on the WGS84 ellipsoid) then you would use a command like:

apltran -igm line01.igm -output line01 UTM30.igm -outproj utm wgs84N 30

Gridding and Mapping the Data

Now that you have an IGM file in the required projection it is time to map the level-1 data to a regular grid. To do this you can use a GIS, some alternative mapping software or the ARSF aplmap package. This will take in the IGM file and level-1 data file and output a map of the level-1 data in the IGM projection.

Using aplmap

The aplmap software is capable of mapping a number of bands of the level-1 data to the projection system of the given IGM file, using the selected interpolation method.

Information on interpolation methods

This section briefly describes the algorithms used for the interpolation of the data within the aplmap software.

Nearest Neighbour

Mapping with nearest neighbour searches the neighbourhood around the grid cell and finds the closest point to this from the IGM file. The corresponding value from the level-1 file is then used for this pixel. This option is selected using -interpolation nearest on the command line.

Inverse Distance Weighted

The inverse distance weighted algorithm first searches for, at most, the X nearest points within the maximum allowed search distance. The number X is specified from the command line in the -interpolation option. Then when these points have been found it calculates a weight for each point and applies this to the value of the data from the level-1 file. The weight for point i is calculated as (distance−2) / (Sum (distance−2)) . This option is selected using -interpolation idw maxpoints where maxpoints is the maximum number of points you want to use.

Bilinear

For the bilinear interpolation algorithm, 4 neighbouring points are required. These are identified by using the level-3 position as an initial search position and finding the 4 nearest points that form a quadrilateral containing this level-3 position. The level-3 point that is being interpolated is found in terms of a proportional distance, U and V, between these 4 quadrilateral vertex points. The values of U and V are calculated using the following formulae (in level-3 space):

P = A + U ∗ (B − A)

Q = D + U ∗ (C − D)

X = P + V ∗ (Q − P)

The values of U and V, which are within the range 0-1, are then used to weight the level-1 pixel values in the interpolation formula:

f(X) = f(A) ∗ (1 − V ) ∗ (1 − U ) + f(B) ∗ (1 − V ) ∗ U + f(D) ∗ (1 − U ) ∗ V + f(C) ∗ U ∗ V

where f(x) is the data value of cell x.

Bilinear interpolation can be selected using -interpolation bilinear on the command line.

Example aplmap commands

For examples specific to your data set please see the read-me document supplied with your ARSF delivery.

To map bands 100, 60 and 30 of the level-1 file to a square pixel size of 5 metres using nearest neighbour interpolation:

aplmap -igm line01 UTM30.igm -lev1 line01 calibrated.bil -mapname line01 mapped.bil -bandlist 100 60 30 -pixelsize 5 5 -interpolation nearest

To map all bands of the level-1 file to a square pixel size of 5m using inverse distance weighted interpolation with 5 points and a maximum interpolation distance of 20m:

aplmap -igm line01 UTM30.igm -lev1 line01 calibrated.bil -mapname line01 mapped.bil -bandlist ALL -pixelsize 5 5 -interpolation idw 5 -maxinterpdistance 20

To map bands 10 to 20 and 50 to 79 inclusive, to a pixel resolution of 1m, using bilinear interpolation:

aplmap -igm line01 OSNG.igm -lev1 line01 calibrated.bil -mapname line01 mapped.bil -bandlist 10-20 50-79 -pixelsize 1 1 -interpolation bilinear