Generating calibration files for Eagle and Hawk

Radiometric, wavelength and miscellaneous other calibration of Eagle and Hawk is undertaken at ARSF-Ops at least annually. This page describes how to use the calibration software written by ARSF-DAN to create the necessary calibration files from the raw calibration data.

Setup

Relevant scripts are held in the arsf-internal SVN repository under tools/libarsfcal.

Note that the various calibration scripts written in Python expect to import a package called "libarsfcal". This means that you must have the directory above the libarsfcal directory in your PYTHONPATH environment variable for the scripts to run. So eg.

1. cd ~
2. svn co libarsfcal
3. PYTHONPATH=$PYTHONPATH:~

Test this has been successful by attempting to run one of the scripts using the -h option. If you've done it right it'll print the usage, if you've done it wrong it'll tell you that there is no module called libarsfcal.

Config file

The master calibration scripts require a config file to supply a lot of their options. An example config file can be found under libarsfcal/supplementary. Keys must be in the format key=value, one per line. The file must contain a [DEFAULT] heading holding default values, an [eagle] heading holding keys for the Eagle calibration and a [hawk] heading holding keys for the Hawk calibration.

Keys required under [DEFAULT]:

• aplcal=<path to aplcal>
• fast_bil_median=<path to fast_bil_median>
• fityk=<path to cfityk>
• linefile=<path to CSV file containing spectral lines>
• lamp_index_file=<path to CSV file matching data file names to spectral lamps>

Keys required under both [eagle] and [hawk]:

• number_of_pixels=<number of spectral pixels on sensor>
• rawdir=<Directory containing raw data files>
• splitdir=<Directory in which to place split light/dark files>
• darkdir=<Directory in which to place dark corrected files>
• averagedir=<Directory in which to place averaged files>
• outdir=<Directory in which to place output files>
• sample_hdr_file=<Path to header file containing old wavelength scale>
• lampfile=<Path to CSV file containing integrating sphere response for this sensor>
• raw_radcal_file=<Path to raw file to start from to create radiometric calibration file>

Example files for linefile, lamp_index_file and lampfile are all under libarsfcal/supplementary.

Wavelength calibration

This must be run first, since the radiometric calibration requires a wavelength calibration to tie the lamp data into. Note for Eagle you may need to re-run gen_bandsets.py (see "Other scripts", below) with different settings in order to obtain bandset files for different spectral binning settings.

Master script: cal_spectral.py

Basic usage of this requires only the name of the config file as an argument. You may also want to use the following arguments:

• -v: Verbose. Will print a lot more stuff but makes it easier to see if it's going wrong.
• -s: Specify whether Eagle or Hawk. Required to run Hawk calibration.
• -x: Comma-separated regex list. Files matching these regexes will not be processed. This may be useful if there are a lot of data files in the specified raw directory (or later stages) that are not relevant to the wavelength calibration.
• -t: Specify a calibration stage to start from. Useful to skip the long-running early stages if you don't need to (for example) split the raw files again when re-running.
• -p: Pauses calibration between stages so output can be easily checked.

Other scripts

The master script calls the following scripts. If the master script is failing you may need to bug-fix one of these, or they can be run individually - this gives more control and flexibility at several stages, but is more complicated.

• common.py: Contains functions that are common to more than one calibration stage
• data_handler.py: Contains functions for handling BIL files
• split_autodark.py: Splits Eagle or Hawk data files containing autodark lines into a light file and a dark file.
• spatial_average.py: Calculates the average of each line for each band
• findbaseline.py: Finds the baseline data value to remove prior to peak fitting
• gen_fityk_script.py: Generates fityk scripts to do peak fitting
• lookup_lamp.py: Looks up which type of spectral lamp was used for a given data file
• checkpeaks.py: Checks spectral peaks found by fityk against known spectral lines and works out corrected pixel numbers
• combine_anchors.py: Combines multiple spectral anchor files into one
• minimise_error.py: Minimises the RMS error of a list of calibration anchor points against known spectral lines
• wavelength_scale.py: Generates a wavelength scale for a given spectral dataset with wavelength anchor points.
• gen_bandsets.py: Generates new bandset (.prn, .bnd and .wls) files from the results of wavelength calibration

Radiometric calibration

This can either be run after the wavelength calibration in order to obtain a cal using an up-to-date wavelength calibration, or it can be run using an old wavelength calibration.

Master script: cal_radiometric.py

Basic usage of this requires only the name of the config file as an argument. You may also want to use the following arguments:

• -v: Verbose. Will print a lot more stuff but makes it easier to see if it's going wrong.
• -s: Specify whether Eagle or Hawk. Required to run Hawk calibration.
• -t: Specify a calibration stage to start from. Useful to skip the long-running early stages if you don't need to (for example) split the raw files again when re-running.
• -p: Pauses calibration between stages so output can be easily checked.
• -w: Specifies a wavelength scale file to use (ie output from wavelength calibration). You shouldn't need to use this if you've just run the wavelength cal, but you might need to if there are multiple wavelength cal files in the output directory or if you are running from an old cal.

Other scripts

The master script calls the following scripts. If the master script is failing you may need to bug-fix one of these, or they can be run individually - this gives more control and flexibility at several stages, but is more complicated.

• common.py: Contains functions that are common to more than one calibration stage
• data_handler.py: Contains functions for handling BIL files
• split_autodark.py: Splits Eagle or Hawk data files containing autodark lines into a light file and a dark file.
• spatial_average.py: Calculates the average of each line for each band
• replace_wavscale.py: Replaces wavelength scale in a given Eagle/Hawk header file with a new wavelength scale
• gen_radcal.py: Generates calibration multipliers given an appropriate BIL input file.

Analysis

There are scripts to generate analysis graphs and files under libarsfcal/analysis. These should be run on the calibration output, please add to them if you think of more tests.