Changes between Initial Version and Version 1 of Procedures/HyperspectralCalibrationCreation


Ignore:
Timestamp:
Mar 21, 2012 11:14:03 AM (8 years ago)
Author:
benj
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/HyperspectralCalibrationCreation

    v1 v1  
     1= Generating calibration files for Eagle and Hawk =
     2
     3Radiometric, wavelength and miscellaneous other calibration of Eagle and Hawk is undertaken at ARSF-Ops at least annually. This page describes how to use the calibration software written by ARSF-DAN to create the necessary calibration files from the raw calibration data.
     4
     5== Setup ==
     6
     7Relevant scripts are held in the arsf-internal SVN repository under tools/libarsfcal.
     8
     9Note that the various calibration scripts written in Python expect to import a package called "libarsfcal". This means that you must have the directory above the libarsfcal directory in your PYTHONPATH environment variable for the scripts to run. So eg.
     10
     11 1. `cd ~`
     12 1. svn co libarsfcal
     13 1. `PYTHONPATH=$PYTHONPATH:~`
     14
     15Test this has been successful by attempting to run one of the scripts using the -h option. If you've done it right it'll print the usage, if you've done it wrong it'll tell you that there is no module called libarsfcal.
     16
     17=== Config file ===
     18
     19The master calibration scripts require a config file to supply a lot of their options. An example config file can be found under libarsfcal/supplementary. Keys must be in the format `key=value`, one per line. The file must contain a `[DEFAULT]` heading holding default values, an `[eagle]` heading holding keys for the Eagle calibration and a `[hawk]` heading holding keys for the Hawk calibration.
     20
     21Keys required under [DEFAULT]:
     22
     23 * aplcal=<path to aplcal>
     24 * fast_bil_median=<path to fast_bil_median>
     25 * fityk=<path to cfityk>
     26 * linefile=<path to CSV file containing spectral lines>
     27 * lamp_index_file=<path to CSV file matching data file names to spectral lamps>
     28
     29Keys required under both [eagle] and [hawk]:
     30
     31 * number_of_pixels=<number of spectral pixels on sensor>
     32 * rawdir=<Directory containing raw data files>
     33 * splitdir=<Directory in which to place split light/dark files>
     34 * darkdir=<Directory in which to place dark corrected files>
     35 * averagedir=<Directory in which to place averaged files>
     36 * outdir=<Directory in which to place output files>
     37 * sample_hdr_file=<Path to header file containing old wavelength scale>
     38 * lampfile=<Path to CSV file containing integrating sphere response for this sensor>
     39 * raw_radcal_file=<Path to raw file to start from to create radiometric calibration file>
     40
     41Example files for linefile, lamp_index_file and lampfile are all under libarsfcal/supplementary.
     42
     43== Wavelength calibration ==
     44
     45This must be run first, since the radiometric calibration requires a wavelength calibration to tie the lamp data into. Note for Eagle you may need to re-run gen_bandsets.py (see "Other scripts", below) with different settings in order to obtain bandset files for different spectral binning settings.
     46
     47'''Master script:''' cal_spectral.py
     48
     49Basic usage of this requires only the name of the config file as an argument. You may also want to use the following arguments:
     50
     51 * -v: Verbose. Will print a lot more stuff but makes it easier to see if it's going wrong.
     52 * -s: Specify whether Eagle or Hawk. Required to run Hawk calibration.
     53 * -x: Comma-separated regex list. Files matching these regexes will not be processed. This may be useful if there are a lot of data files in the specified raw directory (or later stages) that are not relevant to the wavelength calibration.
     54 * -t: Specify a calibration stage to start from. Useful to skip the long-running early stages if you don't need to (for example) split the raw files again when re-running.
     55 * -p: Pauses calibration between stages so output can be easily checked.
     56
     57=== Other scripts ===
     58
     59The master script calls the following scripts. If the master script is failing you may need to bug-fix one of these, or they can be run individually - this gives more control and flexibility at several stages, but is more complicated.
     60
     61 * common.py: Contains functions that are common to more than one calibration stage
     62 * data_handler.py: Contains functions for handling BIL files
     63 * split_autodark.py: Splits Eagle or Hawk data files containing autodark lines into a light file and a dark file.
     64 * spatial_average.py: Calculates the average of each line for each band
     65 * findbaseline.py: Finds the baseline data value to remove prior to peak fitting
     66 * gen_fityk_script.py: Generates fityk scripts to do peak fitting
     67 * lookup_lamp.py: Looks up which type of spectral lamp was used for a given data file
     68 * checkpeaks.py: Checks spectral peaks found by fityk against known spectral lines and works out corrected pixel numbers
     69 * combine_anchors.py: Combines multiple spectral anchor files into one
     70 * minimise_error.py: Minimises the RMS error of a list of calibration anchor points against known spectral lines
     71 * wavelength_scale.py: Generates a wavelength scale for a given spectral dataset with wavelength anchor points.
     72 * gen_bandsets.py: Generates new bandset (.prn, .bnd and .wls) files from the results of wavelength calibration
     73
     74== Radiometric calibration ==
     75
     76This can either be run after the wavelength calibration in order to obtain a cal using an up-to-date wavelength calibration, or it can be run using an old wavelength calibration.
     77
     78 '''Master script:''' cal_radiometric.py
     79
     80Basic usage of this requires only the name of the config file as an argument. You may also want to use the following arguments:
     81
     82 * -v: Verbose. Will print a lot more stuff but makes it easier to see if it's going wrong.
     83 * -s: Specify whether Eagle or Hawk. Required to run Hawk calibration.
     84 * -t: Specify a calibration stage to start from. Useful to skip the long-running early stages if you don't need to (for example) split the raw files again when re-running.
     85 * -p: Pauses calibration between stages so output can be easily checked.
     86 * -w: Specifies a wavelength scale file to use (ie output from wavelength calibration). You shouldn't need to use this if you've just run the wavelength cal, but you might need to if there are multiple wavelength cal files in the output directory or if you are running from an old cal.
     87
     88=== Other scripts ===
     89
     90The master script calls the following scripts. If the master script is failing you may need to bug-fix one of these, or they can be run individually - this gives more control and flexibility at several stages, but is more complicated.
     91
     92 * common.py: Contains functions that are common to more than one calibration stage
     93 * data_handler.py: Contains functions for handling BIL files
     94 * split_autodark.py: Splits Eagle or Hawk data files containing autodark lines into a light file and a dark file.
     95 * spatial_average.py: Calculates the average of each line for each band
     96 * replace_wavscale.py: Replaces wavelength scale in a given Eagle/Hawk header file with a new wavelength scale
     97 * gen_radcal.py: Generates calibration multipliers given an appropriate BIL input file.
     98
     99== Analysis ==
     100
     101There are scripts to generate analysis graphs and files under libarsfcal/analysis. These should be run on the calibration output, please add to them if you think of more tests.