Hyperspectral processing issues
This page details common problems with hyperspectral processing and their solutions.
apl
Dark Frames
If there are missing dark frames for a line extract some from another file. To find a matching file look for a hdr file with matching:
- Bands
- Samples
- fps
- tint
If you cannot find a matching tint value this can be solved later, look for files from the same year and closest julian day possible.
Then run: extract_dark_frames -i VNIR300-08-1.raw -o VNIR300-08-1-dark_frames.bil
This will extract the dark frames to an output file that is then specified in the aplcal section of processing: cal_args_extra = -darkfile VNIR300-08-1-dark_frames.bil
If the tint does not match then you need to scale the dark frames, this in done: dark_frame_scaling.py -i VNIR300-08-1-dark_frames.bil -t 10
This will scale the dark frames values so that the the tint matches the file with missing dark frames.
No matching sync message found in .nav file that matches value in hdr file.
This means that the Specim .nav file has no sync message that matches with that for this flight line. To process the flight line you will need to manually sync the data by applying various SCT offsets and choosing the mapped image that looks the best (no wobbly roads etc).
To do this in the config file set: use_nav=false in the section for the affected flight line.
"There appears to be a non-numeric value in a specim time stamp SPTSMP message in the raw .nav file."
The raw nav files, located in <project directory>/hyperspectral/fenix/*.nav, contain SPTSMP sentences (the ones that begin with "$SPTSMP"), which are timestamp records. Sometimes these sentences will contain non-numeric values, such as "$SPTSMP,207,4094+,1*38". Notice that the third record in this line contains a '+' character at the end of the number. This is what the error message refers to. To correct this, follow these steps:
- Login as arsf and locate the erroneous file.
- Change the permissions on the erroneous file using chmod u+w <erroneous filename> so that it can be edited.
- Locate the value with the spurious character and remove it. Note that this will always be in the second/third number(s) (non-numeric values in the first and last records of the sentence are fine).
- Save the file.
- Revert the permissions to what they were before. If you used the command above, chmod u-w <erroneous filename> will do this.
- Make a note in the ticket saying what you did.
A number of bands in the raw file disagree with the calibration file wavelengths. Number that agree: 0
The wavelengths in the fenix header files are wrong. Replace these using replace_wavscale.py, but save a copy of the original header files in a subdirectory.
If there is an error about the wrong number of bands you are using the wrong textfile. Change to *_vnirbinning4.txt
/users/rsg/arsf/arsf_data/2014/misc/2014_cal_feb_gloucester/software/libarsfcal/replace_wavscale.py -w ~arsf/calibration/2014/fenix/fenix_201405.txt <header_file_to_fix>
az
Sync
If you get something in the log file like:
End of POSATT file with NO Specim sync found no sync within 5.00 secs of raw file header time: 49915.66
then there is no sync info in the nav file for that line and and the range of possible sct values will increase (up to a few seconds). You will need to include 'has_sync = false' in the config file line entry and input a wider range of scts, e.g.
[hawk_-11] ... ... ... has_sync = false sctend = -1 sctincrement = -0.1 sctstart = 1
[hawk_-12] ...
A less likely reason for the above output is that the raw header file in question contains invalid GPS times (either start time or end time) and therefore also results in not lying within the times of the .nav file. If both GPS Starting Position and GPS Start Time are missing (accordingly for End Position/End Time) then the best guess is to replace the invalid time/position with the raw header file from the other sensor (eagle/hawk), making sure that it is the correct corresponding flightline. If only one of the two are missing, time or position, then the navigation files should be used to find the missing data, this is easier done when the position is missing (open .gpb file from applanix/extract) but still possible when the time is missing (.gpb again, but will have to track position).
Turns
If you get something in the log file like:
flight line may have a turn in it heading spread over approximately: 120 degs
run terminated due to turn
then there is a bend in the line that is too sharp. You need to add a -bend flag to the azcorr arguments for the line entry in the config file, e.g.
[eagle_-7] ... ... ... azgcorr_args_extra = -bend
[eagle_-8] ...
Once processed, it is best to exclude the lines which are causing the problems by including a -l flag in azspec followed by the line numbers delineating the part of the line you want to keep, then reprocessing. E.g. if the bend is at the start of the line:
[eagle_-7] ... ... ... azgcorr_args_extra = -bend azspec_args_extra = -l 50 2000
will exclude the first 50 lines. Make sure the second value takes into account dark frames: the number of lines given in the header file will include dark frames and if you want to process to the end of the line you should therefore specify the line number before the figure given in 'autodarkstartline' (also in the header file).