Changes between Version 28 and Version 29 of Procedures/EagleHawkProcessing


Ignore:
Timestamp:
Jul 4, 2011 10:38:08 AM (8 years ago)
Author:
knpa
Comment:

Changed entire page so refers to processing with apl suite. Old text now at: wiki:Procedures/AZEagleHawkProcessing

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/EagleHawkProcessing

    v28 v29  
    1 = Eagle/Hawk Processing =
     1= Eagle/Hawk Processing Guide =
    22
    3 Once the [wiki:Procedures/NewDataArrival data have been unpacked] and the [wiki:Procedures/ProcessingChainInstructions/NavigationProcessing nav data have been processed], the Eagle/Hawk data need to be run through the [wiki:Processing/Flow AZ Systems processing chain] (primarily azspec, azimport, aznav and azgcorr).
     3This guide describes how to process hyperspectral data using the apl suite. This should be used for 2011 data onwards. If you are processing earlier data, see [wiki:Procedures/AZEagleHawkProcessing here] for instructions for processing with the az suite.
    44
    5 == Creating the Config file ==
     5'''All processing in the ~arsf/arsf_data directories should be done as airborne user:'''
    66
    7 generate_runscripts.py is the script used to create the config file, which is in turn used to process the eagle and hawk lines. Before beginning, make sure the raw files are present as well as a processed navigation (sbet) file in applanix/proc and a suitable dem (see below). You may also need to create a text version of the logsheet.
     7If no config file exists (in <proj_dir>/processing/hyperspectral) then, in top project directory, run:
    88
    9 Run with -h flag to get full usage help. An example command is below:
     9`generate_apl_runscripts.py -s s -n <numlines> -j <jday> -y <year>`
    1010
    11 generate_runscripts.py -s s -n 5 -j 196 -y 2010 -l admin/196-10_Eufar10-03.txt
    12  
    13 Beware that the method is not robust and may fill in the wrong data details, especially for the per-line information.
    14 For unknown variables (global or flight line specific) a "?" will be inserted in the config file and should be replaced by hand.
     11This should generate a config file based on the raw data and applanix files, and output it to the processing/hyperspectral directory.
     12The file may need editing, the main parts to check:
    1513
    16 If no log sheet is available, one can use the keywords on the command line to specify the global variables, using "" if requires a space within the name: 
    17 
    18 generate_runscripts.py -s s -n 4 --JDAY=200 --YEAR=2009 --PI="P.I. Name" --SITE="Over There" --PROJCODE=EX01_99
    19 
    20 Adding keywords to the command line and using a logsheet results in the command line keywords taking precedent over the logsheet keyword.
    21 For a full list of keywords and their default values run: generate_runscripts.py -h.
    22 
    23 To use the logsheet just for the global variables you can add to command line --NOPERLINELOG. This will then use the logsheet for global variables, but not for per flight line values. For the per flight line values it will use liblogwriter.py and the raw eagle/hawk header and sbet files to extract average values for the speed/altitude/direction. But using this method will not name the scripts by flight line order on the logsheet, but by the filename of the raw data. Use the --NOPERLINELOG parameter if you wish to manually add the per line data to the config file.
    24 
    25 Use the --EXCLUDE="n1 n2 n3 ... -1 m1 m2 m3... -1" parameter if you want to exclude line numbers from the config file.
    26 In this case -n linesNo, linesNo can be the number of lines with the previously specified ones excluded.
    27 
    28 == Processing the Lines ==
    29 
    30 === Easy Way ===
    31 
    32 You should now have in the root of the project directory a .cfg file named with the year and julian day of the project. In order to run the project on the gridengine with default settings, run:
    33 
    34 `specim_qsub.py <cfg_file>`
    35 
    36 By default this will do timing runs for each line on the gridengine using SCT offsets between -0.1 and 0.1.
    37 
    38 If you have any problems, check the files created in logs. E.g.
    39 
    40 EUFAR10-03_2010-196_eagle_-2.o293411
    41 EUFAR10-03_2010-196_eagle_-2.e293411
    42 
    43 The first one is an output file (hence the 'o') and the second is the error file ('e'). The last part of the name is the grid node job number.
    44 
    45 Check these for errors (look for stars). Check the errors against the [#Problems example errors] below.
    46 
    47 Now you need to find and record the correct SCT value for each flightline. Look for wobbles in straight lines and try to correct them. Once you have a value for each flightline, enter this for the start and end sct variables in the config file for each flightline. Run once more and the script will no longer delete the lev1s. These are the final product to go in the delivery.
    48 
    49 Check against OS vectors if this is a UK project.
    50 
    51 === Old-fashioned way ===
    52 
    53 If you're unlucky and the automated script fails for some reason, you may need to do at least some of the processing the old-fashioned way.
    54 
    55  1. cd to the project directory.
    56  1. Ensure that directories called "logs", "dem", "lev1" and "lev3" have been created.
    57  1. If there isn't one already, create a "calibration" symlink to the calibration data: `ln -s ~arsf/calibration/<year> calibration`.
    58  1. Create a DEM for the project. If it's in the UK you can use [wiki:Processing/NextMapDEMs Nextmap data] (try running `nextmapdem.sh` in the project directory to do it automatically, otherwise see the [wiki:Processing/NextMapDEMs wiki page]).  If it's non-UK, you'll need to use [wiki:Help/LeicaLidarDems LiDAR data] (you may wish to use this anyway if it's available - see [wiki:Processing/CreateTifs here] on how to do this using a script), or failing that [wiki:Processing/SRTMDEMs SRTM 90m data]. Copy it into the dem directory.
    59  1. Create a symlink to the SBET file if there isn't one already.
    60    1. `cd applanix`
    61    1. `ln -s Proc/sbet_01.out apa<year><jday>.sbet`
    62    1. `cd ..`
    63  1. Copy the sample config file from ~arsf/sample_scripts/<year> to the project directory - you need specim_qsub.py, process_specim_line.py and template_specim_config.cfg.
    64  1. Comparing the .cfg file with the logsheet, replace the entries that need to be replaced as appropriate. You should be able to see which bits these are in the sample scripts because they'll have keywords instead of values. You will need to create one cfg file section flightline per sensor.
    65    * Note that dates must be of the form DD/MM/YY or DD/MM/YYYY (must use / as a separator)
    66    * Note that times must be of the form HH:MM:SS (must use : as a separator)
    67  1. Run the processing scripts. You can either do this via the gridengine (recommended) by running specim_qsub.py, or you can do it on your machine one line at a time on your machine by running process_specim_line.py with appropriate arguments for each line/sensor combination from the root of the project directory. If you do the latter you should pipe the output to tee to ensure a log file is generated: `rune/e12301.sh 2>&1 | tee rune/e12301.log`.
    68  1. Check each set of flightlines to work out which has the best timing offset (ie has the straightest roads, etc). Make a note of the timing offset values in the ticket
    69  1. Check against OS vectors
    70 
    71 '''dem?'''
    72 
    73 To return sensible results you will need a dem. One should have already been completed in the unpacking stage. If not, it will need to be created. If the project is in the UK you can use [wiki:Processing/NextMapDEMs Nextmap data] (try running `nextmapdem.sh` in the project directory to do it automatically, otherwise see the [wiki:Processing/NextMapDEMs wiki page]). If it's non-UK, you'll need to use [wiki:Help/LeicaLidarDems LiDAR data] (you may wish to use this anyway if it's available - see [wiki:Processing/CreateTifs here] on how to do this using a script), or failing that [wiki:Processing/SRTMDEMs SRTM 90m data]. Copy it into the dem directory. Once you've done this include the DEM file in the config file by entering "dem=<dem_file_name>" under the DEFAULT section.
    74 
    75 == Problems ==
    76 
    77 ----
    78 
    79 '''Sync'''
    80 
    81 If you get something in the log file like:
    82 
    83 ** End of POSATT file with NO Specim sync found[[BR]]
    84 **     no sync within 5.00 secs of raw file header time: 49915.66
    85 
    86 then there is no sync info in the nav file for that line and and the range of possible sct values will increase (up to a few seconds). You will need to include 'has_sync = false' in the config file line entry and input a wider range of scts, e.g.
    87 
    88 [hawk_-11] [[BR]]
    89 ...[[BR]]
    90 ...[[BR]]
    91 ...[[BR]]
    92 has_sync = false[[BR]]
    93 sctend = -1[[BR]]
    94 sctincrement = -0.1[[BR]]
    95 sctstart = 1[[BR]]
    96 
    97 [hawk_-12][[BR]]
    98 ...
    99 
    100 A less likely reason for the above output is that the raw header file in question contains invalid GPS times (either start time or end time)
    101 and therefore also results in not lying within the times of the .nav file. If both GPS Starting Position and GPS Start Time are missing (accordingly for End Position/End Time)
    102 then the best guess is to replace the invalid time/position with the raw header file from the other sensor (eagle/hawk), making sure that it is the correct corresponding flightline. If only one of the two are missing, time or position, then the navigation files should be used to find the missing data, this is easier done when the position
    103 is missing (open .gpb file from applanix/extract) but still possible when the time is missing (.gpb again, but will have to track position).
    104 
    105 -----------------------------------------------
    106 
    107 '''Turns'''
    108 
    109 If you get something in the log file like:
    110 
    111 ** flight line may have a turn in it **[[BR]]
    112 ** heading spread over approximately:  120 degs **
    113 
    114 ** run terminated due to turn **
    115 
    116 then there is a bend in the line that is too sharp. You need to add a -bend flag to the azcorr arguments for the line entry in the config file, e.g.
    117 
    118 [eagle_-7][[BR]]
    119 ...[[BR]]
    120 ...[[BR]]
    121 ...[[BR]]
    122 azgcorr_args_extra = -bend
    123 
    124 [eagle_-8][[BR]]
    125 ...
    126 
    127 Once processed, it is best to exclude the lines which are causing the problems by including a -l flag in azspec followed by the line numbers delineating the part of the line you want to keep, then reprocessing. E.g. if the bend is at the start of the line:
     14 * project_code
     15 * dem and dem_origin
     16 * transform_projection is correct for the data
    12817
    12918
    130 [eagle_-7][[BR]]
    131 ...[[BR]]
    132 ...[[BR]]
    133 ...[[BR]]
    134 azgcorr_args_extra = -bend[[BR]]
    135 azspec_args_extra = -l 50 2000
     19=== Scripts to process the data: ===
    13620
    137 will exclude the first 50 lines. Make sure the second value takes into account dark frames: the number of lines given in the header file will include dark frames and if you want to process to the end of the line you should therefore specify the line number before the figure given in 'autodarkstartline' (also in the header file).
     21 * To submit jobs to the grid, from the top level directory use: specim_qsub.py <config_file>
     22 * The actual script which does the processing of each job is: process_specim_apl_line.py
     23 * If using SBETs from IPAS to process the hyperspectral make sure to use these lever arm values (referenced from pav80 not GPS antenna):
    13824
    139 --------------------------------------------------
     25Eagle : 0.415 -0.014 -0.129
    14026
    141 For anything else, check: [wiki:Processing/KnownProblems Common or known problems]. This is now a bit out-of-date - if your solution isn't on there then please add it once you find what it is.
     27Hawk  : 0.585 -0.014 -0.129
    14228
    143 ---------
     29And use these boresight values (PRH):
    14430
    145 == Finished? ==
     31Eagle : -0.322 0.175 0.38
    14632
    147 Once you're satisfied with the processed data, you need to [wiki:Procedures/DeliveryCreation create a delivery directory] for it.
     33Hawk : -0.345 0.29 0.35
     34
     35=== Making a Delivery ===
     36
     37Use the make_hyper_delivery.py script to make the delivery directory. Run it from within the main project directory. By default it runs in dry run mode.
     38
     39Use --final if happy with what it says it will do. Use -m <config> to generate screenshots and mosaics
     40
     41To make the readme file use the script: create_latex_hyperspectral_apl_readme.py
     42
     43=== Individual processing stages: ===
     44
     45==== Stage 1: Radiometric Calibration ====
     46The software that performs this is called aplcal. It uses the cal files to calibrate the raw data to level 1b. It strips off the FODIS region (also calibrated to level 1b), "flips" the data such that:
     47 *  eagle runs low->high wavelengths
     48 *  hawk mirrored so that it is comparable to eagle (i.e. things on the left of the aircraft are on the left of the data file)
     49It also applies smear correction to the eagle data. Rather than mask out bad pixels it creates a separate mask file that contains flags for the following:
     50 *  overflows
     51 *  underflows
     52 *  smear affected
     53 *  bad ccd pixel
     54 *  dropped scans
     55Example command to calibrate a file:
     56   `aplcal -input VNIR11092-1.raw -output e09201b.bil -calfile calibration/2011/eagle/SN001 `
     57Example command to calibrate first 2501 lines of file:
     58   `aplcal -input VNIR11092-1.raw -output e09201b.bil -calfile calibration/2011/eagle/SN001 -lines 0 2500`
     59
     60==== Stage 2: Navigation syncing ====
     61This is performed using the aplnav software. It can use navigation data either from an SBET or specim nav file, depending on whether you want to use post-processed or real-time data.
     62It is possible to add a timing offset to shift the navgation data w.r.t the scan lines, or to shift the position w.r.t to attitude. It would be rare to require the latter as this suggests a problem with the SBET file.
     63
     64Lever arms and boresight values are entered here to offset the position and attitude to the eagle/hawk sensor view direction. It is possible to smooth the data (using a triangle filter) and interpolate the scan time data using either linear or spline methods.
     65BIL files containing quality flags for the navigation data are output also.
     66
     67'''IMPORTANT NOTE:'''
     68
     69 * Lever arms are entered as X Y Z in the aircraft coordinates (X +ve forward, Y +ve to starboard, Z +ve down). This is different to using aznav, which used X +ve forward, Y +ve port, Z +ve up.
     70 * Specim have previously said that there is a -0.055 second offset to apply to the sync message. Rather than hard code this into aplnav, you should add it onto the scantimeoffset parameter. This is automatically done using the processing scripts by the sct_global_offset parameter.
     71
     72Example command to sync post-processed navigation data:
     73   `aplnav -nav VNIR092-11-1.nav -sbet sbet_2011092.out -lev1 e092011b.bil -interp Spline -leverarm 0.559 0.015 1.543 -boresight -0.28 0.16 0.38 -scantimeoffset 0.045 -output e092011b_p_sct0.1_nav_post_processed.bil -qualityfile e092011b_nav_post_processed_quality.bil`
     74
     75==== Stage 3: Geocorrection ====
     76The first step of geocorrection is to generate the IGM file using aplcorr. This takes an optional 1-band BIL / BSQ DEM (in WGS-84 Lat/Lon) and the instrument FOV vector file.
     77
     78Example command:
     79   `aplcorr -lev1file e092011b.bil -vvfile eagle_fov_fullccd_vectors.bil -navfile e092011b_p_sct0.1_nav_post_processed.bil -dem ASTER_resampled_0.1667_arcsec.dem  -igmfile e092013b_p_sct0.1.igm`
     80
     81The second step of geocorrection is to re-project the IGM file into the chosen map projection, using apltran.
     82
     83Example commands:
     84   `apltran -inproj latlong WGS84 -igm e092013b_p_sct0.1.igm -outproj utm_wgs84N 31 -output e092013b_p_sct0.1_utm31n.igm`
     85
     86   `apltran -inproj latlong WGS84 -igm e092013b_p_sct0.1.igm -outproj osng /users/rsg/arsf/dems/ostn02/OSTN02_NTv2.gsb -output e092013b_p_sct0.1_utm31n.igm`
     87
     88The third step is then to map it.
     89
     90Example commands:
     91   `aplmap -igm e092013b_p_sct0.1_utm31n.igm -lev1 e092011b.bil -mapname e092013b_utm31.mapped -bandlist 30 15 7`
     92
     93   `aplmap -igm e092013b_p_sct0.1_utm31n.igm -lev1 e092011b.bil -mapname e092013b_utm31.mapped -bandlist 30 15 7` -pixelsize 2 2
     94
     95   `aplmap -igm e092013b_p_sct0.1_utm31n.igm -lev1 e092011b.bil -mapname e092013b_utm31.mapped -bandlist 30 15 7` -oversample 3 3