Changes between Initial Version and Version 1 of Procedures/ProcessingChainInstructions/NavigationProcessing


Ignore:
Timestamp:
May 28, 2010 8:35:32 PM (10 years ago)
Author:
mggr
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/ProcessingChainInstructions/NavigationProcessing

    v1 v1  
     1= Processing navigation data =
     2
     3This page describes the steps necessary to process navigation data, both for Eagle/Hawk data (Applanix) and for the ALS-II Lidar (IPAS).
     4
     5== IPAS processing ==
     6
     7 * [wiki:Processing/IPASPro]
     8 * [wiki:Processing/GrafNavProcessing]
     9
     10== Applanix processing ==
     11
     12to use the automated system the project folder must be in the arsf workspace, it must contain the applanix/raw and rinex directorys as real directorys and the eagle and/or hawk directorys (can be sym linked)
     13the 'win_dispatcher' script must be running on the windows box and the workspace mounted as drive W:\  (if this script is not running it must be started before continueing, see explanation further down page)
     14
     15
     16
     17to use the system simple place the project in the arsf workspace and add the flag 'status_applanix_batch_gen_ready' to the directory
     18
     19{{{
     20touch status_applanix_batch_gen_ready
     21}}}
     22
     23
     24The processing of navigation data through the package of applanix programs has now been fully automated, what follows is a brief description of the new system.
     25The old system was a manual system and is documented [wiki:Processing/Applanix here].  Reading this beforehand will help your understanding of how it has been automated.
     26
     27
     28== win dispatcher ==
     29
     30This python script periodically checks the directorys in a specified directory for a status flag and if it finds one calls another script passing the name of the directory
     31
     32win_dispatcher [<search dir>][<status flag>][<script to call>]
     33
     34generally it should be setup as such
     35{{{
     36 log in on windows box
     37 check W:\ drive is mapped to arsf workspace
     38 run command prompt
     39 cd d:\auto_applanix_scripts
     40 win_dispatcher W:\ status_applanix_processing_ready auto_applanix_dispatcher_link.py
     41}}}
     42
     43at its core the automated system consists of three stages
     44
     45 1. the dynamic detection of information from file names and contents in the project folder which is used to generate batch scripts to control the applanix software and config files
     46 1. the copying of the project to the windows box, execution of the applanix control batch scripts, analysis of the combine separation files and the copying back to the unix system of the results
     47 1. generation of graphs and cleanup
     48
     49
     50== files created (not created during manual processing) ==
     51 '''before processing'''
     52 1. 4 applanix control batch scripts
     53 1. 4 .gnv POSGNSS config files
     54 1. 1 auto.bat file which runs all other batch files
     55
     56 '''after processing'''
     57 1. 1 'combine_seperation_bestguess_values' file which contains the values the 'combine_seperation_analyser.py' script has given each attempt and the one chosen as best
     58 1. 4 sets of POSGNSS output files
     59 1. 4 sets of gnuplot data and config files
     60 1. 4 png files showing the separation graphs
     61 1. 1 sbet file from the run with the best combine separation file
     62
     63
     64== scripts used by stage ==
     65
     66 1. stage 1
     67   1. 'auto_applanix_driver.sh', this script calls 'auto_applanix_batch_config_generater.py' 4 times with different arguments and then calls 'control_batch_generator'
     68   1. 'auto_applanix_batch_config_generater.py', this script dynamically collects the information from file names and contents and uses it to create the applanix batch files and POSGNSS config file. uses several python scripts to perform conversions
     69   1. 'control_batch_generater.py'
     70 1. stage 2
     71   1. 'win_dispatch.py' a python script that should be constantly running on the windows machine. this script checks the specified directory for directorys containing a specific flag and then calls a linking script
     72   1. 'auto_bob_dispatcher_link.py' this script performs all the copying to the windows box, then calls 'auto.bat' script, then copys everything except the RAW files (to preserve the timestamps) back to the arsf workspace
     73   1. 'auto.bat' just calls the other .bat files in the directory
     74   1. 'combine_seperation_analyser.py' this compares the combined seperation of each of the 4 versions and attempts to select the best one, the values used for this attempt are recorded in 'combine_seperation_bestguess_values'. the script then calls the relevant versions sbet creation script. this means that only one sbet is created.
     75 1. stage 3
     76   1. 'gnuploter.py' this script takes the cmb files and uses them to draw graphs for the combined separation of the different versions. it also moves any files in the auto_applanix directory into the correct applanix directorys
     77
     78
     79
     80== flags ==
     81
     82because this system is fully automated it makes use of a series of flags to indicated which stage it is at and whether an error has occurred
     83
     84||'status_applanix_batch_gen_ready'|| this is the first flag and indicates that the project is ready for stage 1||
     85||'status_applanix_queuing'|| this flag shows the scripts have been submitted to gridnodes||
     86||'status_applanix_processing_ready'|| this flag shows that the auto generation of batch and config files is complete and is detected by the 'win_dispatcher.py' script||
     87||'status_applanix_in_progress'|| this flag shows that the applanix programs are currently running on the windows box||
     88||'status_applanix_processing_complete || this flag shows that the nav process has finished but not that the transfer back from the windows box or the clean up has been finished ||
     89||'status_windows_applanix_completed'|| this flag shows that the windows section of the processing has completed successfully||
     90||'status_applanix_gnuplot_queuing'|| this flag shows that the generation of graphs is queued or in progress||
     91
     92||'status_applanix_complete'|| navigation processing complete||
     93
     94== errors ==
     95
     96in the event something goes horribly wrong there are several things that will happen
     97
     98'''command line messages'''
     99
     100when the scripts fail they generally attempt to convey a meaningful message to indicate the cause or point at which the error occurred. additionally in all but the hardest crashes the linux side controller auto_applanix_driver.sh will print out in which script the error occurred.
     101
     102
     103
     104'''error flags'''
     105 
     106||'status_error_auto_applanix_batch_gen_dispatch'|| this indicates a problem dispatching to the grid nodes||
     107||'status_error_gen'|| this indicates an error has occurred during stage one relating to 'auto_applanix_batch_config_generater.sh', in the event of this error check the 'dispatch_summary' file in the project folder and find the last 2 lines which contain the error warning and the location of the error||
     108||'status_error_copytowin'|| this flag indicates that an error occurred while copying the project to the windows box||
     109||'status_error_winprocess'|| for some reason one of the batch files failed to produce an sbet||
     110||'status_error_copytounix'|| this flag indicates that an error occurred while copying the project to the unix file system||
     111||'status_combine_seperation_analyses_error'|| this indicates a problem has occurred in the combine_seperation_analyser.py script on the windows box||
     112||'status_error_gnuplot_dispatch'|| this flag shows that an error occurred submitting the 'gnuploter.py' script to the grid nodes||
     113||'status_error_auto_applanix_batch_gen_dispatch'|| an error has occurred submitting batch gen to gridnodes||
     114
     115'''dispatch summary'''
     116
     117the dispatch summary file is the file into which the output of any script run during the gridnodes processing is sent, this means it contains all the debugging messages and in the case of an unexpected error (one not handled with a proper error message) the error message of the command can be found.
     118
     119
     120== checking ==
     121
     122if you wish to double check the results of the system you can view the separation graphs in the auto_bapplanix folder. If it is necessary the project folder can be manually copied to the windows machine and then the user need only double click on the .gnv file in the auto_applanix folder relating to the version they wish to check. This will open GNSSPOS with all the of settings that would have been used by the automated system, the user can then proceed to alter settings and manually process the flight data.
     123
     124NOTE: if you change settings and then save them this will alter the .gnv config file, this means that until the automatic system is run again from scratch the new settings will be used.
     125
     126
     127== useful info ==
     128
     129the batch generation script 'auto_applanix_batch_config_generater' can be run separately and then the output bat files used to run applanix. This is useful if you wish to try settings
     130that are not tried by the auto_applanix_driver, or wish to run parts of the system manually and parts automatically
     131
     132'''auto_applanix_batch_config_generater arguments'''
     133|| -f [directory]|| this argument allows the project folder to be specified, if absent the current directory is used as the target||
     134|| -u [directory]|| this argument allows the location of the utils folder (holding all the auto applanix associated scripts and files required) to be specified. NOTE: this must be an absolute path||
     135|| -b [name]     || specifies the name of the desired base station within the projects rinex folder, if absent the closest to the average center of all flightlines in the eagle and hawk dirs will be used.||
     136|| -em [float]   ||   sets the elevation mask to be used, default 12.5'||
     137|| -kn ["AUTO", "IONO", "HIGH", "MEDIUM", "LOW"] || sets the KAR L2 NOISE, default "AUTO"||
     138|| -kd1 [float]  ||   set KAR single frequency distance km, default 7.5 km  ||
     139|| -kd2 [float]   ||  set KAR dual frequency distance km, default 30 km ||
     140|| -io ["OFF", "RELATIVE", "FREE"] || set the ionospheric model option, default "FREE" ||
     141|| -bn [int][int][int] ||  this applies a nudge to the xyz coordinates of the base station, this was introduced for the hyquapro tests and should not be used in normal operation as it is only suitable for the boresight little rissiginton project. if it is needed for another project see the comments in the 'cordinate_offsetter.py' script. NOTE. this is not in the python version, if you need this option you will need to use the old bash version ||
     142|| -blind     ||      this argument specifies that the batch file generated should preform the whole applanix chain from extraction to creation of the sbet file without user input. if absent the batch files will contain only commands up to the point where the pretty green lines are displayed. From this point the user can check the separation manually and reprocess and then produce an sbet manually. ||
     143|| -k [name]  ||      this specifies the kernel name used. the kernel is used in the naming of many of the output and results files produced by both the auto_applanix system and applanix itself. ||
     144|| -first     ||    '''MANADATORY''' for the moment. if your feeling confident you can leave this argument off if you know the project folder already contains the extracted and converted files from applanix (as these files don't change there is no reason to reproduce them every time). the absence of the 'first' argument will suppress the creation of the batch scripts which control extraction and conversion. ||
     145
     146
     147