Changes between Initial Version and Version 1 of Procedures/ProcessingChainInstructions/auto_applanix


Ignore:
Timestamp:
Oct 4, 2010 3:33:57 PM (10 years ago)
Author:
knpa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/ProcessingChainInstructions/auto_applanix

    v1 v1  
     1The processing of navigation data through the package of applanix programs has now been fully automated.
     2
     3The old system is documented [wiki:Processing/Applanix here].
     4Reading this beforehand will help your understanding of how it has been automated.
     5
     6**NOTE: If you are processing a non-UK flight, the results may be better if you process the navigation manually**
     7
     8'''Warning:''' the automated system relies heavily on the standardized file system structure which means that incorrect file/folder names or locations cause undefined behaviour (although generally the system just exits if it fails to find something)
     9
     10'''Warning2:''' the automated system can only read base station files in the .yyo format (ascii) you could try converting to this format if you need to process projects from outside the uk (generally have a binary base file)
     11
     12to use the automated system the project folder must be in the arsf workspace, it must contain the applanix/raw and rinex directories as real directories and the eagle and/or hawk directories (can be sym linked)
     13the 'win_dispatcher' script must be running on the windows box and the workspace mounted as drive W:\  (if this script is not running it must be started before continuing, see explanation further down page)
     14
     15
     16to use the system simple place the project in the arsf workspace and add the flag 'status_applanix_batch_gen_ready' to the directory
     17
     18{{{
     19touch status_applanix_batch_gen_ready
     20}}}
     21
     22
     23== win dispatcher ==
     24
     25This python script periodically checks the directories in a specified directory for a status flag and if it finds one calls another script passing the name of the directory
     26
     27win_dispatcher [<search dir>][<status flag>][<script to call>]
     28
     29generally it should be setup as such
     30{{{
     31 log in on windows box
     32 check W:\ drive is mapped to arsf workspace
     33 run command prompt
     34 cd d:\auto_applanix_scripts
     35 win_dispatcher W:\ status_applanix_processing_ready auto_applanix_dispatcher_link.py
     36}}}
     37
     38at its core the automated system consists of three stages
     39
     40 1. the dynamic detection of information from file names and contents in the project folder which is used to generate batch scripts to control the applanix software and config files
     41 1. the copying of the project to the windows box, execution of the applanix control batch scripts, analysis of the combine separation files and the copying back to the unix system of the results
     42 1. generation of graphs and cleanup
     43
     44
     45== files created (not created during manual processing) ==
     46 '''before processing'''
     47 1. 5 applanix control batch scripts
     48 1. 5 .gnv POSGNSS config files
     49 1. 1 auto.bat file which runs all other batch files
     50
     51 '''after processing'''
     52 1. 1 'combine_seperation_bestguess_values' file which contains the values the 'combine_seperation_analyser.py' script has given each attempt and the one chosen as best
     53 1. 5 sets of POSGNSS output files
     54 1. 5 sets of gnuplot data and config files
     55 1. 5 png files showing the separation graphs
     56 1. 1 sbet file from the run with the best combine separation file
     57
     58
     59== scripts used by stage ==
     60
     61 1. stage 1
     62   1. 'auto_applanix_driver.sh', this script calls 'auto_applanix_batch_config_generater.py' 5 times with different arguments and then calls 'control_batch_generator'
     63   1. 'auto_applanix_batch_config_generater.py', this script dynamically collects the information from file names and contents and uses it to create the applanix batch files and POSGNSS config file. uses several python scripts to perform conversions
     64   1. 'control_batch_generater.py'
     65 1. stage 2
     66   1. 'win_dispatch.py' a python script that should be constantly running on the windows machine. this script checks the specified directory for directorys containing a specific flag and then calls a linking script
     67   1. 'auto_applanix_dispatcher_link.py' this script performs all the copying to the windows box, then calls 'auto.bat' script, then copys everything except the RAW files (to preserve the timestamps) back to the arsf workspace
     68   1. 'auto.bat' just calls the other .bat files in the directory
     69   1. 'combine_seperation_analyser.py' this compares the combined separation of each of the 4 versions and attempts to select the best one, the values used for this attempt are recorded in 'combine_seperation_bestguess_values'. the script then calls the relevant versions sbet creation script. this means that only one sbet is created.
     70 1. stage 3
     71   1. 'gnuploter.py' this script takes the cmb files and uses them to draw graphs for the combined separation of the different versions. it also moves any files in the auto_applanix directory into the correct applanix directories
     72
     73
     74
     75== flags ==
     76
     77because this system is fully automated it makes use of a series of flags to indicated which stage it is at and whether an error has occurred
     78
     79||'status_applanix_batch_gen_ready'|| this is the first flag and indicates that the project is ready for stage 1||
     80||'status_applanix_queuing'|| this flag shows the scripts have been submitted to gridnodes||
     81||'status_applanix_processing_ready'|| this flag shows that the auto generation of batch and config files is complete and is detected by the 'win_dispatcher.py' script||
     82||'status_applanix_in_progress'|| this flag shows that the applanix programs are currently running on the windows box||
     83||'status_applanix_processing_complete || this flag shows that the nav process has finished but not that the transfer back from the windows box or the clean up has been finished ||
     84||'status_windows_applanix_completed'|| this flag shows that the windows section of the processing has completed successfully||
     85||'status_applanix_gnuplot_queuing'|| this flag shows that the generation of graphs is queued or in progress||
     86
     87||'status_applanix_complete'|| navigation processing complete||
     88
     89== errors ==
     90
     91in the event something goes horribly wrong there are several things that will happen
     92
     93'''command line messages'''
     94
     95when the scripts fail they generally attempt to convey a meaningful message to indicate the cause or point at which the error occurred. additionally in all but the hardest crashes the linux side controller auto_applanix_driver.sh will print out in which script the error occurred. These messages are printed to the command line but because the scripts are run on the grid nodes the messages are piped into a file called dispatch_summary
     96
     97
     98
     99'''error flags'''
     100 
     101||'status_error_auto_applanix_batch_gen_dispatch'|| this indicates a problem dispatching to the grid nodes||
     102||'status_error_gen'|| this indicates an error has occurred during stage one relating to 'auto_applanix_batch_config_generater.sh', in the event of this error check the 'dispatch_summary' file in the project folder and find the last 2 lines which contain the error warning and the location of the error||
     103||'status_error_copytowin'|| this flag indicates that an error occurred while copying the project to the windows box||
     104||'status_error_winprocess'|| for some reason one of the batch files failed to produce an sbet||
     105||'status_error_copytounix'|| this flag indicates that an error occurred while copying the project to the unix file system||
     106||'status_combine_seperation_analyses_error'|| this indicates a problem has occurred in the combine_seperation_analyser.py script on the windows box||
     107||'status_error_gnuplot_dispatch'|| this flag shows that an error occurred submitting the 'gnuploter.py' script to the grid nodes||
     108||'status_error_auto_applanix_batch_gen_dispatch'|| an error has occurred submitting batch gen to gridnodes||
     109
     110'''dispatch summary'''
     111
     112the dispatch summary file is the file into which the output of any script run during the gridnodes processing is sent, this means it contains all the debugging messages and in the case of an unexpected error (one not handled with a proper error message) the error message of the command can be found.
     113
     114
     115== checking ==
     116
     117if you wish to double check the results of the system you can view the separation graphs in the auto_applanix folder. If it is necessary the project folder can be manually copied to the windows machine and then the user need only double click on the .gnv file in the auto_applanix folder relating to the version they wish to check. This will open POSGNSS with all the of settings that would have been used by the automated system, the user can then proceed to alter settings and manually process the flight data.
     118
     119NOTE: if you change settings and then save them this will alter the .gnv config file, this means that until the automatic system is run again from scratch the new settings will be used.
     120
     121
     122== useful info ==
     123
     124the batch generation script 'auto_applanix_batch_config_generater' can be run separately and then the output bat files used to run applanix. This is useful if you wish to try settings
     125that are not tried by the auto_applanix_driver, or wish to run parts of the system manually and parts automatically
     126
     127'''auto_applanix_batch_config_generater arguments'''
     128|| -f [directory]|| this argument allows the project folder to be specified, if absent the current directory is used as the target||
     129|| -u [directory]|| this argument allows the location of the utils folder (holding all the auto applanix associated scripts and files required) to be specified. NOTE: this must be an absolute path||
     130|| -b [name]     || specifies the name of the desired base station within the projects rinex folder, if absent the closest to the average center of all flightlines in the eagle and hawk dirs will be used.||
     131|| -em [float]   ||   sets the elevation mask to be used, default 12.5'||
     132|| -kn ["AUTO", "IONO", "HIGH", "MEDIUM", "LOW"] || sets the KAR L2 NOISE, default "AUTO"||
     133|| -kd1 [float]  ||   set KAR single frequency distance km, default 7.5 km  ||
     134|| -kd2 [float]   ||  set KAR dual frequency distance km, default 30 km ||
     135|| -io ["OFF", "RELATIVE", "FREE"] || set the ionospheric model option, default "FREE" ||
     136|| -bn [int][int][int] ||  this applies a nudge to the xyz coordinates of the base station, this was introduced for the hyquapro tests and should not be used in normal operation as it is only suitable for the boresight little rissington project. if it is needed for another project see the comments in the 'cordinate_offsetter.py' script. NOTE. this is not in the python version, if you need this option you will need to use the old bash version ||
     137|| -blind     ||      this argument specifies that the batch file generated should preform the whole applanix chain from extraction to creation of the sbet file without user input. if absent the batch files will contain only commands up to the point where the pretty green lines are displayed. From this point the user can check the separation manually and reprocess and then produce an sbet manually. ||
     138|| -k [name]  ||      this specifies the kernel name used. the kernel is used in the naming of many of the output and results files produced by both the auto_applanix system and applanix itself. ||
     139|| -first     ||    '''MANADATORY''' for the moment. if your feeling confident you can leave this argument off if you know the project folder already contains the extracted and converted files from applanix (as these files don't change there is no reason to reproduce them every time). the absence of the 'first' argument will suppress the creation of the batch scripts which control extraction and conversion. ||
     140
     141
     142