Changes between Version 9 and Version 10 of Procedures/ProcessingChainInstructions/NavigationProcessing


Ignore:
Timestamp:
Oct 4, 2010 3:34:28 PM (9 years ago)
Author:
knpa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/ProcessingChainInstructions/NavigationProcessing

    v9 v10  
    88 * [wiki:Processing/GrafNavProcessing]
    99
    10 The processing of navigation data through the package of applanix programs has now been fully automated.
    11 
    12 The old system is documented [wiki:Processing/Applanix here].
    13 Reading this beforehand will help your understanding of how it has been automated.
    14 
    15 **NOTE: If you are processing a non-UK flight, the results may be better if you process the navigation manually**
    16 
    17 '''Warning:''' the automated system relies heavily on the standardized file system structure which means that incorrect file/folder names or locations cause undefined behaviour (although generally the system just exits if it fails to find something)
    18 
    19 '''Warning2:''' the automated system can only read base station files in the .yyo format (ascii) you could try converting to this format if you need to process projects from outside the uk (generally have a binary base file)
    20 
    21 to use the automated system the project folder must be in the arsf workspace, it must contain the applanix/raw and rinex directories as real directories and the eagle and/or hawk directories (can be sym linked)
    22 the 'win_dispatcher' script must be running on the windows box and the workspace mounted as drive W:\  (if this script is not running it must be started before continuing, see explanation further down page)
    23 
    24 
    25 to use the system simple place the project in the arsf workspace and add the flag 'status_applanix_batch_gen_ready' to the directory
    26 
    27 {{{
    28 touch status_applanix_batch_gen_ready
    29 }}}
    30 
    31 
    32 == win dispatcher ==
    33 
    34 This python script periodically checks the directories in a specified directory for a status flag and if it finds one calls another script passing the name of the directory
    35 
    36 win_dispatcher [<search dir>][<status flag>][<script to call>]
    37 
    38 generally it should be setup as such
    39 {{{
    40  log in on windows box
    41  check W:\ drive is mapped to arsf workspace
    42  run command prompt
    43  cd d:\auto_applanix_scripts
    44  win_dispatcher W:\ status_applanix_processing_ready auto_applanix_dispatcher_link.py
    45 }}}
    46 
    47 at its core the automated system consists of three stages
    48 
    49  1. the dynamic detection of information from file names and contents in the project folder which is used to generate batch scripts to control the applanix software and config files
    50  1. the copying of the project to the windows box, execution of the applanix control batch scripts, analysis of the combine separation files and the copying back to the unix system of the results
    51  1. generation of graphs and cleanup
    52 
    53 
    54 == files created (not created during manual processing) ==
    55  '''before processing'''
    56  1. 5 applanix control batch scripts
    57  1. 5 .gnv POSGNSS config files
    58  1. 1 auto.bat file which runs all other batch files
    59 
    60  '''after processing'''
    61  1. 1 'combine_seperation_bestguess_values' file which contains the values the 'combine_seperation_analyser.py' script has given each attempt and the one chosen as best
    62  1. 5 sets of POSGNSS output files
    63  1. 5 sets of gnuplot data and config files
    64  1. 5 png files showing the separation graphs
    65  1. 1 sbet file from the run with the best combine separation file
    66 
    67 
    68 == scripts used by stage ==
    69 
    70  1. stage 1
    71    1. 'auto_applanix_driver.sh', this script calls 'auto_applanix_batch_config_generater.py' 5 times with different arguments and then calls 'control_batch_generator'
    72    1. 'auto_applanix_batch_config_generater.py', this script dynamically collects the information from file names and contents and uses it to create the applanix batch files and POSGNSS config file. uses several python scripts to perform conversions
    73    1. 'control_batch_generater.py'
    74  1. stage 2
    75    1. 'win_dispatch.py' a python script that should be constantly running on the windows machine. this script checks the specified directory for directorys containing a specific flag and then calls a linking script
    76    1. 'auto_applanix_dispatcher_link.py' this script performs all the copying to the windows box, then calls 'auto.bat' script, then copys everything except the RAW files (to preserve the timestamps) back to the arsf workspace
    77    1. 'auto.bat' just calls the other .bat files in the directory
    78    1. 'combine_seperation_analyser.py' this compares the combined separation of each of the 4 versions and attempts to select the best one, the values used for this attempt are recorded in 'combine_seperation_bestguess_values'. the script then calls the relevant versions sbet creation script. this means that only one sbet is created.
    79  1. stage 3
    80    1. 'gnuploter.py' this script takes the cmb files and uses them to draw graphs for the combined separation of the different versions. it also moves any files in the auto_applanix directory into the correct applanix directories
    8110
    8211
    8312
    84 == flags ==
    85 
    86 because this system is fully automated it makes use of a series of flags to indicated which stage it is at and whether an error has occurred
    87 
    88 ||'status_applanix_batch_gen_ready'|| this is the first flag and indicates that the project is ready for stage 1||
    89 ||'status_applanix_queuing'|| this flag shows the scripts have been submitted to gridnodes||
    90 ||'status_applanix_processing_ready'|| this flag shows that the auto generation of batch and config files is complete and is detected by the 'win_dispatcher.py' script||
    91 ||'status_applanix_in_progress'|| this flag shows that the applanix programs are currently running on the windows box||
    92 ||'status_applanix_processing_complete || this flag shows that the nav process has finished but not that the transfer back from the windows box or the clean up has been finished ||
    93 ||'status_windows_applanix_completed'|| this flag shows that the windows section of the processing has completed successfully||
    94 ||'status_applanix_gnuplot_queuing'|| this flag shows that the generation of graphs is queued or in progress||
    95 
    96 ||'status_applanix_complete'|| navigation processing complete||
    97 
    98 == errors ==
    99 
    100 in the event something goes horribly wrong there are several things that will happen
    101 
    102 '''command line messages'''
    103 
    104 when the scripts fail they generally attempt to convey a meaningful message to indicate the cause or point at which the error occurred. additionally in all but the hardest crashes the linux side controller auto_applanix_driver.sh will print out in which script the error occurred. These messages are printed to the command line but because the scripts are run on the grid nodes the messages are piped into a file called dispatch_summary
    105 
    106 
    107 
    108 '''error flags'''
    109  
    110 ||'status_error_auto_applanix_batch_gen_dispatch'|| this indicates a problem dispatching to the grid nodes||
    111 ||'status_error_gen'|| this indicates an error has occurred during stage one relating to 'auto_applanix_batch_config_generater.sh', in the event of this error check the 'dispatch_summary' file in the project folder and find the last 2 lines which contain the error warning and the location of the error||
    112 ||'status_error_copytowin'|| this flag indicates that an error occurred while copying the project to the windows box||
    113 ||'status_error_winprocess'|| for some reason one of the batch files failed to produce an sbet||
    114 ||'status_error_copytounix'|| this flag indicates that an error occurred while copying the project to the unix file system||
    115 ||'status_combine_seperation_analyses_error'|| this indicates a problem has occurred in the combine_seperation_analyser.py script on the windows box||
    116 ||'status_error_gnuplot_dispatch'|| this flag shows that an error occurred submitting the 'gnuploter.py' script to the grid nodes||
    117 ||'status_error_auto_applanix_batch_gen_dispatch'|| an error has occurred submitting batch gen to gridnodes||
    118 
    119 '''dispatch summary'''
    120 
    121 the dispatch summary file is the file into which the output of any script run during the gridnodes processing is sent, this means it contains all the debugging messages and in the case of an unexpected error (one not handled with a proper error message) the error message of the command can be found.
    122 
    123 
    124 == checking ==
    125 
    126 if you wish to double check the results of the system you can view the separation graphs in the auto_applanix folder. If it is necessary the project folder can be manually copied to the windows machine and then the user need only double click on the .gnv file in the auto_applanix folder relating to the version they wish to check. This will open POSGNSS with all the of settings that would have been used by the automated system, the user can then proceed to alter settings and manually process the flight data.
    127 
    128 NOTE: if you change settings and then save them this will alter the .gnv config file, this means that until the automatic system is run again from scratch the new settings will be used.
    129 
    130 
    131 == useful info ==
    132 
    133 the batch generation script 'auto_applanix_batch_config_generater' can be run separately and then the output bat files used to run applanix. This is useful if you wish to try settings
    134 that are not tried by the auto_applanix_driver, or wish to run parts of the system manually and parts automatically
    135 
    136 '''auto_applanix_batch_config_generater arguments'''
    137 || -f [directory]|| this argument allows the project folder to be specified, if absent the current directory is used as the target||
    138 || -u [directory]|| this argument allows the location of the utils folder (holding all the auto applanix associated scripts and files required) to be specified. NOTE: this must be an absolute path||
    139 || -b [name]     || specifies the name of the desired base station within the projects rinex folder, if absent the closest to the average center of all flightlines in the eagle and hawk dirs will be used.||
    140 || -em [float]   ||   sets the elevation mask to be used, default 12.5'||
    141 || -kn ["AUTO", "IONO", "HIGH", "MEDIUM", "LOW"] || sets the KAR L2 NOISE, default "AUTO"||
    142 || -kd1 [float]  ||   set KAR single frequency distance km, default 7.5 km  ||
    143 || -kd2 [float]   ||  set KAR dual frequency distance km, default 30 km ||
    144 || -io ["OFF", "RELATIVE", "FREE"] || set the ionospheric model option, default "FREE" ||
    145 || -bn [int][int][int] ||  this applies a nudge to the xyz coordinates of the base station, this was introduced for the hyquapro tests and should not be used in normal operation as it is only suitable for the boresight little rissington project. if it is needed for another project see the comments in the 'cordinate_offsetter.py' script. NOTE. this is not in the python version, if you need this option you will need to use the old bash version ||
    146 || -blind     ||      this argument specifies that the batch file generated should preform the whole applanix chain from extraction to creation of the sbet file without user input. if absent the batch files will contain only commands up to the point where the pretty green lines are displayed. From this point the user can check the separation manually and reprocess and then produce an sbet manually. ||
    147 || -k [name]  ||      this specifies the kernel name used. the kernel is used in the naming of many of the output and results files produced by both the auto_applanix system and applanix itself. ||
    148 || -first     ||    '''MANADATORY''' for the moment. if your feeling confident you can leave this argument off if you know the project folder already contains the extracted and converted files from applanix (as these files don't change there is no reason to reproduce them every time). the absence of the 'first' argument will suppress the creation of the batch scripts which control extraction and conversion. ||
    149 
    150 
    151