= Processing navigation data =

This page describes the steps necessary to process navigation data, both for Eagle/Hawk data (Applanix) and for the ALS-II Lidar (IPAS).

== IPAS processing ==

 * [wiki:Processing/IPASPro]
 * [wiki:Processing/GrafNavProcessing]

== Applanix processing ==

The processing of navigation data through the package of applanix programs has now been fully automated, what follows is a brief description of the new system.
The old system was a manual system and is documented [wiki:Processing/Applanix here].  Reading this beforehand will help your understanding of how it has been automated.

'''Warning:''' the automated system relies heavily on the standardized file system structure which means that incorrect file/folder names or locations cause undefined behaviour (although generally the system just exits if it fails to find something)

'''Warning2:''' the automated system can only read base station files in the .yyo format (ascii) you could try converting to this format if you need to process projects from outside the uk (generally have a binary base file)

to use the automated system the project folder must be in the arsf workspace, it must contain the applanix/raw and rinex directories as real directories and the eagle and/or hawk directories (can be sym linked)
the 'win_dispatcher' script must be running on the windows box and the workspace mounted as drive W:\  (if this script is not running it must be started before continuing, see explanation further down page)


to use the system simple place the project in the arsf workspace and add the flag 'status_applanix_batch_gen_ready' to the directory

{{{
touch status_applanix_batch_gen_ready
}}}


== win dispatcher ==

This python script periodically checks the directories in a specified directory for a status flag and if it finds one calls another script passing the name of the directory

win_dispatcher [<search dir>][<status flag>][<script to call>]

generally it should be setup as such
{{{
 log in on windows box
 check W:\ drive is mapped to arsf workspace
 run command prompt
 cd d:\auto_applanix_scripts
 win_dispatcher W:\ status_applanix_processing_ready auto_applanix_dispatcher_link.py
}}}

at its core the automated system consists of three stages

 1. the dynamic detection of information from file names and contents in the project folder which is used to generate batch scripts to control the applanix software and config files
 1. the copying of the project to the windows box, execution of the applanix control batch scripts, analysis of the combine separation files and the copying back to the unix system of the results
 1. generation of graphs and cleanup


== files created (not created during manual processing) ==
 '''before processing'''
 1. 5 applanix control batch scripts
 1. 5 .gnv POSGNSS config files
 1. 1 auto.bat file which runs all other batch files

 '''after processing'''
 1. 1 'combine_seperation_bestguess_values' file which contains the values the 'combine_seperation_analyser.py' script has given each attempt and the one chosen as best
 1. 5 sets of POSGNSS output files 
 1. 5 sets of gnuplot data and config files
 1. 5 png files showing the separation graphs
 1. 1 sbet file from the run with the best combine separation file


== scripts used by stage ==

 1. stage 1
   1. 'auto_applanix_driver.sh', this script calls 'auto_applanix_batch_config_generater.py' 5 times with different arguments and then calls 'control_batch_generator'
   1. 'auto_applanix_batch_config_generater.py', this script dynamically collects the information from file names and contents and uses it to create the applanix batch files and POSGNSS config file. uses several python scripts to perform conversions
   1. 'control_batch_generater.py'
 1. stage 2
   1. 'win_dispatch.py' a python script that should be constantly running on the windows machine. this script checks the specified directory for directorys containing a specific flag and then calls a linking script
   1. 'auto_applanix_dispatcher_link.py' this script performs all the copying to the windows box, then calls 'auto.bat' script, then copys everything except the RAW files (to preserve the timestamps) back to the arsf workspace
   1. 'auto.bat' just calls the other .bat files in the directory
   1. 'combine_seperation_analyser.py' this compares the combined separation of each of the 4 versions and attempts to select the best one, the values used for this attempt are recorded in 'combine_seperation_bestguess_values'. the script then calls the relevant versions sbet creation script. this means that only one sbet is created.
 1. stage 3
   1. 'gnuploter.py' this script takes the cmb files and uses them to draw graphs for the combined separation of the different versions. it also moves any files in the auto_applanix directory into the correct applanix directories 



== flags ==

because this system is fully automated it makes use of a series of flags to indicated which stage it is at and whether an error has occurred

||'status_applanix_batch_gen_ready'|| this is the first flag and indicates that the project is ready for stage 1||
||'status_applanix_queuing'|| this flag shows the scripts have been submitted to gridnodes||
||'status_applanix_processing_ready'|| this flag shows that the auto generation of batch and config files is complete and is detected by the 'win_dispatcher.py' script||
||'status_applanix_in_progress'|| this flag shows that the applanix programs are currently running on the windows box||
||'status_applanix_processing_complete || this flag shows that the nav process has finished but not that the transfer back from the windows box or the clean up has been finished ||
||'status_windows_applanix_completed'|| this flag shows that the windows section of the processing has completed successfully||
||'status_applanix_gnuplot_queuing'|| this flag shows that the generation of graphs is queued or in progress||

||'status_applanix_complete'|| navigation processing complete|| 

== errors ==

in the event something goes horribly wrong there are several things that will happen

'''command line messages'''

when the scripts fail they generally attempt to convey a meaningful message to indicate the cause or point at which the error occurred. additionally in all but the hardest crashes the linux side controller auto_applanix_driver.sh will print out in which script the error occurred. These messages are printed to the command line but because the scripts are run on the grid nodes the messages are piped into a file called dispatch_summary



'''error flags'''
 
||'status_error_auto_applanix_batch_gen_dispatch'|| this indicates a problem dispatching to the grid nodes||
||'status_error_gen'|| this indicates an error has occurred during stage one relating to 'auto_applanix_batch_config_generater.sh', in the event of this error check the 'dispatch_summary' file in the project folder and find the last 2 lines which contain the error warning and the location of the error||
||'status_error_copytowin'|| this flag indicates that an error occurred while copying the project to the windows box||
||'status_error_winprocess'|| for some reason one of the batch files failed to produce an sbet||
||'status_error_copytounix'|| this flag indicates that an error occurred while copying the project to the unix file system||
||'status_combine_seperation_analyses_error'|| this indicates a problem has occurred in the combine_seperation_analyser.py script on the windows box||
||'status_error_gnuplot_dispatch'|| this flag shows that an error occurred submitting the 'gnuploter.py' script to the grid nodes||
||'status_error_auto_applanix_batch_gen_dispatch'|| an error has occurred submitting batch gen to gridnodes||

'''dispatch summary'''

the dispatch summary file is the file into which the output of any script run during the gridnodes processing is sent, this means it contains all the debugging messages and in the case of an unexpected error (one not handled with a proper error message) the error message of the command can be found.


== checking ==

if you wish to double check the results of the system you can view the separation graphs in the auto_applanix folder. If it is necessary the project folder can be manually copied to the windows machine and then the user need only double click on the .gnv file in the auto_applanix folder relating to the version they wish to check. This will open POSGNSS with all the of settings that would have been used by the automated system, the user can then proceed to alter settings and manually process the flight data. 

NOTE: if you change settings and then save them this will alter the .gnv config file, this means that until the automatic system is run again from scratch the new settings will be used.


== useful info == 

the batch generation script 'auto_applanix_batch_config_generater' can be run separately and then the output bat files used to run applanix. This is useful if you wish to try settings
that are not tried by the auto_applanix_driver, or wish to run parts of the system manually and parts automatically

'''auto_applanix_batch_config_generater arguments'''
|| -f [directory]|| this argument allows the project folder to be specified, if absent the current directory is used as the target||
|| -u [directory]|| this argument allows the location of the utils folder (holding all the auto applanix associated scripts and files required) to be specified. NOTE: this must be an absolute path||
|| -b [name]     || specifies the name of the desired base station within the projects rinex folder, if absent the closest to the average center of all flightlines in the eagle and hawk dirs will be used.||
|| -em [float]   ||   sets the elevation mask to be used, default 12.5'||
|| -kn ["AUTO", "IONO", "HIGH", "MEDIUM", "LOW"] || sets the KAR L2 NOISE, default "AUTO"||
|| -kd1 [float]  ||   set KAR single frequency distance km, default 7.5 km  ||
|| -kd2 [float]   ||  set KAR dual frequency distance km, default 30 km ||
|| -io ["OFF", "RELATIVE", "FREE"] || set the ionospheric model option, default "FREE" ||
|| -bn [int][int][int] ||  this applies a nudge to the xyz coordinates of the base station, this was introduced for the hyquapro tests and should not be used in normal operation as it is only suitable for the boresight little rissington project. if it is needed for another project see the comments in the 'cordinate_offsetter.py' script. NOTE. this is not in the python version, if you need this option you will need to use the old bash version ||
|| -blind     ||      this argument specifies that the batch file generated should preform the whole applanix chain from extraction to creation of the sbet file without user input. if absent the batch files will contain only commands up to the point where the pretty green lines are displayed. From this point the user can check the separation manually and reprocess and then produce an sbet manually. ||
|| -k [name]  ||      this specifies the kernel name used. the kernel is used in the naming of many of the output and results files produced by both the auto_applanix system and applanix itself. ||
|| -first     ||    '''MANADATORY''' for the moment. if your feeling confident you can leave this argument off if you know the project folder already contains the extracted and converted files from applanix (as these files don't change there is no reason to reproduce them every time). the absence of the 'first' argument will suppress the creation of the batch scripts which control extraction and conversion. ||