Changes between Initial Version and Version 1 of Procedures/DeliveryCreation/pythonlibrary


Ignore:
Timestamp:
Jan 21, 2014 10:58:05 AM (6 years ago)
Author:
mark1
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/DeliveryCreation/pythonlibrary

    v1 v1  
     1---------------------------------
     2== Make delivery instructions ==
     3--------------------------
     4
     5Delivery creation is now via a python library rather than individual scripts. This should make it easier to perform individual steps to the creation of the deliveries. The library files are located in ''~arsf/usr/lib/python/site-packages/arsf_delivery_library/'' with functions separated into library scripts based on the delivery data type. These files are:
     6
     7basic_functions.py - functions that are required for multiple delivery types
     8hyperspectral.py - functions for creating hyperspectral deliveries
     9lidar.py - functions for creating lidar deliveries
     10rcdcamera.py - functions for creating RCD camera deliveries
     11
     12To use or run any of the functions just import the required module and run the function, e.g.:
     13
     14{{{
     15import arsf_delivery_library.basic_functions
     16arsf_delivery_library.basic_functions.CreateDeliveryStructure(deltype='hyperspectral',delpath='/path/to/delivery',maketoplevel=True,onlyaddmissing=True)
     17}}}
     18
     19For 'normal' use there is a wrapper script that is run to generate a default delivery for each sensor type. This is the make_arsf_delivery.py script. It makes assumptions about the layout of the data (i.e. where in the system it is), uses default parameters for certain processes and should replace all previous delivery creation scripts. It can be run more than once if required (e.g. if something breaks the first run) and the steps to be performed (or missed out) can be selected. By default it will run in dry-run mode, although this mode of operation may fail certain tests because directories have not yet been created.
     20
     21When the script is run it will clearly state which part of the processing it is starting, so as to make it as easy as possible to correct mistakes or bug fix the script.
     22
     23---------------------------------
     24=== Example - create a hyperspectral delivery ===
     25-----------------------------------------
     26
     27{{{
     28make_arsf_delivery.py --projectlocation /users/rsg/arsf/arsf_data/2013/flight_data/arsf_internal/GB13_00-2013_308a_Little_Riss/ \
     29                      --deliverytype hyperspectral
     30}}}
     31
     32will (assuming everything is 'good' about the project) output information about commands it will run. As the delivery directory does not exist there will appear to be many errors, so if the output of the first stage looks correct (the STRUCTURE section) then run that step only to create the structure.
     33
     34{{{
     35make_arsf_delivery.py --projectlocation /users/rsg/arsf/arsf_data/2013/flight_data/arsf_internal/GB13_00-2013_308a_Little_Riss/ \
     36                      --deliverytype hyperspectral --steps STRUCTURE --final
     37}}}
     38
     39This will have now created the empty delivery structure. The script can be run again in dry run mode now but stating that STRUCTURE does not need to be run.
     40
     41{{{
     42make_arsf_delivery.py --projectlocation /users/rsg/arsf/arsf_data/2013/flight_data/arsf_internal/GB13_00-2013_308a_Little_Riss/ \
     43                      --deliverytype hyperspectral --notsteps STRUCTURE
     44}}}
     45
     46Hopefully the output will be a list of commands the script will run, separated up into sections based on the step names. If everything looks to be correct then the ''--final'' can be added to the command line to run the delivery creation.
     47
     48The output text should be inspected for error messages. These should be obvious and appear as yellow text against a red background. If problems occur then individual steps can be run after the problem has been fixed (note that you may need to also run steps after the one that failed). Also note that GRASS outputs a lot of progress messages to standard error - these will appear as errors (in yellow + red on terminal) but are not actually errors.
     49
     50---------------------------------
     51=== A not-so-straight-forward example ===
     52---------------------------------
     53
     54For cases which are not so straight forward there are some extra options that can be used in ''make_arsf_delivery.py'', or you can use the library functions directly.
     55
     56For example, to work with projects not in the repository, you will need to specify project details (year, julian day, project code and sortie) using the ''--projectinfo'' keyword. This is because the database can not be queried.
     57
     58If multiple configfiles exist the correct one to use can be specified using ''--configfile''.
     59
     60To create a delivery in a different place to the default (processing/delivery/...) you can use ''--deliverylocation'' to specify the delivery name and file path.
     61
     62If you feel unsure about anything or want to be cautious then you can run each step at a time by specifiying it with the ''--steps'' keyword. A list of the steps (in the order they're run) can be seen in the script ''--help'' - the ordering is important as the step may rely on something from a previous step.
     63
     64