| Version 4 (modified by asm, 9 years ago) (diff) |
|---|
Make delivery instructions
Delivery creation is now via a python library rather than individual scripts. This should make it easier to perform individual steps to the creation of the deliveries. The library files are located in ~arsf/usr/lib/python/site-packages/arsf_delivery_library/ with functions separated into library scripts based on the delivery data type. These files are:
basic_functions.py - functions that are required for multiple delivery types hyperspectral.py - functions for creating hyperspectral deliveries lidar.py - functions for creating lidar deliveries rcdcamera.py - functions for creating RCD camera deliveries
To use or run any of the functions just import the required module and run the function, e.g.:
import arsf_delivery_library.basic_functions arsf_delivery_library.basic_functions.CreateDeliveryStructure(deltype='hyperspectral',delpath='/path/to/delivery',maketoplevel=True,onlyaddmissing=True)
For 'normal' use there is a wrapper script that is run to generate a default delivery for each sensor type. This is the make_arsf_delivery.py script. It makes assumptions about the layout of the data (i.e. where in the system it is), uses default parameters for certain processes and should replace all previous delivery creation scripts. It can be run more than once if required (e.g. if something breaks the first run) and the steps to be performed (or missed out) can be selected. By default it will run in dry-run mode, although this mode of operation may fail certain tests because directories have not yet been created.
When the script is run it will clearly state which part of the processing it is starting, so as to make it as easy as possible to correct mistakes or bug fix the script.
Example - create a hyperspectral delivery
make_arsf_delivery.py --projectlocation /users/rsg/arsf/arsf_data/2013/flight_data/arsf_internal/GB13_00-2013_308a_Little_Riss/ \
--deliverytype hyperspectral
will (assuming everything is 'good' about the project) output information about commands it will run. As the delivery directory does not exist there will appear to be many errors, so if the output of the first stage looks correct (the STRUCTURE section) then run that step only to create the structure.
make_arsf_delivery.py --projectlocation /users/rsg/arsf/arsf_data/2013/flight_data/arsf_internal/GB13_00-2013_308a_Little_Riss/ \
--deliverytype hyperspectral --steps STRUCTURE --final
This will have now created the empty delivery structure. The script can be run again in dry run mode now but stating that STRUCTURE does not need to be run.
make_arsf_delivery.py --projectlocation /users/rsg/arsf/arsf_data/2013/flight_data/arsf_internal/GB13_00-2013_308a_Little_Riss/ \
--deliverytype hyperspectral --notsteps STRUCTURE
Hopefully the output will be a list of commands the script will run, separated up into sections based on the step names. If everything looks to be correct then the --final can be added to the command line to run the delivery creation.
The output text should be inspected for error messages. These should be obvious and appear as yellow text against a red background. If problems occur then individual steps can be run after the problem has been fixed (note that you may need to also run steps after the one that failed). Also note that GRASS outputs a lot of progress messages to standard error - these will appear as errors (in yellow + red on terminal) but are not actually errors.
*Note: If creating lidar delivery is complaining about dem, try running:
make_arsf_delivery.py --deliverytype lidar --steps STRUCTURE --lidardeminfo resolution=2 inprojection=UTM31N --final
or
make_arsf_delivery.py --projectlocation ./ --deliverytype lidar --steps DEMGENERATION APLDEMGENERATION --lidardeminfo inprojection=UTM31N --final
A not-so-straight-forward example
For cases which are not so straight forward there are some extra options that can be used in make_arsf_delivery.py, or you can use the library functions directly.
For example, to work with projects not in the repository, you will need to specify project details (year, julian day, project code and sortie) using the --projectinfo keyword. This is because the database can not be queried.
If multiple configfiles exist the correct one to use can be specified using --configfile.
To create a delivery in a different place to the default (processing/delivery/...) you can use --deliverylocation to specify the delivery name and file path.
If you feel unsure about anything or want to be cautious then you can run each step at a time by specifiying it with the --steps keyword. A list of the steps (in the order they're run) can be seen in the script --help - the ordering is important as the step may rely on something from a previous step.