A guide for unpacking – Project Set Up
On this page you will find practical instructions about how to proceed in case new data comes to our facilities (via SATA disk or other method) in order to do the Project Set Up.
First of all, copy everything without modification of the creation times to the desired unpacking directory. To do this rsync the data across as "arsf", for example:
rsync -av –progress * ~arsf/arsf_data/2017/unpacking/
If you do not know how to access the data in order to do the above transfer, please visit:
Files should be copying now and this is likely to take time. In the meantime, you might want to start making notes of the julian day and project code, summary, scientific aim… to fill later on the database. It is wise to make a note in a text editor to copy paste into the wiki once everything is finished.
Once data has been copied, you will need to go to the unpacking directory and fix all undesirable characters that will cause trouble for our scripts. Just use:
fix_undesirable_chars.py -p . -v
Check the output and you can either copy paste those commands or much easier, use the above command with -x instead of -v.
Also remove executable permissions from all the files - none of these files should be executable. Type following in the top-level directory of the project.
find . -type f -exec chmod a-x {} \;
Now that you and I have the data placed in the system it is time to change the directory hierarchy to fit our standards. You, who should be arsf user, copy the unpacking config file to the unpacking directory and name it with year, julian and sortie. Easy: unpacking_yyyyjjjs.cfg→ Example year 2020, julian 367 & sortie z => unpacking_2020367z.cfg In short, use:
cp ~arsf/templates/config_files/processing_config/unpacking/unpacking_template.cfg ~arsf/arsf_data/yyyy/flight_data/unpacking_yyyyjjjs.cfg
You have finished all preparations but before moving files, you will need to edit the config file and open it in the text editor of your choice. Need to edit at least:
project_in_path = project_out_path = year = jday = sortie = #Write <empty> without '< >' if there is no sortie project_code = site =
You can gather all the information from data filenames, logsheet or/and the application form. If in need of further assistance, ask a veteran member of the team and ultimately write a mail to NERC-ARF Ops.
You will need to check and edit if necessary all “_in” paths but should not change anything on “_out” if the format is the usual. If no data has been record for some of the inputs, fill those entries with the keywords "empty" or "absent" in the inputs. Once happy with the config file you are ready to use unpack_project. This script will create the new top project directory and all other directories that arsf is the owner as well as moving the files. Try before with symlink. Run first:
unpack_project.py -c ‘config_path’ -s
Check then the output and you are ready to use:
unpack_project.py -c ‘the_config.cfg’ --final
This will create a log file with all the changes made and will have the same name that the config file but with the extension ‘.log’. I suggest you keep track on this log and use it when using posterior commands on this guide. Have a look to the old project directory and make sure there is not any essential file that has not been moved to the right directory. If so, solve this manually before continuing and make a note in the log.
The next step won't run unless the flight has been added to the processing status page.
Before renaming files and directories create the processing directories as airborne user. Go to the new project directory and as arsf and add group write permissions to the root directory only (not recursively); then change user to airborne and use:
build_structure.py -p . --only-missing
Renaming Fenix and Owl files
Once the files have been moved and the directory structure created the Fenix and Owl files need to be renamed according to our naming conventions.
Fenix
Check that all files look OK (no missing data, does everything looks fine compared to the logsheet… the usual stuff.). The script for renaming files will point out this so you are meant to make sure the output is correct. Once ready, go to the top project directory and run:
fenix_rename.py -c ‘config_path.cfg’ -p . -l ‘./admin/should_move_log_here.log’
Pay close attention and check carefully the output. If there is no major problems, you can use –final. I suggest you use as the log file the file created from unpack_project.py so we have all the information in the same file that should later be placed into project/admin directory. If at any time you prefer to write year, julian and sortie you might do so instead of using the config file. Using the config file is recommended for simplicity. The script will check that the files have the correct julian day, year and will look for missing files (the same will be done later for the Owl sensor).
Everything is sorted now for the Fenix sensor.
Owl
The steps for the Owl are almost the same. You can run from top project directory:
owl_rename.py -c ‘config_path.cfg’ -p . -l ‘./admin/should_move_log_here.log’
Do your best once more and check carefully the output. If the starting time is still written on the hdr files the script should no find any problem ordering and renaming files. If not, the script will not allow you to continue. If no major problems then use --final
All files and directories are now sorted and everything has been unpacked. You only need to run the final checks from the top project directory.
run_unpacked_checks.py -p . -c ‘config_path.cfg’ -l ‘./admin/should_move_log_here.log’
It is possible to manually specify the year and day rather than using a config file.
At this stage the project set up is done! Before leaving, do not forget to remove writing permissions. As arsf user, go to the top project directory and use:
chmod a-w -R .
This will give errors for the folders owned by "airborne" but this isn't anything to worry about.
There are still a few tasks that remains to consider the project fully unpacked and ready to start processing. You will need to go back in the wiki and follow the instructions for: generate a logsheet, tickets and tracking, KML overview and emailing the PI.
Go back to the main unpacking instructions here: Procedures/NewDataArrival