A guide for unpacking – Project Set Up

On this page you will find practical instructions about how to proceed in case new data comes to our facilities (via SATA disk or other method) in order to do the Project Set Up.

First of all, copy everything without modification of the creation times to the desired unpacking directory. That is to say, copy all to the unpacking dir in its corresponding year in the arsf_data directory. To do this you will need to be arsf user and give you write permissions to the directory and use:

rsync -av –progress  *   ~arsf/arsf_data/2017/unpacking/

Files are copying now and this is likely to take time so come back here once the data has been successfully copied. In the meantime, you might want to start making notes of the julian day and project code, summary, scientific aim… to fill later on the database. It is wise to make a note somewhere to copy paste into the wiki once everything is finished.

Once data has been copied, you will need to go to the unpacking directory and fix all undesirable characters that will cause trouble for our scripts. Just use:

fix_undessirable_chars.py -p .  -v

Check the output and you can either copy paste those commands or much easier, use the above command with -x instead of -v.

Now that you and I have the data placed in the system it is time to change the directory hierarchy to fit our standards. You, who should be arsf user, copy the unpacking config file to the unpacking directory and name it with year, julian and sortie. Easy: unpacking_yyyyjjjs.cfg→ Example year 2020, julian 367 & sortie z => unpacking_2020367z.cfg In short, use:

cp  ~arsf/templates/config_files/processing_config/unpacking/unpacking_template.cfg  ~arsf/arsf_data/yyyy/flight_data/unpacking_yyyyjjjs.cfg

Well done, you have finished all preparations. It is about time to start the real thing. You will need to edit the config file and open it in the text editor of your choice. You will see there that you will need to edit at least:

project_in_path =
project_out_path =  
year = 
jday = 
sortie = #Leave it empty if there is no sortie
project_code = 
site = 

I will assume that if you have been entrusted with the unpacking procedure of new data you are a trustworthy colleague that knows already where to get the information from. It will usually be described in the data filenames, the logsheet or/and the application form. It is known that in the past some projects not flown by us were difficult to decipher and there was nothing left to throw some light about those fields. If so, asking to a colleague has always opened many doors. I suggest you first try to approach a veteran member of the team (they are all good people and easy to talk to but somebody involved in creating quotes will provide more expertise).

Make sure that all folder paths named there correct initial (_in) paths. You will need to check and edit if necessary all “_in” paths but should not change anything on “_out” if the format is the usual. Once happy with the config file you are ready to use unpack_project. This script will create the new top project directory and all other directories that arsf is the owner as well as moving the files. Try before with symlink. Run first:

unpack_project.py -c ‘config_path’ -s

Check then the output and you are ready to use:

unpack_project.py -c ‘the_config.cfg’ --final

This will create a log file with all the changes made and will have the same name that the config file but with the extension ‘.log’. I suggest you keep track on this log and use it when using posterior commands on this guide. Have a look to the old project directory and make sure there is not any essential file that has not been moved to the right directory. If so, solve this manually and make a note in the log.

Let’s pretend you have created the new project and moved all files. Sure you have anyway. Before renaming files and directories perhaps you would like to create the processing directories as airborne user. You can do this any time later as well but be sure not to forget. Go to the new project directory and as arsf give write permissions to the group; then change user to airborne and use: build_structure.py -p . --only-missing

Renaming Fenix and Owl files

Now, let me catch up with my notes… oh yes, you can now rename Fenix and Owl files. Let’s start with the Fenix as traditionally is easier. Check that all files look OK (no missing data, does everything looks fine compared to the logsheet… the usual stuff.) The script for renaming files will point out this so you are meant to compare with what you can see. Are you ready then? Very well, go to the top project directory and run:

fenix_rename.py –config_path ‘config_path.cfg’ –path .

If at anytime you prefer to write year, julian and sortie you might do so instead of using the config file. Using the config file is recommended for simplicity. Check carefully the output. Come on, this is important now so pay attention and check carefully the output. Once done, you can use –final. I suggest you use as the log file the file created from unpack_project.py so we have all the information in the same file that should later be placed into project/admin directory. Use:

fenix_rename.py  -c ‘config_path.cfg’  -p . -l ‘./admin/should_move_log_here.log’ --final

And there it is, everything sorted for the Fenix.

Let’s move to the Owl. The steps are almost the same. You can run from top project directory:

owl_rename.py --config_path ‘config_path.cfg’  --path .

You will have to do your best once more and check carefully the output. If the starting time is still written on the hdr files the script should no find any problem ordering and renaming files. If not, the script will not allow you to continue. If no mayor problems then use --final

All files and directories are now sorted, everything has been unpacked and we are almost done. You only need to run the final checks from the top project directory. You can also select to save the output in a logfile so use the same than in the previous stages if you want to do so. Again, you can use year, julian and sortie instead of config file (have a look at the help). As an example run:

run_unpacked_checks.py -p ‘project_path’ -y yyyy -j jjj -s s

At this stage the project set up is done, congratulations!. However, there are still a few tasks that remains to consider the project unpacked. You will need to go back to the wiki and follow the instructions for: generate a logsheet, tickets and tracking, KML overview and emailing the PI.