Changes between Version 5 and Version 6 of Procedures/NewDataArrival/ProjectSetUp


Ignore:
Timestamp:
Feb 3, 2017 12:41:29 PM (2 years ago)
Author:
asm
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Procedures/NewDataArrival/ProjectSetUp

    v5 v6  
    2727}}}
    2828
    29 Well done, you have finished all preparations. It is about time to start the real thing. You will need to edit the config file and open it in the text editor of your choice. You will see there that you will need to edit at least:
     29You have finished all preparations but before moving files, you will need to edit the config file and open it in the text editor of your choice.  Need to edit at least:
    3030{{{
    3131project_in_path =
     
    3333year =
    3434jday =
    35 sortie = #Leave it empty if there is no sortie
     35sortie = #Write <empty> without '< >' if there is no sortie
    3636project_code =
    3737site =
    3838}}}
    39 I will assume that if you have been entrusted with the unpacking procedure of new data you are a trustworthy colleague that knows already where to get the information from. It will usually be described in the data filenames, the logsheet or/and the application form. It is known that in the past some projects not flown by us were difficult to decipher and there was nothing left to throw some light about those fields. If so, asking to a colleague has always opened many doors. I suggest you first try to approach a veteran member of the team (they are all good people and easy to talk to but somebody involved in creating quotes will provide more expertise).
     39You can gather all the information from data filenames, logsheet or/and the application form. If in need of further assistance, ask a veteran member of the team and ultimately write a mail to NERC-ARF Ops.
    4040
    41 Make sure that all folder paths named there correct initial (_in) paths. You will need to check and edit if necessary all “_in” paths but should not change anything on “_out” if the format is the usual.
     41You will need to check and edit if necessary all “_in” paths but should not change anything on “_out” if the format is the usual. If no data has been record for some of the inputs, fill those entries with the keywords "empty" or "absent" in the inputs.
    4242Once happy with the config file you are ready to use unpack_project. This script will create the new top project directory and all other directories that arsf is the owner as well as moving the files. Try before with symlink.  Run first:
    4343{{{
    4444unpack_project.py -c ‘config_path’ -s
    4545}}}
    46 Check then the output and  you are ready to use:
     46Check then the output and you are ready to use:
    4747{{{
    4848unpack_project.py -c ‘the_config.cfg’ --final
    4949}}}
    50 '''This will create a log file''' with all the changes made and will have the same name that the config file but with the extension ‘.log’. I suggest you keep track on this log and use it when using posterior commands on this guide. Have a look to the old project directory and make sure there is not any essential file that has not been moved to the right directory. If so, solve this manually and make a note in the log.
     50'''This will create a log file''' with all the changes made and will have the same name that the config file but with the extension ‘.log’. I suggest you keep track on this log and use it when using posterior commands on this guide. Have a look to the old project directory and '''make sure there is not any essential file that has not been moved to the right directory. If so, solve this manually before continuing and make a note in the log.'''
    5151
    52 Let’s pretend you have created the new project and moved all files. Sure you have anyway. Before renaming files and directories perhaps you would like to create the processing directories as airborne user. You can do this any time later as well but be sure not to forget. Go to the new project directory and as arsf give write permissions to the group; then change user to airborne and use:
     52Before renaming files and directories perhaps you would like to create the processing directories as airborne user. You can do this any time later as well.Go to the new project directory and as arsf and ive write permissions to the group; then change user to '''airborne''' and use:
    5353build_structure.py -p .  --only-missing
    5454
     
    5656
    5757=== Renaming Fenix and Owl files ===
    58 Now, let me catch up with my notes… oh yes, you can now rename Fenix and Owl files. Let’s start with the Fenix as traditionally is easier. Check that all files look OK (no missing data, does everything looks fine compared to the logsheet… the usual stuff.) The script for renaming files will point out this so you are meant to compare with what you can see. Are you ready then? Very well, go to the''' top project directory''' and run:
     58Now, let me catch up with my notes… oh yes, you can now rename Fenix and Owl files. Let’s start with the Fenix as traditionally is easier. Check that all files look OK (no missing data, does everything looks fine compared to the logsheet… the usual stuff.). The script for renaming files will point out this so you are meant to make sure the output is correct. Once ready, go to the''' top project directory''' and run:
    5959{{{
    6060fenix_rename.py –config_path ‘config_path.cfg’ –path .
    6161}}}
    62 If at anytime you prefer to write year, julian and sortie you might do so instead of using the config file. Using the config file is recommended for simplicity. The script will check that the files have the correct julian day, year and will look for missing files (the same will be done later for the Owl sensor). '''Pay close attention and check carefully the output'''. Once done, you can use –final. I suggest you use as the log file the file created from unpack_project.py so we have all the information in the same file that should later be placed into project/admin directory.  Use:
     62If at any time you prefer to write year, julian and sortie you might do so instead of using the config file. Using the config file is recommended for simplicity. The script will check that the files have the correct julian day, year and will look for missing files (the same will be done later for the Owl sensor). '''Pay close attention and check carefully the output'''. If there is no major problems, you can use –final. I suggest you use as the log file the file created from unpack_project.py so we have all the information in the same file that should later be placed into project/admin directory.  Use:
    6363{{{
    6464fenix_rename.py  -c ‘config_path.cfg’  -p . -l ‘./admin/should_move_log_here.log’ --final
    6565}}}
    66 Everything is sorted for the Fenix sensor.
     66Everything is sorted now for the Fenix sensor.
    6767
    6868Let’s move to the Owl. The steps are almost the same. You can run from top project directory:
     
    7070owl_rename.py --config_path ‘config_path.cfg’  --path .
    7171}}}
    72 You will have to do your best once more and check carefully the output. If the starting time is still written on the hdr files the script should no find any problem ordering and renaming files. If not, the script will not allow you to continue. If no mayor problems then use {{{ --final}}}
     72Do your best once more and check carefully the output. If the starting time is still written on the hdr files the script should no find any problem ordering and renaming files. If not, the script will not allow you to continue. If no major problems then use {{{ --final}}}
    7373
    74 All files and directories are now sorted, everything has been unpacked and we are almost done. You only need to run the final checks from the top project directory. You can also select to save the output in a logfile so use the same than in the previous stages if you want to do so. Again, you can use year, julian and sortie instead of config file (have a look at the help). As an example run:
     74All files and directories are now sorted and everything has been unpacked and we are almost done. You only need to run the final checks from the top project directory. You can also select to save the output in a logfile so use the same than in the previous stages if you want to do so. Again, you can use year, julian and sortie instead of config file (have a look at the help). As an example run:
    7575{{{
    7676run_unpacked_checks.py -p ‘project_path’ -y yyyy -j jjj -s s
    7777}}}
    7878
    79 At this stage the project set up is done, congratulations!. However, there are still a few tasks that remains to consider the project unpacked. You will need to go back to the wiki and follow the instructions for: generate a logsheet, tickets and tracking, KML overview and emailing the PI.
     79At this stage the project set up is done! Before leaving, do not forget to remove writing permissions. As '''arsf''' user, go to the top project directory and use:
     80{{{
     81chmod ugo-w  -R  .
     82}}}
     83
     84There are still a few tasks that remains to consider the project fully unpacked and ready to start processing. You will need to go back in the wiki and follow the instructions for: generate a logsheet, tickets and tracking, KML overview and emailing the PI. You can go back here:  [wiki:Procedures/NewDataArrival]