Changes between Version 5 and Version 6 of Procedures/NewDataArrival/ProjectSetUp
- Timestamp:
- Feb 3, 2017, 12:41:29 PM (7 years ago)
Legend:
- Unmodified
- Added
- Removed
- Modified
-
Procedures/NewDataArrival/ProjectSetUp
v5 v6 27 27 }}} 28 28 29 Well done, you have finished all preparations. It is about time to start the real thing. You will need to edit the config file and open it in the text editor of your choice. You will see there that you will need to edit at least:29 You have finished all preparations but before moving files, you will need to edit the config file and open it in the text editor of your choice. Need to edit at least: 30 30 {{{ 31 31 project_in_path = … … 33 33 year = 34 34 jday = 35 sortie = # Leave it emptyif there is no sortie35 sortie = #Write <empty> without '< >' if there is no sortie 36 36 project_code = 37 37 site = 38 38 }}} 39 I will assume that if you have been entrusted with the unpacking procedure of new data you are a trustworthy colleague that knows already where to get the information from. It will usually be described in the data filenames, the logsheet or/and the application form. It is known that in the past some projects not flown by us were difficult to decipher and there was nothing left to throw some light about those fields. If so, asking to a colleague has always opened many doors. I suggest you first try to approach a veteran member of the team (they are all good people and easy to talk to but somebody involved in creating quotes will provide more expertise).39 You can gather all the information from data filenames, logsheet or/and the application form. If in need of further assistance, ask a veteran member of the team and ultimately write a mail to NERC-ARF Ops. 40 40 41 Make sure that all folder paths named there correct initial (_in) paths. You will need to check and edit if necessary all “_in” paths but should not change anything on “_out” if the format is the usual.41 You will need to check and edit if necessary all “_in” paths but should not change anything on “_out” if the format is the usual. If no data has been record for some of the inputs, fill those entries with the keywords "empty" or "absent" in the inputs. 42 42 Once happy with the config file you are ready to use unpack_project. This script will create the new top project directory and all other directories that arsf is the owner as well as moving the files. Try before with symlink. Run first: 43 43 {{{ 44 44 unpack_project.py -c ‘config_path’ -s 45 45 }}} 46 Check then the output and 46 Check then the output and you are ready to use: 47 47 {{{ 48 48 unpack_project.py -c ‘the_config.cfg’ --final 49 49 }}} 50 '''This will create a log file''' with all the changes made and will have the same name that the config file but with the extension ‘.log’. I suggest you keep track on this log and use it when using posterior commands on this guide. Have a look to the old project directory and make sure there is not any essential file that has not been moved to the right directory. If so, solve this manually and make a note in the log.50 '''This will create a log file''' with all the changes made and will have the same name that the config file but with the extension ‘.log’. I suggest you keep track on this log and use it when using posterior commands on this guide. Have a look to the old project directory and '''make sure there is not any essential file that has not been moved to the right directory. If so, solve this manually before continuing and make a note in the log.''' 51 51 52 Let’s pretend you have created the new project and moved all files. Sure you have anyway. Before renaming files and directories perhaps you would like to create the processing directories as airborne user. You can do this any time later as well but be sure not to forget. Go to the new project directory and as arsf give write permissions to the group; then change user to airborneand use:52 Before renaming files and directories perhaps you would like to create the processing directories as airborne user. You can do this any time later as well.Go to the new project directory and as arsf and ive write permissions to the group; then change user to '''airborne''' and use: 53 53 build_structure.py -p . --only-missing 54 54 … … 56 56 57 57 === Renaming Fenix and Owl files === 58 Now, let me catch up with my notes… oh yes, you can now rename Fenix and Owl files. Let’s start with the Fenix as traditionally is easier. Check that all files look OK (no missing data, does everything looks fine compared to the logsheet… the usual stuff.) The script for renaming files will point out this so you are meant to compare with what you can see. Are you ready then? Very well, go to the''' top project directory''' and run:58 Now, let me catch up with my notes… oh yes, you can now rename Fenix and Owl files. Let’s start with the Fenix as traditionally is easier. Check that all files look OK (no missing data, does everything looks fine compared to the logsheet… the usual stuff.). The script for renaming files will point out this so you are meant to make sure the output is correct. Once ready, go to the''' top project directory''' and run: 59 59 {{{ 60 60 fenix_rename.py –config_path ‘config_path.cfg’ –path . 61 61 }}} 62 If at any time you prefer to write year, julian and sortie you might do so instead of using the config file. Using the config file is recommended for simplicity. The script will check that the files have the correct julian day, year and will look for missing files (the same will be done later for the Owl sensor). '''Pay close attention and check carefully the output'''. Once done, you can use –final. I suggest you use as the log file the file created from unpack_project.py so we have all the information in the same file that should later be placed into project/admin directory. Use:62 If at any time you prefer to write year, julian and sortie you might do so instead of using the config file. Using the config file is recommended for simplicity. The script will check that the files have the correct julian day, year and will look for missing files (the same will be done later for the Owl sensor). '''Pay close attention and check carefully the output'''. If there is no major problems, you can use –final. I suggest you use as the log file the file created from unpack_project.py so we have all the information in the same file that should later be placed into project/admin directory. Use: 63 63 {{{ 64 64 fenix_rename.py -c ‘config_path.cfg’ -p . -l ‘./admin/should_move_log_here.log’ --final 65 65 }}} 66 Everything is sorted for the Fenix sensor.66 Everything is sorted now for the Fenix sensor. 67 67 68 68 Let’s move to the Owl. The steps are almost the same. You can run from top project directory: … … 70 70 owl_rename.py --config_path ‘config_path.cfg’ --path . 71 71 }}} 72 You will have to do your best once more and check carefully the output. If the starting time is still written on the hdr files the script should no find any problem ordering and renaming files. If not, the script will not allow you to continue. If no mayor problems then use {{{ --final}}}72 Do your best once more and check carefully the output. If the starting time is still written on the hdr files the script should no find any problem ordering and renaming files. If not, the script will not allow you to continue. If no major problems then use {{{ --final}}} 73 73 74 All files and directories are now sorted ,everything has been unpacked and we are almost done. You only need to run the final checks from the top project directory. You can also select to save the output in a logfile so use the same than in the previous stages if you want to do so. Again, you can use year, julian and sortie instead of config file (have a look at the help). As an example run:74 All files and directories are now sorted and everything has been unpacked and we are almost done. You only need to run the final checks from the top project directory. You can also select to save the output in a logfile so use the same than in the previous stages if you want to do so. Again, you can use year, julian and sortie instead of config file (have a look at the help). As an example run: 75 75 {{{ 76 76 run_unpacked_checks.py -p ‘project_path’ -y yyyy -j jjj -s s 77 77 }}} 78 78 79 At this stage the project set up is done, congratulations!. However, there are still a few tasks that remains to consider the project unpacked. You will need to go back to the wiki and follow the instructions for: generate a logsheet, tickets and tracking, KML overview and emailing the PI. 79 At this stage the project set up is done! Before leaving, do not forget to remove writing permissions. As '''arsf''' user, go to the top project directory and use: 80 {{{ 81 chmod ugo-w -R . 82 }}} 83 84 There are still a few tasks that remains to consider the project fully unpacked and ready to start processing. You will need to go back in the wiki and follow the instructions for: generate a logsheet, tickets and tracking, KML overview and emailing the PI. You can go back here: [wiki:Procedures/NewDataArrival]