Changes between Version 119 and Version 120 of Procedures/NewDataArrival


Ignore:
Timestamp:
Sep 29, 2010, 12:52:05 PM (14 years ago)
Author:
knpa
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • Procedures/NewDataArrival

    v119 v120  
    1717 * Currently located ~arsf/docs/flight_data_status/YYYY/flight_data_status_YYYY.xls
    1818
    19 == Move to permanent PML data storage ==
    20 
    21 Move to appropriate location in the repository (~arsf/arsf_data/2009/flight_data/...)
    22  * ensure the project directory names conform to the standard - '''PROJECTCODE-YYYY_JJJxx_SITENAME''', e.g. ''GB07_07-2007_102a_Inverclyde'', ''boresight-2007_198'', etc
    23 
    24 ** IMPORTANT ** [[BR]]
    25 If there is no space in the repository then it will need to go into archive-over-flow temporarily (~arsf/arsf_data/YYYY/flight_data/archive-overflow). Make a note in ticket (once you've created it!) saying that this project is in non-backed up space.
    26 
    2719The next stage can be manual or semi-automatic:
    2820
    2921=== Semi-scripted method ===
    30   * in the directory above the new project directories (eg '../flight_data/unpacking/') run 'unpack_folder_structure.py'
    31     * by default, it runs safely in a dry-run mode and will output the commands it will run to the terminal. Check these look ok.
    32     * if happy, either re-run with --execute (and optionally --verbose)
     22  * In the directory above the new project directories (eg '.../flight_data/unpacking/') run 'unpack_folder_structure.py' [[BR]] NOTE: If you want to just run this on just one project (and as we usually only unpack one at a time) then you will need to move this project into a temporary containing folder when running this script (will change script at some point so can specify a single project).
     23    * By default, it runs safely in a dry-run mode and will output the commands it will run to the terminal. Check these look ok.
     24    * If happy, either re-run with --execute (and optionally --verbose)
    3325    * Each project directory should be re-formatted to the current standard
    34   * in each project directory run 'unpack_file_check.py -l <admin/logsheet.doc(.txt)>'
     26  * In each project directory run 'unpack_file_check.py -l <admin/logsheet.doc(.txt)>'
    3527    * This will convert .doc logsheet to .txt, or use the .txt if one available. NOTE to convert to .doc requires ooffice macro
    3628    * Will then do various checks of data against logsheet as listed below. Information will be output to terminal. Important (error) messages are printed again at the end.
    … …  
    4739    * This will generate a qsub command script for use to submit processing of all eagle, hawk, atm and casi flight lines to the grid nodes.
    4840
    49 
    50 
    5141=== Non-scripting  method ===
    52  * prune empty directories (check these aren't stubs for later data first) (use `rmdir *` in the project directory)
    53  * rename any capitalised subdirectories
    54  * move DCALM* files to applanix/raw
    55  * remove any spaces, brackets or other Unix-upsetting characters in filenames
     42 * Prune empty directories (check these aren't stubs for later data first) (use `rmdir *` in the project directory)
     43 * Rename any capitalised subdirectories
     44 * Move DCALM* files to applanix/raw
     45 * Remove any spaces, brackets or other Unix-upsetting characters in filenames
    5646   * use `find -regex '.*[^-0-9a-zA-Z/._].*' | ~arsf/usr/bin/fix_naughty_chars.py`
    5747   * gives suggested commands, but check before pasting commands!
    58  * remove executable bit on all files
     48 * Remove executable bit on all files
    5949   * use `find -type f -exec chmod a-x {} \;`
    60  * remove group & other write permission
     50 * Remove group & other write permission
    6151   * `chmod go-w . -R`
    6252 * Convert the .doc logsheet to .pdf
    6353   * `ooffice -invisible  "macro:///Standard.Module1.SaveAsPDF(FULL_PATH_TO_FILE)"`
     54
     55== Move to permanent PML data storage ==
     56
     57Move to appropriate location in the repository (~arsf/arsf_data/2009/flight_data/...)
     58 * ensure the project directory names conform to the standard - '''PROJECTCODE-YYYY_JJJxx_SITENAME''', e.g. ''GB07_07-2007_102a_Inverclyde'', ''boresight-2007_198'', etc
     59
     60** IMPORTANT ** [[BR]]
     61If there is no space in the repository then it will need to go into archive-over-flow temporarily (~arsf/arsf_data/YYYY/flight_data/archive-overflow). Make a note in ticket (once you've created it!) saying that this project is in non-backed up space.
    6462
    6563== Verification ==
    … …  
    9088== Tickets and tracking ==
    9189Raise a [http://arsf-dan.nerc.ac.uk/trac/newticket new trac ticket] (type 'flight processing') for the new data.
    92  * ticket summary should be of the form '''BGS07/02, flight day 172/2007, Keyworth'''
    93  * add short version of scientific purpose to guide processing (check ARSF application in ~arsf/arsf_data/2009/ARSF_Applications)
    94  * note arrival time of data
    95  * set priority of ticket from project grading (try [wiki:Internal/ProjectGradings] or the application, or hassle ARSF-Ops)
    96  * note any specific comments that might help with processing
    97  * owner should be blank
    98  * ticket body should contain:
     90 * Ticket summary should be of the form '''BGS07/02, flight day 172/2007, Keyworth'''
     91 * Add short version of scientific purpose to guide processing (check ARSF application in ~arsf/arsf_data/2009/ARSF_Applications)
     92 * Note arrival time of data
     93 * Set priority of ticket from project grading (try [wiki:Internal/ProjectGradings] or the application, or hassle ARSF-Ops)
     94 * Note any specific comments that might help with processing
     95 * Owner should be blank
     96 * Ticket body should contain:
    9997{{{
    10098Data location: ~arsf/arsf_data/2009/flight_data/..... FILL IN